Keyu Tian @keyutian
Master's student @ Peking University linkedin.com/in/keyu-tian/?… Joined October 2021-
Tweets24
-
Followers62
-
Following26
-
Likes39
SparK - [ICLR'23 Spotlight] The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" github.com/keyu-tian/SparK
: The first BERT-style pretraining on CNNs! #nlp #cnn reddit.com/r/MachineLearn…
画像認識の事前学習で、ViTは一部の入力をMaskし、それを予測するタスクが成功していたがConvは隣り合うパッチ間で重複があり情報が漏れ成功してなかった。SparKは出力も入力と同じマスクを維持するSparseConvを使うことで問題を解決、また復号器側も階層を持ちさらに改善 openreview.net/forum?id=NRxyd…
How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7
[CV] Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling K Tian, Y Jiang, Q Diao, C Lin, L Wang, Z Yuan [Peking University & Bytedance Inc & University of Oxford] (2023) arxiv.org/abs/2301.03580 #MachineLearning #ML #AI #CV [1/2]
🤩The first successful BERT-style #SelfSupervisedLearning on any convolutional network! #ResNet now enjoys masked autoencoding! 🚀A breakthrough paper "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" by @keyutian et al. deepai.org/publication/de…
Designing BERT for convolutional networks: sparse and hierarchical masked modeling Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan tl;dr: create image, which looks to CNN same, as transformers -> MIM starts working arxiv.org/abs/2301.03580…
Daisy-mae Cofer @MaeCofer67348
90 Followers 5K Following
Hongwei Yi @HongweiYi2
2K Followers 3K Following PhD student at @MPI_IS, working on Human-Scene Interaction.
Sen Mei @Masonfake
34 Followers 1K Following
Zeyi Sun @sunzeyi6
23 Followers 94 Following Research Intern in Shanghai AI Lab in CV PhD student in SJTU
Peilun @PEILUN10
6 Followers 59 Following A Doctoral student || Medical AI, M.Res @imperialcollege || B.Eng @unibirmingham
Tongtian Zhu @Raiden13238619
81 Followers 289 Following 🎓 Phd student @ Zhejiang_University (@ZJU_China) 💡 Deep learning theory; Decentralized learning
🦖 @KarlMarxSweetie
38 Followers 296 Following
Miles Yan @MgYuanYan
15 Followers 115 Following
Jiarui Jin @jerryjiaruijin
39 Followers 258 Following Visiting Ph.D. Student @UCL, Ph.D. Student @sjtu1896, working on IR with Structure Mining
Darius @lam_darius
1K Followers 2K Following Scale Maximalist | Prev Cerebras Systems, Harvard | Views my own
YouJiacheng @YouJiacheng
46 Followers 214 Following
Ollin Boer Bohan @madebyollin
2K Followers 1K Following "Stack more layers" enthusiast. Made sdxl-vae-fp16-fix, taesd, that pokemon-emulation-via-dnn thing.
Blue Sky @DongRunpei
178 Followers 928 Following
fuzzy_stonk @FuzzyRoam
61 Followers 1K Following
Haiyang Wang @haiyang73756134
20 Followers 54 Following PhD Student @pku1898, focusing on topics in the foundation model and network architecture designing. Work for AGI.
AK @_akhaliq
310K Followers 3K Following AI research paper tweets, ML @Gradio (acq. by @HuggingFace 🤗) dm for promo follow on Hugging Face: https://t.co/q2Qoey80Gx
abisoye @__bisoye__
123 Followers 322 Following Computer science undergrad🤓👨🏽💻|| Futarian|| Data science/ML
charmier @MingruiChenX
23 Followers 239 Following
Sarah @turner66sarah
176 Followers 3K Following
Mohammad Bayazi @653mjd
305 Followers 747 Following Ph.D. student in Artificial Intelligence and Brain-Computer Interface @med_umontreal and @mila_quebec | CEO of Hummingbird AI | Vice-chair @mtlieee | 🧠🔁🖥️
Wanru Zhao (Looking f.. @Renee42581826
513 Followers 2K Following Postgraduate Student @CaMLSys @Cambridge_CL | Ex-Intern @DGLGraph @AWS and @CambridgeJBS | Do not go gentle into that good night 🧗
Jude Gomila @judegomila
15K Followers 3K Following Explorer of universes. Previously founded Golden & Heyzap. Early $ into Linear Mercury Gusto Astranis Solugen Ironclad Benchling Boom Stoke and many unicorns.
Noah Jones @noahcjones
229 Followers 1K Following Ph.D Candidate @MIT | Affective Computing. Interested in reliable machine learning and multimodal transfer learning for healthcare applications.
li xi xi @lixixi75382146
0 Followers 13 Following
Bin Yan @BinYanMaster
4 Followers 40 Following PhD candidate at Dalian University of Technology. Intern at Microsoft Research Asia and ByteDance. My research interests are Computer Vision and Deep Learning.
zbz @zbz58071410
0 Followers 511 Following
Siyuan Li @LupinLSY
72 Followers 271 Following Third-year CS PhD. Student at @Westlake_Uni and @ZJU_China || Advisor: Stan Z. Li || Research interest: Self-sup & semi-sup learning, Network design, and AI4S.
Bohang Zhang @ICLR 20.. @bohang_zhang
645 Followers 230 Following Phd in @pku1898, focusing on topics in ML, GNN and LLM through an expressivity perspective
Di Chang @DiChang10
701 Followers 1K Following PhD @CSatUSC｜BSc @TU_Muenchen｜BEng @dlut1949｜Previous @TikTok_US @EPFL｜ Working on 3D Computer Vision, Generative Model ｜17‘ Camaro SS 1LE
Khoa Tuan Nguyen @Khoa_NguyenTuan
484 Followers 5K Following Enjoy learning new things. PhD student at Ghent University Global Campus in Korea.
Alfredo LF @ajlf88
443 Followers 5K Following Caribeño Colombiano; Me gusta Hablar y Escuchar; Orgulloso Hijo de la Universidad Pública; No hay Preguntas estúpidas; Amante de la Naturaleza; Ser Curioso...
Material Scientist @Mtrl_Scientist
48K Followers 3K Following Founder #MaterialIndicators | Professional shitposter Married to @count_grumpula Telegram: https://t.co/QOjdZ72iG7
Anna Woodard @foozlefoo
260 Followers 2K Following Then: particle physics @CMSExperiment @cern. Now: post-doc @DSI_UChicago deep learning + medical imaging, cat whisperer
EP @EP225654
162 Followers 5K Following
LifeAsTuesday @TuesdayAs
11 Followers 959 Following
Shengjie Luo @Roger98079446
273 Followers 454 Following PhD Student @pku1898, interested in Machine Learning
Jiaming Song @baaadas
5K Followers 992 Following Chief Scientist @LumaLabsAI. Working on visual generative AI. Were @NVIDIA @Stanford @OpenAI @MetaAI
YouJiacheng @YouJiacheng
46 Followers 214 Following
Jude Gomila @judegomila
15K Followers 3K Following Explorer of universes. Previously founded Golden & Heyzap. Early $ into Linear Mercury Gusto Astranis Solugen Ironclad Benchling Boom Stoke and many unicorns.
Wanru Zhao (Looking f.. @Renee42581826
513 Followers 2K Following Postgraduate Student @CaMLSys @Cambridge_CL | Ex-Intern @DGLGraph @AWS and @CambridgeJBS | Do not go gentle into that good night 🧗
Mohammad Bayazi @653mjd
305 Followers 747 Following Ph.D. student in Artificial Intelligence and Brain-Computer Interface @med_umontreal and @mila_quebec | CEO of Hummingbird AI | Vice-chair @mtlieee | 🧠🔁🖥️
Gradio @Gradio
35K Followers 12 Following Build and share machine learning demos in 3 lines of Python. Part of the @Huggingface family 🤗
Bin Yan @BinYanMaster
4 Followers 40 Following PhD candidate at Dalian University of Technology. Intern at Microsoft Research Asia and ByteDance. My research interests are Computer Vision and Deep Learning.
Siyuan Li @LupinLSY
72 Followers 271 Following Third-year CS PhD. Student at @Westlake_Uni and @ZJU_China || Advisor: Stan Z. Li || Research interest: Self-sup & semi-sup learning, Network design, and AI4S.
Bohang Zhang @ICLR 20.. @bohang_zhang
645 Followers 230 Following Phd in @pku1898, focusing on topics in ML, GNN and LLM through an expressivity perspective
Hongyang Li @francislee2020
615 Followers 98 Following Recruiting PhD/Postdoc. 🇭🇰 Incoming Assistant Prof @HKUniversity. PI @OpenDriveLab. We only live once, if we do it right, once is enough. Opinions my own.
Anna Woodard @foozlefoo
260 Followers 2K Following Then: particle physics @CMSExperiment @cern. Now: post-doc @DSI_UChicago deep learning + medical imaging, cat whisperer
Shengjie Luo @Roger98079446
273 Followers 454 Following PhD Student @pku1898, interested in Machine Learning
Tian Han @s85302019
5 Followers 47 Following
Hanyu @Hanyu32509610
1 Followers 2 Following
Xuxin Cheng @Xuxin_Cheng_PKU
9 Followers 123 Following Master Student @ PKU ＆ Research Intern @ Bytedance AI Lab
Chen Lin @WillLin1028
43 Followers 39 Following Ph.d Student, University of Oxford Torr Vision Group Studying Graph Deep Learning and Computational Chemistry
Jeff Liang @LiangJeff95
216 Followers 560 Following Ph.D. candidate @utexasece. Ex- @AIatMeta @Tsinghua_Uni
DeepAI @DeepAI
51K Followers 2K Following DeepAI is an experimental AI Product Lab. Sharing cool research at @arxiv_daily For product support email [email protected]
AK @_akhaliq
310K Followers 3K Following AI research paper tweets, ML @Gradio (acq. by @HuggingFace 🤗) dm for promo follow on Hugging Face: https://t.co/q2Qoey80Gx
Sebastian Raschka @rasbt
267K Followers 906 Following Machine learning & AI researcher writing at https://t.co/A0tXWzG1p5. LLM research engineer @LightningAI. Previously stats professor at UW-Madison.
Di Chang @DiChang10
701 Followers 1K Following PhD @CSatUSC｜BSc @TU_Muenchen｜BEng @dlut1949｜Previous @TikTok_US @EPFL｜ Working on 3D Computer Vision, Generative Model ｜17‘ Camaro SS 1LE
It's December! And I was just reminiscing about all the things that happened in / defined AI in 2023, putting together a short list of keywords that were top of my mind (in no particular order). 1) LLM efficiency & adapter methods: One of the biggest research threads has been to…
Overhauled the "Understanding LLMs -- A Cross-Section of the Most Relevant Literature To Get Up to Speed" reading list! Now featuring parameter-efficiency methods, and latest studies and insights via Pythia, among others to round it from 13 to 20! magazine.sebastianraschka.com/p/understandin…
What an awesome week for open source and the PyTorch ecosystem with three big launches! - PyTorch 2.0 - Lightning Trainer 2.0 for PyTorch - Fabric for PyTorch! Just updated my "faster PyTorch" article to include the latest tools! 🔗 sebastianraschka.com/blog/2023/pyto…
Btw if you have an hour to spare on the weekend, I am covering how to use the Trainer to fit PyTorch models in the newly released Unit 5: lightning.ai/pages/courses/…
What an awesome week for open source and the PyTorch ecosystem with three big launches! - PyTorch 2.0 - Lightning Trainer 2.0 for PyTorch - Fabric for PyTorch! Just updated my "faster PyTorch" article to include the latest tools! 🔗 sebastianraschka.com/blog/2023/pyto…
PS: our original submission in Oct. 2022 was titled "Sparse and Hierarchical Masked Modeling for Convolutional Representation Learning"; we've changed it to the new one to more clearly express our vision for bringing the powerful BERT-style self-supervised learning to CNNs.
[ICLR'23 Spotlight] SparK: The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" github.com/keyu-tian/SparK
SparK - [ICLR'23 Spotlight] The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" github.com/keyu-tian/SparK
When I asked ChatGPT the "Why do ViTs need more data than ConvNets" question this morning, I saw the biggest blunder yet > "Fewer parameters: Vision transformers have fewer parameters than CNNs, this means that they require more data [...]." (via leanpub.com/machine-learni…)
: The first BERT-style pretraining on CNNs! #nlp #cnn reddit.com/r/MachineLearn…
In image recognition pretraining, ViT succeeds w/ masked autoencoding, but Conv does not. SparK uses sparse conv (outputs keep the same mask pattern) to prevent information leakage between overlapped patches and uses the hierarchical decoder. openreview.net/forum?id=NRxyd…
@keyutian Thank you, and happy lunar new year! 🥳
@keyutian Big congratulations!! 🎉🎊 This is a groundbreaking paper 🔥 I'm excited to try applying your SparK to medical imaging, where our small datasets make ViTs pretty challenging. I'm looking forward to you posting the fine-tuning and viz code 🙏🙏🙏
Now this paper has been accepted to #ICLR2023 as a spotlight🌟! I truly appreciate everyone's efforts. Check out our repo github.com/keyu-tian/SparK for the latest demo video and updates! Title: "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7
@svpino 💯! And don’t forget to supercharge your ResNet with masked autoencoding!
How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7
I have to study for my final exam now,and I hope everyone can pass the test:)
画像認識の事前学習で、ViTは一部の入力をMaskし、それを予測するタスクが成功していたがConvは隣り合うパッチ間で重複があり情報が漏れ成功してなかった。SparKは出力も入力と同じマスクを維持するSparseConvを使うことで問題を解決、また復号器側も階層を持ちさらに改善 openreview.net/forum?id=NRxyd…
@kushirosea @rasbt That paper relied on ViTs. The critical difference here is they have adapted the method for CNNs.
@reidatcheson @rasbt Thanks for your interest! We are writing a google Colab tutorial to make it easy to play with our pre-trained models (e.g. to reconstruct the images you upload). We'll release it before this weekend and hope it can be helpful.
@rasbt Thanks for sharing! Some other exciting things: 1. This generative pretraining surpasses contrastive learning. 2. CNNs gain more from pretraining than Transformers. 3. SparK may be more general than ConvNextV2: it doesn't modify CNN's architecture and works well on ConvNextV1.
I've been playing with this lately
How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7
Trends for United States
You might like
