Hao Liu @haoliuhl
machine learning, neural networks. phd student @berkeley_ai. https://t.co/ZNJawlrerS Joined September 2018-
Tweets267
-
Followers4K
-
Following155
-
Likes381
How do LLMs scale to million token context window? Ring Attention is a nice trick to parallelize long sequence across devices and rotate them in a ring with zero overhead scaling. In our new blog, we cover the tricks behind this magic. It looks like this (1/5🧵)
How do state-of-the-art LLMs like Gemini 1.5 and Claude 3 scale to long context windows beyond 1M tokens? Well, Ring Attention by @haoliuhl presents a way to split attention calculation across GPUs while hiding the communication overhead in a ring, enabling zero overhead scaling
Large world models by @haoliuhl et al. is worth checking out if you like the Gemini-1.5 and Sora results. It has 1M context window, generates videos, and is open sourced. 1. LWM shows high recall over 1M context window, and performs exceptionally well with video chat as well.…
Current works are restricted to short sequences of texts and images, limiting their ability to model the world. Presenting Large World Model (LWM): capable of processing long text, images, videos of over 1M tokens (and *no* lost in the middle!) Project: largeworldmodel.github.io
Current works are restricted to short sequences of texts and images, limiting their ability to model the world. Presenting Large World Model (LWM): capable of processing long text, images, videos of over 1M tokens (and *no* lost in the middle!) Project: largeworldmodel.github.io
Gemini 1.5 Pro is an impressive work. Sadly, the technical report is very very sparse on details. But this paper from yesterday gives a glimpse on how one may train for extra long context: x.com/haoliuhl/statu…
Gemini 1.5 Pro is an impressive work. Sadly, the technical report is very very sparse on details. But this paper from yesterday gives a glimpse on how one may train for extra long context: x.com/haoliuhl/statu…
AK @_akhaliq
310K Followers 3K Following AI research paper tweets, ML @Gradio (acq. by @HuggingFace 🤗) dm for promo follow on Hugging Face: https://t.co/q2Qoey80Gx
Andrej Karpathy @karpathy
978K Followers 904 Following 🧑🍳. Previously Director of AI @ Tesla, founding team @ OpenAI, CS231n/PhD @ Stanford. I like to train large deep neural nets 🧠🤖💥
Jim Fan @DrJimFan
229K Followers 3K Following @NVIDIA Sr. Research Manager & Lead of Embodied AI (GEAR Lab). Creating foundation models for Humanoid Robots & Gaming. @Stanford Ph.D. @OpenAI's first intern.
Eric Jang @ericjang11
69K Followers 3K Following physical AGI at 1X. Author of "AI is Good for You" https://t.co/eFg4WXhg0p
Danijar Hafner @danijarh
14K Followers 868 Following Building AI that makes autonomous decisions using world models, artificial curiosity, and temporal abstraction @DeepMind
Misha Laskin @MishaLaskin
8K Followers 174 Following Staff Research Scientist @DeepMind. Previously @berkeley_ai. YC alum.
Yuandong Tian @tydsh
16K Followers 801 Following Research Scientist and Senior Manager in Meta AI (FAIR). AI-guided Optimization and Representation Learning. Novelist in spare time. PhD in @CMU_Robotics.
Richard Socher @RichardSocher
101K Followers 970 Following CEO @youSearchEngine Investing at @aixventuresHQ Before: Stanford Adj Prof in AI/NLP, Chief Scientist at Salesforce, MetaMind
Eugene Vinitsky @EugeneVinitsky
13K Followers 2K Following Anti-cynic. Artificial narrow intelligence. Autonomous vehicles, multi-agent learning, and transportation. RS at Apple, Asst. Prof at @nyutandon. He/him.
Kyunghyun Cho @kchonyc
61K Followers 2K Following a combination of a mediocre scientist, a mediocre manager, a mediocre advisor & a mediocre PC at @nyuniversity (@CILVRatNYU) & @genentech (@PrescientDesign).
Abhishek Gupta @abhishekunique7
5K Followers 639 Following Assistant Professor at University of Washington. I like robots, and reinforcement learning. Previously: post-doc at MIT, PhD at Berkeley
rohan anil @_arohan_
12K Followers 2K Following Principal Engineer, @GoogleDeepMind Gemini. prev PaLM-2. Tinkering with optimization and distributed systems. opinions are my own.
Horace He @cHHillee
23K Followers 449 Following Working at the intersection of ML and Systems @ PyTorch "My learning style is Horace twitter threads" - @typedfemale
Victor Zhong @hllo_wrld
4K Followers 450 Following ML+NLP assistant prof @UWCheritonCS. Formerly @MSFTResearch @MetaAI, @SFResearch via @MetamindIO, @uwnlp, @StanfordNLP, @eceuoft.
Dhaval Adjodah @_dval_
495 Followers 2K Following funding AI+X research @SchmidtFutures Past: @MIT @medialab @Mila_Quebec @NYU_Courant @worldbank send me spicy RKHS memes
pengch fan @FanPengch
216 Followers 6K Following
Amir M. Tavakkoli @AmirTheWanderer
131 Followers 2K Following PhD Student @ University of Utah @KahlertSoC | High-performance Computing, Compilers, Hardware-software Co-design | Classical Music 🎻🎹 and Physics 🪐
Philip Meier @Ch1nges
697 Followers 3K Following Manager at https://t.co/lJ90WLptXr | Assoc. Researcher @hiig_berlin | Liberal Thinker @fdp | Co-Hosting the #TalkingAboutPlatforms podcast | Meat Lover | Optimist
Hongwei Yi @HongweiYi2
2K Followers 3K Following PhD student at @MPI_IS, working on Human-Scene Interaction.
Capybara ai @capybara_ai
61 Followers 499 Following Capybara doing PhD@TsinghuaCS, checkout my blog @ https://t.co/2Iz05C84xd. Interested in Reinforcement Learning, LLM-based Agents, Alignment.
anon @almostschurlie
1 Followers 24 Following
Keane Moraes @lordvader_31
287 Followers 1K Following cs+cogsci student at uw. incoming dl @nvidia. i also blog about cinema - https://t.co/OGS3KXN5qE
arreycah @RK2930757112999
82 Followers 870 Following
Dhruv Thanki @DhruvThank
15 Followers 385 Following Humanoid Controls and Autonomy @ Boardwalk Robotics Inc.
Mojtaba Vàlipour @ValipourMojtaba
391 Followers 3K Following CS PhD at @UWaterloo, Founding Engineer at Coastal Carbon and part time Researcher at @huawei Noah’s Arc Lab, prev enjoyed my time at @oraclelabs, & @CVC_UAB
Dana Mahmood @deordered
9 Followers 650 Following Fine-tuning AI models oftentimes & practicing philosopher at other times.
panjun003 @panjun003
17 Followers 179 Following
Daanish @danishabbir
624 Followers 5K Following elk again. before: startup founder, ml eng (e.g. @nvidia), ee + english (@stanford)
Xuhui Zhang @XuhuiZhangXHZ
4 Followers 236 Following
Janhavee Shinde @SJanhavee
56 Followers 2K Following
Hamza Alsbaihi @hamza_alsbaihi
523 Followers 2K Following Pursing Master in Data Science at TU Wien. Interested in Data Science and AI.
Nadeesha Amarasinghe @nadeesha99
2K Followers 240 Following AI Inference + Infra @Tesla_AI prev @Apple, @Nvidia. Learned stuff at @UofT.
Huy Tran @huytransformer
92 Followers 3K Following
Dong Zhang @dongzha35524835
86 Followers 259 Following Speech Language Models | MS Student at FudanNLP Lab @FudanUniv | Looking for Ph.D. in 2025 fall
Roberta @JHGwcnObA65A2
20 Followers 1K Following I am a lively and cheerful girl who would like to meet a good friend.
aVerity @AVerityjane
4 Followers 144 Following
HinePo @Hine__Po
187 Followers 439 Following Head of AI & Data. Data science tech lead. Chemical engineer. Kaggle Competitions Expert (top 1%).
Michael Auger @michaelc0des
28 Followers 95 Following
Anonymous Founder @anonymfounder
383 Followers 7K Following My startup diary. Leading the way in innovation and industry transformation. From startups to marketing, finance to entrepreneurship........and cryptocurrency.
Eddy Emmanuel @youngboi_eddy
113 Followers 432 Following Machine learning //Artificial intelligence//crypto enthusiast. GitHub: https://t.co/pLyM6JSfh5 LinkedIn:https://t.co/iLoDYlwIXD
Richard Gibbons @RichardGibbonsX
377 Followers 1K Following Founder @DigitalApplied. Digital Marketing & Transformation | AI | SEO | PPC | Social Media | Web Development | eCommerce | Automation | CRM & Analytics.
Abcd @Abcd271582
46 Followers 245 Following
WaSaBi @dtabait
39 Followers 315 Following
Tony AdAstra @tonyadastra
121 Followers 699 Following Self-starter, Well-wisher, Tech Enthusiast ⚡️ where there’s a will, there’s a way
زِرِنگ @premature79
402 Followers 917 Following
Viacheslav Sinii @ummagumm_a
56 Followers 260 Following
Yuh Dean Tsai @ydtsaia
23 Followers 40 Following
サッカーインフ.. @footballinflu
574 Followers 278 Following 海外でプレーする日本人サッカー選手の情報や、海外の反応をブログでまとめています。ツイッターでは選手の小ネタや豆情報なんかをつぶやいていきます。
Pensé FFun @inftyCategory
108 Followers 6K Following
Redie @rediejarvis
8 Followers 117 Following
ma @ma52987379
0 Followers 120 Following
Alpay Ariyak @AlpayAriyak
1K Followers 2K Following AI @RunPod_io | Lead: @OpenChatDev (600k+ downloads on HuggingFace🤗)
Shamik Bose @BoseShamik
356 Followers 510 Following PhD, Senior Researcher XAI | Will talk at length about the harms and considerations for the current state of AI | Views my own | he/him
Harsh Desai @dreamerharsh
1 Followers 3K Following
Yann LeCun @ylecun
711K Followers 718 Following Professor at NYU. Chief AI Scientist at Meta. Researcher in AI, Machine Learning, Robotics, etc. ACM Turing Award Laureate.
Jim Fan @DrJimFan
229K Followers 3K Following @NVIDIA Sr. Research Manager & Lead of Embodied AI (GEAR Lab). Creating foundation models for Humanoid Robots & Gaming. @Stanford Ph.D. @OpenAI's first intern.
Danijar Hafner @danijarh
14K Followers 868 Following Building AI that makes autonomous decisions using world models, artificial curiosity, and temporal abstraction @DeepMind
Yi Tay @YiTayML
29K Followers 97 Following chief scientist / cofounder @RekaAILabs 🫠 past: research scientist @google brain 🤯 currently learning to be a dad 🍼
Soumith Chintala @soumithchintala
186K Followers 877 Following Cofounded and lead @PyTorch at Meta. Also dabble in robotics at NYU. AI is delicious when it is accessible and open-source.
Misha Laskin @MishaLaskin
8K Followers 174 Following Staff Research Scientist @DeepMind. Previously @berkeley_ai. YC alum.
Yuandong Tian @tydsh
16K Followers 801 Following Research Scientist and Senior Manager in Meta AI (FAIR). AI-guided Optimization and Representation Learning. Novelist in spare time. PhD in @CMU_Robotics.
Richard Socher @RichardSocher
101K Followers 970 Following CEO @youSearchEngine Investing at @aixventuresHQ Before: Stanford Adj Prof in AI/NLP, Chief Scientist at Salesforce, MetaMind
Kyunghyun Cho @kchonyc
61K Followers 2K Following a combination of a mediocre scientist, a mediocre manager, a mediocre advisor & a mediocre PC at @nyuniversity (@CILVRatNYU) & @genentech (@PrescientDesign).
Marc G. Bellemare @marcgbellemare
13K Followers 350 Following CSO & co-founder, Reliant AI. Ex RL research lead at Google Brain, DeepMind. Known for Atari 2600 RL benchmark, Distributional RL (MIT Press 2023).
Abhishek Gupta @abhishekunique7
5K Followers 639 Following Assistant Professor at University of Washington. I like robots, and reinforcement learning. Previously: post-doc at MIT, PhD at Berkeley
rohan anil @_arohan_
12K Followers 2K Following Principal Engineer, @GoogleDeepMind Gemini. prev PaLM-2. Tinkering with optimization and distributed systems. opinions are my own.
PyTorch @PyTorch
379K Followers 77 Following Tensors and neural networks in Python with strong hardware acceleration. PyTorch is an open source project at the Linux Foundation. #PyTorchFoundation
Horace He @cHHillee
23K Followers 449 Following Working at the intersection of ML and Systems @ PyTorch "My learning style is Horace twitter threads" - @typedfemale
Naveen Rao @NaveenGRao
28K Followers 785 Following VP GenAI @Databricks. Former CEO/cofounder MosaicML & Nervana/IntelAI. Neuro + CS. I like to build stuff that will eventually learn how to build other stuff.
Simon Guo 🦝 @simonguozirui
1K Followers 4K Following Incoming CS PhD student @Stanford and curr training models at @cohere | 🎓 @Berkeley_EECS | prev built things at @ @anyscalecompute @nvidia
Kilian Haefeli @khshind
232 Followers 341 Following Exploring crevasses of Deep Learning at ETH Zurich & UofT | Previously: @Aleph__Alpha, @Logitech, and exfounder at Airica
Denny Zhou @denny_zhou
9K Followers 420 Following @GoogleDeepMind founder & lead of Reasoning Team. Build LLMs to reason. Opinions my own.
Jack Rae @drjwrae
9K Followers 353 Following Principal Scientist @ Google DeepMind Work on Gemini 💎♊ Compression is all you need LLMs (e.g. Gopher, Chinchilla, Gemini) 💼 Past: OpenAI, Quora
Rohan Taori @rtaori13
2K Followers 1K Following phd student @StanfordAILab🌲| proud @Cal alum 🐻 | prev taught w @BerkeleyML
Ulyana Piterbarg @ulyanapiterbarg
256 Followers 328 Following reasoning and decision-making @CILVRatNYU @nyuniversity @MSFTResearch, alum of @MITMath
Tianyi Zhang @Tianyi_Zh
1K Followers 613 Following iterating ... I used to train more language models but am working on agents now
Xuechen Li @lxuechen
2K Followers 900 Following Building intelligence @xai. PhD @Stanford. Undergrad @UofT. Worked at @GoogleAI @MSFTResearch @Vectorinst. I go by Chen.
Tongzhou Wang @TongzhouWang
1K Followers 1K Following representation of type 1→2 agi @mit Ex @pytorch @MetaAI @berkeley_ai
Zhuohan Li @zhuohan123
3K Followers 686 Following CS PhD Student 👨🏻💻 @ UC Berkeley 🌁 🤖️ Machine Learning Systems
Erich Elsen @erich_elsen
2K Followers 260 Following Adept. Previously Deepmind, Google Brain, Baidu SVAIL. LLMs, exascale computing, systems research, GPU nerd.
Xinyun Chen @xinyun_chen_
4K Followers 840 Following Research Scientist at @GoogleDeepMind. PhD from @Berkeley_EECS.
Yang You @YangYou1991
8K Followers 386 Following Presidential Young Professor at @NUSingapore. @Forbes 30 under 30. Ph.D. from @UCBerkeley. Founder, President and Chairman of @HPCAITech and Colossal-AI.
Dacheng Li @DachengLi177
619 Followers 476 Following Intelligence. PhD @Berkeley_EECS @lmsysorg @ucbrise @berkeley_ai, Prev. @Google @SCSatCMU.
Evgenii Nikishin @nikishin_evg
1K Followers 845 Following PhD student @Mila_Quebec working on AI agents. Past: a research scientist intern @GoogleDeepMind London #StopWar 🇺🇦
Sheng Shen @shengs1123
1K Followers 539 Following Ph.D. student @berkeley_ai; Building 🦙@MetaAi; Former @MSFTResearch, @allen_ai, @GoogleDeepMind
Johan S. Obando 👍�.. @johanobandoc
1K Followers 2K Following Graduate student @Mila_Quebec @UMontrealDIRO | RL/Deep Learning/AI | De Cali/Colombia pal’ Mundo 🇨🇴 | #JuntosProsperamos⚡#TogetherWeThrive| 🌱🌎
Peter J. Liu @peterjliu
4K Followers 2K Following Research Scientist @ Google B̵r̵a̵i̵n̵ DeepMind, frontier language models research (aka chatbot engineer). Opinions are my own. 🤖🔄🚀
Scott Reed @scott_e_reed
16K Followers 387 Following Research Scientist at NVIDIA working on generalist embodied agent research
Qinyuan Ye @qinyuan_ye
2K Followers 1K Following 👩💻 Ph.D. student @nlp_usc @CSatUSC @USC_ISI | 🐾 Teaching machines to be more versatile and curious.
Tao Xu @txhf
6K Followers 888 Following Learning Machine at OpenAI, previously Airbnb, Quora, Facebook and Microsoft.
Tri Dao @tri_dao
18K Followers 364 Following Incoming Asst. Prof @PrincetonCS, Chief Scientist @togethercompute. Machine learning & systems.
Matei Zaharia @matei_zaharia
39K Followers 1K Following CTO at @Databricks and CS prof at @UCBerkeley. Working on data+AI, including @ApacheSpark, @DeltaLakeOSS, @MLflow, https://t.co/94gROE5Xa0. https://t.co/nmRYAKG0LZ
Eric Wallace @Eric_Wallace_
6K Followers 1K Following Researcher at OpenAI working to make language models more trustworthy, secure, and private.
Shikun Liu @liu_shikun
990 Followers 737 Following Ph.D. student at the Dyson Robotics Lab at Imperial College.
CLS @ChengleiSi
2K Followers 3K Following vibing @stanfordnlp | real AGI is the friends we made along the way
Charlie Snell @sea_snell
4K Followers 5K Following PhD @berkeley_ai & student researcher @GoogleDeepMind. My friend told me to tweet more. I stare at my computer a lot and make things
Huazhe Harry Xu @HarryXu12
2K Followers 895 Following Hi, I like reinforcement learning, robots, and video games:) I am an amateur pianist. Assistant Prof at Tsinghua; Postdoc at Stanford; Ph.D. at Berkeley
Amber Xie @amberxie_
433 Followers 171 Following 🤖🤖 @Stanford CS PhD. Previously @berkeley_ai MS, BA.
Alex Li @alexlioralexli
633 Followers 344 Following PhD student in ML at @mldcmu. Prev: @AIatMeta and undergrad @berkeley_ai
Brandon Trabucco @brandontrabucco
390 Followers 235 Following AI/ML PhD Student at @mldcmu advised by @rsalakhu, recipient of the @NDSEG Fellowship, musician https://t.co/qWxtgOEnAJ
Mohit Shridhar @mohito1905
1K Followers 1K Following Research Scientist at @Dyson. @uwcse PhD in Robotics.
Jacky Liang @jackyliang42
4K Followers 735 Following Research Scientist @GoogleDeepMind working on foundation models for robotics. PhD @CMU_Robotics @iamlab_cmu. Prev. intern @NVIDIAAI. Writes @Last_Week_in_AI
Ademi Adeniji @AdemiAdeniji
385 Followers 180 Following PhD @UCBerkeley. Prev @NVIDIAAI, @Stanford '21. Reinforcement Learning, Robot Learning, Behavior Foundation Models.
Victoria X Lin @VictoriaLinML
3K Followers 760 Following Research Scientist @AIatMeta Foundational AI Research • ex-@SFResearch • PhD @uwcse 📜 https://t.co/j6QTac5q0r
Carlo Sferrazza @carlo_sferrazza
470 Followers 202 Following Postdoc at @berkeley_ai. PhD from @eth_en. Robotics, Artificial Intelligence, Tactile Sensing.
Dinghuai Zhang 张鼎.. @zdhnarsil
2K Followers 1K Following PhD student at @Mila_Quebec. Ex intern at FAIR Labs @MetaAI. Previous math undergraduate at @PKU1898.
Boyuan Chen @BoyuanChen0
2K Followers 275 Following PhD student @MIT_CSAIL @MITEECS, Ex @GoogleDeepMind, @berkeley_ai; Doing AI & Robotics. Foundational model for decision making. World models and robotics.I don’t think everyone has comprehended the massive disruption and distortion that is going to happen in the Gen AI market due to Llama3. Moats will be destroyed and investments will go to zero. Just like everything in Gen AI, this will all happen fast.
Spent the last few weeks working on this blog! When I first read the Ring Attention paper, I kind of get the concept yet not really. Diving into the details from the math to compute was incredibly rewarding for our understanding, and hope it be a fun read for you too!
How do state-of-the-art LLMs like Gemini 1.5 and Claude 3 scale to long context windows beyond 1M tokens? Well, Ring Attention by @haoliuhl presents a way to split attention calculation across GPUs while hiding the communication overhead in a ring, enabling zero overhead scaling
Me, @simonguozirui & @bonniesjli and I spent the past few weeks really understanding the concept from math to device. Check out our blog for detailed walkthroughs and discussions: coconut-mode.com/posts/ring-att… Let us know if you have any thoughts or feedback!
Each device can send/receive KV blocks while computing local result, this completely overlaps communication with computation! Check out our blog for a detailed walkthrough on each trick! Fun writing with @khshind and @simonguozirui (5/5🧵) coconut-mode.com/posts/ring-att…
How do LLMs scale to million token context window? Ring Attention is a nice trick to parallelize long sequence across devices and rotate them in a ring with zero overhead scaling. In our new blog, we cover the tricks behind this magic. It looks like this (1/5🧵)
How do state-of-the-art LLMs like Gemini 1.5 and Claude 3 scale to long context windows beyond 1M tokens? Well, Ring Attention by @haoliuhl presents a way to split attention calculation across GPUs while hiding the communication overhead in a ring, enabling zero overhead scaling
Large world models by @haoliuhl et al. is worth checking out if you like the Gemini-1.5 and Sora results. It has 1M context window, generates videos, and is open sourced. 1. LWM shows high recall over 1M context window, and performs exceptionally well with video chat as well.…
@haoliuhl @wilson1yan @matei_zaharia @pabbeel Super impressive project!
I did it at the end 😎 Large World Model with 1M context size ready for you on @ollama 🔥🔥🔥 RingAttention to the MAX! Text-Chat 1M model uploaded! Thanks for the model to @haoliuhl @matei_zaharia @pabbeel and @wilson1yan Go and play with it: ollama.com/ifioravanti/lwm
Context length keeps getting longer!! With ability to inject ~1 hr videos, short term memory seems within reach. What about long term memories? Could be game changing for continual agents 🧠
Current works are restricted to short sequences of texts and images, limiting their ability to model the world. Presenting Large World Model (LWM): capable of processing long text, images, videos of over 1M tokens (and *no* lost in the middle!) Project: largeworldmodel.github.io
This guy is a genius who is seeing the future with his systems. First he produced all the best results in unsupervised RL. Then he cracked near-infinite context transformers, which he has now applied to video with @wilson1yan👏 Excited to make this large model colossal at Clone!
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
Gemini 1.5 Pro is an impressive work. Sadly, the technical report is very very sparse on details. But this paper from yesterday gives a glimpse on how one may train for extra long context: x.com/haoliuhl/statu…
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
This is one of the coolest LLM papers I've seen in a bit. A win for open science. Good work doesn't flourish with closed source in-company models. Props to Berkeley AI! (From a former undergrad Berkeley AI researcher)
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
Current works are restricted to short sequences of texts and images, limiting their ability to model the world. Presenting Large World Model (LWM): capable of processing long text, images, videos of over 1M tokens (and *no* lost in the middle!) Project: largeworldmodel.github.io
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
@haoliuhl excited to see advances in multimodal approaches. Did you consider audio as input and output modality as well? What would be the challenges with that?
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
New paper on long context world models with @haoliuhl !
We are excited to share Large World Model (LWM), a general-purpose 1M context multimodal autoregressive model. It is trained on a large dataset of diverse long videos and books using RingAttention, and can perform language, image, and video understanding and generation.
@matei_zaharia @haoliuhl @wilson1yan Congrats, looks amazing.
@haoliuhl @wilson1yan @matei_zaharia @pabbeel Great work, congrats! @haoliuhl @wilson1yan
World Model on Million-Length Video And Language With RingAttention Open-sources 7B models capable of processing long text documents and videos of over 1M tokens proj: largeworldmodel.github.io abs: arxiv.org/abs/2402.08268
Super cool work by @haoliuhl and @wilson1yan using Ring Attention. A multimodal 7B model with 1M context length.
World Model on Million-Length Video And Language With RingAttention Open-sources 7B models capable of processing long text documents and videos of over 1M tokens proj: largeworldmodel.github.io abs: arxiv.org/abs/2402.08268
Trends for United States
You might like
