Shuvendu Roy @ShuvenduBikash

by the way. recently wrote a paper on this! for transformers, the number is about 3.6 bits-per-parameter so you would need 25GB ÷ 3.6 bits ≈ 56.9B parameters to exactly memorize Wikipedia that’s a pretty big model actually

prerat @prerationalist

2 weeks ago

by the way. recently wrote a paper on this! for transformers, the number is about 3.6 bits-per-parameter so you would need 25GB ÷ 3.6 bits ≈ 56.9B parameters to exactly memorize Wikipedia that’s a pretty big model actually https://t.co/CJXFMAOieC

113 78 4K 514K 1K

RLPT: Reinforcement Learning on Pre-Training Data • RL directly on pre-train data (no human labels) • Next-segment reasoning objective (ASR + MSR tasks) → self-supervised rewards • Gains on Qwen3-4B: +3.0 MMLU, +8.1 GPQA-Diamond, +6.6 AIME24, +5.3 AIME25

Thinking Augmented Pre-training "we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens…

How do language models actually develop their capabilities during pre-training? We need mechanistic insights into what's happening inside! We used crosscoders to track linearly interpretable features across 32 training snapshots, revealing a surprising two-phase learning process.

This MIT paper just broke my brain. Everyone keeps saying LLMs can't do real logical reasoning. Turns out we've just been teaching them wrong this whole time. These researchers built something called PDDL-INSTRUCT that actually teaches models to think through planning problems…

Meta Superintelligence Labs presents MetaEmbed: Scalable multimodal retrieval • Flexible late interaction via Meta Tokens • Test-time scaling: trade off retrieval accuracy vs efficiency • SOTA on MMEB + ViDoRe, robust up to 32B models • Matryoshka training → coarse-to-fine…

Ever wondered how Energy-Based Models (EBMs) work and how they differ from normal neural networks? ☕️We go over EBMs and then dive into the Energy-Based Transformers paper to make LLMs that refine guesses, self-verify, and could adapt compute to problem difficulty. (link👇)

Top AI Papers of The Week (September 15-21): - K2-Think - DeepDive - AgentScaler - Shutdown Resistance in LLMs - Is In-Context Learning Learning? - Towards a Physics Foundation Model - Retrieval and Structuring Augmented Generation with LLMs Read on for more:

🚨Brilliant New @AIatMeta Superintelligence Labs Paper. It asks a simple question: "Can inference compute substitute for missing supervision?" And the big deal is that this paper shows you don’t need humans to provide labels or feedback in reinforcement learning anymore.…

🇨🇳China unveils world's first brain-like AI Model SpikingBrain1.0 Upto 100X faster while being trained on less than 2% of the data typically required. Designed to mimic human brain functionality, uses much less energy. A new paradigm in efficiency and hardware independence.…

Rohan Paul @rohanpaul_ai

3 weeks ago

🇨🇳China unveils world's first brain-like AI Model SpikingBrain1.0 Upto 100X faster while being trained on less than 2% of the data typically required. Designed to mimic human brain functionality, uses much less energy. A new paradigm in efficiency and hardware independence.… https://t.co/8NA1PE4fkU

7 35 197 229K 181

Download Image

HUGE AI breakthrough from META. This can change everything (in AI industry) 30x Faster LLMs, 16x Bigger Contexts, Zero Accuracy Loss 👀 Meta Superintelligence Labs is clearly already cooking. "The core problem with long context is simple: making a document 2x longer can make…

Jackson Atkins @JacksonAtkinsX

4 weeks ago

HUGE AI breakthrough from META. This can change everything (in AI industry) 30x Faster LLMs, 16x Bigger Contexts, Zero Accuracy Loss 👀 Meta Superintelligence Labs is clearly already cooking. "The core problem with long context is simple: making a document 2x longer can make… https://t.co/fLO7gWQGOE

25 145 854 245K 832

Download Image

10 latest Preference Optimization techniques ▪️ Pref-GRPO ▪️ PVPO (Policy with Value PO) ▪️ DCPO (Dynamic Clipping PO) ▪️ ARPO (Agentic Reinforced PO) ▪️ GRPO-RoC (Resampling-on-Correct) ▪️ TreePO ▪️ DuPO ▪️ TempFlow-GRPO ▪️ MixGRPO ▪️ MaPPO (Maximum a Posteriori PO) Save the…

This is probably one of THE most important paper of the last few months. Small language models are sufficiently powerful, operationally suitable, and economical Agentic tasks. - Phi-2 matches 30 billion models running 15x faster. - Serving a 7 billion parameter small language…

Universal Deep Research NVIDIA recently published another banger tech report! The idea is simple: allow users to build their own custom, model-agnostic deep research agents with little effort. Here is what you need to know:

Fine-tuning LLM Agents without Fine-tuning LLMs Catchy title and very cool memory technique to improve deep research agents. Great for continuous, real-time learning without gradient updates. Here are my notes:

NVIDIA research just made LLMs 53x faster. 🤯 Imagine slashing your AI inference budget by 98%. This breakthrough doesn't require training a new model from scratch; it upgrades your existing ones for hyper-speed while matching or beating SOTA accuracy. Here's how it works:…

A Deep Dive into RL for LLM Reasoning Provides a roadmap for practitioners applying RL for LLM reasoning. Nice to have some of the latest techniques in one place.

Efficient Agents This is a great study full of insights on how to build efficient agents. If you are looking to reduce costs with AI agents, don't miss it. Pay attention to this one, devs! Here are my notes:

🚨This week's top AI/ML research papers: - Mixture-of-Recursions - Scaling Laws for Optimal Data Mixtures - Training Transformers with Enforced Lipschitz Constants - Reasoning or Memorization? - How Many Instructions Can LLMs Follow at Once? - Chain of Thought Monitorability -…