Seongjun Ahn @elixir_devv

The most important AI paper of 2025 might have just dropped. NVIDIA lays out a framework for Small Language Model agents that could outcompete LLMs. Here’s the full breakdown (and why it matters):

“Everyone knows” what an autoencoder is… but there's an important complementary picture missing from most introductory material. In short: we emphasize how autoencoders are implemented—but not always what they represent (and some of the implications of that representation).🧵

This is one of the craziest ideas I've ever seen. He converted a drawing of a bird into a spectrogram (PNG -> Soundwave) then played it to a Starling who sung it back reproducing the PNG. Using the birds brain as a hard drive with 2mbps read write speed. youtube.com/watch?si=HMtVd…

keyboard_arrow_left Previous keyboard_arrow_right Next

I noticed that @OpenAI added learnable bias to attention logits before softmax. After softmax, they deleted the bias. This is similar to what I have done in my ICLR2025 paper: openreview.net/forum?id=78Nn4…. I used learnable key bias and set corresponding value bias zero. In this way,…

keyboard_arrow_left Previous keyboard_arrow_right Next

OpenAI @OpenAI

a month ago

I noticed that @OpenAI added learnable bias to attention logits before softmax. After softmax, they deleted the bias. This is similar to what I have done in my ICLR2025 paper: openreview.net/forum?id=78Nn4…. I used learnable key bias and set corresponding value bias zero. In this way,… https://t.co/PwxukdR5lR

1K 3K 20K 6.5M 4K

@aryehazan @bremen79 The way I did it was progressively asking it to breakdown the problem into self-contained sub-questions that could be used to make a determination. I'd feed each one to a fresh context gemini and report back. This particular subquestion produced the counter example in response.

Reward hacking is challenging when fine-tuning few-step Diffusion models. Direct fine-tuning on rewards can create artifacts that game metrics while degrading visual quality. We propose Noise Hypernetworks as a theoretically grounded solution, inspired by test-time optimization.

Compressing infinite eigenvals in finite dim of matrices will implement real intelligence stuffs

Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit A method to replace tokenizers post-hoc by reconstructing unseen token embeddings using (OMP): Two-phase process - Represent each new token as a sparse combo of shared tokens in donor embedding space -…

𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8

3 months ago

Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit A method to replace tokenizers post-hoc by reconstructing unseen token embeddings using (OMP): Two-phase process - Represent each new token as a sparse combo of shared tokens in donor embedding space -… https://t.co/BqqjZCV3zq

0 0 8 3K 1

Download Image

Semantics in language is naturally hierarchical, but attempts to interpret LLMs often ignore this. Turns out: baking semantic hierarchy into sparse autoencoders can give big jumps in interpretability and efficiency. Thread + bonus musings on the value of SAEs:

keyboard_arrow_left Previous keyboard_arrow_right Next

✨New preprint: Dual-Process Image Generation! We distill *feedback from a VLM* into *feed-forward image generation*, at inference time. The result is flexible control: parameterize tasks as multimodal inputs, visually inspect the images with the VLM, and update the generator.🧵

Alignment reseaches are far more fall behind the other LLM fields and this is hazardous.

We introduce Reinforcing "Cognitive Experts" – a new approach to enhance reasoning in MoE-based Large Reasoning Models (LRMs) 🌟. Thanks to Tencent's support, we had the opportunity to explore the inner workings of ultra-large models like DeepSeek-R1-671B and Qwen3-235B. By…

it's been more than a decade since KD was proposed, and i've been using it all along .. but why does it work? too many speculations but no simple explanation. @_sungmin_cha and i decided to see if we can come up with the simplest working description of KD in this work. we ended…

keyboard_arrow_left Previous keyboard_arrow_right Next

Microsoft's Distilled Decoding is now on Hugging Face It distills Image AR models (like VAR and LlamaGen) to generate images in just one step, resulting in up to 218x faster image generation!

Is Compression Really Linear with Code Intelligence? This paper revisits the link between compression and code LLM capability. Prior work claimed a linear BPC–performance relationship but tested weak models on Python completions. The authors introduce a broader framework: -…

I want to share my latest (very short) blog post: "Active Learning vs. Data Filtering: Selection vs. Rejection." What is the fundamental difference between active learning and data filtering? Well, obviously, the difference is that: 1/11

Geometry-aware immunology models are replacing sequence-only epitope pickers ImmunoStruct folds every peptide-MHC with AlphaFold2, blends the 3-D graph with sequence chemistry, and doubles precision on true T-cell hits. Multimodal, structure-aware AI will script the next…

𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗮𝘀 𝗖𝗼𝗱𝗲 We see more tools enabling you to create software architecture and other diagrams as code. The main benefit of using this concept is that 𝗺𝗼𝘀𝘁 𝗼𝗳 𝘁𝗵𝗲 𝗱𝗶𝗮𝗴𝗿𝗮𝗺𝘀 𝗮𝘀 𝗰𝗼𝗱𝗲 𝘁𝗼𝗼𝗹𝘀 𝗰𝗮𝗻 𝗯𝗲 𝘀𝗰𝗿𝗶𝗽𝘁𝗲𝗱 𝗮𝗻𝗱…