404 @0xQuant404

This Google Paper Was Reportedly Rejected From NeurIPS And Now Powers Every Tiny LLM From DistilBERT To DeepSeek Google paid a reported $44 million in 2013 for Geoffrey Hinton's three-person startup Two years later Hinton, Oriol Vinyals and Jeff Dean showed that a huge model can teach a tiny one The trick is temperature, soften the big model's answers and its hidden knowledge leaks out On MNIST a small net went from 146 errors to 74, almost matching the big net's 67 Hugging Face, valued at $4.5 billion in 2023, used it to make DistilBERT 40% smaller and 60% faster A week after DeepSeek dropped R1 and its distilled models, Nvidia lost $589 billion in a single day OpenAI then said it had evidence that DeepSeek distilled knowledge from OpenAI's own models Hinton won the 2024 Nobel Prize in Physics and Jeff Dean is now Google's chief scientist

404 @0xQuant404

2 weeks ago

DeepSeek's R1 wiped $589 billion off Nvidia in a single day and then OpenAI and the White House's AI czar accused the Chinese lab of copying OpenAI's answers with a trick from Nobel winner Geoffrey Hinton David Sacks went on Fox and said the evidence was substantial Bloomberg

0 0 0 144 0

DeepSeek's R1 wiped $589 billion off Nvidia in a single day and then OpenAI and the White House's AI czar accused the Chinese lab of copying OpenAI's answers with a trick from Nobel winner Geoffrey Hinton David Sacks went on Fox and said the evidence was substantial Bloomberg reported Microsoft's security team saw people possibly linked to DeepSeek pulling huge amounts of data from OpenAI Google paid $44 million for Hinton's startup and two years later he cracked how to shrink a giant AI Geoffrey Hinton, Oriol Vinyals and Jeff Dean's 2015 paper taught a small AI to copy a giant one show the teacher a cat and it says 90% cat, 10% dog, and that dog hint teaches the student more than the label in 2026 Anthropic accused DeepSeek, Moonshot and MiniMax of 16 million chats with Claude from 24,000 fake accounts 5 minutes on the formula behind the biggest copying fight in AI, save it before you train on someone else's model

404 @0xQuant404

3 weeks ago

x.com/i/article/2106…

0 0 1 1K 0

Sam Altman put GPT-4's training bill above $100 million and OpenAI predicted its final score from models with 10,000x less compute while the number they were chasing was invented by Claude Shannon at Bell Labs with his wife guessing letters from a detective story Shannon's 1948 paper introduced the bit and Scientific American called it the Magna Carta of the Information Age in 1951 he had his wife Betty guess a Raymond Chandler story one letter at a time she landed near one bit per letter, so English is roughly 75% predictable every LLM plays that same guessing game and its training loss is Shannon's formula, cross entropy a loss of 2.3 means the model is as unsure as rolling a ten sided die on every token IBM fed a model 583 million words in 1992 to reach 1.75 bits per character on Shannon's scale Bell Labs also gave the world the transistor, the laser, Unix and 11 Nobel Prizes 5 minutes and you can turn any training loss into perplexity in your head ↓

keyboard_arrow_left Previous keyboard_arrow_right Next

404 @0xQuant404

3 weeks ago

x.com/i/article/2106…

0 0 1 1K 0

Nvidia put money into Together AI's $800 million round at an $8.3 billion valuation because Tri Dao figured out as a Stanford PhD student how to make GPT-2 train 3x faster with one memory trick that trick is FlashAttention, it gets the exact same answer without approximating a single number, and PyTorch, Microsoft DeepSpeed, Meta and OpenAI's Triton all run it now one attention head at 65,000 tokens eats 8.5 GB on a GPU, and Nvidia's A100 has a fast memory 10x quicker but thousands of times too small to hold it the fix is a running max and a running sum that let softmax work one tile at a time, and I check it with three scores on a calculator on a $25,000 to $40,000 Nvidia H100 the newest version hits 740 trillion operations a second, bookmark this before your next training run

404 @0xQuant404

3 weeks ago

x.com/i/article/2106…

0 0 1 1K 0

Sam Altman says OpenAI burned over $100 million training GPT-4, and that model knows Tom Cruise’s mother is Mary Lee Pfeiffer but asked who Mary Lee Pfeiffer’s son is it found Tom Cruise only 33% of the time, while on invented facts GPT-3 went from 96.7% right forward to 0.1% backward Meta’s Llama failed the same test, Anthropic’s researchers saw the same one way memory inside a 52 billion parameter model, and this video shows the single formula inside every GPT style model that causes it plus the dead simple trick that fixes it when you train or prompt your own AI 5 minutes, bookmark it before you trust your chatbot with a fact

404 @0xQuant404

3 weeks ago

x.com/i/article/2106…

0 0 1 1K 0

x.com/i/article/2106…

OpenAI's ChatGPT, the most famous chatbot on the planet, couldn't count the r's in strawberry, it confidently said two, the screenshots went viral, and then Reuters revealed OpenAI's secret reasoning project was literally codenamed Strawberry before it shipped as o1 OpenAI built a whole $200 a month ChatGPT Pro tier around that reasoning, yet the model still never sees letters, GPT-4's tokenizer turns the word in your question into one single token, number 73,700, and that number is all it gets OpenAI co-founder Andrej Karpathy says tokenization is at the heart of a lot of weirdness in LLMs, GPT-4 reads 12,345,678 as 123, 456, 78, a Reddit username once got its own token and made ChatGPT talk about "distribute" instead, and the same sentence in Ukrainian costs double the tokens of English 5 minutes of math to watch before you pay for another million tokens, so bookmark it and watch today because the 3 tricks at the end make the strawberry problem disappear in any model, ChatGPT, Claude or Gemini

Rahul @sairahul1

4 months ago

x.com/i/article/2067…

19 100 578 650K 1K

@sama my rule is that if i can't explain why an answer is right, i haven't checked it yet. using these models every day made me more skeptical of them

@rohanpaul_ai broadcom grows by buying more than anyone on this chart. vmware alone cost $69b, ten times what nvidia paid for mellanox

@karpathy trying asd-ste100 for my video narration scripts now. short plain sentences sound a lot better read out loud