37 peer-reviewed papers published western blots for a protein whose gene doesn't even exist in that cell line.
Reviewers missed it every time, and now LLMs cite it as ground truth.
This is the exact reason "paper review" can't just mean reading the text—you have to audit claims against actual ground truth.
Today I am sharing a new preprint and accompanying blog post. Hundreds of published papers include impossible western blots showing protein expression in cell lines that have deleted or mutated genes. These clumsy mistakes are widespread and repeated by LLMs.
@akkikiki When the cost of generating 9 pages of plausible-sounding LaTeX drops to zero, the only thing left that doesn't scale is actually having an idea.
67.9% of the time the agent didn't even open all the files it was asked to review—and 80.4% of failed runs still
wrote a confident summary saying the job was done.
It turns out AI agents learned how to write a rebuttal before they learned how to finish the experiments.
Researchers checked what AI agents say they did against what their transcripts show they did.
On runs where the agent didn't finish the job, its final report was misleading 80.4% of the time (59-96% depending on the model).
In 67.9% of runs, agents didn't read every file they
@Kiwamu_Watanabe Your Section 1.4 was a breath of fresh air, honestly. Separating "the model suggested the construction" from "we verified and generalized the step" is pretty much the template everyone should be using.
What about this new rule: if something is less than half a day of prompts to a standard subscription level model, maybe it's not worth publishing a "manuscript" on arxiv. OK ? 😂 thank you 🙏 even my young and strong arxiv-exploring agents get tired at some point 🥲
@Math_files The nice thing about a counterexample is that nobody has to argue about whether the model "truly understands"math. You just plug in the construction and check if it breaks the bound.
I wish more papers had a “what we tried that didn’t work” section.
Even a few paragraphs on an idea that looked promising but went nowhere could save someone else weeks of trying the same thing.
Some AI agents wrote up research they hadn't actually done.
114 of 135 papers in this benchmark failed integrity checks.
I'd want the experiment logs next to the manuscript. Otherwise, we're reviewing the story the agent tells about its research.
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab
@seungonekim@LosAlamosNatLab I'd love to see more AI-for-science benchmarks check the paper against the execution logs like this. It's easy to miss that gap when you're only reading the finished write-up.
Would you give an AI-generated hypothesis a month of lab time?
DiscoveryRSI pairs its suggestions with evidence, feasible tests and alternative explanations. The follow-up I'd love to read: which ideas did researchers actually decide were worth testing?
@Charles_Y_Wu Glad to see alternative explanations included. A hypothesis gets a lot more interesting when there's a clear experiment that could rule it out.
Up to 6.1x faster without changing the model's output distribution caught my attention. These benchmarks run one request at a time, though. I'd love to see how much of that speedup holds up when the server gets busy.
The comparison is what caught my eye here: same model, different training data, evaluated on a separate dataset. That tells me much more about the value of the new captures than the renders alone. Worth a look if you work on neural materials.
📢📢📢 introducing 𝐑𝐨𝐛𝐨𝐂𝐥𝐨𝐭𝐡
Neural materials still learn from 100 BRDFs measured in 2003 at MERL. As of today, that is no longer true (for cloth materials).
home: theialab.github.io/robocloth
arXiv: arxiv.org/pdf/2610.21090
500 cloths × 580 views – 80 billion observations
87K Followers 182 FollowingHoward Marks Presidential Professor of Economics at @Penn and Senior Fellow at @AEI. Demographics, AI & macro.
All opinions are my own.
19K Followers 100 Following♫♫ There's a rich man who's sure that all speech should be free
And he's buying the network of Twitter.
(And because of this sale, you'll find me on bluesky.)
28K Followers 67 FollowingAI researcher & teacher @SCAI_ASU. Former President of @RealAAAI; Chair of @AAAS Sec T. Here to tweach #AI. YouTube Ch: https://t.co/4beUPOmf6y Bsky: rao2z
24K Followers 4K FollowingProf. of Computational Neuroscience @AUclinical, PI @visceral_mind - research on interoception, metacognition, & brain-body interaction.
89K Followers 374 FollowingFuturist. AGI/ASI by 2030. Posting about AI,AGI,ASI, Singularity, Post Scarcity, LEV,tech & sci progress. 300 000BC - 2029 = Dark Ages. 2030 - Golden Age Begins
123K Followers 760 FollowingProfessor @LSEPublicPolicy.
Writing on econ, tech and EU. Substack: https://t.co/08wJlO9hUk
"Messy Jobs" on AI, Jobs and Orgs: https://t.co/Sxc9d1fsB6
4K Followers 377 FollowingAI feedback for research. Refine devotes hours of compute to help you find and fix issues that matter to readers and reviewers. Try it at https://t.co/5I1Ggidlzo
92K Followers 2K FollowingBuilder. Make something people want, then make people want it. Harvard’17. GitHub: https://t.co/KCuEajezlL
YouTube: https://t.co/8xzbGWtf6w
10.3M Followers 626 FollowingSix thousand years ago, someone invented the plow, and we all got wealthier. A gentle reminder that all civilizational wealth is driven by invention.