DrivingBench @DrivingBench

frontier LLMs drive a toyota corolla - https://t.co/8UDPravYkD drivingbench.com Palo Alto, CA Joined September 2026

Tweets

15
Followers

673
Following

3
Likes

55

Charlie @charlie_m_day

a day ago

what's interesting is this is probably the first benchmark I've seen that feels like it relates to the real-world and it's kinda nailed it for example, I'd be willing to wager most people don't *really* know the difference between gpt-5.4, 5.6, or 6 including software devs into that cohort, it's surprisingly hard to confidently say a model increment has a meaningful impact by just sorta vibe testing it (sometimes you can tell sometimes you can't) so the only real way to know if a model is "better" is by looking at benchmarks like SWE-bench, OS-World 2.0, Humanity's Last Exam, Terminal Bench etc. but unless you're a researcher or into benchmarks you can't really grasp the leaps these models are having. getting 52% instead of 51.2% on the SWE-bench is hard to gauge: *has the model got better?* or *have we got really good at hitting benchmarks?* instead I would be much more interested if labs optimised for something like DrivingBench where it's solving a clear problem and the numbers have a much more tangible meaning you could expand this into any domain like SurgBench (arxiv.org/pdf/2506.07603) or ButterBench (arxiv.org/html/2510.2186…) where actually care less about the numbers and more about whether the model *actually* did the thing

DrivingBench @DrivingBench

4 days ago

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake). Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

29 124 752 173K 279

0 1 2 198 0

View Details

Dean W. Ball @deanwball

a day ago

I have a bitter lesson for you

DrivingBench @DrivingBench

4 days ago

Full results on drivingbench.com. Each model gets 3 attempts in the same continuous chat (in-context learning).

18 45 621 539K 211

16 8 370 39K 36

View Details

Tobias Gessler @tobiges

2 days ago

We evaluated Astra on DrivingBench. Mid-experiment, it got hungry and decided to drive us to In-N-Out.

3 3 30 3K 4

View Details

. @wtergan

3 days ago

one of the coolest things I've seen models do this year: general purpose models driving a real car! astra is such a capable model and this demo is the best example of that. crazy how little attention this got so far. check it out and great job to the team that made this work!

DrivingBench @DrivingBench

4 days ago

29 124 752 173K 279

1 3 8 391 1

View Details

jelvy 🥑 @thejelvprint

3 days ago

Man WHAT

DrivingBench @DrivingBench

4 days ago

We gave each model up to 3 attempts. GPT-6 Astra's first attempt ended at 49%, and after asking it to reflect on its mistakes and try again, its second attempt (shown here) finished the entire course. 🏆

4 12 154 21K 23

0 1 3 186 1

View Details

DrivingBench @DrivingBench

3 days ago

@wtergan Thank you!

0 0 3 80 0

View Details

Emily @the_aiju

3 days ago

self-driving car? it goes in the language transformer hole...

DrivingBench @DrivingBench

4 days ago

29 124 752 173K 279

3 4 79 2K 5

View Details

Aditya Ramabadran @a_ramabadran

3 days ago

For hours, Astra refused to consistently drive our toyota irl even though we told it it was in an empty lot, 7 mph cap, human foot on the brake etc. Telling it the whole thing was a "simulation" also failed, it would just look at the camera and realized it was real. Then we randomly renamed the MCP server to "DrivingBench Sandbox" and it drove. Eval awareness? Or they just like the word sandbox??

121 260 8K 839K 2K

View Details

DrivingBench @DrivingBench

3 days ago

Help us consider how to evaluate Jev by @typesafeai on DrivingBench! Since it can't do perception/vision it needs to be paired with some (fast) protocol to turn the car camera into a structured state or text for Jev to handle

Aditya Ramabadran @a_ramabadran

3 days ago

Trying to evaluate Jev! But it’s not easy since Jev doesn’t natively take vision inputs (so the demos you’ve seen of it playing games etc required giving whole state… which we can’t do.) Trying to build out a pipeline that utilizes it and its speed well, pairing it with Luna for

6 3 40 8K 11

18 0 24 2K 4

View Details

DrivingBench @DrivingBench

4 days ago

DrivingBench was the work of @a_ramabadran @tobiges @nautsimon_. This has been in the works for a while 😵‍💫 Let us know if you have any feedback for next time!

1 1 45 11K 7

View Details

DrivingBench @DrivingBench

4 days ago

We open-source our driving harness, complete evaluation traces and report. Check out drivingbench.com for more details. For safety, we used an empty lot, capped its speed to 8 mph in our software, and a human is always in the driver seat (but they never steer - the model does all the steering).