Jacob Lin @JacobLinCool

We ran it on StreamDecisionBench and it now tops the in-force accuracy leaderboard. There is still a gap to frontier LLMs running with no reasoning on untimed accuracy. Would love to see how far that gap can be closed without giving up the latency advantage. jacoblincool.github.io/StreamDecision…

keyboard_arrow_left Previous keyboard_arrow_right Next

@singularity_bly Looking forward to the latency improvements! GLiDE’s strong untimed accuracy gives it plenty of room to improve its in-force score. Thanks for the heads-up on fine-tuning too!

If a model is going to make decisions inside a live application, static accuracy is not enough. StreamDecisionBench evaluates the decision actually in force as the state keeps evolving. We tested decision models, general LLMs, and open-weight models. Here’s the leaderboard. 👇

@perplexitydevs We benchmarked Decider on StreamDecisionBench, and it takes #1 on in-force accuracy! We were also impressed by how consistently fast the responses were, usually arriving in well under a second. x.com/JacobLinCool/s…

Jacob Lin @JacobLinCool

3 weeks ago

We benchmarked @perplexity_ai’s Perplexity Decider 27B on StreamDecisionBench across voice-controlled presentations, IDE debugging, assembly instructions, and support-call workflows. Decider ranks #1 on our primary metric, in-force accuracy, with 63.37% normalized log-AUC. Its

keyboard_arrow_left Previous keyboard_arrow_right Next

0 0 0 57 0

We benchmarked @perplexity_ai’s Perplexity Decider 27B on StreamDecisionBench across voice-controlled presentations, IDE debugging, assembly instructions, and support-call workflows. Decider ranks #1 on our primary metric, in-force accuracy, with 63.37% normalized log-AUC. Its untimed accuracy is 70.00%. While the model thinks, new speech, logs, and events keep arriving. The application keeps using its previous decision until a newer response arrives. We measure how often the decision actually in use is correct, including time spent waiting and acting on outdated information. Explore the results: jacoblincool.github.io/StreamDecision…

keyboard_arrow_left Previous keyboard_arrow_right Next

GLiDE is now on StreamDecisionBench! It scored 39.43% normalized log-AUC on in-force accuracy, our primary metric, and 80.49% untimed accuracy, the highest among the decision models we tested. We observed ~5.5s median latency and ~17.3s p95. Its strong untimed accuracy makes latency an interesting area to explore for improving the in-force score. Full results and measurement details: github.com/JacobLinCool/S…

We benchmarked all six vLLM Decision 2.0 models on StreamDecisionBench, across presenter control, live debugging, assembly coaching, and call assist. Lux-9B came out strongest on the in-force score, while Vega-27B showed strong untimed accuracy but dropped much more once latency was counted. Code and full results are here github.com/JacobLinCool/S…

keyboard_arrow_left Previous keyboard_arrow_right Next

@singularity_bly Already on it. We’re running GLiDE now. Will share the results soon!

A bit more on how we designed SDB. One thing we spent a lot of time on was task selection. Existing evaluations often reuse LLM benchmarks, but those can mix tasks that should be answered immediately with tasks that genuinely benefit from more reasoning. For us, speed needed to matter to the task itself. Presenter control, live debugging, assembly coaching, and call assist all sit close to a human in the loop. If a decision arrives seconds late, the user may already have moved on. We also tried to avoid tasks that reduce to semantic similarity. There are already plenty of benchmarks for clustering, classification, and semantic matching. We were more interested in tasks where following the given rules and instructions actually matters. At the same time, we wanted the tasks themselves to stay on the fast side. Our operational check was whether a strong frontier model could solve them almost immediately. Astra does not have a no-reasoning setting, so we used its lowest setting. Across three passes it reached 99.86% mean untimed accuracy. In the original pass it used 773 reported reasoning tokens over 480 requests, about 1.6 per request, effectively answering without deliberation. That is the kind of task we wanted for SDB. Not necessarily the hardest task, but the right task for decision models, where judgment and speed both matter. We are still benchmarking new decision models as they come out. Right now we are running @perplexity_ai’s Decider and @fastinoAI’s GLiDE. More results soon.

Jacob Lin @JacobLinCool

3 weeks ago

If a model is going to make decisions inside a live application, static accuracy is not enough. StreamDecisionBench evaluates the decision actually in force as the state keeps evolving. We tested decision models, general LLMs, and open-weight models. Here’s the leaderboard. 👇

6 3 25 12K 27

@DopeRakshith Added! Wity-1 is now on the leaderboard in both modes. Across SDB’s streaming decision tasks, including presentation navigation and incident response, auto mode improves accuracy but has higher latency. Code and evaluation details are available at github.com/JacobLinCool/S…

keyboard_arrow_left Previous keyboard_arrow_right Next

@EldanRing Done! We ran both Winnow models. Winnow-12B looks really strong for a 12B model. Both results are now up on the SDB leaderboard. Updated leaderboard: jacoblincool.github.io/StreamDecision…

@EldanRing Already running them now. Will share the Winnow-12B / Winnow-E4B results as soon as they’re done!

What should we test next? We’ll keep adding new APIs and open-weight models to SDB. If we missed one, reply or open an issue. We’re also looking forward to evaluating OpenAI’s Decisions API once it’s broadly available. Details + leaderboard + code: jacoblincool.github.io/StreamDecision…

That model was Kev. We first tested Kev-4B. Its result was promising enough that we went back for Kev-9B and Kev-27B. Kev-27B reached 62.15 on SDB, the only open-weight model we tested so far in the same range as Jev at 59.63. Really impressive work, @jaredpalmer.

BUILD-AND-FIND uses a builder and finder protocol. One agent builds a repo, and another recovers the intended design choices. We measure recovery quality and inspection effort, with the latter reflecting repository-level understandability. 📄 arxiv.org/abs/2605.06136

Excited to share that BUILD-AND-FIND has been accepted to #NeurIPS2026! As agents take on long-term codebase maintenance, working code is only part of the story. We ask whether future agents can recover the intent behind earlier design choices and how much effort that takes.

The core is also open source and available as a framework-agnostic browser package on npm: hear-my-site Give it browser-owned audio streams and a Gemini or OpenAI live transcription provider, and it exposes live transcripts to agents through WebMCP. npmjs.com/package/hear-m…

The demo uses WebRTC + @CloudflareDev Workers & Durable Objects. Media stays P2P. The server never handles audio or transcripts, only signaling and short-lived credentials. …y-site-meeting-demo.jacob.workers.dev

I've played with WebMCP for the last 3 days, and I think it has huge potential when the browser is directly part of the harness. But it's clearly an evolving standard. The biggest gap is multimodal resources: it only works with strings. That's what Hear My Site tries to solve.