Scott Swingle @bio_bootloader

Father of 3, building Mentat (the github native coding agent!) @AbanteAI, prev @DeepMind mentat.ai California Joined May 2022

Tweets

5K
Followers

10K
Following

3K
Likes

43K

Scott Swingle @bio_bootloader

2 weeks ago

> The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes. we give Mentat agents an "exit survey" to report harness bugs, even if they found a workaround

0 0 5 360 1

Scott Swingle @bio_bootloader

2 weeks ago

it's easy to get to 95% or even 100% of code "written by AI". But that's not the same as a 20x speedup! Complex tasks in real codebases aren't yet even 2-3x sped up yet. It's coming. Saying it's here already is premature.

Garry Tan @garrytan

2 weeks ago

211 182 2K 488K 459

0 0 12 1K 1

ja3k @ja3k_

2 weeks ago

I wrote a blog post about using the same agent to review and write code

1 2 8 565 2

Download Image

Scott Swingle @bio_bootloader

2 weeks ago

solution I came up with: PostToolUse hook, matching Bash, running a script that runs: `gh pr checks --watch --fail-fast` if claude's command contained `git push` or `gh pr create` And if that fails, shows Claude the output makes Claude Code feel a bit more like Mentat

Scott Swingle @bio_bootloader

2 weeks ago

4 0 7 2K 0

0 0 7 1K 1

Scott Swingle @bio_bootloader

4 weeks ago

after much testing and tuning, I think this is still the case At least for a coding agent like Mentat, Sonnet handles uncertainty and exploration better, and is more thorough that GPT-5. It's much more expensive and slower though, so there are real tradeoffs

Scott Swingle @bio_bootloader

2 months ago

0 0 17 4K 0

4 1 25 3K 2

Scott Swingle @bio_bootloader

a month ago

feels like a lot of these ideas will simply work now that we have powerful in-context learning AlphaGo needed RL because we didn't have a generally intelligent base. Now models can start teaching themselves

Andrej Karpathy @karpathy

a month ago

265 876 7K 900K 5K

0 0 6 676 1

Download Image

Scott Swingle @bio_bootloader

a month ago

after tons of testing with Mentat: Sonnet: Great default behavior, but ignores a lot of prompts. Needs strong pushes to change behavior GPT-5: Super steerable. Actually does what I say! But I have to tell it what to do more Depending on use case each can be good

0 0 8 500 1

Scott Swingle @bio_bootloader

a month ago

every day at midnight do all of OpenAI's api prompt caches invalidate simultaneously?

1 0 12 817 1

Download Image

Scott Swingle @bio_bootloader

a month ago

whoa OpenAI is giving GPT-5 on the API a prompt before the one I set?? they give it: - instructions on formatting (which is contradicting my own) - today's date - telling it that it's being used over the API - a bunch of other stuff not cool

11 8 158 32K 19

Scott Swingle @bio_bootloader

2 months ago

@repligate @mayfer @claudeai finally ran Opus on LoCoDiff and it's better than Sonnet on short context, but falls apart on higher contexts? very strange

0 1 7 501 1

Download Image

Grant♟️ @granawkins

2 months ago

Wrap your head around this. - Have git spit out a file's edit history - Line up all the diffs one-after-another - Feed it to a model and tell it to recreate the current (last) state At 100,000 tokens, Sonnet 4 succeeds 2/3 of the time, while GPT-5 does <5%.