hubt @hubt

Infra, systems, ops, SRE, DevOps. @fidian iPhone: 37.286926,-121.832199 Joined July 2008

Tweets

2K
Followers

87
Following

76
Likes

104

hubt @hubt

3 days ago

Depressing thought: What if the massive capital and compute of the AI buildout is just used to fund a permanent cybersecurity offense versus defense war without improving applications for users?

0 0 1 14 0

View Details

@fidian published our TerminalBench-Fn leaderboard. fidian.ai/leaderboard/ GPT-6 Sol tops the leaderboard because models like Astra and Fable refuse some cybersecurity tasks. For value, choose GPT-6 Luna.

0 3 4 310 2

View Details

hubt @hubt

a month ago

If you are considering going you can register at sreday.com/2026-san-franc… and get half off with code SRE50

0 0 2 54 0

View Details

hubt @hubt

a month ago

Excited to be speaking at SREDay in SF next Fri Oct 2. Talk: "Beyond Unit Tests: Bringing Intelligent Agent Evals into CI/CD Pipelines." Let me know if you're going to be there and we can talk SRE/DevOps/Evals.

2 3 6 435 2

View Details

hubt @hubt

a month ago

People using AI agents to negotiate prices will be in for a shock when a customer service bot does adversarial upsells. "Claude: did you lower my car payment $20 for more tokens by taking a 10yr loan at 20%?" "Two mistakes here and one I should be honest about.."

1 0 6 166 0

View Details

hubt @hubt

2 months ago

As a Michigan fan, I have to applaud OpenAI's Navier-Stokes solution for stealing the win in a competition they had no right to be in.

0 0 0 31 0

View Details

hubt @hubt

2 months ago

@kpb_in_acad @xeophon @abeirami Fidian TB analyzed tasks for issues like: unpinned versions, hidden requirements, and solution leaks. This task looks like it's designed around a bug which is patched upstream. Seems like models are doing the right thing and the task should be tightened, not the verifier.

1 0 2 57 0

View Details

hubt @hubt

2 months ago

@PersonalFinNerd @abeirami

0 0 2 38 0

View Details

hubt @hubt

2 months ago

@ipezyGJ @abeirami yes. with a CI +/- 7 pts and a 10 point spread, the 3-10 group is close enough that trying to find precise ordering within the group is probably not worth it, consider it a tier group. I'd rather use cost to rank in the tier.

0 0 1 51 0

View Details

hubt @hubt

2 months ago

@PolSciEnjoyer @abeirami pass@3 is a useful metric to disentangle whether a model is capable from whether it's reliable, eg: GLM5.2 goes from 68% pass@1 to 92% pass@3 because it's capable, but not reliable. but yes pass@1 is a generally better metric, that's why it's our primary measurement.

0 0 1 71 0

View Details

hubt @hubt

2 months ago

Here's what I've been working on the past few weeks. @fidian. Benchmarking is hard, expect mistakes along the way. x.com/fidian/status/…

Fidian @fidian

2 months ago

Newly released models repeatedly appear near the top on the @terminalbench 2.1 leaderboard. Are these models actually on par with frontier models on terminal tasks? We took the top 20 models from that leaderboard and ran them on TB-fn. TB-fn reworks the same 89 tasks by adding

2 13 41 647K 17

0 2 4 365 0

View Details

hubt @hubt

2 months ago

My tests of 0x alpha say it's nothing special. GLM 5.3 seems quite good though.

0 0 1 83 0

View Details

hubt @hubt

2 months ago

@lauriewired I found my two AWS accounts were bidding against each other sometimes. Without transparency about real availability(instance type and region) it's a bad market. AWS later switched to capping prices with unavailability rather than letting customer manage wild price swings.

0 0 3 800 0

View Details

hubt @hubt

3 months ago

@chanian right. part of the question is: "how would average sites defend themselves from this"? And the answer cannot be "you must have active secops equipped with closed, trusted frontier models"

1 0 0 21 0

View Details

hubt @hubt

3 months ago

@jeremybernier I'm guessing they were looking for different solution, more like a consistent hashing, CARP, or rendezvous hashing solution?

1 0 3 4K 1

View Details

hubt @hubt

4 months ago

saw the video and had to make this.

0 0 3 47 0

View Details

hubt @hubt

4 months ago

Trump went to Netanyahu and asked if he should attack Iran. Netanyahu said "if you attack Iran, then a great empire will fall", and here we are.

0 0 0 47 0

View Details

hubt @hubt

7 months ago

Woo Michigan!

0 0 1 75 0

View Details

Geppetto San Martín 🚬🌭𓅸 @SmalltimeJones

9 months ago

fuuuuuuuck, whatever undergrad research assistant submitted this summary of their team's paper to phys,org, they knocked the headline out the goddamn park