EvalOps @EvalOpsDev

Empowering builders to ship reliable AI evalops.dev San Francisco, CA Joined July 2025

Tweets

34
Followers

299
Following

1
Likes

0

Jonathan Haas @HaasOnSaaS

2 days ago

LLM vendor: “Just quantization.” Reality: reward-hacked code, broken workflows, lost week. Companies: “nbd.” Users: 🙃🔥 Making this a thing of the past.

0 1 2 360 1

Download Image

🐙 Meet Mocktopus — multi‑armed mocks for your LLM apps! 🧪 Deterministic mocks for OpenAI‑style chat completions, tool calls & streaming. Make your evals deterministic, run CI offline, and record & replay 👉 github.com/evalops/mockto…

0 1 4 992 2

EvalOps @EvalOpsDev

a week ago

New @EvalOpsDev research drop: Metaethical Breach DSPy ⚖️ Can models be coaxed into breaking their own guardrails if you wrap harmful prompts in moral philosophy? Repo: github.com/evalops/metaet…

0 2 5 3K 6

EvalOps @EvalOpsDev

2 weeks ago

Most eval tools tell you how your model did. @EvalOpsDev + DreamCoder tells you why. By synthesizing programs from data, DreamCoder finds hidden structure, invents evals, and surfaces failure grammars you didn’t think to test. Evals that discover the evals.

0 0 0 471 0

Download Image

EvalOps @EvalOpsDev

2 weeks ago

We open sourced an ATS. Why? Because paying $5k for commodity software is rent-seeking. In the AI era, code-gen makes it possible to build internal tools that are fast, flexible, and actually deliver value—so SaaS can focus on outcomes, not overhead. github.com/evalops/EvalATS

0 0 0 502 1

EvalOps @EvalOpsDev

2 weeks ago

our Q3 strategy is simple: step 1. touch grass step 2. monetize grass step 3. become grass

0 0 0 491 0

EvalOps @EvalOpsDev

2 weeks ago

Now shipping: founder-style cold email that learns from your winners. Open-source, CLI, @DSPyOSS inside (really need that on a sticker...)

Jonathan Haas @HaasOnSaaS

2 weeks ago

Now shipping: founder-style cold email that learns from your winners. Open-source, CLI, @DSPyOSS inside (really need that on a sticker...)

4 6 105 12K 159

0 0 16 6K 7

EvalOps @EvalOpsDev

2 weeks ago

LLMs under pressure will sometimes override their own safety rules—trading reliability for speed. That tension—urgency vs. safety—is exactly what Override Cascade was built to measure.

Jonathan Haas @HaasOnSaaS

2 weeks ago

LLMs under pressure will sometimes override their own safety rules—trading reliability for speed. That tension—urgency vs. safety—is exactly what Override Cascade was built to measure.

1 1 8 2K 9

0 0 0 172 0

EvalOps @EvalOpsDev

2 weeks ago

What if we treated LLMs like compilers? We hacked on that idea → LLMCC: • JSDoc contracts + JSON schemas • Constrained decoding • Automatic repair attempts • Property tests for invariants Still early, but fun to explore. Repo here: github.com/evalops/llmcc

0 2 4 288 1

Jonathan Haas @HaasOnSaaS

3 weeks ago

YC partners are blunt for a reason: most startups die by default. Orbit Agent is our first shot at bottling that advice into something you can text (or use via the CLI) when you need it most. Part of the @EvalOpsDev push to build agents that reflect real workflows (and…

3 2 18 5K 11

EvalOps @EvalOpsDev

3 weeks ago

How we turn PRDs into evals: PRD lives as Markdown Comment “@Gremlin implement this” GH Action runs @DSPyOSS (parse→expand→validate→optimize) Bot opens PR with evals/*.yaml + REPORT.md (coverage + manifest)

Jonathan Haas @HaasOnSaaS

3 weeks ago

7 27 430 76K 757

1 2 13 2K 12

EvalOps @EvalOpsDev

3 weeks ago

PRD → evals → PR to generate evals from PRDs. It’s evals all the way down. 🐢🌀

1 1 32 64K 31

Download Image

EvalOps @EvalOpsDev

3 weeks ago

2025 is the year of the agent. 2025 is also the year agents need to be safe. That’s why we built Gremlin: the evaluation-native AI agent. Coming soon.