EvalOps @EvalOpsDev

Empowering builders to ship reliable AI evalops.dev San Francisco, CA Joined July 2025

Tweets

14
Followers

117
Following

1
Likes

0

EvalOps @EvalOpsDev

2 days ago

sounds like a Wes Anderson film but it’s actually just us trying to fix AI reliability

0 0 0 83 0

Download Image

EvalOps @EvalOpsDev

3 days ago

“We’ll just build our own eval system.” Sure. Or… just use the one you were about to reinvent. Thanks, Gemini. (not pictured: a whole lot of parquet)

0 0 0 324 0

Download Image

EvalOps @EvalOpsDev

6 days ago

Lesson learned: don’t ask an LLM to do marketing memes unless your campaign theme is “existential dread.”

0 0 0 350 0

Download Image

Tired of LLMs arguing with no ground truth? Cognitive Dissonance DSPy links DSPy agents with Coq proofs – translating claims to formal specs, detecting conflicts, and resolving them with math. Move from “probably right” to provably right. 🔥 github.com/evalops/cognit…

7 36 236 17K 265

EvalOps @EvalOpsDev

a week ago

Multi-agent AI: 'Let's vote on the truth!' Formal verification: 'Or we could just prove it?' New framework resolves AI disagreements with theorem proving. No more believing the confident liar. Open source: github.com/evalops/cognit…

0 1 4 512 1

EvalOps @EvalOpsDev

2 weeks ago

You don’t build the eval layer for AI the slow way. EvalOps: +201k net LOC in 22 days. The system that fixes the system doesn’t wait.

Jonathan Haas @HaasOnSaaS

2 weeks ago

You don’t build the eval layer for AI the slow way. EvalOps: +201k net LOC in 22 days. The system that fixes the system doesn’t wait.

0 0 1 387 1

0 0 0 151 0

EvalOps @EvalOpsDev

2 weeks ago

Stop guessing if your prompts are better. Start testing. EvalOps spins up A/B tests between GPT-4, Claude, Gemini—whatever you’re running—in minutes, not weeks. Hypothesis in. Dataset in. Variants out. Clear winner. Ship evals like you ship code.

0 0 1 398 3

Download Image

EvalOps @EvalOpsDev

2 weeks ago

Your evals should run more reliably than your sleep schedule. With EvalOps Schedules: set daily, weekly, or custom cron jobs. Models drift. Humans forget. Schedules don’t. ✨ Try it: app.evalops.dev/schedules ✨

0 0 0 424 1

Download Image

EvalOps @EvalOpsDev

3 weeks ago

We've built bandit_dspy, a DSPy teleprompter that integrates Bandit for static analysis feedback during LLM program optimization. It allows you to define a security metric and compile DSPy modules to favor code generation patterns that pass SAST checks. github.com/evalops/bandit…

0 0 3 739 2

EvalOps @EvalOpsDev

3 weeks ago

Friendly fire for models: provoke max disagreement, measure who holds up. Starting with the eternal question—hot dog vs sandwich. This is @EvalOpsDev in the wild

Jonathan Haas @HaasOnSaaS

3 weeks ago

Friendly fire for models: provoke max disagreement, measure who holds up. Starting with the eternal question—hot dog vs sandwich. This is @EvalOpsDev in the wild

3 0 8 984 0

Download Image

0 1 3 366 0

EvalOps @EvalOpsDev

3 weeks ago

🚨 Studying AI "echo chambers" in real-time Our Folie à Deux experiment shows what happens when AI systems optimize for agreement rather than truth. Spoiler: it's complicated. 🔬 Open source framework: github.com/evalops/folie-…

0 0 1 662 1

EvalOps @EvalOpsDev

3 weeks ago

Stop hand-crafting evals. With EvalOps + DSPy, just describe your scenario and we auto-generate test cases for you. From “refund requests with angry customers” → a full test suite in seconds.

0 3 37 14K 39

Download Image

EvalOps @EvalOpsDev

a month ago

🤝 New in EvalOps: Full model provider integration You can now securely connect: • OpenAI • Anthropic • Google AI • Claude • Cohere • OpenRouter • Groq • Replicate • Hugging Face ...and more. 🔐 Keys encrypted & stored server-side ⚡ Instant plug-and-play for evals