Agents do deep research, but they don’t generalize: each task needs its own definition of quality.
Small eval beats a big rewrite. Read how we made it happen. jetty.io/guides/small-e…#evals#hermesagent#aievals
When everyone is writing evaluations, the tools need to change. I've been running experiments with audio and video generation to illustrate what hill climbing looks like. DM me for demo.
This is great to see! We explored using harbor as an integration format and it works great for some teams, but found that it was often a bit overkill for a simple eval task. With @_jettyio we made the choice of enabling users bringing their own docker containers to AI Runbooks, embedding the evals in the task description and beefing up our skill creation pipeline.
We've updated our @eve integration to support the new built-extension format. Bring your own agent and have @_jettyio run evals online. Check out our bandit example: github.com/jettyio/jetty-…
A green test suite is a vector of zeros: every passing test says "none of the failures I already knew about happened." It has no direction. An eval rubric does. That's why evals are not tests.
Evals are not tests:
open.substack.com/pub/lebensold/…
1K Followers 985 FollowingHelping you hill-climb your agentic system @ Jetty. AI has an evaluation problem and I’m trying to fix it. PhD in privacy and ML, ex-Meta AI, Google.
4K Followers 330 FollowingCo-author of Apprenticeship Patterns. Learning to listen. Making things happen. Telling better stories.
Devrel at @SierraPlatform
33K Followers 448 FollowingCo-founder, Recursive. Professor, CS, U. British Columbia. CIFAR AI Chair, Vector Institute. | ML, AI, deep RL, deep learning, AI-Generating Algorithms (AI-GAs)
47K Followers 397 FollowingAPI orchestration platform for AI agents and modern applications. GraphQL and REST connectivity. Trusted by enterprise leaders. 🚀
57K Followers 92 FollowingCreator of @elixirlang. Chief Adoption Officer at @dashbit, where we build https://t.co/FK8F4URbVG and https://t.co/xncEVrvWml.
322K Followers 5K FollowingCloudflare is the world’s leading #ConnectivityCloud, and we have our eyes set on an ambitious goal — to help build a #BetterInternet.
1K Followers 985 FollowingHelping you hill-climb your agentic system @ Jetty. AI has an evaluation problem and I’m trying to fix it. PhD in privacy and ML, ex-Meta AI, Google.