Scholar + Therapist • Freshprogresswellness @freshprogress

Scholar/therapist | #phdstudent Researching Growth After Trauma • Lovely & Fresh Podcast. Views are my own. Want a session: [email protected] bit.ly/1C0YRce Joined June 2015

Tweets

10K
Followers

492
Following

1K
Likes

21K

HK Motte @kman62

6 days ago

How Anything Uses Evals to Build Better AI Agents create.xyz/blog/how-creat…

0 1 0 30 1

Are you obsessed about evals?! Work with Anthropic on: • ai safety level assessments • advanced capabilities and safety metrics • infra, tools, and methods for building evals anthropic.com/news/a-new-ini… Here's are some principles of good evals:

2 29 235 19K 214

Download Image

ben (is hiring engineers) @benhylak

3 weeks ago

i just wrote my first long-form post on evals. what they are, where they work, where they don't, and what i think the future looks like. if you're building an ai product, you should read this.

30 27 495 104K 721

Download Image

Hamel Husain @HamelHusain

3 months ago

I use lots of eval tools. One of my favorites is open source & created by JJ Allaire ( @fly_upside_down ) called Inspect. Inspect is used by Anthropic, Deepmind, Grok and others. I invited JJ to our evals course and took detailed notes: hamel.dev/notes/llm/eval… 1/3

8 28 266 25K 380

elvis @omarsar0

3 weeks ago

Improving the reliability of multi-agent systems is extremely hard. And it's a must when deploying AI agents. Galileo offers one of the most comprehensive agent eval solutions I've seen. Here is how they help devs and huge companies deploy reliable AI agents:

10 12 93 16K 106

Download Gif

Eugene Yan @eugeneyan

a year ago

What are some good resources on automated eval/llm-as-judge? • Constitutional AI - Harmlessness from AI feedback: arxiv.org/abs/2212.08073 • ChatGpt for summary factual inconsistency: arxiv.org/abs/2303.15621 • Human-like summary eval with chatgpt: arxiv.org/abs/2304.02554 •…

11 48 263 50K 462

EVAL Engine @Eval_Engine

8 months ago

💡EVAL Engine: Why It Matters AI agent evaluation tools are increasingly important in Web2 but are not yet as prevalent as in Web3. They are essential for improving customer support, search algorithms, content moderation, and fraud detection. Challenges include data silos,…

2 16 42 8K 0

Tae Soo Kim @tae_skim

2 years ago

Improving LLM outputs is difficult because evaluating these outputs is difficult... Introducing EvalLM ⚗️ - an interactive system for evaluating LLM outputs on user-defined and application-specific criteria 📏 🔗: evallm.kixlab.org 📄: arxiv.org/abs/2309.13633 (1/n)

6 51 278 47K 145

Download Video

Anoop Thomas Mathew @atmb4u

2 days ago

1/ You’ve shipped SaaS. You know APIs, QA, SLOs. Agents feel the same… until they don’t. Agentic Software = eval_loop((context, tools) → outcome) Took 3 weeks to condense my learnings so far. Here’s the small set of switches I wish I’d flipped sooner. 🧵

1 1 2 12K 0

Download Image

Diamond Bishop 🤖 @diamondbishop

4 months ago

hopped on the AI Adoption Playbook podcast recently and had a good chat. learn about AI eval and real applied AI agent use cases while getting to listen to my voice for almost an hour, win/win

2 2 18 16K 4

Download Video

Peter Yang @petergyang

a month ago

"Everyone says AI evaluations are important, so let's actually build one live from scratch.” Here's my new episode with @_amankhan (Arize) where we build AI evals for a customer support agent live, including: ✅ Creating the eval criteria ✅ Labeling the golden dataset ✅…

5 10 92 37K 167

Download Video

Logan Graham @logangraham

a year ago

Evals are critical for measuring AI capabilities + safety. If you're building evals, I'd like you to apply for our support. Here are some we wish existed. anthropic.com/news/a-new-ini…

6 60 350 108K 274

Lance Martin @RLanceMartin

4 months ago

Agents from scratch This repo covers the basics of building agents: + Fundamentals + Build an agent + Agent eval + Agent w/ human-in-the-loop + Agent w/ long-term memory Builds to a deployable agent to run your email Code (all open source): github.com/langchain-ai/a…

9 110 708 52K 1K

Download Image

Andrew Ng @AndrewYNg

7 months ago

New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with @arizeai and taught by @JohnGilhuly, Head of Developer…

137 236 1K 124K 1K

Download Video

Paweł Huryn @PawelHuryn

4 weeks ago

AI Evals are critical for AI PMs and Engineers. But many still confuse them with unit tests. By popular request, I removed a paywall from the @HamelHusain's Mastering AI Evals, A Complete Guide. Key insights, free guide, and massive resources: 🧵

9 63 321 30K 649

Download Image

Aadit Sheth @aaditsh

6 days ago

What AI Can't Do docs.google.com/document/d/19l…

13 35 201 110K 286

Aakash Gupta @aakashg0

3 days ago

80% of AI PMs lack this skill: AI security. Here's a free masterclass from Okta's VP of PM ($16B): 1. What AI PMs Need to Know about Security 2. How to Protect Yourself (If You Are Hacked) 3. How to PM in the AI Era 🎥 Watch Now: youtube.com/watch?v=4dWbwG… Or Listen: 🎧…

0 3 6 2K 20

Download Video

Chris Laub @ChrisLaubAI

3 days ago

Claude > ChatGPT Claude > Grok Claude > Gemini But 99.9% of the users don't know how to get 100% accurate results from Claude. To fix this you need to learn how to write prompts Claude. Here's a complete guide on how to prompts for Claude using XML tags to get best results:

29 37 324 33K 563

Download Image

Chris Laub @ChrisLaubAI

3 days ago

Hiring isn’t real anymore

0 1 4 386 1

Download Image

🔥 Matt Dancho (Business Science) 🔥 @mdancho84

a day ago

🚨The AI agent handbook Google just dropped a 46-page playbook on how to build and use agents. This is what you need to know (and how to get it 100% free):