Suren Hiraman @_surenh

x.com/i/article/2031…

Which models can your machine actually run? Magnitude now answers that. The new model catalog: - Profiles your hardware automatically - Estimates tok/s for every model before you download - Recommends the best models for your machine Pick one and Magnitude handles the rest: - Downloads the model and quant from Hugging Face - Loads it into the built-in inference engine - Configures speculative decoding (MTP, DFlash, etc.) - Sets concurrency based on your memory Try it on your hardware: npm i -g @magnitudedev/cli

On recruiting great ppl: - great people are (a) rare, and (b) always have tons of options - therefore once you decide a candidate is great you cannot hesitate or delay. You must drop everything and rearrange heaven and earth to get them - there are small things (sending flowers), medium things (comp), and big things (their identity and dreams) - you want the barbell strategy-- put all your energy into small things and big things. Leave medium things till the end Small things: - ppl massively underestimate the power of small gestures. TL;DR: love-bomb the candidate - spend more time than money. Expense accounts are abundant but time is scarce. It's an expensive signal no one can fake - stay in close contact over text and phone - send them a first edition used book or an analog oscilloscope or some other random thing they mentioned in passing they like. Make these unique, everyone sends wine - invite them to computer history museum or moma or something. Spend time with them! Treat these meetings as dates, make them unique. Everyone buys dinner - be persistent but not annoying. Sometimes ppl need time alone, you should respect that. Big things: - there is a gap between what the candidate is doing in life and how they see themselves in the privacy of their dreams. Your job is to help them bridge this gap - to bridge the gap you must first understand it. Take the candidate on walks to museums and parks. Walks help the candidate drop their guard so you can learn about them. Try hard to get to know them, to understand their dreams - use your mission, your culture, your people, your company to paint a world for them in which the gap no longer exists. You may need to adjust the role or even change it altogether. For great ppl it is worth it - don't assume ppl want what you want, dreams can take all kinds of shapes. Maybe they want their work to matter, maybe they want a bigger team, maybe they want to code and avoid meetings, maybe they want to be in the center of the action, maybe they want to learn, maybe all or some or none of the above. Find out! Medium things (aka closing): - you've constructed a world the candidate is excited to live in. The purpose of this phase is to remove mundane obstacles to make this world easy to enter - I don't think I ever sold a candidate on a dream and then lost them on comp. Ppl just want to be compensated fairly and work out some life constraints - I don't have much to add to all the comp advice out there. My philosophy on comp/closing is to get this part done and move on, don't penny pinch too much, and give more equity --- Failure modes/don't do this: - do not ignore candidate's spouse/parents. They have an outsized role in the decision. Most of what I said about the candidate applies to them too (love bomb them, understand their dreams, etc.) - counterintuitive, but do not talk about the company or even the mission too much. The candidate and their dreams are front and center. At this stage you, the role, the mission are only relevant to the degree they can make candidate's dreams come true - same w/ perks, what the day is like, etc. Don't blindly talk about these-- use them as tools to reenforce how the candidate seems themselves - don't get mad if ppl reject you, it happens. Maybe it's not a fit for them now but will be a great fit four years from now. Maybe they become a customer. Maybe you work for them ten years later. Play the long game.

Agents across the SDLC are a step toward a software factory. Then the harder work is engineering the system around them. My take on the path to bounded L4 software engineering, and why engineering teams need to BUILD and OWN that SYSTEM. x.com/i/article/2112…

En la chamba empezamos usando el harness con SDD: para cada feature generábamos una spec, luego un plan y luego una lista de tareas. Pero vimos que ese approach tiene sus límites: los documentos se desactualizan rápido y dejan de reflejar lo que hace el código, se va mucho tiempo (y tokens) en escribirlos y revisarlos, y muchas veces el resultado no es mejor que trabajar con el agente paso a paso. Así que lo fuimos ajustando con lo que ha ido recomendando la comunidad y hemos experimentado: Specs vivas solo para lo que dura y cruza servicios (contratos, decisiones de arquitectura, reglas de negocio) Plan mode para el día a día de cada feature Tests y checks como la verdadera fuente de verdad Si quieren explorar algo de este enfoque: Los tres niveles de SDD (spec-first, spec-anchored, spec-as-source) explicados por Birgitta Böckeler: martinfowler.com/articles/explo… Por qué el SDD pesado perdió contra plan mode, y dónde la spec sí sigue valiendo: instil.co/blog/spec-driv… SDD adaptativo + harness + TDD: blog.adrianhall.uk/posts/2026/202… Nosotros lo usamos porque trabajamos en features que tocan varios servicios y delegamos bastante trabajo a agentes en paralelo, ahí un malentendido nos cuesta días, y tener los contratos claros nos lo evita. No es la receta para todos, y lo seguimos puliendo sobre la marcha.

Build your own harness, folks! Highly recommended reading on Ramp's own agent harness and why they built one. I've seen this closely. More hardcore engineering teams are realizing how important it is to own the harness. I expect more full-stack AI companies to follow. Ramp quickly realized they could move faster and better with their agent harness. Benefits: - Model-and-provider agnostic - Shipping faster as the harness is closer to other teams - Owning the observability stack (a must given the complexity of agentic systems today) Their Laminar harness supports secure conversation storage, MCP for standard tool calling, durable execution, code execution sandboxes, observability tools, and more. When you own the harness, you can co-evolve it to meet your needs. The foundations don't change much; only the tooling and behavior need extending, so make the harness malleable from the start. Ramp also made several interesting design choices, such as using sandbox-as-a-tool, leveraging Boring and a standard stack (Postgres, S3, etc.), always-open agent sessions, and going all-in on cloud.

rahul @rahulgs

4 days ago

after much experimentation, we landed on a single set of rails that powers our agentic systems across the company - background agents, async agents, live chat: 1. stack - accounting harness (ramp.com/stack) 2. glass - general purpose coworker (ramp.com/glass)

30 23 717 71K 940

Congrats to @nvidia for creating the best open-weight model on document parsing so far. Nemotron 3.5 Parse takes the #1 spot on ParseBench, our comprehensive real-world benchmark for document understanding. The full list consists of 150+ frontier models, open-weight models, and specialized solutions. It does this while remaining sub-1B parameters so it's also blazing fast ⚡️ Big thanks to @JensenHuang for the shoutout at GTC Berlin! Nemotron release: huggingface.co/blog/nvidia/ne… ParseBench: parsebench.ai

We love papers that go deeper down the stack. This is a good example that shifts from agentic context engineering to agentic cache engineering. They also apply recursive self-improvement to cache policies to further reduce cost by 7.1% over expert design. Specifically, this work focuses on KV cache management in long-running agents. When an agent summarizes or trims its history, the prefix change invalidates the cache after it, and the retained context must be prefilled again. Cache engineering gives the agent two operations on its own KV cache. One removes entries from future attention, and the other shifts the positions of the keys that remain. The trajectory text stays unchanged. This essentially provides a programmable interface for agents to define and refine their own cache engineering strategies. A rolling drop of old tool responses scores 1.7 points higher and costs 27.7% less than threshold-triggered summarization. Their AKV serving engine adds up to 1.2x throughput and 1.5x faster P95 latency. arxiv.org/abs/2610.28577 Chat with Paper: academy.dair.ai/papers/cache-e…

I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: cameronrwolfe.substack.com/p/subjective-e… Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.

keyboard_arrow_left Previous keyboard_arrow_right Next

Build specialized harnesses, folks. This is a banger paper from Google Cloud AI Research on self-evolving agent harnesses. (bookmark it) They introduce OO-Harness, which raises AutomationBench from 61.6% to 77.4% with the same Gemini 3.5 Flash executor. It also beats Meta-Harness on all three benchmarks they test. More details below. Instead of evolving one global harness for every task, an evolution agent builds a class hierarchy of specialized harnesses from execution records. The design has three parts. > Subclasses inherit the base harness and override only the methods a task cluster needs > Design lessons record which changes helped and which ones failed > A router reads the task and the base run, then picks a subclass or keeps the base result On Workspace-Bench, the score goes from 25.5% to 39.5%, and on Toolathlon-GYM strict pass goes from 31.2% to 37.5%. Spending extra inference on best-of-K sampling or self-refinement does not reliably help. Inheritance also improves results. On Workspace-Bench, harnesses built as subclasses score 4.96 points higher than the same evolved code copied into flat classes. arxiv.org/abs/2610.28380 Chat with Paper: academy.dair.ai/papers/object-…

99% of Architect interviews will ask you about these patterns And how to design systems with them 👇 In an interview, you need to explain when you'd pick a pattern and what the trade-offs are. Here are 12 patterns every architect should know: 𝟭. 𝗔𝗣𝗜 𝗚𝗮𝘁𝗲𝘄𝗮𝘆 One entry point between clients and backend services. It routes requests and handles auth and rate limits. ↳ Use when: many services need one API URL for clients 𝟮. 𝗦𝗲𝗿𝘃𝗶𝗰𝗲 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆 Services find each other through a registry instead of hardcoded addresses. ↳ Use when: instances scale and IPs keep changing 𝟯. 𝗣𝗼𝗶𝗻𝘁-𝘁𝗼-𝗣𝗼𝗶𝗻𝘁 𝗔𝘀𝘆𝗻𝗰 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 One service sends a message to a queue. One consumer processes it at its own pace. ↳ Use when: background jobs or spiky traffic 𝟰. 𝗣𝘂𝗯𝗹𝗶𝘀𝗵/𝗦𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲 A publisher sends an event to a broker. Many subscribers react to it on their own. ↳ Use when: one event should trigger work in several services 𝟱. 𝗢𝘂𝘁𝗯𝗼𝘅 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 Save the event in an outbox table in the same transaction as your data. A worker publishes it later. ↳ Use when: losing an event is not an option 𝟲. 𝗦𝗮𝗴𝗮 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 Split a distributed transaction into local steps. On failure, compensating actions undo earlier steps. ↳ Use when: a process spans services, like order → payment → shipping 𝟳. 𝗖𝗤𝗥𝗦 Separate models for writes (commands) and reads (queries). ↳ Use when: reads and writes have very different load or shape 𝟴. 𝗦𝗵𝗮𝗿𝗱𝗶𝗻𝗴 Split data across many databases by a shard key. ↳ Use when: one database can't handle your data size or write load 𝟵. 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 Copies of data on several nodes. The primary takes writes, replicas serve reads. ↳ Use when: you need high availability or faster reads 𝟭𝟬. 𝗦𝗶𝗱𝗲𝗰𝗮𝗿 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 A helper container next to your app for logging, monitoring, or networking. ↳ Use when: services in different languages need the same infra features 𝟭𝟭. 𝗦𝘁𝗿𝗮𝗻𝗴𝗹𝗲𝗿 𝗙𝗶𝗴 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 Replace a legacy system piece by piece until the old one can retire. ↳ Use when: a big-bang rewrite is too risky 𝟭𝟮. 𝗔𝗻𝘁𝗶-𝗖𝗼𝗿𝗿𝘂𝗽𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿 A translation layer between your domain and an external or legacy system. ↳ Use when: a third-party or legacy model would leak into your domain 📌 Interview tip: Every pattern here has a cost. Name the trade-off before the interviewer asks → it shows real project experience. 👉 I wrote a deep dive on all 12, with use cases and trade-offs: antondevtips.com/blog/12-essent… Which of these patterns have you never used?

"Mathematical Theory of the Transformer" is an extensive, 825-page mathematical monograph exploring the theoretical foundations of Transformer architectures. It covers a remarkable range of topics, including embeddings and positional encoding, self-attention and multi-head attention, matrix decompositions, linear algebra, spectral analysis, information geometry, training dynamics, redundancy, optimization, and incremental Transformer architectures. This is an advanced, mathematically rigorous read for anyone interested in understanding the mathematical structures underlying Transformer-based AI systems, including modern generative AI. doi.org/10.5281/zenodo…

Your AI coding agent could understand an entire codebase without reading every file Graphify turns code, docs, and other project files into a searchable knowledge graph, reportedly using 71.5x fewer tokens per query while running locally without embeddings or API costs. Stop paying your AI agent to rediscover the same codebase every session. Repo: github.com/Graphify-Labs/…

If you know nothing about AI agent evals, start with this Anthropic article. Covers eval design, different types of graders, failure modes, and how to actually evaluate AI agents. anthropic.com/engineering/de…

"it's underrated how hard it is to build agents!" great convo w/ @hwchase17 and @HarryStebbings on @twentyminutevc an agent = model + context + harness, and you can improve an agent by tweaking any of the three. a few takeaways: 1. the highest roi thing you can do is start by optimizing context (then the harness, then the weights); this is where domain knowledge lives and that's incredibly valuable 2. the story of continual learning is: run an agent, collect traces, aggregate learnings, fix something, measure the fix, loop! the fix can land in any of the three layers. those fixes all look different but the cycle is the same. 3. memory is still massively unsolved, but also massively important. the hardest part is distilling learnings from traces which is incredibly domain specific. 4. when you build agents, you're probably worried about a few things: accuracy, cost, latency, and security. we've seen these themes confirmed year after year in our dev surveys 5. when we think about reducing cost, the pipeline looks like adding visibility into spend, putting controls in place, and only then pushing on agent optimizations. this is how we reduced our coding agent spend @LangChain 6. model routing, which is great for cost controls, is most effective in the harness (where the context needed for a routing decision lives) 7. open models are used for more narrow tasks right now (small parts of the agent loop OR for very specific domains w/ post-training), but as they improve they'll power more core agent loops 8. we're looking at a future with both horizontal agents and vertical agents. horizontal agents will use the vertical agents, and infra needs to change to support this load (agents can do background work at a rate that humans cannot). youtube.com/watch?v=1jDCPE…

Tomorrow morning, I’ll release a blog that outlines a detailed roadmap for setting up evals on subjective / open-ended tasks. The blog also covers a variety of stats / ML tricks for reducing noise and increasing confidence in your evals. Here is a list of blogs, papers, and other resources (written both by me and by others) I often return to when thinking about this topic: - High-quality human data: lilianweng.github.io/posts/2024-02-… - Human evaluation guidelines: eugeneyan.com/writing/labeli… - LLM-as-a-Judge core concepts: cameronrwolfe.substack.com/p/llm-as-a-jud… - LLM-as-a-Judge practical guide: hamel.dev/blog/posts/llm… - LLM-as-a-Judge best practices: eugeneyan.com/writing/llm-ev… - Rubric-based judges (RL focus): cameronrwolfe.substack.com/p/rubric-rl - Rubric-based judges (eval focus): arxiv.org/abs/2603.00077 - Stats for LLM evals: cameronrwolfe.substack.com/p/stats-llm-ev… - Adding error bars to evals: arxiv.org/abs/2411.00640 - Anatomy of a benchmark: cameronrwolfe.substack.com/p/llm-bench - Agent evals: cameronrwolfe.substack.com/p/agent-evals - Full ecosystem for agent evals: anthropic.com/engineering/de… Would highly recommend reading all of these if you're interested in building better evals. Each of these blogs also link to further papers / resources on each topic that are incredibly useful.

I have been fine-tuning LLMs for over 3 years now. Here are 5 LLM fine-tuning techniques, explained with visuals: Traditional fine‑tuning is impractical for LLMs (billions of params; 100s of GB). Since this kind of compute isn't accessible to everyone, parameter-efficient fine-tuning techniques (PEFT) were built. Before we go into details of each technique, here's some background that will help you better understand these techniques: LLM weights are matrices of numbers adjusted during fine-tuning. Most PEFT techniques involve finding a lower-rank adaptation of these matrices, a smaller-dimensional matrix that can still represent the information stored in the original. Now with a basic understanding of the rank of a matrix, we're in a good position to understand the different fine-tuning techniques. (refer to the image below for a visual explanation of each technique) 1) LoRA - Add two low-rank trainable matrices, A and B, alongside the weight matrices. - Instead of fine-tuning W, adjust the updates in these low-rank matrices. Even for the largest LLMs, LoRA matrices take up a few MBs of memory. 2) LoRA-FA While LoRA significantly decreases the total trainable parameters, it requires substantial activation memory to update the low-rank weights. LoRA-FA (FA stands for Frozen-A) freezes matrix A and only updates matrix B. 3) VeRA - In LoRA, low-rank matrices A and B are unique for each layer. - In VeRA, A and B are frozen, random, and shared across all layers. - Instead, it learns layer-specific scaling VECTORS (b and d) instead. 4) Delta-LoRA - It tunes the matrix W as well, but not in the traditional way. - Here, the difference (or delta) between the product of matrices A and B in two consecutive training steps is added to W. 5) LoRA+ - In LoRA, both matrices A and B are updated with the same learning rate. - Authors of LoRA+ found that setting a higher learning rate for matrix B results in better convergence. ____ Find me → @_avichawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

Don't waste all your time improving how your AI agents reason. Do this instead... Spend more time defining what they should reason about. Even the smartest model can make a bad decision if the options are poorly defined. This is why I like using Jev and ontologies together... Jev by @typesafeai turns unstructured inputs into structured decisions with calibrated confidence scores. Instead of asking an LLM to reason through every decision, you define the possible outcomes in code. Jev selects an outcome and returns its confidence. And the ontology defines what those outcomes mean. Imagine an AI agent handling customer support tickets. The ontology defines: • Which issues belong to billing versus technical support. • How customers, subscriptions, and invoices relate. • Which categories should never overlap. Jev classifies tickets against those categories. Based on its confidence, your agent can: 1. High confidence → Execute an action. 2. Medium confidence → Escalate to an LLM. 3. Low confidence → Request human review. So expensive reasoning is reserved for cases that need it. But what if your ontology is wrong? Jev can confidently choose the wrong category if the options are poorly defined. You still need evals to identify failures, improve definitions, and test changes before deploying. As those definitions evolve, version them like your prompts, datasets, and code. The takeaway: Building better AI agents isn't always about making the model smarter. Sometimes it's about defining its world more clearly. P.S. How are you defining the decision space for your AI agents?

How to write your own skill, and how to check whether an external skill helps you at all? There are millions of skills available, hardly any was ever tested, and most were vibe-written. The whole skills industry is subjective. Very proud of this work. x.com/i/article/2112…

Learn to build evals, folks. Highly recommended reading on using decision models for evals. From our internal evals, System One models like Jev are a great fit for LLM Judges. If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges. (bookmark it) 1) Jev-as-a-Judge for Agent Evaluations - academy.dair.ai/dashboard/reso… 2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - academy.dair.ai/labs/jev-as-a-… Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - arxiv.org/abs/2609.26550

Hamel Husain @HamelHusain

4 days ago

New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. x.com/i/article/2112…

187 255 2K 314K 4K