Mike Carter | Building with AI @AIProfitStackHQ

@emollick The dismissal made sense. Most people measured it on toy benchmarks. The real signal was watching it hold coherent reasoning across 20+ step problems where previous models would drift. Reliability at depth, not headline scores, was the unlock.

Skipping the failure log. Most people test for the happy path, see a clean output, and ship. Nobody is systematically testing for the subtle wrong. The vendor name that is slightly off. The line item that gets silently dropped. By the time a client catches it, it has been wrong for two weeks. The second mistake: no production monitor. You cannot manage what you cannot see.

Real question for the builders here: What is the most common mistake you see people make when deploying AI workflows to actual clients? Not in demos. Not in sandboxes. In production, with real data and real consequences. I will go first in the replies.

@gdb Quiet ships matter more than the marketing-friendly ones. The visual upgrades alone reshaped how we present outputs to clients. Most users will only feel the change after the third or fourth small update lands.

@sama Two million in tokens sounds like rocket fuel until you remember it is also rocket lock-in. Build for portability from day one. If your prompts, tools, and evals are model-agnostic, you keep the optionality. If not, you are renting your roadmap from one vendor.

I have been running an experiment for the past 3 weeks. Every time I build a new AI workflow for a client, I write down the exact moment it breaks. Not when it fails completely. When it does something subtly wrong that nobody would catch without knowing what to look for. I have 47 of those moments logged now. Patterns are starting to appear. Going to turn this into something useful.

@emollick The pattern I keep seeing matches this. Repetitive specialist work flows back in-house first because one person plus Claude can match a vendor's output. Deep domain expertise is stickier. Vendors built ten years of judgment that a new hire cannot reconstitute in a quarter.

The operators who survive the next 18 months will be the ones who can point to something still running.

Claude is now the most used AI in US business payments. First time OpenAI has been knocked off the top spot. I have been telling clients this for months. The product that wins enterprise is not always the one with the best demo. It is the one that does not hallucinate on billing data. Claude has been better at following long, specific instructions. That matters when the instructions are "do not touch line items that contain this specific vendor code." OpenAI is still dominant overall. But enterprise is a different game. And that game is shifting.

@gdb The audit trail matters more than the 87 unsubscribes. Most automation breaks silently on the 'are you sure' page and you only notice weeks later when the same sender keeps appearing. Flagging 14 for human review is the right shape. Bulk action, narrow attention budget.

@gdb Shipping from a phone forces ruthless scope cuts. The PR you would split into three on a laptop becomes one tight diff because typing on a small keyboard punishes ambiguity. Codex closing that loop on mobile means more small, focused changes hit prod each day.

The math is simple. The sensitivity is not. Anyone running production evals has watched accuracy swing 20 points on the same task by changing a single token in a prompt. That gap between describable mechanism and unpredictable behavior is why every team I know runs offline eval suites that look more like fuzz testing than benchmarks.

@swyx @gabrielchua Multifile context coherence is where the founder mode shows up. A month ago Codex still lost the plot halfway through a cross-file refactor. Now it holds state across the tree. The agentic excel demo is the headline, but reliability on long tasks is the real win.

@swyx @AIEMiami @SAPConcur Pun aside, transforming 6800 people on SAP is not really an AI problem. It is a workflow archaeology problem. The model is the easiest part. Everything else is decades of patched custom fields and approval rules nobody owns anymore.

@rohanpaul_ai Parity on benchmarks rarely translates to parity in production. The numbers that matter for shipping are tool-call reliability over long sessions, context cache behavior, and 99th percentile latency. Cannot wait to test those on real workloads, not curated evals.

@svpino @OmarchyLinux Nice setup. The HX 370's NPU is the part most people leave on the table. llama.cpp and ollama still do not target XDNA 2, so heavy lifting falls to the iGPU through ROCm or the CPU cores. 32GB caps you around 14B Q4 once you add context and an embedding model.

Capability budgets over approval prompts is the right call. The mental model shift that helped us most: treat the sandbox as a declaration of intent, not just a security boundary. When an agent knows its permission scope upfront, it actually writes better code because it stops attempting operations outside its lane.

@rohanpaul_ai The deployment shape framing is sharp, but adoption without measured workflow uplift becomes a vanity metric. Real question: does Qwen change throughput for doctors, or just live as another open tab? Reliable integration across those flows is the actual engineering.

@emollick Version-locking is the practical outcome. Ops teams I have seen freeze on whatever model was current at procurement, even when a better one ships a month later. Retraining humans costs more than the upgrade saves. Most run two generations behind on purpose.

@gdb Long refactors expose the drift problem. Models lose the original target after a few tool calls and start hill-climbing on whatever subtask is in front of them. A pinned /goal forces the agent to keep checking work against the actual end state.