Nick @hodnick

If you learn AI Infrastructure Engineering, you will never be unemployed again. Stage 1: Systems & GPU Foundations - Learn: C++, Rust, CUDA memory hierarchy, PCIe vs NVLink, thread blocks and warp scheduling. - Practice: Write a custom CUDA kernel for matrix multiplication from scratch and profile it against cuBLAS. - Why: You cannot optimize what you do not understand at the silicon level. Stage 2: Transformer Inference Physics - Learn: Prefill vs Decode phases, arithmetic intensity, memory bandwidth bottlenecks, KV-cache math. - Practice: Profile a HuggingFace model with Nsight Systems to identify the exact memory bottleneck during token generation. - Why: LLM inference is almost always memory-bound, not compute-bound. Stage 3: Modern Serving Engines & Batching - Learn: vLLM, SGLang, PagedAttention, continuous batching, chunked prefill. - Practice: Deploy a 70B model and tune chunk sizes to maximize throughput without starving decode requests during long-context prefills. - Why: Naive batching leaves 60% of GPU VRAM wasted. PagedAttention fixes this. Stage 4: KV-Cache & Memory Optimization - Learn: Prefix caching, KV quantization, CPU offloading, multi-turn cache reuse. - Practice: Build a routing proxy that directs requests with identical system prompts to the same replica to share KV blocks. - Why: Reused cache is free speed. It can slash Time-To-First-Token (TTFT) by 80%. Stage 5: Quantization & Compression - Learn: FP8, INT4, AWQ, GPTQ, Sparsity, TensorRT-LLM calibration. - Practice: Serve a model in FP8 vs FP16 and benchmark the exact perplexity drop against the latency and VRAM gains. - Why: Quantization is the only way to fit frontier models on edge GPUs and protect your margins. Stage 6: Kernel-Level Engineering - Learn: Triton, FlashAttention, CUDA graphs, operator fusion. - Practice: Write a fused Triton kernel for RMSNorm or Softmax and benchmark it against the native PyTorch implementation. - Why: Python overhead kills inference. Fused kernels save milliseconds that compound at scale. Stage 7: Distributed Inference & Parallelism - Learn: Tensor parallelism, Pipeline parallelism, Expert parallelism (for MoE models). - Practice: Shard a 405B parameter model across 8 nodes and measure the communication overhead between GPUs. - Why: Single-GPU inference is dead for frontier models. You must master the shard. Stage 8: Speculative Decoding - Learn: Draft-target models, Medusa heads, acceptance rates, n-gram drafting. - Practice: Build a pipeline where a local 3B model drafts tokens for a cloud 70B model to verify in parallel. - Why: 2x decode speed at zero quality cost is the closest thing to a free lunch in inference. Stage 9: Multi-Node & Hardware Interconnects - Learn: NCCL, RDMA, InfiniBand, NVLink, Disaggregated Prefill/Decode architectures. - Practice: Set up a multi-node cluster and profile the network latency of tensor parallelism across nodes vs within a single node. - Why: Network latency is the new GPU bottleneck. Disaggregating prefill and decode is the 2026 meta. Stage 10: Cluster Orchestration & GPU Scheduling - Learn: Kubernetes GPU operators, Ray, Slurm, MIG (Multi-Instance GPU) partitioning, KEDA. - Practice: Build a queue-based autoscaler that spins up spot GPUs based on pending inference requests and drains them when empty. - Why: Idle H100s burn $3+/hr. FinOps and scheduling are now core infra responsibilities. Stage 11: AI Gateways, Routing & Observability - Learn: TTFT/ITL SLOs, semantic routing, DCGM metrics, OpenTelemetry for LLMs. - Practice: Build a gateway that routes simple queries to a quantized local model and complex reasoning to a frontier API based on prompt complexity. - Why: Routing protects your margins and DCGM metrics tell you when your GPUs are silently throttling. Stage 12: Public Benchmarks & Teardowns - Learn: Reproducible methodology, latency/throughput Pareto curves, cost-per-token analysis. - Practice: Publish a teardown comparing vLLM vs SGLang vs TensorRT-LLM on your specific hardware with full configs. - Why: Public proof of hardware mastery gets you hired instantly by top AI labs. Wrappers are a commodity. AI Infrastructure is the physics of scale. The modern AI Infra Engineer builds the silicon nervous system for global intelligence. Bookmark and Repost!

Don't waste 2 years learning to become an AI agentic engineer in 2026. Andrew Ng, the godfather of AI, gave the complete playbook to become one from scratch. 1 hour course. Free: • 00:00 - AI agent basics • 12:12 - AI Agentic workflows & design patterns • 53:27 - Practical tips for building AI agents • 1:20:30 - self-improving AI agent loops • 1:30:19 - multi-agent AI systems I watched it last night. Halfway through, I realized I could get into Anthropic in weeks, not years. Bookmark now. Watch it. Then build your own AI agent

Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel. - Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all. - Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract. - Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code. - And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while. The funnel is driven by a combination of factors: 1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel. 2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel. 3. Access. Free-tier, paid-tier, internal. So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.

Andrej Karpathy @karpathy

6 months ago

Judging by my tl there is a growing gap in understanding of AI capability. The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is

1K 3K 22K 6.4M 12K

joder, GitHub está lleno de cosas gratis que casi nadie está usando. los 10 repos que están explotando este mes, de lo más fácil a lo más técnico: lo que puede usar cualquiera 1. VoiceStudio: clona voces, dobla vídeos y transcribe en 646 idiomas, gratis y sin salir de tu ordenador (57K ⭐) github.com/debpalash/Voic… 2. ppt-master: le pasas un PDF o un Word y te devuelve un PowerPoint de verdad, editable y con animaciones (59K ⭐) github.com/hugohe3/ppt-ma… 3. gods-eye-view de @bilawalsidhu: un simulador de satélite espía en tu navegador, con aviones, barcos y cámaras reales sobre un globo 3D (50K ⭐) github.com/bilawalsidhu/g… 4. REA: le enseñas a tu IA una función de otra app y te explica cómo está hecha, sin tener el código (65K ⭐) github.com/morluto/rea 5. diagram-design de @cathrynlavery: pides un diagrama con palabras y te lo dibuja con tu marca, 44 tipos (49K ⭐) github.com/cathrynlavery/… 6. paperclip: como tener tu propia empresa de agentes autónomos trabajando para ti, la app open source para dirigirlos como si fueran tu plantilla (99K ⭐) github.com/paperclipai/pa… para que Claude Code trabaje mejor 7. skills de @mattpocockuk: las que usa a diario un ingeniero de verdad, pequeñas y fáciles de adaptar (284K ⭐) github.com/mattpocock/ski… 8. ECC: skills, memoria y seguridad en un pack para Claude Code, Codex, Cursor y OpenCode (276K ⭐) github.com/affaan-m/ECC 9. andrej-karpathy-skills: un solo archivo con 4 reglas sacadas de lo que Karpathy dice que hacen mal los modelos programando (218K ⭐) github.com/multica-ai/and… 10. claude-mem: guarda lo que hizo tu agente, lo resume y se lo pasa a la siguiente sesión, para que no empiece de cero (99K ⭐) github.com/thedotmack/cla… compártelo si te ha parecido interesante ♻️ y guárdalo antes de perderlo 🔖

Microsoft has launched a new container system for AI agents across Windows, Linux and macOS. It’s called Microsoft Execution Containers, or MXC, and it lets developers control which files, networks and system resources an agent can access. An agent can edit your project and run its tools while being blocked from personal files or changing configurations it only needs to read. These restrictions are enforced outside the agent’s control, so it can’t grant itself extra permissions. MXC uses platform-specific sandboxes, including Bubblewrap on Linux, through a unified JSON configuration and SDK. GitHub Copilot, OpenAI Codex, Replit and OpenClaw already support it, with Claude Code support planned. Source: blogs.windows.com/windowsdevelop…

keyboard_arrow_left Previous keyboard_arrow_right Next

Here's a topic list for my chat with Lex: 1:22 - First computer 7:00 - Dropping out of high school 14:41 - Joining Microsoft 16:53 - MS-DOS 20:05 - Windows 95 26:52 - The man behind Windows 31:48 - Debugging 37:05 - Task Manager 42:14 - 3D Pinball: Space Cadet 47:13 - Start menu and taskbar 58:12 - Blue Screen of Death 1:00:21 - Best programmers 1:08:22 - Scariest time of Dave's life 1:15:50 - Best Windows version 1:17:40 - Slot machines 1:21:23 - Autism and ADHD 1:40:43 - Fastest programming language 1:44:48 - Future of programming

Lex Fridman @lexfridman

a year ago

Here's the links for my conversation with Dave Plummer (@davepl1968): YouTube: youtube.com/watch?v=HsLgZz… Spotify: open.spotify.com/show/2MAi0BvDc… Podcast: lexfridman.com/podcast

29 71 426 303K 107

Andrej Karpathy: "Prompting is going away. Delete everything, keep Graph." In 1 hour he shows how to build Graphs, and why it's the only thing that will be left standing at the end LLMs → Prompts → Agents → Graphs a prompt gets you one answer and then forgets the job an agent finishes the job and then forgets what it learned a graph keeps every finished job and turns it into the starting point of the next one the first 10 minutes alone are worth more than most paid courses watch the lecture first, then save the full guide on Graphs below ↓

Hanako @hanakoxbt

2 months ago

x.com/i/article/2091…

81 358 2K 9.8M 10K

Really good read. This resonates with my work in industrial AI: so much of the context lives with the people doing the work. Good FDEs earn their trust, understand their constraints, and build something they can actually use and own arenamag.com/articles/forwa…

Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction and recap 01:52 Interpreting basic GRPO training metrics 06:34 Planned improvements to GRPO 08:58 Running longer training jobs with Python scripts 13:39 Running the baseline GRPO training script 17:29 Loading and plotting training logs 19:29 Diagnosing unstable training 23:55 Evaluating checkpoints on MATH-500 26:26 Downloading existing checkpoints 30:09 Tracking advantage statistics 34:53 Understanding entropy 40:32 Computing entropy in PyTorch 44:17 Interpreting entropy values 48:58 Adding entropy tracking to GRPO 53:36 Analyzing advantage and entropy metrics 56:18 Stabilizing GRPO with clipped policy ratios 1:03:27 Implementing the clipped policy loss 1:09:39 Analyzing clipped policy training results 1:11:25 KL divergence and reward hacking 1:15:12 Adding a KL loss term 1:20:34 Limitations of the simplified KL loss 1:23:04 Format rewards and think tags 1:25:47 Adding special tokens to the tokenizer 1:30:29 Implementing the format reward 1:35:56 Analyzing format reward training 1:38:25 Rewarding format only for correct answers 1:40:48 Further GRPO improvements from research 1:45:43 Next steps and distillation