llm-d 1.0: Open Source Inference Gets Its Production Moment
Training a frontier model makes the headlines. Serving one — affordably, at scale, without melting the ops team — is the quieter problem that decides whether AI actually runs in production. This week, the open-source project aimed at that second half shipped its 1.0.
What llm-d is. Announced October 20 at the PyTorch Conference North America, llm-d 1.0 is a Kubernetes-native stack for distributed inference. The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, and is now hosted as a CNCF sandbox project with more than 500 contributors across 25-plus organizations. Its founding slogan — "any model, any accelerator, any cloud" — is now a deployment target, not a pitch. llm-d sits in front of existing engines like vLLM and SGLang and coordinates a cluster of model servers into a single serving pool that runs on NVIDIA, AMD, and Intel GPUs plus Google TPUs.
Why it matters now. Agentic workloads changed what inference traffic looks like. A year ago, most of it was chat: one prompt in, one response out. Today, multi-step agents fan out across tools, share long prefixes, and arrive in bursts. With naive round-robin routing, replicas recompute the same prompts and duplicate the same KV cache — accelerator memory fills with copies, and one bursty agent loop can stall interactive users.
llm-d's answer is a set of techniques it has been refining since launch: prefix-cache-aware routing (send a request to the replica that already holds its context), KV-cache offloading across GPU, CPU, and storage, prefill/decode disaggregation, wide expert parallelism for MoE models, and batch and async inference.
What 1.0 adds. The release is less about raw speed than surviving the real world. v1.0 brings multi-tenant quality of service — priority- and fairness-aware flow control so latency-sensitive requests and bursty agents share a fleet without starving each other — plus zero-downtime operations: router high availability, rolling updates, and a DisaggregatedSet component that coordinates prefill and decode replicas as one unit through upgrades. There's a stable v1 configuration API, and "agent-aware serving": session-aware routing and turn-priority scheduling so agents reuse cached context across turns instead of recomputing it.
The numbers. Backers are putting specifics on the table. Recent demonstrations showed cost reductions of five to ten times for context-heavy agentic workloads and three times greater output throughput. In one benchmark-style demo, researchers served GLM 5.2, an open-weight mixture-of-experts model of roughly 753 billion parameters, on widely deployed NVIDIA GPUs, sustaining over 6.6 million output tokens per minute at peak while running up to 3,000 concurrent coding agents with zero preemptions. And IBM Research used llm-d to run quantum error-correction experiments at 11.5 times lower inference cost than an all-proprietary configuration, with a prefix-cache hit rate around 98% and a blended serving cost of about $0.11 per million tokens.
What it means. TechTarget's framing is apt: llm-d aims to do for distributed inference on GPUs what Kubernetes did for web apps — squeeze more work out of scarce, expensive hardware. That matters most for agents, which read far more context than they generate; context reuse is where the money leaks. Open serving stacks like this also give teams a credible alternative to renting closed APIs, and keep sovereign or on-prem fleets viable.
The honest caveat: llm-d remains a CNCF sandbox project — incubation and graduation still lie ahead. But 1.0 is the milestone that matters for production bets, and it landed just in time to shape the economics of the agent era.