Teams shipping agentic coding workflows aren’t stalled by a percent‑point change in perplexity; they’re blocked by cold KV caches, memory‑bandwidth limits, and orchestration that fails to keep prefixes hot.
What happened
NVIDIA published engineering posts and tutorials showing system work — not just model tweaks — unlocks agent throughput and safety. Their FP8 recipe moves linear layers to FP8, yielding much higher peak throughput versus BF16 and reducing bytes per parameter when generation is memory‑bandwidth bound. Dynamo maps cache placement and routing to keep conversation prefixes hot, reporting 85–97% per‑worker cache hits, a 97.2% aggregate hit rate across teammates, and an 11.7x read/write ratio that defines a write‑once/read‑many pattern. NemoClaw packages orchestration, sandboxing, and lifecycle tools to run Nemotron models locally and safely on DGX Spark. NVIDIA’s red team disclosed an AGENTS.md supply‑chain attack on Codex workflows, showing infrastructure choices change the threat model for agentic systems.
“To make these workloads viable, researchers and engineers are turning to low-precision datatypes like FP8 to boost performance in training and throughput-oriented generation.”
— developer.nvidia.com
Why it matters
At production scale, sessions are dominated by repeated reads of the same conversation prefix and by hardware’s ability to meet strict latency budgets. Dynamo’s numbers show sessions behave like WORM workloads where maximizing cache reuse is the operational priority. Moving linear layers to FP8 delivers roughly 2x peak throughput versus BF16 and cuts parameter bytes when memory bandwidth limits generation; that translates directly into more concurrent agents, lower tail latency, and cheaper GPU hours — outcomes that matter more to users than marginal perplexity gains. Orchestration stacks like NemoClaw close the gap between open models and production by handling routing, sandboxing, and lifecycle ops that would otherwise be baked into managed services. The wins compound: better cache routing raises hit rates, which amplifies FP8 and bandwidth gains and reduces evictions and costly retries. For teams building or hosting coding agents, the competitive multiplier sits in low‑level systems engineering, not the latest marginally stronger code model.
“Maximizing cache reuse rate across all workers and keeping KV blocks warm and routable is the central optimization target for agentic inference.”
— developer.nvidia.com
Counterpoint
Model quality still matters for correctness and reducing revision churn. NVIDIA’s posts acknowledge numeric challenges with low‑precision RL, and the AGENTS.md example shows infrastructure introduces attack surfaces. Model advances plus rigorous FP8 validation and hardened orchestration remain complementary.
What to watch
How robust are FP8 recipes across instruction‑tuned and reasoning models at scale? Which eviction and routing policies best preserve cross‑session prefixes under heavy swarm workloads? When will production stacks expose standardized cache‑aware SLAs and measurable memory‑bandwidth budgets?
