MLCommons and Silicon Valley Systems Consortium Release MLPerf Inference v6.0 Benchmarks for Agentic Tool Calling

Systems architects and performance engineers across the MLCommons Inference Working Group—spanning hardware, cloud, and research teams in San Francisco, San Jose, and Santa Clara—have ratified and published the October 2026 MLPerf Inference v6.0 benchmark harness, introducing the industry's first peer-reviewed evaluation track for multi-turn agentic tool calling and prefix-cached retrieval-augmented generation (RAG). Announced during a Silicon Valley benchmark committee briefing on Thursday, October 22, 2026, the updated specification moves datacenter AI benchmarking beyond isolated single-prompt question answering to reflect real production agent loops.

In earlier MLPerf generations, language-model inference servers were evaluated primarily on independent, uncorrelated input prompts, where every request computed its key-value (KV) attention state from scratch. By contrast, production coding assistants and enterprise workflow agents execute five to twenty sequential tool-calling turns per session, repeatedly appending JSON function outputs to a shared system prompt and conversation history ranging from 16,000 to 64,000 tokens.

To capture this operational regime accurately, the MLPerf v6.0 LoadGen traffic generator streams stateful multi-turn session traces that reward inference engines capable of radix-tree prefix KV-cache reuse across HBM, host DRAM, and NVMe tiers. Submissions must simultaneously satisfy a strict 99th-percentile Time-to-First-Token (TTFT) ceiling of 450 milliseconds, an Inter-Token Latency (ITL) limit of 25 milliseconds, and deterministic JSON schema validation scores on the reference tool-use evaluation dataset.

Initial reference runs across liquid-cooled multi-node GPU and custom ASIC racks in San Jose and Santa Clara demonstrated that hierarchical prefix caching and disaggregated prefill routing boost sustained agent sessions per second by more than 3.1x per kilowatt. Open-source LoadGen containers and calibration weights are available at mlcommons.org.