AgentPerf, developed by Artificial Analysis, introduces the first benchmark specifically tailored for agentic AI, offering developers, enterprises, and infrastructure providers a definitive means to evaluate systems designed for agentic workloads. The initial results reveal that the NVIDIA Blackwell Ultra NVL72 platform excels, exhibiting up to 20 times more agents per megawatt compared to the NVIDIA Hopper in the tasks tested.
Agentic AI differs significantly from conversational AI; whereas a single request for chat completion equates to one LLM call and a quick response, agentic AI functions more like a relay that breaks down a goal into multiple steps and continues processing until completion. This means chaining together dozens or even hundreds of LLM calls, each transferring an expanding context to the next, supplemented with tool calls that include coding, database querying, and browsing at each phase. The increase in complexity is multiplicative rather than simply additive.
This distinction is crucial for accurately gauging performance. Traditional AI inference benchmarks focus on measuring the speed of individual LLM calls, which are not suitable for evaluating agentic workloads where numerous LLM calls, tool delays, and growing contexts challenge accelerated computing systems in unique ways.
For organizations that develop and deploy agents on a large scale, understanding agent responsiveness, the number of concurrent agents that can be utilized, and the productive output achievable per investment in AI infrastructure is imperative.
The NVIDIA GB300 NVL72 shows remarkable efficiency, outperforming its predecessor by 20 times in agent capacity per megawatt in this initial benchmark phase. Utilizing the DeepSeek V4 Pro, a cutting-edge mixture-of-experts (MoE) model, this system achieves superior performance thanks to a tightly integrated design that connects 72 GPUs within a single rack, allowing large models like DeepSeek V4 Pro to execute their tasks efficiently at scale.
CUDA kernel technology enhances performance by overlapping communication with computation, thus mitigating the latency typically associated with coordinating multiple experts. The NVIDIA TensorRT LLM optimally manages efficiency as the number of concurrent agent sessions increases, separating input processing from output generation to facilitate independent optimization.
AgentPerf’s methodology is meticulously crafted to align with the dynamics of agentic AI as it functions in real-world scenarios. It is based on actual coding agent flows: agents undertake tasks, read and write code, and execute commands, all derived from authentic public code sources spanning over 12 programming languages. The lengthy sequences, tool usage patterns, and delays simulated in the benchmark replicate genuine coding environments.
This benchmarking quantifies how many agentic tasks can be simultaneously processed on a given platform while adhering to specific performance metrics. Tool calls are modeled rather than actually executed, ensuring that the performance variations directly reflect the capabilities of accelerated computing.
The outcomes from AgentPerf significantly influence infrastructure decisions, highlighting how many concurrent agentic tasks can efficiently run per accelerator and per megawatt. These metrics are vital for enterprises eager to scale AI agents, as they indicate the potential productivity from their infrastructural investments.
NVIDIA's ecosystem partners, such as Baseten, DeepInfra, and Together AI, are already leveraging the Blackwell platform to manage agentic workloads with frontier models like DeepSeek V4 Pro, facilitating the deployment of production-level agentic applications.
Together AI's Cursor platform harnesses NVIDIA Blackwell to enable real-time inference for coding tasks, where agents identify bugs, create features, and refactor code while developers engage in parallel work. Meanwhile, DeepInfra drives Pam.ai, an AI solution for the automotive sector that utilizes agents to schedule service appointments, manage inbound calls, and conduct outbound sales campaigns, all powered by NVIDIA Blackwell.
As NVIDIA and the broader open-source community work towards enhancing inference software, the performance of agentic workloads is set to improve steadily. The new NVIDIA Vera Rubin architecture is now operational, poised to address the escalating demands of agentic AI at scale.
For further insights into AgentPerf’s methodologies and NVIDIA's comprehensive optimizations for agentic AI, check out the detailed technical blog.
