On Friday, Nvidia unveiled intriguing findings indicating that the software harness surrounding an AI model plays a much more significant role than the model itself when it comes to executing long-term tasks. The harness represents the software framework that integrates memory management, operational rules, and tools necessary to empower a raw model to function autonomously.
In essence, researchers found that by employing a tailored harness that effectively manages memory and incorporates a supervisory component, they enabled Claude Opus 5 to score a perfect 100% on the interactive reasoning benchmark ARC-AGI-3. This benchmark involves a series of 2D games without any instructions, requiring the model to independently learn how to play and win, similar to human capabilities. Previously, without the harness, Opus 5 only managed a score of 30%, which, while the highest among the models tested, highlights the dramatic impact of the right harness.
Nvidia’s findings reinforce the notion that, although the model selection is important, the configuration of the harness—essentially the infrastructure that supports the model—plays a crucial role, particularly for complex, long-horizon tasks. The harness effectively manages memory, contextual understanding, and feedback loops that transform a simple model into a fully functioning agent.
Adel El Hallak, Nvidia’s vice president of product in AI, emphasized this distinction when speaking with TechCrunch. He noted that many in the industry view an agent merely as an API for the model, but it encompasses much more. An agent consists of both the model and the supporting harness, which includes an array of tools, runtimes, and libraries.
Long-horizon tasks require a series of sequential decisions that may unfold over extended periods, contrasting with shorter tasks where an AI responds to a prompt. The challenge of guiding AI toward successful completion of these lengthy tasks without distraction is a central goal in agent research.
For context, previous research from Microsoft highlighted that 19 large language models (LLMs) struggled with long-horizon tasks, particularly in document editing, producing numerous errors. Such performance would be unacceptable for human workers.
Nvidia's decision to utilize the ARC-AGI-3 benchmark is noteworthy, considering its significance; achieving a 100% score indicates that the model can outperform human players in these games. This development was especially frustrating for OpenAI, whose models achieved scores of less than 10% on the same benchmark. Following their disappointing results, OpenAI also conducted research and found that minor adjustments to their harness settings resulted in improved performance, though still far from the perfect score achieved by Nvidia's researchers.
One key element identified by Nvidia was the inclusion of a supervisory agent that functions like a CEO, guiding the primary agent when it deviates from its intended path. El Hallak elaborated on this concept, suggesting that the supervisor nudges the agent back on track when necessary or prompts it to revisit previous strategies that may have been overlooked.
Although the idea of a supervising agent isn't entirely new, many current harnesses rely on a single layer of support, such as Claude Code or Codex. Nvidia developed its own advanced harness known as Agentic Variation Operators (AVO). It’s essential to clarify that this does not represent a new product from Nvidia; rather, they provide a variety of open-source technologies for constructing harnesses under the Nemo brand, combining both commercial and freely accessible components.
Nvidia's findings further contribute to the ongoing discussion that an AI model's performance is influenced significantly by its harness. Data from July published by Databricks illustrated how variances in harnesses could lead to substantial differences in operational costs, underscoring the point that users need to consider the harness in addition to the model.
Finally, Nvidia advocates for an open harness approach, which allows users greater control and adaptability. El Hallak stated that harnesses should enable users to fine-tune various settings to enhance accuracy, highlighting the potential benefits of an open agent ecosystem to propel advancements in AI securely.
