Artificial intelligence is evolving beyond the confines of chatbots and digital assistants, venturing into physical spaces such as laboratories, factories, and hospitals. A new breed of AI agents is emerging, collaborating with humans to enhance their understanding of the environment, facilitate access to information, and enable immediate action.
Creating agentic systems that seamlessly integrate various models, skills, tools, and runtime capabilities for hands-on tasks presents significant challenges. To function efficiently in dynamic, real-world settings, these AI agents must transcend mere response generation.
Similar to human employees, these agents require a deep well of knowledge, specialized tools, and expertise to interpret video, audio, and sensor data. They must navigate swiftly changing conditions and spatial contexts, extract information from enterprise systems, analyze the best course of action, and utilize software tools to accomplish tasks—all while maintaining low latency and supporting the user without causing distractions.
NVIDIA XR AI is a robust developer library designed to aid in the creation of these intelligent applications. By integrating inputs from augmented reality (AR) glasses and extended reality (XR) devices with advanced AI models, enterprise data, and high-speed computing, NVIDIA XR AI empowers agents to perceive, reason, and act seamlessly within workflows.
This platform serves as a foundational resource for developers to craft or link skills and tools tailored for enterprise XR applications. It streamlines the incorporation of multimodal perception, enterprise knowledge retrieval, reasoning models, and agent orchestration, making it easier to produce spatially aware, multimodal AI agents that provide rapid, context-aware support in AR and XR environments.
NVIDIA XR AI consolidates four essential capabilities:
1. It captures real-world data from AR and XR devices, encompassing video, audio, depth, pose, and sensor inputs. 2. It links agents to specialized tools and services, such as NVIDIA Metropolis for visual AI and video analysis, as well as NVIDIA NeMo Retriever for enterprise information retrieval and enhanced generation. 3. It supports a diverse ecosystem of AI models, featuring NVIDIA Nemotron reasoning models, NVIDIA Cosmos Reason, and additional compatible foundational models. 4. It integrates agent orchestration and accelerated runtime services, facilitating the transition from prototype development to full-scale production.
With the NVIDIA NeMo Agent Toolkit, developers can enable tool usage, design reasoning workflows, and coordinate multiple agents. Additionally, NVIDIA's accelerated computing platforms—like the NVIDIA DGX Spark, DGX Station, and RTX PRO systems—provide the necessary infrastructure for running inferences across cloud, data center, and edge environments.
Collectively, these features empower AI agents to comprehend their environments, tap into enterprise knowledge, analyze complex tasks, and offer contextual assistance in real time.
