Skip to content

OpenXR Operators#

Authors: Connor Smith (NVIDIA), Rafael Wiltz (NVIDIA), Mimi Liao (NVIDIA)
Supported platforms: x86_64
Language: C++, Python
Last modified: August 7, 2025
Latest version: 1.0.0
Minimum Holoscan SDK version: 3.1.0
Tested Holoscan SDK versions: 3.1.0
Contribution metric: Level 4 - Experimental

A collection of Holoscan operators for XR (Extended Reality) integration, enabling real-time XR applications with Vulkan graphics and CUDA compute capabilities.

Overview#

The XR operators provide a complete framework for building Holoscan applications that can render to XR headsets. The implementation leverages the OpenXR standard for cross-platform XR support, Vulkan for graphics, and CUDA for compute operations, enabling high-performance real-time XR applications.

Core Components#

Below is a diagram showing how to use these operators in a typical XR workflow:

Typical XR Operator Workflow

Frame Management Operators#

XrBeginFrameOp#

  • Synchronizes application with XR device timing
  • Calls OpenXR xrWaitFrame (blocks until the next frame is ready and returns a predicted display time)
  • Calls OpenXR xrBeginFrame (signals the start of rendering for the new frame)
  • Outputs frame state for downstream operators

XrEndFrameOp#

  • Calls OpenXR xrEndFrame to submit composition layers to XR device

XrSession#

The central resource that manages the OpenXR session lifecycle:

  • Initialization: Creates OpenXR instance, system, session, and Vulkan graphics context
  • Plugin System: Supports extensible plugins for additional XR functionality
  • View Space Management: Provides reference and view spaces for pose tracking
  • Graphics Integration: Manages Vulkan instance, device, and queue creation

Rendering Infrastructure#

XrSwapchainCuda#

  • Manages OpenXR swapchains with CUDA interop
  • Provides Vulkan/CUDA memory sharing via external memory
  • Supports multiple formats: RGBA8, depth buffers
  • Handles GPU synchronization between CUDA and Vulkan

XrCompositionLayerManager#

  • Manages color and depth swapchains
  • Creates composition layer storage for frame rendering
  • Provides simplified interface for XR rendering

XrCompositionLayerProjectionStorage#

  • Stores projection layer views and depth information
  • Supports stereo rendering with side-by-side layout
  • Handles depth range and near/far plane configuration

Input Tracking#

XrHandTracker#

  • Implements OpenXR hand tracking extension
  • Provides joint locations for both hands
  • Supports 26 hand joints per hand
  • Real-time pose tracking with validation

Features#

  • Session Management: Create and manage XR sessions, including initialization, lifecycle, and management of reference/view spaces.
  • Frame Operations: Handle XR frame begin and end operations to synchronize rendering and presentation with the XR device.
  • CUDA Swapchain: Enable direct GPU processing of OpenXR swapchain images using CUDA, supporting efficient zero-copy memory sharing between CUDA and Vulkan.
  • Layer Composition and Stereo Rendering: Collect and manage all layers (color, depth, etc.) to be displayed, with built-in support for stereo rendering using side-by-side layouts.
  • Input Systems: Provide 6DOF pose tracking and hand tracking (OpenXR EXT_hand_tracking).
  • Performance and Efficiency: Leverage efficient CUDA-Vulkan interoperability, GPU synchronization, and external memory/semaphore primitives for high-performance XR rendering.

Dependencies#

External Libraries#

  • OpenXR SDK: Cross-platform XR API
  • OpenXR-Hpp: C++ headers for OpenXR
  • GLM: Mathematics library for graphics
  • Vulkan: Graphics API for XR rendering
  • CUDA: Compute API for GPU operations

Holoscan Requirements#

  • Minimum SDK Version: 3.3.0
  • Tested Versions: 3.3.0
  • Platforms: x86_64

Usage Example#

For practical usage examples, see the following applications that utilize these XR operators:

  • xr_holoviz: Simple XR visualization using Holoviz.
  • xr_gsplat: XR rendering of 3D scenes using Gaussian Splatting.

API Reference#

C++#

holoscan::IXrPlugin#

Methods#
Method Description
IXrPlugin(std::shared_ptr< holoscan::XrSession > xr_session) noexcept(false)
std::vector<std::string_view> get_required_instance_extensions()=0
void on_session_created()=0

holoscan::ops::XrBeginFrameOp#

Inherits from: holoscan::Operator

Methods#
Method Description
void setup(OperatorSpec &spec) override
void compute(InputContext &input, OutputContext &output, ExecutionContext &context) override

holoscan::XrCompositionLayerManager#

Inherits from: holoscan::Resource

Methods#
Method Description
void setup(ComponentSpec &spec) override
void initialize() override
holoscan::Tensor acquire_color_swapchain_image()
holoscan::Tensor acquire_depth_swapchain_image()
void release_swapchain_images(cudaStream_t cuda_stream)
std::shared_ptr<XrCompositionLayerProjectionStorage> create_composition_layer(const xr::FrameState &xr_frame_state)

holoscan::ops::XrEmptyCompositionLayerOp#

Inherits from: holoscan::Operator

Methods#
Method Description
void setup(OperatorSpec &spec) override
void compute(InputContext &input, OutputContext &output, ExecutionContext &context) override

holoscan::ops::XrEndFrameOp#

Inherits from: holoscan::Operator

Methods#
Method Description
void setup(OperatorSpec &spec) override
void compute(InputContext &input, OutputContext &output, ExecutionContext &context) override

holoscan::XrHandTracker#

Inherits from: holoscan::Resource, holoscan::IXrPlugin

Methods#
Method Description
XrHandTracker(std::shared_ptr< holoscan::XrSession >, xr::HandEXT hand)
std::vector<std::string_view> get_required_instance_extensions() override
void on_session_created() noexcept(false) override
std::optional<std::vector<xr::HandJointLocationEXT>> locate_hand_joints() noexcept(false)

holoscan::XrSession#

Inherits from: holoscan::Resource

Methods#
Method Description
void setup(ComponentSpec &spec) override
void initialize() override
void register_plugin(std::reference_wrapper< IXrPlugin > plugin)
xr::SpaceLocation locate_view_space() const noexcept(false)
xr::Session & get()
xr::Instance & instance()
xr::DispatchLoaderDynamic & dispatch()
xr::Space & reference_space()
xr::Space & view_space()
const std::vector<xr::ViewConfigurationView> & view_configurations()
xr::ViewConfigurationDepthRangeEXT view_configuration_depth_range()
xr::EnvironmentBlendMode blend_mode()
vk::raii::PhysicalDevice & vk_physical_device()
vk::raii::Device & vk_device()
uint32_t vk_queue_family_index()
vk::raii::Queue & vk_queue()

holoscan::XrSwapchainCuda#

Methods#
Method Description
XrSwapchainCuda(XrSession &session, Format format, uint32_t width, uint32_t height)
holoscan::Tensor acquire()
void release(cudaStream_t cuda_stream)
xr::Swapchain & get()
uint32_t width()
uint32_t height()