Skip to content

HSB RoCE Receiver Operator with DOCA GPUNetIO#

Authors: Holoscan Team (NVIDIA)
Supported platforms: aarch64
Language: C++
Last modified: March 25, 2026
Latest version: 0.1.0
Minimum Holoscan SDK version: 4.0.0
Tested Holoscan SDK versions: 4.0.0
Contribution metric: Level 1 - Highly Reliable

This GPU-resident Holoscan operator receives messages over RoCE (RDMA over Converged Ethernet) directly to GPU memory using NVIDIA DOCA GPUNetIO for GPU-side completion queue (CQ) polling. By performing CQ polling entirely on the GPU, this operator eliminates the CPU from the RDMA receive critical path, enabling fully GPU-resident frame reception.

This operator is an alternative to the no-host-metadata (NMD)-based hsb_roce_receiver_nmd receiver. It can be used as a drop-in replacement when the --docagpunetio flag is passed to the imx274_gpu_resident application.

Overview#

The DocaRoceReceiverOp is a GPU-resident Holoscan operator that:

  • Receives messages via RDMA directly to GPU memory
  • Polls the RDMA Completion Queue (CQ) from a CUDA kernel running inside a Holoscan SDK GPU-resident graph, with no CPU involvement in the data and control path
  • Manages the full DOCA verbs lifecycle (CQ, QP, ring buffer, GPU export)

Requirements#

  • Holoscan SDK 4.0.0 or later
  • NVIDIA DOCA SDK 3.2.1+ with GPUNetIO support
  • Holoscan Sensor Bridge (HSB) 2.5.0 or later
  • CUDA-capable GPU with GPUDirect RDMA support (Ampere or later, e.g. RTX A6000)
  • aarch64 platform (e.g. IGX Orin)
  • RoCE-capable NIC (e.g., ConnectX)
  • libibverbs (InfiniBand Verbs library)

Usage#

C++#

#include <hsb_roce_receiver_doca_gpunetio/doca_roce_receiver_op.hpp>

// In your application setup:
auto receiver = make_operator<hololink::operators::DocaRoceReceiverOp>(
    "doca_roce_receiver",
    holoscan::Arg("hololink_channel", data_channel),
    holoscan::Arg("device_start", start_fn),
    holoscan::Arg("device_stop", stop_fn),
    holoscan::Arg("frame_size", frame_size),
    holoscan::Arg("ibv_name", "roceP5p3s0f0"),
    holoscan::Arg("ibv_port", 1u),
    holoscan::Arg("pages", 2u)
);

Parameters#

Parameter Type Default Description
hololink_channel DataChannel* - Pointer to Hololink DataChannel object
device_start std::function - Function called to start the sensor device
device_stop std::function - Function called to stop the sensor device
frame_size size_t 0 Size of one frame in bytes
ibv_name string "roceP5p3s0f0" InfiniBand Verbs device name
ibv_port uint32_t 1 Port number of IBV device
pages uint32_t 2 Number of ring buffer pages

Build#

This operator is built as part of the HoloHub build system when the doca mode is selected:

./holohub build imx274_gpu_resident doca

Or enable the operator explicitly:

./holohub build imx274_gpu_resident --build-with hsb_roce_receiver_doca_gpunetio

Dependencies#

  • hololink::core - Core hololink library
  • hololink::common - Common hololink utilities
  • holoscan::core - Holoscan SDK
  • DOCA SDK - NVIDIA DOCA verbs, GPUNetIO, and common libraries
  • ibverbs - InfiniBand Verbs library
  • CUDA - CUDA driver and runtime APIs
  • fmt - Formatting library

API Reference#

C++#

hololink::operators::DocaCq#

Methods#
Method Description
DocaCq(uint32_t cqe_num, struct doca_gpu *gdev, struct doca_dev *ndev, struct doca_uar *uar, struct doca_verbs_context *vctx, bool umem_cpu)
doca_error_t create()
struct doca_verbs_cq * get() const

hololink::operators::DocaQp#

Methods#
Method Description
DocaQp(uint32_t wqe_num, struct doca_gpu *gdev, struct doca_dev *ndev, struct doca_uar *uar, struct doca_verbs_context *vctx, struct doca_verbs_pd *vpd, struct doca_verbs_cq *cq_rq, struct doca_verbs_cq *cq_sq, bool umem_cpu)
doca_error_t create(struct doca_verbs_context *verbs_ctx, size_t frame_size)
doca_error_t create_ring(size_t stride_sz, unsigned stride_num, struct ibv_pd *ibv_pd)
doca_error_t connect(struct doca_verbs_gid &doca_rgid, uint32_t gid_index, uint32_t dest_qp_num)
struct doca_verbs_qp * get() const
struct doca_gpu_verbs_qp * get_gpu() const
struct doca_gpu_dev_verbs_qp * get_gpu_dev() const

hololink::operators::DocaRoceReceiverOp#

Inherits from: holoscan::GPUResidentOperator

GPU-resident operator for DOCA GPUNetIO RoCE reception. Used as the data-ready handler: its kernel runs inside the CUDA Graph WHILE node, performs a non-blocking CQ check for an RDMA Write completion, and returns CTRL_DATA_NOT_READY if none is available yet. When a completion arrives it selects the received frame (writing its address to chosen_frame_memory_) and marks CTRL_DATA_READY so the rest of the pipeline proceeds. All DOCA resource creation (verbs context, CQ, QP, ring buffer, GPU export) and HSB authentication/configuration happen in initialize().

Methods#
Method Description
void setup(holoscan::OperatorSpec &spec) override
void initialize() override
void start() override
void stop() override
void compute(holoscan::InputContext &, holoscan::OutputContext &, holoscan::ExecutionContext &) override
void set_on_chosen_frame_memory_ready(std::function< void(unsigned char **)> callback) Register a callback invoked during initialize() once chosen_frame_memory_ has been allocated. The callback receives the device pointer-to-pointer so the caller can publish it to the pipeline's frame source operator.