StreamingClient Operator #

Authors: Holoscan Team (NVIDIA)
Supported platforms: x86_64, aarch64
Language: C++, Python
Last modified: November 3, 2025
Latest version: 1.1.0
Minimum Holoscan SDK version: 3.5.0
Tested Holoscan SDK versions: 3.5.0
Contribution metric: Level 1 - Highly Reliable

The VideoStreamingClientOp class implements a Holoscan operator that provides bidirectional video streaming capabilities with the following key components:

Configuration and Initialization:
Configurable parameters for frame dimensions (width, height), frame rate (fps), server connection (IP, port)
Input/output ports for frame data using GXF entities
Support for both sending and receiving frames through separate flags

Frame Processing Pipeline: - Input handling: Receives frames as GXF entities containing H.264 encoded video tensors - Frame conversion: Converts input tensors to VideoFrame objects with BGRA format - Memory management: Implements safe memory handling with bounds checking and zero-padding - Output generation: Creates GXF entities with properly configured tensors for downstream processing

Streaming Protocol Implementation: - Bidirectional streaming support through StreamingClient class - Frame callback system for receiving frames - Frame source system for sending frames - Connection management with server including timeout handling

📚 Related Documentation: - Main Operators README - Setup, dependencies, NGC downloads, and Python examples - Client Application README - Complete client application with usage examples - Server Operator README - Companion server operator documentation - Testing Documentation - Integration testing and verification

Architecture Overview#

The StreamingClient operator integrates with the Holoscan Client Cloud Streaming library to provide seamless video streaming capabilities:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                           Holoscan Application                                  │
├─────────────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐    ┌─────────────────────────────────────────────────────┐ │
│  │   Input Source  │    │              VideoStreamingClientOp                      │ │
│  │                 │    │                                                     │ │
│  │  • V4L2 Camera  │───▶│  ┌─────────────────┐    ┌─────────────────────────┐ │ │
│  │  • Video File   │    │  │  Frame Convert  │    │    VideoFrame Object   │ │ │
│  │  • Tensor Data  │    │  │  BGR → BGRA     │───▶│    • Width/Height      │ │ │
│  │                 │    │  │  Validation     │    │    • Pixel Data        │ │ │
│  └─────────────────┘    │  └─────────────────┘    │    • Format (BGRA)     │ │ │
│                         │                         │    • Timestamp         │ │ │
│  ┌─────────────────┐    │                         └─────────────────────────┘ │ │
│  │  Output Sink    │◀───┤                                      │              │ │
│  │                 │    │                                      ▼              │ │
│  │  • HoloViz      │    │  ┌─────────────────────────────────────────────────┐ │ │
│  │  • File Writer  │    │  │         Holoscan Client Cloud Streaming        │ │ │
│  │  • Next Op      │    │  │                                                 │ │ │
│  └─────────────────┘    │  │  ┌─────────────────┐    ┌─────────────────────┐ │ │ │
│                         │  │  │ StreamingClient │    │   Network Protocol  │ │ │ │
│                         │  │  │                 │    │                     │ │ │ │
│                         │  │  │ • sendFrame()   │───▶│  • Cloud Streaming │ │ │ │
│                         │  │  │ • Callbacks     │    │  • Signaling       │ │ │ │
│                         │  │  │ • Connection    │    │  • Media Transport  │ │ │ │
│                         │  │  │   Management    │    │  • Encryption       │ │ │ │
│                         │  │  └─────────────────┘    └─────────────────────┘ │ │ │
│                         │  │                                      │          │ │ │
│                         │  └──────────────────────────────────────┼──────────┘ │ │
│                         │                                         │            │ │
│                         └─────────────────────────────────────────┼────────────┘ │
└───────────────────────────────────────────────────────────────────┼──────────────┘
                                                                    │
                          ┌─────────────────────────────────────────┼──────────────┐
                          │                    Network                             │
                          │                                                        │
                          │  ┌─────────────────────────────────────────────────┐   │
                          │  │              Streaming Server                   │   │
                          │  │                                                 │   │
                          │  │  • Holoscan Server Cloud Streaming             │   │
                          │  │  • Multi-client support                        │   │
                          │  │  • Bidirectional communication                 │   │
                          │  │  • Frame processing and relay                  │   │
                          │  └─────────────────────────────────────────────────┘   │
                          └────────────────────────────────────────────────────────┘

Component Interactions#

Input Processing: The operator receives video frames from upstream Holoscan operators (V4L2, video replayer, etc.)
Frame Conversion: Input tensors are converted to VideoFrame objects with proper format validation and memory management
Cloud Streaming Integration: The VideoFrame is passed to the Holoscan Client Cloud Streaming library via StreamingClient::sendFrame()
Network Transport: The cloud streaming library handles:
Protocol implementation
Signaling and connection establishment
Media encoding and transport
Security and encryption
Bidirectional Communication: Frames received from the server are processed through callbacks and converted back to Holoscan tensors
Output Generation: Processed frames are emitted as GXF entities for downstream operators (HoloViz, file writers, etc.)

Requirements & Setup#

For complete setup instructions including: - Holoscan SDK 3.5.0 and CUDA 12.x requirements - NGC binary downloads (client streaming binaries) - Build troubleshooting

See the Main Operators README for detailed setup instructions.

Camera Setup and Testing#

This section provides detailed technical camera configuration for the StreamingClient operator. For application-level camera setup and quick start instructions, see the Application README.

Testing Your V4L2 Camera#

Before using the streaming client with your camera, verify it's working properly:

# Check available video devices
ls -la /dev/video*

# Get camera information
v4l2-ctl --device=/dev/video0 --info

# List supported formats and resolutions
v4l2-ctl --device=/dev/video0 --list-formats-ext

# Test camera capture (replace resolution as needed)
v4l2-ctl --device=/dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=MJPG --stream-mmap --stream-count=10

Configuring Camera Resolution#

The streaming client applications use YAML configuration files to set camera parameters. Edit the appropriate config file:

For video_streaming_client:#

Edit ../../../applications/video_streaming/video_streaming_client/cpp/video_streaming_client_demo.yaml:

# V4L2 camera configuration
v4l2_source:
  device: "/dev/video0"        # Camera device path
  width: 1280                  # Camera resolution width
  height: 720                  # Camera resolution height  
  frame_rate: 30               # Camera frame rate
  pixel_format: "MJPG"         # Pixel format (MJPG recommended for higher resolutions)
  # Optional camera settings:
  # exposure_time: 100         # Exposure time in multiples of 100μs
  # gain: 10                   # Camera gain value

Recommended Settings by Camera Type#

For Logitech HD Pro Webcam C920: - 1280x720 @ 30fps MJPG - Best balance of quality and performance - 1920x1080 @ 30fps MJPG - High quality (higher bandwidth) - 640x480 @ 30fps YUYV - Low bandwidth testing

General Guidelines: - Use MJPG format for resolutions above 640x480 for better performance - Use YUYV format for lower resolutions or when uncompressed data is needed - Start with 30 FPS and adjust based on your system performance - Match the resolution between client and server applications

Troubleshooting Camera Issues#

Camera not detected:

# Check camera permissions
sudo usermod -a -G video $USER
# Log out and back in, then test again

Poor performance: - Try lower resolution (e.g., 640x480) - Switch from YUYV to MJPG format - Reduce frame rate to 15 or 24 FPS

Format not supported:

# Check what formats your camera actually supports
v4l2-ctl --device=/dev/video0 --list-formats-ext | grep -E "Size:|Interval:"

FrameSaver Utility Class#

The FrameSaverOp is a utility operator that can save video frames to disk for debugging and analysis purposes. This operator is not integrated into the main streaming client but can be used as a standalone debugging tool.

Features#

Frame Capture: Saves individual video frames to disk
Multiple Formats: Supports both raw binary (.raw) and BGR format (.bgr) output
GPU/CPU Support: Automatically handles frames from both GPU and CPU memory
Configurable Output: Customizable output directory and filename patterns
Data Analysis: Includes frame content analysis and logging

Usage#

Basic Configuration#

# Add FrameSaverOp to your Holoscan application
frame_saver:
  output_dir: "debug_frames"           # Directory to save frames
  base_filename: "frame_"              # Base filename for saved frames
  save_as_raw: false                   # false = .bgr format, true = .raw format

Integration Example#

#include "frame_saver.hpp"

// In your application setup
auto frame_saver = make_operator<holoscan::ops::FrameSaverOp>(
    "frame_saver",
    holoscan::Arg("output_dir", std::string("debug_frames")),
    holoscan::Arg("base_filename", std::string("frame_")),
    holoscan::Arg("save_as_raw", false)
);

// Connect to your frame source
add_flow(frame_saver, source, ('input_frames', 'output_frames'));

Parameters#

Parameter	Type	Default	Description
`output_dir`	string	"output_frames"	Directory where frames will be saved
`base_filename`	string	"frame_"	Base name for saved frame files
`save_as_raw`	bool	false	Whether to save as raw binary (.raw) or BGR format (.bgr)

Output Files#

BGR Format (.bgr): Standard BGR pixel format, can be opened with image viewers
Raw Format (.raw): Binary data, useful for debugging memory layouts
Naming: frame_000001.bgr, frame_000002.bgr, etc.

Debugging Features#

The FrameSaver includes built-in debugging features:

Content Analysis: Logs frame size, zero-pixel count, and data validity
Memory Handling: Automatically copies GPU frames to CPU before saving
Error Handling: Comprehensive error reporting for file operations

Example Output#

Frame 0 data analysis: size=1843200, all_zeros=false, non_zero_count=95
Saved frame 0 to debug_frames/frame_000001.bgr

Building the FrameSaver#

To use the FrameSaver in your application, you'll need to:

Include the source files in your CMakeLists.txt:

add_library(frame_saver
  frame_saver.cpp
  frame_saver.hpp
)

Link against required libraries:

target_link_libraries(frame_saver
  holoscan::core
  CUDA::cudart
)

Use Cases#

Debugging: Save frames at specific points in your pipeline
Analysis: Examine frame content and format
Testing: Verify frame data integrity
Development: Visual inspection of processed frames

Python Bindings & Applications#

For Python usage, application examples, and testing: - Main Operators README - Python bindings overview and setup - Client Application README - Complete Python client implementation - Testing Documentation - Integration testing guide

StreamingClient Operator#