Interactively Debugging a Holoscan Application ¶

Authors: Tom Birdsong (NVIDIA)
Supported platforms: x86_64, aarch64
Last modified: March 18, 2025
Language: C++
Latest version: 0.1.0
Minimum Holoscan SDK version: 2.2.0
Tested Holoscan SDK versions: 2.2.0, 2.3.0
Contribution metric: Level 1 - Highly Reliable

Holoscan SDK is a platform for rapidly developing low-latency AI pipelines. As part of software development we often find it useful to inspect pipeline operations and data contexts during execution. This tutorial walks through a few common scenarios to illustrate how common command line interface tools can be used in debugging an application based on Holoscan SDK.

Tutorial Sections¶

Background¶

What is Debugging?¶

Software debugging is the process of identifying and rectifying issues in software. - Just-in-time debugging lets us pause execution and inspect the state while a program is running. For a C++ library such as Holoscan SDK, we rely on debugging symbols generated by the compiler at runtime to map the execution state back to human-readable source code, which lets us understand what the program was doing when paused. Tools such as gdb and Visual Studio Code allow us to manage application execution by setting breakpoints and watches, as well as inspect the program state such as local variable values when paused. - Logging is one process through which we record events that occur during real-time execution of a pipeline without interrupting the flow of execution. Log messages can provide a view of the application state and flow that is limited to the messages the developer chooses to log. Holoscan SDK supports logging based on popular libraries such as spdlog and the Python logging module.

In general, both logging and just-in-time debugging are useful tools for prototyping and general development, while logging is usually better suited for application deployment. In this guide, we focus on using just-in-time debugging tools to inspect applications built on Holoscan SDK.

What are debugging symbols?¶

Debugging symbols are generated by a C++ compiler at build time to provide additional information for application debugging such as file names and line numbers. Debugging symbols must be present for debugging tools to provide meaningful information to a developer when inspecting an application's execution state.

Holoscan SDK uses the following build profiles in its run script: - release: Instructs the C++ compiler to optimize where possible and not generate debugging symbols. We use this build type to create minimum-footprint binaries such as those in the Holoscan SDK Debian package or Python wheel distributions. - rel-debug: Instructs the C++ compiler to optimize where possible and generate debugging symbols. We use this build type to create developer-friendly binaries packaged in the Holoscan SDK NGC containers. Debug symbols may occasionally have inaccuracies due to release optimizations. - debug: Instructs the C++ compiler to minimize optimizations and prioritize debugging symbol accuracy. As of v2.3, we do not release Holoscan SDK builds of this type, but we do provide a run script to help developers generate their own debug builds on demand.

What are some common tools I can use for debugging my application?¶

There are a wide variety of free and/or open source software tools available for general C++ and Python debugging, including: - NVIDIA's debugging solutions, including NVIDIA NSight and CUDA-GDB; - The GNU project DeBugger (GDB); - The built-in Python Debugger (pdb) module; - Microsoft Visual Studio Code, with a wide variety of community extensions.

In this tutorial we will focus on the GDB and pdb command line tools. These require minimal setup and can be run via a simple terminal without a dedicated display ("headless"). For advanced development we recommend reviewing Visual Studio Code Development Containers with custom launch profiles, with HoloHub support coming soon.

References¶

To get started with debugging your Holoscan SDK application, visit the Debugging user guide section for common topics, including: - Generating debug symbols with VSCode - Live debugging for C++ and Python applications - Inspecting application crashes - Profiling and code coverage

Visit the Logging user guide section for a thorough overview on how to set up Holoscan SDK logging in your C++ or Python application.

Prerequisites¶

The steps for getting started with gdb depend on how you are consuming Holoscan SDK. - We encourage using Holoscan SDK containers from NGC for development and we take this approach for most of the tutorial. If you are using an NGC container for Holoscan SDK v2.3.0 or later, you already have access to debug symbols and can get started right away. - If you are using an older Holoscan SDK container from NGC, or if you are consuming Holoscan SDK through another means such as Debian packages, Python wheels, or custom installation, you will need to build Holoscan SDK with debugging symbols in order to step through Holoscan SDK code during debugging. Jump to the Legacy Holoscan SDK section to get started.

Review the HoloHub Prerequisites along with the Endoscopy Tool Tracking requirements C++, Python to get started before continuing.

If you have previously built the Endoscopy Tool Tracking application, you should clear your build directory before proceeding.

./run clear-cache

Debugging a C++ Application with `gdb`¶

Background¶

GDB (the GNU project DeBugger) is a widely used tool for headless just-in-time debugging of C++ applications. GDB distributions are available for most Holoscan SDK supported platforms and do not require a display to set up. Refer to the GDB User Manual to get started.

Getting Started¶

For this tutorial we will debug the Endoscopy Tool Tracking application. The tutorial debug_gdb.sh script is a self-contained example that will build the C++ application and launch into a gdb debugging session.

Run the script to get started:

./tutorials/cli_debugging/debug_gdb.sh

The script runs through the following steps: 1. Builds the tutorial container environment with gdb based on the Holoscan SDK NGC container. 2. Builds the Endoscopy Tool Tracking application in the container environment. By default we use the debug build mode to generate detailed debugging symbols for the Endoscopy Tool Tracking application. Note that this does not regenerate build symbols for Holoscan SDK, which are already packaged in Holoscan SDK binaries in rel-debug mode. 3. Launches the Endoscopy Tool Tracking application with gdb. This step prefixes the launch command given by ./run launch endoscopy_tool_tracking to run with gdb. The command sets a few actions for gdb to take on startup: - Sets a breakpoint in the main function of Endoscopy Tool Tracking; - Runs the program with custom arguments until the breakpoint is hit; - Sets a breakpoint in Holoscan SDK's add_flow function.

At this point gdb enters an interactive session where we can inspect the Endoscopy Tool Tracking program state and advance execution.

GDB can also be used to interactively debug C++ symbols underlying a Holoscan Python pipeline. Run the tutorial debug_gdb.sh script to inspect the Endoscopy Tool Tracking Python application:

./tutorials/cli_debugging/debug_gdb.sh debug python

When the python argument is provided, the script builds the Endoscopy Tool Tracking application with Python bindings and initiates debugging with GDB from the Python script entrypoint. Once symbols are loaded in GDB, we can set breakpoints and inspect the underlying state of Holoscan SDK C++ operator implementations at runtime.

From this point we recommend referring to the GDB Manual or online tutorials to get started with interactive debugging commands.

Frequently Asked Questions and Troubleshooting¶

How can I verify that Holoscan SDK debugging symbols have been loaded in `gdb`?¶

gdb loads debugging symbols for Holoscan SDK only when the application loads Holoscan SDK binaries. Before that time, we can set breakpoints in Holoscan SDK files, but gdb will not understand them yet.

We can use info sharedlibrary to inspect the shared libraries that have been dynamically loaded for execution.

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7fc5090  0x00007ffff7fee315  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7ca3ba0  0x00007ffff7ec655d  Yes         /opt/nvidia/holoscan/lib/libholoscan_op_aja.so.2
0x00007ffff7f90ac0  0x00007ffff7faee1f  Yes         /opt/nvidia/holoscan/lib/libholoscan_op_video_stream_replayer.so.2
0x00007ffff7bd9810  0x00007ffff7bf4e3f  Yes         /opt/nvidia/holoscan/lib/libholoscan_op_video_stream_recorder.so.2
...

We can use info sources to inspect the Holoscan SDK symbols are available. Note: Source paths are loaded from Holoscan SDK binaries and respect source file locations at the time the Holoscan SDK distribution was built. These paths may not reflect your filesystem if you have mounted holoscan-sdk somewhere other than /workspace/holoscan-sdk.

(gdb) info sources /workspace/holoscan-sdk/src/core
/workspace/holoscan-sdk/src/core/application.cpp, /workspace/holoscan-sdk/src/core/services/generated/system_resource.grpc.pb.cc,
/workspace/holoscan-sdk/src/core/services/generated/system_resource.pb.h, /workspace/holoscan-sdk/src/core/services/generated/system_resource.pb.cc,
/workspace/holoscan-sdk/src/core/services/generated/result.grpc.pb.cc, /workspace/holoscan-sdk/src/core/services/generated/result.pb.h,
...

How can I pause a running Holoscan SDK application for debugging?¶

After launching the application with gdb as done in debug_gdb.sh, use continue to allow the application to run. Then, press Ctrl+C to force the application to pause when you want to enter interactive debugging. Use backtrace to view stack frames at the point where the application paused.

How can I manage breakpoints to pause the application sometime in the future?¶

To add a breakpoint in holoscan/operators/format_converter/format_converter:

(gdb) break /workspace/holoscan-sdk/src/operators/format_converter/format_converter.cpp:compute"

To list all current breakpoints:
```
(gdb) info breakpoints
```

To remove breakpoints:

(gdb) delete <number(s) of breakpoint(s)>

How can I inspect local variables?¶

Use the info command to inspect the values of local variables.

(gdb) info locals

How can I attach to a running Holoscan SDK session?¶

Do the following to attach to a HoloHub application (C++ or Python):

Launch the container with root permissions and start the process in the background:

./dev_container launch --img holohub:debugging --as_root

# Run inside the container
>>> ./run launch endoscopy_tool_tracking &

Press Ctrl+C to return to your interactive shell

Find the process ID (PID) of the running application:

# Inside the container
>>> ps -ef | grep endoscopy_tool_tracking
user+     292     203 28 13:17 pts/9    00:00:04 /workspace/holohub/build/endoscopy_tool_tracking/applications/endoscopy_tool_tracking/cpp/endoscopy_tool_tracking --data /workspace/holohub/data/endoscopy

Attach to the running process with gdb -p:
```
# Inside the container
>>> gdb -p 292
```

From this point you can use Ctrl+C to pause application execution, then use the interactive GDB console to set breakpoints and inspect application state as usual.

I see a `gdb` Python Exception in the `gdb` log.¶

gdb relies on a Python module for operations such as unwinding. If the Python module is not properly referenced in the container, you may see gdb errors appear in the console log such as the following:

"Python Exception <class 'NameError'>: Installation error: gdb._execute_unwinders function is missing"

To resolve the error, update your PYTHONPATH variable to include your GDB Python directory:

PYTHONPATH=${PYTHONPATH}:/usr/share/gdb/python

Debugging a Python application with `pdb`¶

Background¶

The Python Debugger module is a built-in interactive debugging tool for Python programs. Similar to gdb, it supports setting breakpoints for interactive, headless, just-in-time debugging. You can use pdb to debug Holoscan SDK Python programs on any platform supported by Holoscan SDK.

Holoscan SDK Python libraries serve as wrappers around Holoscan SDK C++ libraries. While pdb may load C++ symbols, it is not well suited for setting breakpoints or stepping into underlying C++ code. pdb is best suited for debugging Holoscan SDK operators whose implementation lies in a Python-native compute method.

Prerequisites¶

The pdb module is built in to modern Python versions. No additional installation is required.

We will continue to use the Holoscan SDK v2.3 development container from NGC for this section, which includes pre-installed versions of Python and Holoscan SDK.

Getting Started¶

We will continue to debug the Endoscopy Tool Tracking application. The tutorial debug_pdb.sh script is a self-contained example that will build the application and launch into a pdb debugging session.

Run the script to get started:

./tutorials/cli_debugging/debug_pdb.sh

The script runs through the following steps: 1. Builds the HoloHub container environment based on the Holoscan SDK NGC container. 2. Builds the Endoscopy Tool Tracking Python application in the container environment. By default we use the debug build mode to generate detailed C++ debugging symbols for the Endoscopy Tool Tracking application. 3. Launches the Endoscopy Tool Tracking application with pdb, which is invoked via the Python interpreter:

python3 -m pdb <command --args ...>

This command launches the Python version of the Endoscopy Tool Tracking application with a breakpoint set on the very first line. From this point you can set additional breakpoints, inspect application state, and control program execution.

For instance, the following snippet sets a breakpoint and then inspects the value of self.source.lower() during pipeline setup in the app compose method:

(Pdb) break endoscopy_tool_tracking.py:77
Breakpoint 1 at /workspace/holohub/applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py:77
(Pdb) continue
...
> /workspace/holohub/applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py(77)compose()
-> if self.source.lower() == "aja":
(Pdb) p self.source.lower()
'replayer'
(Pdb)
...

You can also add a breakpoint directly in the .py source code before running a program by adding a breakpoint() statement.

From here we recommend referring to pdb documentation for common commands and debugging strategies.

Debugging Symbols for Legacy Holoscan SDK Versions¶

Starting from Holoscan SDK v2.3, we package debugging symbols as part of the libraries distributed in NGC Holoscan Containers, with the goal of improving ease of development and debugging. This change comes at a small cost to size and performance of the Holoscan SDK binary distribution. But what about older versions of Holoscan SDK containers?

If you are using a legacy Holoscan SDK container (earlier than v2.3) for your development, your container does not come with debugging symbols pre-packaged. However, you can rebuild the Holoscan SDK from its source code to enable interactive debugging. We provide utilities as part of Holoscan SDK open source code to help you generate debugging symbols that will allow you to use tools such as gdb and pdb with legacy Holoscan SDK.

The following script provides the necessary steps to rebuild Holoscan SDK version with debugging symbols and then set up for debugging with gdb:

./tutorials/cli_debugging/debug_legacy.sh

The script does the following: 1. Builds the specified Holoscan SDK version with the specified build type in a temporary tutorial folder. Refer to background discussion for an overview of the different build types. 2. Builds the Endoscopy Tool Tracking application against the custom Holoscan SDK debug build. 3. Runs the Endoscopy Tool Tracking application with gdb for interactive debugging.

For convenience, the debug_legacy.sh script mounts your custom Holoscan SDK installation at /opt/nvidia/holoscan, the default library path in the Holoscan SDK container. This effectively hides the Holoscan SDK build otherwise distributed inside the container and instead makes your custom debugging build available to build downstream applications.

Frequently Asked Questions and Troubleshooting¶

How can I use a custom container path for my Holoscan SDK debugging build other than `/opt/nvidia/holoscan`?¶

You can choose to mount your custom Holoscan SDK debugging build at another path in the container with the Docker -v option or the HoloHub dev_container script --local_sdk_root or --mount-volume options. If you are mounting your build at a custom path in the Holoscan SDK container for general development, consider the following details when building and debugging: - LD_LIBRARY_PATH is an environment variable with a list of locations to look up for dynamic loading. By default the Holoscan SDK container sets LD_LIBRARY_PATH to include /opt/nvidia/holoscan, and then the HoloHub run script sets it again when launching an application. Edit this variable and launch your application directly to load libraries from your custom mount by default. - RPATH or RUNPATH is an ELF header field that embeds shared library lookup locations in an executable. HoloHub applications set RPATH to include /opt/nvidia/holoscan by default. Edit the value of CMAKE_INSTALL_RPATH in CMakeLists.txt to remove the reference to /opt/nvidia/holoscan or reference your preferred mount path.

How can I launch the tutorial application?¶

You can simply re-run the tutorial script to rebuild and relaunch the application:

./tutorials/cli_debugging/debug_legacy.sh

Alternatively, run the following to relaunch the application in the debugging container without rebuilding:

# Find the custom Holoscan SDK debugging build
INSTALL_DIR=$(realpath $(find ./tutorials/cli_debugging/tmp -type d -name "install-*"))

# Launch the debugging container
./dev_container launch --docker_opts "-v $INSTALL_DIR:/opt/nvidia/holoscan --security-opt seccomp=unconfined" --img holohub:debugging

# Inside the container
>>> gdb -q \
    -ex "break main" \
    -ex "run --data /workspace/holohub/data/endoscopy" \
    -ex "break /workspace/holoscan-sdk/src/core/application.cpp:add_flow" \
    /workspace/holohub/build/endoscopy_tool_tracking/applications/endoscopy_tool_tracking/cpp/endoscopy_tool_tracking

Refer to the debug_legacy.sh script for more details.

Logging¶

Just-in-time debugging is not well suited to problems that require real-time performance analysis. Logging is usually the better choice to debug performance related issues in your Holoscan application.

The Holoscan SDK User Guide Logging section presents a detailed overview of how to get started with logging from your application.

Logging from a C++ application¶

C++ applications based on Holoscan should use the HOLOSCAN_LOG_LEVEL environment variable or holoscan::set_log_level function to set the global level of detail to log in the application. You can add inline macros such as HOLOSCAN_LOG_INFO AND HOLOSCAN_LOG_TRACE in your application code to print out log messages at runtime according to the current logging level of detail.

export HOLOSCAN_LOG_LEVEL="Debug"
./run launch endoscopy_tool_tracking

Logging from a Python application¶

Python applications based on Holoscan should use the Python logging module. Holoscan observes the standard logging module interface with statements such as logger.info and logger.debug. Refer to the Python logging module for more information.

Interactively Debugging a Holoscan Application¶

Tutorial Sections¶

Background¶

What is Debugging?¶

What are debugging symbols?¶

What are some common tools I can use for debugging my application?¶

References¶

Prerequisites¶

Debugging a C++ Application with gdb¶

Background¶

Getting Started¶

Frequently Asked Questions and Troubleshooting¶

How can I verify that Holoscan SDK debugging symbols have been loaded in gdb?¶

How can I pause a running Holoscan SDK application for debugging?¶

How can I manage breakpoints to pause the application sometime in the future?¶

How can I inspect local variables?¶

How can I attach to a running Holoscan SDK session?¶

I see a gdb Python Exception in the gdb log.¶

Debugging a Python application with pdb¶

Background¶

Prerequisites¶

Getting Started¶

Debugging Symbols for Legacy Holoscan SDK Versions¶

Frequently Asked Questions and Troubleshooting¶

How can I use a custom container path for my Holoscan SDK debugging build other than /opt/nvidia/holoscan?¶

How can I launch the tutorial application?¶

Logging¶

Logging from a C++ application¶

Logging from a Python application¶

Interactively Debugging a Holoscan Application ¶

Debugging a C++ Application with `gdb`¶

How can I verify that Holoscan SDK debugging symbols have been loaded in `gdb`?¶

I see a `gdb` Python Exception in the `gdb` log.¶

Debugging a Python application with `pdb`¶

How can I use a custom container path for my Holoscan SDK debugging build other than `/opt/nvidia/holoscan`?¶