Skip to content

UCX-based Distributed Endoscopy Tool Tracking#

Authors: Holoscan Team (NVIDIA)
Supported platforms: x86_64, aarch64
Language: Python
Last modified: August 5, 2025
Latest version: 1.0
Minimum Holoscan SDK version: 2.1.0
Tested Holoscan SDK versions: 2.1.0
Contribution metric: Level 1 - Highly Reliable

Digital endoscopy is a key technology for medical screenings and minimally invasive surgeries. Using real-time AI workflows to process and analyze the video signal produced by the endoscopic camera, this technology helps medical professionals with anomaly detection and measurements, image enhancements, alerts, and analytics.


Fig. 1 Endoscopy (laparoscopy) image from a cholecystectomy (gallbladder removal surgery) showing AI-powered frame-by-frame tool identification and tracking. Image courtesy of Research Group Camma, IHU Strasbourg and the University of Strasbourg (NGC Resource)

The Distributed Endoscopy Tool Tracking application provides an example of how an endoscopy data stream can be captured and processed using the C++ or Python APIs on multiple hardware platforms.

The Distributed Endoscopy Tool Tracking application is very similar to the Endoscopy Tool Tracking application but divides all operators into three fragments as depicted below. This allows fragments to run on different systems. For example, one can run the Video Input Fragment and the Visualization Fragment on a radiology workstation while the Inference Fragment runs on a more powerful system with multiple GPUs. Refer to the Holoscan SDK User Guide for more information on distributed applications.

Video Stream Replayer Input#


Fig. 2 Tool tracking application workflow with replay from file

The pipeline uses a recorded endoscopy video file (generated by convert_video_to_gxf_entities script) for input frames. Each input frame in the file is loaded by Video Stream Replayer and passed to the following two pipeline branches:

  • In the first branch, the input frames are directly passed to Holoviz in the Visualization Fragment for rendering in the background.
  • In the second branch, the frames go through the Format Converter in the Inference Fragment to convert the data type of the image from uint8 to float32 before it is fed to the tool tracking model (with Custom TensorRT Inference). The result is then ingested by the Tool Tracking Postprocessor which extracts the masks, points, and text from the inference output, before Holoviz renders them as overlays in the Visualization Fragment .

Dev Container#

To start the the Dev Container, run the following command from the root directory of Holohub:

./holohub vscode

VS Code Launch Profiles#

C++#

There are several launch profiles configured for the C++ version of this application:

  1. (gdb) ucx_endoscopy_tool_tracking/cpp (all fragments): Launches all fragments in a single process in debug mode.
  2. (gdb) ucx_endoscopy_tool_tracking/cpp - video_in fragment: Starts the video_in fragment in debug mode.
  3. (gdb) ucx_endoscopy_tool_tracking/cpp - inference fragment: Starts the inference fragment in debug mode.
  4. (gdb) ucx_endoscopy_tool_tracking/cpp - viz fragment: Starts the viz fragment in debug mode.
  5. (compound) ucx_endoscopy_tool_tracking/cpp: Starts #2, #3, #4 in sequence.

Python#

There are several launch profiles configured for this application:

  1. (debugpy) ucx_endoscopy_tool_tracking/python (all fragments): Launches all fragments in a single process in debug mode.
  2. (debugpy) ucx_endoscopy_tool_tracking/python - video_in fragment): Launches all fragments in a single process in debug mode.
  3. (pythoncpp) ucx_endoscopy_tool_tracking/python - video_in fragment: Starts the video_in fragment in debug mode.
  4. (debugpy) ucx_endoscopy_tool_tracking/python - inference fragment: Starts the video_in fragment in debug mode.
  5. (pythoncpp) ucx_endoscopy_tool_tracking/python - inference fragment: Starts the inference fragment in debug mode.
  6. (debugpy) ucx_endoscopy_tool_tracking/python - viz fragment: Starts the inference fragment in debug mode.
  7. (pythoncpp) ucx_endoscopy_tool_tracking/python - viz fragment: Starts the viz fragment in debug mode.
  8. (compound) ucx_endoscopy_tool_tracking/python: Starts #2, #4, #6 in sequence.

Note: Launch profiles prefixed with debugpy enables debugging of Python code only. Use pythoncpp to debug both C++ and Python code.