Skip to content

Multi AI SSD Detection and MONAI Endoscopic Tool Segmentation#

Authors: Holoscan Team (NVIDIA)
Supported platforms: x86_64, aarch64
Language: Python
Last modified: May 13, 2025
Latest version: 1.0
Minimum Holoscan SDK version: 0.6.0
Tested Holoscan SDK versions: 0.6.0
Contribution metric: Level 2 - Trusted

In this application, we show how to build a Multi AI application with detection and segmentation models, write postprocessing operators using CuPy and NumPy in Python tensor interop and MatX library (An efficient C++17 GPU numerical computing library with Python-like syntax) in C++ tensor interop, and pass multiple tensors from postprocessing to Holoviz.


Fig. 1 Endoscopy (laparoscopy) image from a cholecystectomy (gallbladder removal surgery) showing tool detection and segmentation results from two concurrently executed AI models. Image courtesy of Research Group Camma, IHU Strasbourg and the University of Strasbourg (NGC Resource)

Please refer to the README under ./app_dev_process to see the process of developing the applications.

The application graph looks like:

Model#

We combine two models from the single model applications SSD Tool Detection and MONAI Endoscopic Tool Segmentation:

Data#

📦️ (NGC) Sample App Data for AI-based Endoscopy Tool Tracking

Requirements#

Ensure you have installed the Holoscan SDK via one of the methods specified in the SDK user guide.

The directory specified by --data at app runtime is assumed to contain three subdirectories, corresponding to the three NGC resources specified in Model and Data: endoscopy, monai_tool_seg_model and ssd_model. These resources will be automatically downloaded to the holohub data directory when building the application.

Building the application#

The repo level build command 

./run build multiai_endoscopy
will build one of the cpp apps post-proc-cpu.

Running the application#

Python Apps#

To run the Python application, you can make use of the run script 

./run launch multiai_endoscopy python
Alternatively, to run this application, you'll need to configure your PYTHONPATH environment variable to locate the necessary python libraries based on your Holoscan SDK installation type.

You should refer to the glossary for the terms defining specific locations within HoloHub.

If your Holoscan SDK installation type is:

  • python wheels:
export PYTHONPATH=$PYTHONPATH:<HOLOHUB_BUILD_DIR>/python/lib
  • otherwise:

export PYTHONPATH=$PYTHONPATH:<HOLOSCAN_INSTALL_DIR>/python/lib:<HOLOHUB_BUILD_DIR>/python/lib
Next, run the application: 
cd <HOLOHUB_SOURCE_DIR>/applications/multiai_endoscopy/python
python3 multi_ai.py --data <DATA_DIR>

C++ Apps#

There are three versions of C++ apps, with the only difference being that they implement the inference post-processing operator DetectionPostprocessorOp in different ways:

  • post-proc-cpu: Multi-AI app running the inference post-processing operator on the CPU using std features only.
  • post-proc-matx-cpu: Multi-AI app running the inference post-processing operator on the CPU using the MatX library).
  • post-proc-matx-gpu: Multi-AI app running the inference post-processing operator on the GPU using MatX (CUDA).

To run post-proc-cpu, since it already gets built with ./run build multiai_endoscopy: 

./run launch multiai_endoscopy cpp

For the other two C++ applications, you'll need to build these without the run script as follows.

To run post-proc-matx-cpu or post-proc-matx-gpu, first navigate to the app directory.

cd cpp/post-proc-matx-cpu

Next we need to configure and build the app.

Configuring#

First, create a build folder:

mkdir -p build

then run CMake configure with:

cmake -S . -B build

Unless you make changes to CMakeLists.txt, this step only needs to be done once.

Building#

The app can be built with:

cmake --build build

or equally:

cd build
make

Running#

You can run the app with:

./build/multi_ai --data <DATA_DIR>