MedSegAgent: A Universal and Scalable Multi-Agent System for Instructive Medical Image Segmentation

This work has been accepted by the IEEE Journal of Biomedical and Health Informatics (JBHI). Paper: IEEE Xplore | PDF.

Overview of the MedSegAgent framework: natural language query parsing, coarse-to-fine dataset matching, and final segmentation with result integration.

MedSegAgent is a multi-agent system for instructive medical image segmentation. Instead of training one universal segmentation model, it orchestrates specialized dataset-specific models through natural language understanding, coarse-to-fine dataset matching, and execution-time result integration.

Key Features:

• Universal & Scalable: Tackle diverse medical image segmentation tasks using natural language instructions.
• Precise Automation: Automatically selects the most suitable segmentation models.
• Enhanced Robustness: Improves reliability through multi-model integration and ensemble capabilities.

Overview

MedSegAgent parses a free-form segmentation request, filters candidate datasets from modality to anatomy to label, and then runs the matched segmentation models. The current repository organizes each integrated dataset with a standardized JSON metadata entry in dataset/, making it straightforward to extend the model library with new tasks.

In the paper setting, MedSegAgent integrates 23 datasets and supports 343 segmentation targets across CT, MRI, PET/CT, and ultrasound-related scenarios.

Skill

This repository also includes a reusable skill at skills/medsegagent-nnunet-runner/ for inspecting nnUNet_results, selecting a deployed task, and running or preparing nnUNet inference.

The skill is intended for environments that already have nnUNet v2 available. To run inference, the target environment must provide a configured nnUNet_results path and usable nnUNet CLI or Python API support.

Demo Webapp

webapp/ contains a standalone, ChatGPT-style chat demo: type a free-text segmentation request and upload a CT/MRI .nii.gz volume, and it runs real TotalSegmentator inference (modality auto-detected from the volume) and shows the result overlaid in-browser via Niivue. See webapp/README.md for setup and details. Once dependencies are installed (pip install -r webapp/requirements.txt), launch it with:

uvicorn webapp.backend.main:app --host 0.0.0.0 --port 8000 --reload

Thttp://localhost:8000/`.

Supported Datasets

The repository currently includes metadata for the following datasets and targets. The summary below is adapted from the dataset table in the paper. The links point to official dataset access pages rather than direct-download mirrors hosted by this repository; some datasets still require registration, challenge participation, or a signed data-use agreement before download.

Quick Start:

Setup your environment

conda create -n medsegagent python=3.12
conda activate medsegagent
pip install uv
uv pip install -r requirements.txt

Then set your LLM service API keys like OAI_CONFIG_LIST.example, the config file should be named OAI_CONFIG_LIST.

[
    {
        "model": "Qwen/Qwen2.5-32B-Instruct",
        "api_key": "<Your API KEY>",
        "base_url": "https://api.siliconflow.cn/v1",
        "tags": ["silicon"]
    },
    {
        "model": "gpt-4o-2024-08-06",
        "api_key": "<Your API KEY>",
        "base_url": "<your BASE URL>",
        "tags": ["openai"]
    }
]

Test script

Start from eval_example.sh to try our Coarse-to-Fine setting of seg model selection.

python ./evaluate.py \
    --test_file_path "model_selection_test_case.jsonl" \
    --test_pattern "C2F"  \
    --model "gpt-4o-2024-08-06"    \
    --log_to_file

Acknowledgments

This project builds on the open contributions of the medical image segmentation community. We gratefully acknowledge the creators and maintainers of the public datasets integrated in MedSegAgent, whose annotations, benchmarks, and challenge platforms make this system possible.

We also acknowledge nnU-Net, which provides the strong self-configuring segmentation framework used for the dataset-specific models in our study.

TODO

• Upload the trained segmentation models.

Citation

If you use MedSegAgent in your research, please cite:

@ARTICLE{11455620,
  author={Huang, Ziyan and Wang, Haoyu and Ye, Jin and Ji, Yuanfeng and Hu, Xiaowei and Liu, Lihao and Yang, Zhikai and Li, Wei and Hu, Ming and Su, Yanzhou and Li, Tianbin and Gu, Yun and Zhang, Shaoting and Qiao, Yu and Gu, Lixu and He, Junjun},
  journal={IEEE Journal of Biomedical and Health Informatics},
  title={MedSegAgent: A Universal and Scalable Multi-Agent System for Instructive Medical Image Segmentation},
  year={2026},
  volume={},
  number={},
  pages={1-12},
  keywords={Image segmentation;Medical diagnostic imaging;Filtering;Solid modeling;Natural languages;Computed tomography;Computational modeling;Accuracy;Multi-agent systems;Liver;Universal Medical Image Segmentation;Multi-Agent System;Natural Language Instruction},
  doi={10.1109/JBHI.2026.3677444}
}