MedSegAgent Demo Webapp

A ChatGPT-style chat UI where you type a natural-language segmentation request (e.g. "肝臓をセグメンテーションして" / "segment the liver") and upload a CT or MRI volume (.nii.gz). The app parses the query with an LLM, resolves the target structure against this repo's dataset/TotalSegmentator_v2.json / TotalSegmentator_MRI.json label maps, runs real TotalSegmentator inference (total / total_mr tasks, Apache-2.0, no license key needed), and shows the result overlaid on the volume with Niivue.

This is purely additive: it does not modify anything in agents/, utils/, dataset/, or the root requirements.txt.

Scope: single-target queries only (e.g. "segment the liver"), CT and MRI only. TotalSegmentator's fast (3mm) mode is used for every request; there is no quality toggle.

Result display follows the familiar image-chat pattern (ChatGPT/Gemini style): each completed turn shows a small static thumbnail inline in its chat bubble; tapping it opens a modal with the full interactive Niivue viewer (multiplanar view, scrollable slices). There is one shared Niivue canvas for the whole page, reused for both off-screen thumbnail rendering and the on-screen modal.

Features

• Natural-language query in English or Japanese ("segment the liver" / "肝臓をセグメンテーションして") -- an LLM extracts the target structure.
• Modality (CT vs. MRI) is auto-detected from the uploaded volume's own voxel intensities, not from the query text (a query often won't mention modality at all even though the file is unambiguously one or the other).
• Real segmentation via TotalSegmentator, not a mock -- the whole task is run and the requested structure is isolated into its own mask file.
• Chat-style result: a tappable thumbnail per turn, expanding into an interactive multiplanar volume viewer with an overlay on/off toggle.

Project layout

webapp/
├── backend/
│   ├── main.py               # FastAPI app: /api/chat, /api/jobs/{id}, static file serving
│   ├── config.py              # paths, model name, fast-mode default
│   ├── query_parser.py        # LLM call: free text -> {modality, target}
│   ├── modality_detector.py   # CT vs MR from voxel intensities
│   ├── label_matcher.py       # target string -> TotalSegmentator label name/id
│   ├── segmentation_runner.py # runs TotalSegmentator (GPU-serialized)
│   ├── mask_extractor.py      # isolates just the target label into its own file
│   └── jobs.py                # in-memory job status store
├── frontend/
│   ├── index.html / app.js / style.css   # vanilla JS chat UI + Niivue viewer
└── data/                       # per-job uploads/outputs (gitignored) + smoke-test samples

Setup

cd /home/nlab/MedSegAgent
source .venv/bin/activate

# Check for dependency conflicts with the pinned research deps
# (pyautogen==0.2.35, spacy==3.8.7, ...) before installing for real.
# If this wants to change already-installed torch/nnunetv2/spacy/pyautogen
# versions, stop and reconsider (e.g. a separate venv) rather than proceeding.
pip install --dry-run -r webapp/requirements.txt

pip install -r webapp/requirements.txt

OAI_CONFIG_LIST at the repo root must already contain a working entry for the model named in webapp/backend/config.py (MODEL_NAME, default "deepseek-v4-flash").

Run

cd /home/nlab/MedSegAgent
source .venv/bin/activate
uvicorn webapp.backend.main:app --host 0.0.0.0 --port 8000 --reload

Thttp://localhost:8000/` (WSL2 forwards this to the Windows host automatically).

Smoke test

Grab TotalSegmentator's own bundled test fixtures (small, real NIfTI volumes, safe to use for a demo):

mkdir -p webapp/data/samples
curl -L -o webapp/data/samples/example_ct_sm.nii.gz \
  https://raw.githubusercontent.com/wasserth/TotalSegmentator/master/tests/reference_files/example_ct_sm.nii.gz
curl -L -o webapp/data/samples/example_mr_sm.nii.gz \
  https://raw.githubusercontent.com/wasserth/TotalSegmentator/master/tests/reference_files/example_mr_sm.nii.gz

Backend only (no browser needed):

curl -F "query=segment the liver" \
     -F "file=@webapp/data/samples/example_ct_sm.nii.gz" \
     http://localhost:8000/api/chat
# -> {"job_id": "...", ...}
curl http://localhost:8000/api/jobs/<job_id>
# poll until "status": "done"

Then repeat with a Japanese query (肝臓をセグメンテーションして) and with the MRI fixture (query=segment the spleen, file=@webapp/data/samples/example_mr_sm.nii.gz) to exercise both language handling and the MRI path.

Finally, drive the same three queries from the browser UI and confirm the Niivue canvas shows a plausible overlay, and the "オーバーレイを表示" checkbox toggles it on/off.

Known limitations

• Every request segments the entire task (all ~117 CT / ~56 MRI structures), not just the requested one. An earlier version used TotalSegmentator's roi_subset to restrict inference to just the target structure for speed, but testing found that roi_subset's preliminary low-resolution "rough crop" detection pass can silently miss small structures (reproduced: roi_subset=["spleen"] on total_mr came back completely empty, while roi_subset=["liver"] on the same task worked). Running the full task every time is slower but has been reliably correct; see segmentation_runner.py's docstring.
• Single-user/single-GPU demo: concurrent requests are serialized on a GPU lock (segmentation_runner.GPU_LOCK), not parallelized.
• Job data (webapp/data/<job_id>/) is cleaned up automatically after ~1 hour (config.JOB_MAX_AGE_SECONDS), but is otherwise unbounded during that window -- fine for a demo, not for unattended long-term operation.
• No auth, no persistence across server restarts (jobs are in-memory).