# MedSegAgent Demo Webapp

A ChatGPT-style chat UI where you type a natural-language segmentation
request (e.g. "肝臓をセグメンテーションして" / "segment the liver") and
upload a CT or MRI volume (`.nii.gz`). The app parses the query with an LLM,
resolves the target structure against this repo's
[`dataset/TotalSegmentator_v2.json`](../dataset/TotalSegmentator_v2.json) /
[`TotalSegmentator_MRI.json`](../dataset/TotalSegmentator_MRI.json) label
maps, runs real [TotalSegmentator](https://github.com/wasserth/TotalSegmentator)
inference (`total` / `total_mr` tasks, Apache-2.0, no license key needed),
and shows the result overlaid on the volume with
[Niivue](https://niivue.com/).

This is purely additive: it does not modify anything in `agents/`, `utils/`,
`dataset/`, or the root `requirements.txt`.

Scope: single-target queries only (e.g. "segment the liver"), CT and MRI
only. TotalSegmentator's `fast` (3mm) mode is used for every request; there
is no quality toggle.

Result display follows the familiar image-chat pattern (ChatGPT/Gemini
style): each completed turn shows a small static thumbnail inline in its
chat bubble; tapping it opens a modal with the full interactive Niivue
viewer (multiplanar view, scrollable slices). There is one shared Niivue
canvas for the whole page, reused for both off-screen thumbnail rendering
and the on-screen modal.

## Features

- Natural-language query in English or Japanese ("segment the liver" /
  "肝臓をセグメンテーションして") -- an LLM extracts the target structure.
- Modality (CT vs. MRI) is auto-detected from the uploaded volume's own
  voxel intensities, not from the query text (a query often won't mention
  modality at all even though the file is unambiguously one or the other).
- Real segmentation via TotalSegmentator, not a mock -- the whole task is
  run and the requested structure is isolated into its own mask file.
- Chat-style result: a tappable thumbnail per turn, expanding into an
  interactive multiplanar volume viewer with an overlay on/off toggle.

## Project layout

```
webapp/
├── backend/
│   ├── main.py               # FastAPI app: /api/chat, /api/jobs/{id}, static file serving
│   ├── config.py              # paths, model name, fast-mode default
│   ├── query_parser.py        # LLM call: free text -> {modality, target}
│   ├── modality_detector.py   # CT vs MR from voxel intensities
│   ├── label_matcher.py       # target string -> TotalSegmentator label name/id
│   ├── segmentation_runner.py # runs TotalSegmentator (GPU-serialized)
│   ├── mask_extractor.py      # isolates just the target label into its own file
│   └── jobs.py                # in-memory job status store
├── frontend/
│   ├── index.html / app.js / style.css   # vanilla JS chat UI + Niivue viewer
└── data/                       # per-job uploads/outputs (gitignored) + smoke-test samples
```

## Setup

```bash
cd /home/nlab/MedSegAgent
source .venv/bin/activate

# Check for dependency conflicts with the pinned research deps
# (pyautogen==0.2.35, spacy==3.8.7, ...) before installing for real.
# If this wants to change already-installed torch/nnunetv2/spacy/pyautogen
# versions, stop and reconsider (e.g. a separate venv) rather than proceeding.
pip install --dry-run -r webapp/requirements.txt

pip install -r webapp/requirements.txt
```

`OAI_CONFIG_LIST` at the repo root must already contain a working entry for
the model named in `webapp/backend/config.py` (`MODEL_NAME`, default
`"deepseek-v4-flash"`).

## Run

```bash
cd /home/nlab/MedSegAgent
source .venv/bin/activate
uvicorn webapp.backend.main:app --host 0.0.0.0 --port 8000 --reload
```

Then open `http://localhost:8000/` (WSL2 forwards this to the Windows host
automatically).

## Smoke test

Grab TotalSegmentator's own bundled test fixtures (small, real NIfTI
volumes, safe to use for a demo):

```bash
mkdir -p webapp/data/samples
curl -L -o webapp/data/samples/example_ct_sm.nii.gz \
  https://raw.githubusercontent.com/wasserth/TotalSegmentator/master/tests/reference_files/example_ct_sm.nii.gz
curl -L -o webapp/data/samples/example_mr_sm.nii.gz \
  https://raw.githubusercontent.com/wasserth/TotalSegmentator/master/tests/reference_files/example_mr_sm.nii.gz
```

Backend only (no browser needed):

```bash
curl -F "query=segment the liver" \
     -F "file=@webapp/data/samples/example_ct_sm.nii.gz" \
     http://localhost:8000/api/chat
# -> {"job_id": "...", ...}
curl http://localhost:8000/api/jobs/<job_id>
# poll until "status": "done"
```

Then repeat with a Japanese query (`肝臓をセグメンテーションして`) and with
the MRI fixture (`query=segment the spleen`,
`file=@webapp/data/samples/example_mr_sm.nii.gz`) to exercise both language
handling and the MRI path.

Finally, drive the same three queries from the browser UI and confirm the
Niivue canvas shows a plausible overlay, and the "オーバーレイを表示"
checkbox toggles it on/off.

## Known limitations

- Every request segments the *entire* task (all ~117 CT / ~56 MRI
  structures), not just the requested one. An earlier version used
  TotalSegmentator's `roi_subset` to restrict inference to just the target
  structure for speed, but testing found that `roi_subset`'s preliminary
  low-resolution "rough crop" detection pass can silently miss small
  structures (reproduced: `roi_subset=["spleen"]` on `total_mr` came back
  completely empty, while `roi_subset=["liver"]` on the same task worked).
  Running the full task every time is slower but has been reliably correct;
  see `segmentation_runner.py`'s docstring.
- Single-user/single-GPU demo: concurrent requests are serialized on a GPU
  lock (`segmentation_runner.GPU_LOCK`), not parallelized.
- Job data (`webapp/data/<job_id>/`) is cleaned up automatically after ~1
  hour (`config.JOB_MAX_AGE_SECONDS`), but is otherwise unbounded during
  that window -- fine for a demo, not for unattended long-term operation.
- No auth, no persistence across server restarts (jobs are in-memory).