Part of the Detecture monorepo — this is the evaluation component. The model under evaluation lives at ../Qwen2SAM_Detecture/; the data pipeline that produced its training set lives at ../DetectureMiner/.

Unified benchmark harness that runs four method families on four datasets under a strict, dataset-dependent fairness protocol. One command reproduces every row of the paper's main comparison table.

A fifth row (TextureSAM) is included by copying the published paper numbers; we do not re-run that checkpoint.

Per-dataset "regime" — controls which oracle hints (if any) each method receives. Enforced by the DISPATCH table inside master_runner.py so no ad-hoc prompt can silently leak K_GT back in.

Every method's output is scored through the same metrics_utils.py scorer — Softmax + static dustbin + scipy.optimize.linear_sum_assignment on 1 − IoU + ARI on the predicted partition — so mIoU and ARI are directly comparable across methods and datasets.

cd ~/Detecture
pip install -r requirements.txt      # root requirements.txt

# SAM3 — clone separately and point to it (not pip-installable).
git clone https://github.com/facebookresearch/sam3.git ~/sam3
pip install -e ~/sam3
export SAM3_ROOT=~/sam3               # optional; defaults to ~/sam3

# For SA2VA: install a stub flash_attn (no real kernels) so the model loads.
# Minimal stub — a package directory named `flash_attn` with the names listed
# in eval_vlm_end2end.py (index_first_axis, pad_input, unpad_input, etc.).
# Details inline in that file.

HuggingFace weights download on first run:

• Qwen/Qwen3-VL-8B-Instruct
• ByteDance/Sa2VA-4B pinned to revision b5ffed22 (eval_vlm_end2end.py)
• IDEA-Research/grounding-dino-tiny
• SAM3 via build_sam3_image_model(load_from_HF=True)

All four datasets use a unified metadata JSON. Paths are read from datasets_config.yaml; the defaults expect:

Override the root with DETECTURE_DATASETS_ROOT=/some/where or edit the config. See the root README for Hugging Face download instructions.

Unified metadata entry:

{
  "image_path": "/abs/path/image.jpg",
  "id": "sample_id",
  "textures": [
    {"description": "…", "mask_path": "/abs/path/mask_0.png"},
    {"description": "…", "mask_path": "/abs/path/mask_1.png"}
  ]
}

# Everything (4 methods × 4 datasets = 16 cells)
python master_runner.py

# One method, all datasets
python master_runner.py --model detecture

# One dataset, all methods (smoke)
python master_runner.py --dataset RWTD --limit 10

# One specific cell
python master_runner.py --model sa2va --dataset STLD --limit 10

# Preview commands without spawning anything
python master_runner.py --dry-run

Each (model × dataset) cell spawns as a subprocess — clean GPU state per model (Detecture alone occupies ~16 GB in bf16 and cannot co-reside with a second backbone).

Outputs land under results/<model>/<dataset>/zero_shot_results.json. The runner also writes a combined results/paper_benchmark_summary.json with the dispatch metadata and per-cell summaries.

With best.pt + SAM3 + Grounding-DINO-tiny + SA2VA-4B (rev b5ffed22):

Detecture leads mIoU on every multi-texture route. CAID (single "water surface" class) is where SAM3 wins honestly — noted in the paper as the expected cost of generality.

Aggregate + paper-ready CSV / LaTeX:

python aggregate_results.py --csv results/summary.csv --latex results/summary.tex

• All CLI defaults resolve through Path.home() / environment variables — no absolute paths anywhere in the codebase.
• Sa2VA's HF revision is pinned to b5ffed22 in eval_vlm_end2end.py so upstream modeling changes don't silently alter numbers.
• Each subprocess logs the exact command it ran into the per-cell JSON summary block for traceability.