Virtues — Multiplex Imaging (IMC / MCD) Foundation Model
Wrapper skill around the VirTues (virtues) foundation model for multiplexed spatial
proteomics, plus a parameterised CLI that reproduces the upstream batch pipeline
(MCD → TIFF → spora → VirTues → per-ROI results → merged UMAP).
> Verification scope: only MCD/IMC has been validated. The source notebook
> (batch_all_mcd_all_rois.ipynb) was run end-to-end on two IMC .mcd files. Other
> multiplex platforms that spora-io nominally supports (CODEX, CyCIF, MIBI) have NOT
> been tested — treat every non-IMC path as unverified.
Overview
Upstream
| Resource | Location |
|---|---|
| Code (MIT) | https://github.com/bunnelab/virtues |
| Model weights | https://huggingface.co/bunnelab/virtues (virtues-sp32, virtues-sp31, virtues-imc14) |
| Paper | https://www.nature.com/articles/s41586-026-10884-y (arXiv 2501.06039) |
| spora [io] loader | https://github.com/bunnelab/spora-io |
| spora [data] format | https://spora.epfl.ch/datasets.html |
| Raw data used here | Zenodo record 15495774 (5269a.mcd, 6362a.mcd, ~2.6 GB total) |
VirTues is a marker-aware masked-autoencoder ViT for spatial proteomics. From one backbone it
supports, and this skill covers, four downstream uses:
- Marker reconstruction / inpainting — hide a marker (fully or partially) and predict it.
Diagnostic only; not a prerequisite for anything else. - Cell instance + cell-type segmentation — a segmentation head (
VirtuesSegmentationHead,
built on instanseg) predicts instance masks and a semantic cell-type map. - Cell summary tokens — a 512-D VirTues representation per segmented cell.
- Cross-ROI UMAP — one UMAP over all cells of all ROIs, with per-cell provenance.
Use when you have multiplex imaging (IMC .mcd from a Hyperion/Fluidigm-type instrument),
want marker-aware representations / cell typing without a panel-specific model, or want
patient/ROI-comparable cell embeddings for biomarker work.
Do not use when you only have bulk RNA-seq, or you cannot obtain a GPU: the reference run
required CUDA (Tesla T4, 15 GB) and whole-tissue segmentation of one 1500×1500 ROI takes
~8–10 minutes at ~6 it/s (3481 tiles).
The design principle of the upstream project is that VirTues never reads raw TIFF/RAW: all
imaging must first be materialised as spora [data] (OME-Zarr v3 image + parquet metadata +
npz masks). That conversion step is the bulk of the work.
Inputs
1. spora [data] dataset — the "three-piece set"
Layout used by the source project (27-channel panel generated with Steinbock):
<SPORA_DATASETS_DIR>/
└── nsclc_spora_dataset/
├── metadata/
│ ├── tissues.parquet # tissue_id, patient_id, modality, sample_id, roi_id,
│ │ # source_roi, source_mcd, source_file
│ └── roi_manifest.parquet # sample_id, roi_id, source_mcd, source_file, tissue_id
├── imc/
│ ├── channels.parquet # channel_name, uniprot_id, qc_pass, is_nuclear_marker
│ ├── channels_per_tissue.parquet # tissue_id × channel availability (all True here)
│ └── 1_0mpp/
│ ├── images/<tissue_id>.ome.zarr/0 # Zarr v3 array, dimension_names=("c","y","x")
│ └── standardization/quantile_clipping_log1p/uq_0.99_image/
│ ├── image_level_upper_quantiles.parquet # per-ROI 0.99 quantile
│ ├── global_level_means.parquet # dataset-global mean
│ └── global_level_stds.parquet # dataset-global std
└── segmentations/1_0mpp/
├── tissue_masks/ # npz tissue masks (may be empty → full-frame mask)
└── cell_masks/instances_virtues/<tissue_id>.npz # written by this pipeline
- OME-Zarr v3:
zarr.open_group(path, mode="w", zarr_format=3), one array"0"with
dimension_names=("c","y","x"), chunks(1, min(512,H), min(512,W)), plus minimal
OME-NGFF 0.5multiscalesattrs. spora reads<tissue_id>.ome.zarr/0. - parquet: tissue/channel provenance.
roi_idis zero-padded to 3 digits. - npz masks:
tissue_masks/<tissue_id>.npz(optional) and
cell_masks/instances_virtues/<tissue_id>.npzwith keymask(int instance ids). - Standardization trio:
quantile_level=image,stats_level=global— i.e. each ROI gets
its own 0.99 quantile, but mean/std are global across all ROIs so ROIs are comparable.
2. Marker panel + ESM-2 marker embeddings
For every measured marker a protein embedding must exist, named [UniProtID].pt, stored
directly in the embeddings directory (not in a model-name sub-directory):
python -m virtues.utils.download_fastas \
--output_dir <FASTA_DIR> --input <uniprot_ids.csv> --id_column uniprot_id
python -m virtues.utils.compute_esm_embeddings \
--input_dir <FASTA_DIR> --output_dir <DATASET>/marker_embeddings \
--device cuda --model esm2_t30_150M_UR50D
Reference panel: 20 UniProt IDs → 20 .pt files → embedding matrix (20, 640)
(ESM-2 esm2_t30_150M_UR50D). The loader returns both the stacked matrix
(load_marker_embeddings) and a uniprot_id → row index dict (load_marker_embedding_dict);
the dict is used to build the per-channel midxs tensor.
Panel actually used (Steinbock name → canonical name → UniProt):
RAW_TO_CANONICAL = {"a-SMA":"ACTA2","Vimentin":"VIM","CD14":"CD14","CD16":"FCGR3A",
"Pan-K":"PanCK","PD-L1":"CD274","CD57":"CD57","CD11c":"ITGAX","FoxP3":"FOXP3",
"CD4":"CD4","E-Cadh":"CDH1","CD68":"CD68","CD20":"MS4A1","CD8a":"CD8A","PD-1":"PDCD1",
"GrzB":"GZMB","Ki-67":"MKI67","Coll_I":"COL1A1","CD3":"CD3E","CD45RO":"PTPRC",
"HLA_DR":"HLA-DRA","H3":"H3C1","DNA1":"DNA1","DNA2":"DNA2",
"CSK1":"CSK1","CSK2":"CSK2","CSK3":"CSK3"} # 27 channels total
UNIPROT_MAP = {"ACTA2":"P62736","VIM":"P08670","CD14":"P08571","FCGR3A":"P08637",
"PanCK":None,"CD274":"Q9NZQ7","CD57":None,"ITGAX":"P20702","FOXP3":"Q9BZS1",
"CD4":"P01730","CDH1":"P12830","CD68":"P34810","MS4A1":"P11836","CD8A":"P01732",
"PDCD1":"Q15116","GZMB":"P10144","MKI67":"P46013","COL1A1":"P02452","CD3E":"P07766",
"PTPRC":"P08575","HLA-DRA":"P01903","H3C1":"P68431","DNA1":None,"DNA2":None,
"CSK1":None,"CSK2":None,"CSK3":None} # None → dropped by uniprot_filtered
None UniProt ⇒ channel is dropped when kind="uniprot_filtered" is requested. With
replace_nuclear_uniprot_ids=True, the 3 nuclear channels (H3C1, DNA1, DNA2) are all
re-mapped to P68431, and the uniprot mask ends up with 22 model-visible channels.
3. Model checkpoints
| File | Role |
|---|---|
virtues-sp32/model.safetensors |
backbone (marker-aware ViT), loaded with safetensors.torch.load_file |
virtues-sp32/segmentation-head.safetensors |
instance + semantic head, load_state_dict(..., strict=False) |
virtues-sp32 (32 datasets) is CC BY-NC 4.0; virtues-sp31 is MIT — pick per licence needs.
virtues-imc14 has no segmentation head.
4. Raw MCD (only if you must start from scratch)
# Zenodo record 15495774 — 2 MCD files (~2.6 GB) + metadata workbooks
wget -c "https://zenodo.org/records/15495774/files/5269a.mcd?download=1" -O 5269a.mcd
wget -c "https://zenodo.org/records/15495774/files/6362a.mcd?download=1" -O 6362a.mcd
wget -c "https://zenodo.org/records/15495774/files/IMC_panel.xlsx?download=1" -O IMC_panel.xlsx
wget -c "https://zenodo.org/records/15495774/files/IMC_images.xlsx?download=1" -O IMC_images.xlsx
wget -c "https://zenodo.org/records/15495774/files/_md_Outcome.xlsx?download=1" -O md_Outcome.xlsx
steinbock preprocess imc panel # writes panel.csv next to raw/
steinbock preprocess imc images --hpf 50 # MCD → multi-channel TIFF, one per ROI
Steps
Step 0 — environment
export SPORA_DATASETS_DIR=/path/to/spora # root containing <dataset_name>/
export HF_HOME=/path/to/cache/huggingface # + XDG_CACHE_HOME / TORCH_HOME as needed
git clone https://github.com/bunnelab/virtues.git && cd virtues && pip install -e .
git clone https://github.com/bunnelab/spora-io.git && cd spora-io && pip install -e .
conda activate /path/to/env/virtues # env created by upstream setup.sh
Step 1 — MCD → TIFF → spora [data]
Run Steinbock (Step 0 of Inputs §4), build a ROI manifest by regex-parsing
^(?P<sample>.+)_(?P<roi>\d+)\.(tif|tiff)$, then write OME-Zarr + parquet per ROI. Keep
sample_id / roi_id / source_mcd / source_file in tissues.parquet from the very beginning —
this is what later makes every cell in the UMAP traceable back to its MCD and ROI.
img = tifffile.imread(tiff_path) # Steinbock IMC TIFF: (C, H, W)
img = np.asarray(img[selected_indices], dtype=np.float32) # keep the 27 panel channels
root = zarr.open_group(str(output_path), mode="w", zarr_format=3)
root.create_array("0", data=img, chunks=(1, min(512, h), min(512, w)),
dimension_names=("c", "y", "x"))
root.attrs["ome"] = {"version": "0.5", "multiscales": [{"name": tissue_id, "axes": [
{"name": "c", "type": "channel"},
{"name": "y", "type": "space", "unit": "micrometer"},
{"name": "x", "type": "space", "unit": "micrometer"}],
"datasets": [{"path": "0",
"coordinateTransformations": [{"type": "scale", "scale": [1.0, 1.0, 1.0]}]}]}]}
channels_df = pd.DataFrame({ # → imc/channels.parquet
"channel_name": channel_names_batch,
"uniprot_id": [UNIPROT_MAP[x] for x in channel_names_batch],
"qc_pass": True,
"is_nuclear_marker": [x in {"H3C1", "DNA1", "DNA2"} for x in channel_names_batch]})
Re-runnable by design: an existing <tissue_id>.ome.zarr is skipped unless you force
(FORCE_SPORA_CONVERT=True); existing ROIs keep their old tissue_id (matched on
source_file); _0000/_0001 style ids from earlier single-ROI runs coexist with
<prefix>_<sample>_roi<roi> ids assigned to new ROIs.
Step 2 — standardization statistics
cd <spora-io> && export SPORA_DATASETS_DIR=/path/to/spora
python scripts/compute_standardization_stats.py \
--dataset-name nsclc_spora_dataset --modality imc \
--method quantile_clipping_log1p --quantile-level image --stats-level global \
--upper-quantile 0.99 --resolution 1.0 [--overwrite]
Add --overwrite whenever a new ROI appears: the stats must cover all ROIs, otherwise
per-ROI quantiles and global mean/std are inconsistent and old segmentation / cell tokens
become incomparable. QC gate — all three tables must be NaN-free:
image_level_upper_quantiles.parquet → (n_tissues, n_channels); global_level_means.parquet
and global_level_stds.parquet → (1, n_channels). Reference: (8, 27), (1, 27), (1, 27),
0 NaN.
Step 3 — preflight: embeddings + checkpoints
required = set(pd.read_parquet(DATASET_PATH / "imc" / "channels.parquet")["uniprot_id"]
.dropna().astype(str))
available = {p.stem for p in PATH_MARKER_EMBEDDINGS.glob("*.pt")}
missing = sorted(required - available) # must be empty
# reference: required = 20, available = 20
for p in [MODEL_CKPT, SEGMENTATION_CKPT]: # both must exist locally
assert p.exists(), f"missing weights: {p} (compute nodes are usually offline)"
Step 4 — Tesla T4 / Turing attention fallback
T4 (compute capability 7.5) does not support the FlashAttention varlen kernel VirTues uses.
Monkey-patch the kernel at runtime only — training weights are untouched:
import torch.nn.functional as F
import virtues.modules.layers.attention_flashattention as attention_impl
def sdpa_varlen_qkvpacked_func(qkv, cu_seqlens, max_seqlen, dropout_p=0.0,
softmax_scale=None, causal=False, deterministic=False, **kwargs):
if causal:
raise NotImplementedError("Non-causal VirTues attention only.")
outputs = []
for start, end in zip(cu_seqlens.detach().cpu().tolist()[:-1],
cu_seqlens.detach().cpu().tolist()[1:]):
block = qkv[start:end] # (L, 3, heads, head_dim)
q = block[:, 0].permute(1, 0, 2).unsqueeze(0)
k = block[:, 1].permute(1, 0, 2).unsqueeze(0)
v = block[:, 2].permute(1, 0, 2).unsqueeze(0)
kw = {"dropout_p": dropout_p, "is_causal": False}
if softmax_scale is not None:
kw["scale"] = softmax_scale
outputs.append(F.scaled_dot_product_attention(q, k, v, **kw)
.squeeze(0).permute(1, 0, 2).contiguous())
return qkv.new_empty((0, qkv.shape[2], qkv.shape[3])) if not outputs \
else torch.cat(outputs, dim=0)
if torch.cuda.is_available() and torch.cuda.get_device_capability(0)[0] < 8:
attention_impl.flash_attn_varlen_qkvpacked_func = sdpa_varlen_qkvpacked_func
Step 5 — load backbone + segmentation head (once for all ROIs)
conf = OmegaConf.load(VIRTUES_ROOT / "configs" / "base_config.yaml")
marker_embeddings = load_marker_embeddings(str(PATH_MARKER_EMBEDDINGS))
marker_embedding_dict = load_marker_embedding_dict(str(PATH_MARKER_EMBEDDINGS))
model = MultiplexVirtues(
use_default_config=False, custom_config=None,
prior_bias_embeddings=marker_embeddings,
prior_bias_embedding_type="esm", prior_bias_embedding_fusion_type="add",
patch_size=conf.model.patch_size, model_dim=conf.model.model_dim,
feedforward_dim=conf.model.feedforward_dim,
encoder_pattern=conf.model.encoder_pattern, num_encoder_heads=conf.model.num_encoder_heads,
decoder_pattern=conf.model.decoder_pattern, num_decoder_heads=conf.model.num_decoder_heads,
num_hidden_layers=conf.model.num_decoder_hidden_layers,
positional_embedding_type=conf.model.positional_embedding_type,
dropout=conf.model.dropout, group_layers=conf.model.group_layers,
norm_after_encoder_decoder=conf.model.norm_after_encoder_decoder, verbose=False)
state = load_file(str(MODEL_CKPT), device="cpu") # CPU first → lower peak VRAM
model.load_state_dict(state); del state
model = model.to(DEVICE).eval()
segmentation_model = VirtuesSegmentationHead(virtues_model=model, dim_out=5, num_celltypes=9)
seg_state = load_file(str(SEGMENTATION_CKPT), device="cpu")
segmentation_model.load_state_dict(seg_state, strict=False); del seg_state
segmentation_model = segmentation_model.to(DEVICE).eval()
Step 6 — dataset, then per-ROI segmentation
dataset = MultiplexImagingDataset(
name="nsclc_spora_dataset", path=str(DATASET_PATH), modality="imc",
standardization="quantile_clipping_log1p/uq_0.99_image", resolution=1.0,
tile_size=conf.data.tile_size, # 128
disable_quantile_mask=True, filter_list=["gaussian_blur"],
use_mean_std=True, replace_nuclear_uniprot_ids=True)
for tissue_id in dataset.get_tissue_ids():
tissue = dataset.get_tissue(tissue_id, kind="uniprot_filtered")
x = tissue.image # (22, H, W), e.g. (22, 1500, 1500)
channel_names = np.asarray(tissue.channel_names).astype(str)
uniprot_ids = np.asarray(tissue.uniprot_ids).astype(str)
midxs = torch.tensor([marker_embedding_dict[u] for u in uniprot_ids], dtype=torch.long)
with torch.inference_mode(), torch.amp.autocast("cuda", enabled=(DEVICE == "cuda")):
pred_instance, semantic_logits = segmentation_model.segment_tissue(
x.to(DEVICE), midxs.to(DEVICE),
tile_size=128, overlap=32, batch_size=1,
large_tissue_threshold=1024) # force large-tissue stitching for 1500×1500
pred_instance_np = remove_small_cells(pred_instance.detach().cpu().numpy(), min_cell_size=15)
semantic_cpu = semantic_logits.detach().float().cpu()
semantic_cpu[[0, 7], :, :] = float("-inf") # drop "None" and "Unknown"
phenotype_map = assign_cell_types(pred_instance_np, torch.argmax(semantic_cpu, dim=0).numpy())
Cell-type mapping emitted by the head (9 classes; 0 and 7 are suppressed):
CELL_TYPE_MAPPING = {0:"None", 1:"B cell", 2:"CD4 T cell", 3:"CD8 T cell",
4:"Fibroblasts / Stroma", 5:"Myeloid", 6:"Tumor", 7:"Unknown",
8:"Vessel / Endo."}
Abort on pred_instance_np.max() == 0 (a known "Too many seeds" failure) instead of carrying an
empty mask into the token stage. Reuse an existing <tissue_id>.npz only if it is non-empty
and standardization was not recomputed.
Step 7 — cell tokens, then merged UMAP
Order matters: pad first, then standardize (the official cell-token path), and read the
un-padded image with preprocess=False:
tissue_raw = dataset.get_tissue(tissue_id, kind="uniprot_filtered", preprocess=False)
segmentation_mask = dataset.get_cell_instance_mask(
tissue_id, use_instances_from_virtues=True).mask
pad = 120
x_pad = torch.nn.functional.pad(tissue_raw.image, (pad,)*4, mode="constant", value=0)
mask_pad = torch.nn.functional.pad(segmentation_mask, (pad,)*4, mode="constant", value=0)
x_std, refined_mask = dataset.standardizer.apply(
x_pad, tissue_id, tissue_raw.measured_mask, tissue_raw.image_loading_mask)
with torch.inference_mode():
cell_ids, cell_tokens, crop_tokens, crop_indices = compute_cell_tokens(
model, x_std, midxs, mask_pad,
device=DEVICE, tile_size=128, patch_size=int(conf.model.patch_size),
stride=42, chunk_size=8) # reference: (18626, 512) tokens per ROI
Save per-ROI cell_tokens.parquet (columns virtues_000…virtues_511 + provenance
global_cell_id / tissue_id / sample_id / roi_id / source_mcd / source_file / cell_id +
predicted_cell_type), then merge every ROI and fit one UMAP:
from umap import UMAP
umap_model = UMAP(n_components=2, random_state=42)
cell_umap = umap_model.fit_transform(all_tokens[feature_cols].to_numpy(dtype=np.float32))
Assert global_cell_id is unique before the fit — duplicates mean broken provenance.
Running the bundled CLI
scripts/virtues_mcd_inference.py packages Steps 3–7 with no hardcoded paths. Every path
(including the SPORA_DATASETS_DIR root) is a flag:
python3 scripts/virtues_mcd_inference.py --help
python3 scripts/virtues_mcd_inference.py \
--spora-dir /path/to/spora \
--dataset-name nsclc_spora_dataset \
--virtues-root /path/to/virtues \
--checkpoint /path/to/ckpt/virtues-sp32/model.safetensors \
--segmentation-checkpoint /path/to/ckpt/virtues-sp32/segmentation-head.safetensors \
--marker-embedding /path/to/spora/nsclc_spora_dataset/marker_embeddings \
--output-dir /path/to/out \
--batch all --roi all \
--tile-size 128 --overlap 32 --batch-size 1 \
--device auto --stride 42 --chunk-size 8
Useful flags: --skip-reconstruction, --skip-segmentation, --skip-cell-tokens,
--skip-umap to run only part of the chain; --check-only for a cheap environment/asset audit
that imports nothing heavy and exits non-zero under --strict. --output-dir is optional — omit
it and nothing is written; give it and results land in <out>/Img (figures), <out>/Stat
(tables) and <out>/Rdata (objects/tensors), see Outputs.
Parameters
Values are the defaults of the validated run (notebook) and the CLI defaults.
| Parameter | Default | Where it comes from | Notes |
|---|---|---|---|
batch (MCD/sample) |
all (validated: 5269a, 6362a) |
CLI / ROI manifest | 2 MCDs, ~2.6 GB raw |
roi |
all (validated: 5269a 002–003, 6362a 001–005 → 7 ROIs) |
CLI / TARGET_ROIS |
sparse ROIs are skipped upstream |
--output-dir |
none |
CLI | output root (default: none); when given, creates Img/, Stat/, Rdata/ |
device |
auto → cuda (validated: Tesla T4, cc 7.5) |
CLI / torch.cuda.is_available() |
CPU is technically possible but impractical |
model checkpoint |
virtues-sp32/model.safetensors |
local file / HF bunnelab/virtues |
CC BY-NC 4.0 |
segmentation checkpoint |
virtues-sp32/segmentation-head.safetensors |
local file / HF | load with strict=False |
patch_size |
8 |
conf.model.patch_size |
mask grid = tile/8 |
model_dim |
512 |
conf.model.model_dim |
= cell-token dimension |
marker embedding model |
esm2_t30_150M_UR50D |
compute_esm_embeddings --model |
640-D per marker; 20 markers → (20, 640) |
marker embedding dir |
<dataset>/marker_embeddings |
CLI | .pt files must sit directly in it |
tile_size |
128 |
conf.data.tile_size / segmentation / cell tokens |
128×128 crops in demos |
segmentation overlap |
32 |
notebook | |
segmentation batch_size |
1 |
notebook (T4 VRAM) | 3481 tiles per 1500×1500 ROI |
large_tissue_threshold |
1024 |
notebook | 1500 px ROI hits the default 1500 boundary → must force stitching |
min_cell_size |
15 |
remove_small_cells |
drops tiny instances |
pad_size (cell tokens) |
120 |
official cell-token demo | padding gives edge cells spatial context |
stride (cell tokens) |
42 |
notebook | sliding-window stride |
chunk_size (cell tokens) |
8 |
notebook | 15 GB T4; drop to 4 on OOM |
num_celltypes / dim_out |
9 / 5 |
VirtuesSegmentationHead |
9 semantic classes, 5 output dims |
standardization |
quantile_clipping_log1p/uq_0.99_image |
spora-io script | quantile per image, mean/std global |
resolution |
1.0 (1_0mpp) |
dataset + stats + masks | keep consistent everywhere |
modality |
imc |
dataset + stats | tissues.parquet filtered on modality == "imc" |
filters |
["gaussian_blur"] |
dataset | |
disable_quantile_mask / use_mean_std / replace_nuclear_uniprot_ids |
True / True / True |
dataset | nuclear channels → P68431 |
| channels | 27 panel → 20 with UniProt → 22 model-visible | channels.parquet + uniprot_filtered |
|
| reconstruction target | CD3E in batch mode, CD8A in the single-ROI demo |
notebook | center crop of each ROI |
| masking ratio (partial) | (0.6, 1.0) |
generate_mask(C, H//8, W//8, masking_ratio=…) |
mask is on the patch grid |
| UMAP | n_components=2, random_state=42 |
notebook | 89208 cells × 512-D → (89208, 2) |
Reference numbers from the validated MCD run
7 ROIs → 89208 cells → 512-D tokens → one 2-D UMAP. Reconstruction of a fully hidden CD3E
(center 128×128): Pearson r 0.10–0.80, MAE 0.03–0.38 depending on the ROI. Cell counts per ROI:
18626, 12875, 14527, 1682, 13190, 14936, 13372.
Outputs
--output-dir is optional (default: none → nothing is written to disk). When it is given,
the script creates <out>/{Img,Stat,Rdata} immediately (idempotent) and writes everything below
into exactly one of the three:
Path (relative to --output-dir) |
Content |
|---|---|
Stat/<tissue_id>_virtues_predicted_cell_types.parquet |
one row per cell: cell_id, area_pixels, predicted_cell_type_id/_type, tissue_id, sample_id, roi_id, source_mcd, source_file, global_cell_id |
Rdata/<tissue_id>_virtues_phenotype_map.npz |
per-pixel phenotype id map |
Stat/<tissue_id>_cell_tokens.parquet |
512-D VirTues tokens + provenance per cell (table) |
Rdata/<tissue_id>_cell_tokens.pt |
the same 512-D tokens as a torch object (tissue_id, sample_id, roi_id, cell_ids, cell_tokens) |
Stat/all_rois_cell_phenotypes.parquet |
merged phenotype tables |
Stat/all_rois_cell_tokens.parquet |
merged 512-D token matrix (89208 × 512 here) |
Stat/all_rois_cell_umap.parquet |
UMAP1, UMAP2 + full provenance (roi_label = sample_ROIxxx) |
Img/all_rois_cell_umap_celltype_and_roi.png |
two-panel UMAP: by predicted cell type, by source ROI |
Img/segmentation_plots/<tissue_id>.png |
3-panel IMC / instances / phenotype per ROI |
Stat/all_rois_reconstruction_metrics.parquet |
MAE / RMSE / Pearson per ROI for the hidden marker |
Rdata/<tissue_id>_<marker>_center_crop.npz |
measured vs predicted crop |
<dataset>/segmentations/1_0mpp/cell_masks/instances_virtues/<tissue_id>.npz |
instance mask, reused as input by the token step (stays in the dataset: spora reads it back) |
<dataset>/metadata/roi_manifest.parquet |
MCD → sample → ROI → TIFF → tissue_id |
Rule of thumb: Img/ = figures, Stat/ = tables and text, Rdata/ = objects and tensors.
At the end of a run (and after --check-only) the CLI prints a per-directory file/size summary
of the three folders. <out>/.cache is the default --cache-dir and is not a result directory.
Pitfalls
- GPU is required in practice. The validated run used a Tesla T4 (15 GB). Whole-tissue
segmentation is ~8–10 min per 1500×1500 ROI (3481 tiles, ~6 it/s); 7 ROIs plus token
extraction is hours of GPU time. On Turing (cc < 8) apply the SDPA fallback of Step 4 or
attention will fail. - Weights are not on PyPI. Download
virtues-sp32(or-sp31) from
https://huggingface.co/bunnelab/virtues and keep bothmodel.safetensorsand
segmentation-head.safetensors;virtues-imc14ships no segmentation head. - Raw data is not in the repo. MCD/IMC data comes from Zenodo record 15495774.
- This container cannot run real inference.
torch,virtuesandspora_ioare absent,
and HuggingFace / Zenodo are not on the download allowlist — the CLI's--check-onlymode is
the only thing that can be exercised here; there is no GPU either. - Only MCD/IMC was ever validated (2 MCDs, 7 ROIs). CODEX, CyCIF, MIBI are untested — the
loader advertises them, the weights were trained on diverse technologies, but no end-to-end run
exists. Verify the panel mapping and the 1.0 µm resolution assumption per platform. - Marker embeddings must lie directly in the directory —
<dir>/*.pt. Files left inside a
esm2_t30_150M_UR50D/sub-directory makeload_marker_embeddingsraise
FileNotFoundError("No .pt files directly in …"). Move them up one level. compute_esm_embeddingscan OOM on the GPU for long proteins: edit
return_contacts=True→Falseinvirtues/utils/compute_esm_embeddings.py(or use
--device cpu).- The mask is defined on the patch grid, not the pixel grid. For a 128×128 crop with
patch_size=8the mask is(C, 16, 16);generate_mask(C=…, H=tile//patch, W=tile//patch).
Passing pixel dims silently changes the masking ratio. - A 1500×1500 ROI lands exactly on the default
large_tissue_threshold=1500, producing
boundary artefacts; the validated run forceslarge_tissue_threshold=1024. - Recompute standardization when ROIs change. New ROI ⇒
--overwrite, then segmentation
and cell tokens must be recomputed, otherwise ROIs are on different scales and the UMAP is
not comparable. The notebook tracks this viaSTANDARDIZATION_RECOMPUTED. pred_instance.max() == 0is a real failure mode ("Too many seeds") — raise, do not feed an
empty mask intocompute_cell_tokens.remove_small_cellsand the phenotype assignment are sequential:assign_cell_typestakes
the already-cleaned instance mask, andsemantic_logits[[0, 7]] = -infmust be applied first so
None/Unknownnever win the argmax.- Cell-token ordering: pad → standardize →
compute_cell_tokens, with
dataset.get_tissue(..., preprocess=False). Standardizing before padding, or letting the
dataset preprocess, gives a different (wrong) normalization. random_state=42in UMAP disables parallelism (n_jobsoverridden to 1); drop the seed if
you need speed and can accept non-reproducible coordinates.channels_per_tissue.parquetmust list every ROI × every channel as available; missing entries
make a channel silently disappear for some ROI and then fail themidxslookup.- The
tiling/1_0mpp/...warnings ("0 tiles", "No tile coordinates") are benign for this
whole-tissue workflow —get_tileis simply unused. - Two spora-io kwargs used by the notebook,
path=and
get_cell_instance_mask(use_instances_from_virtues=True), do not appear in the public
spora-io README (UNVERIFIED against docs; only verified against the notebook run). Same for
disable_quantile_mask,filter_list,use_mean_stdand the private
dataset._refine_channel_metadata.
Environment
Upstream environment (validated run)
which conda # /tools/software/Miniconda/miniconda3/condabin/conda
conda activate <path>/env/virtues # Python 3.12, created by the virtues `source setup.sh`
export SPORA_DATASETS_DIR=<spora root>
This container
| Item | Value |
|---|---|
| Python | /opt/conda/bin/python3 → 3.12.14 (first python3 on PATH) |
| site-packages | /opt/user_python_base/lib/python3.12/site-packages (also /opt/user_python_libs, /opt/public_python_libs) |
| Installed | numpy 2.4.6, pandas 2.3.3, matplotlib 3.11.0, zarr 3.2.1, tifffile 2026.6.1, safetensors 0.8.0, huggingface_hub 1.22.0, scikit-learn 1.9.0, umap-learn 0.5.12, scanpy 1.12.1, anndata 0.12.16, squidpy 1.8.3 |
| Missing | torch, omegaconf, virtues, spora_io |
| GPU | none |
Note: umap-learn and scikit-learn are present in this container, so the UMAP /
logistic-probe tail of the pipeline would import fine — everything up to and including
compute_cell_tokens still needs torch.
Installation (not verifiable in this container — no PyPI packages, no network to GitHub/HF/Zenodo)
virtues and spora_io are not on PyPI; both must be cloned and installed editable:
git clone https://github.com/bunnelab/virtues.git && cd virtues && pip install -e .
git clone https://github.com/bunnelab/spora-io.git && cd spora-io && pip install -e .
# torch + omegaconf come with the upstream requirements (version_requirements.txt)
Verify with python3 -c "import virtues, spora_io, torch, omegaconf". None of the three
network steps (GitHub clone, HuggingFace weights, Zenodo raw data) is reachable from this
container, so this skill's script has only been checked with py_compile, --help and
--check-only here.
Scripts
| File | Purpose |
|---|---|
scripts/virtues_mcd_inference.py |
CLI: load VirTues weights + spora dataset, per-ROI segmentation, optional reconstruction, 512-D cell tokens, merged UMAP. All paths are flags; no hardcoded directories. |