Multi-Camera Person Tracking

This skill runs as a persistent background service that tracks persons across
multiple RTSP cameras with persistent global IDs — the same physical person
keeps one ID while crossing cameras, with brief disappearances tolerated. It
exposes 6 tools via a loopback HTTP API for the Agent to query tracking data
and trigger trajectory analysis on demand.

The tracker runs passively: it only accumulates events. The Agent decides
when and what to analyze by calling tools — there are no automatic VLM calls.

Architecture

                        Agent (decision maker)
                              │
                     HTTP tool calls (6 tools)
                              │
┌─────────────────────────────▼─────────────────────────────┐
│              Tracking Service (background)                 │
│                                                            │
│  Camera 1 ──→ YOLO track() ──→ OSNet ReID ──┐              │
│  Camera 2 ──→ YOLO track() ──→ OSNet ReID ──┤→ cross-cam   │
│  Camera 3 ──→ YOLO track() ──→ OSNet ReID ──┘   association│
│                                                    │       │
│                                                    ▼       │
│                                              EventStore    │
│                                              (append-only, │
│                                               memory+JSONL)│
│                                                    │       │
│                                         API Server (:8765) │
└────────────────────────────────────────────────────────────┘

Read frames are per-camera threads; inference + association run serially in the
main loop (shared model). Details: references/configuration.md.

When to Use

• "Where is everyone right now?" / "Who is in the living room?" → live snapshot
• "What did person 1 do today?" / "Where did they go?" → timeline + trajectory analysis
• "What happened while I was away?" → event log queries
• Suspicious-activity review: a person appeared at an unusual time or place

When NOT to Use

• Single-camera coarse spatial events (pet on table, zone intrusion) → use the
  sibling skill rtsp-spatial-detector instead.
• Face recognition / identity naming — this skill assigns anonymous global IDs,
  it cannot say who a person is.
• No RTSP streams available — camera URLs come from the user or
  @xpai-camera-control.

Quick Start

# 1. One-time: fetch + verify YOLO weights (not bundled, no URLs embedded;
#    slow/blocked networks: import an existing copy via --from, see below)
python scripts/setup_model.py

# 2. Configure cameras + topology in scripts/settings.json (see below)

# 3. Start the background service (tracking mode is the default in settings)
python scripts/run.py --mode tracking

# 4. Verify it is up (any terminal)
curl http://127.0.0.1:8765/api/status

Optional flags: --show (OpenCV preview), --mjpeg-port 8090 (browser view),
--frame-skip N, --conf X, --model PATH. The service keeps running; the
Agent queries the API whenever needed.

Tools Overview

Base URL http://127.0.0.1:8765 (loopback-only; host/port configurable via
api_server in settings.json). All responses are JSON.

Full request/response reference with all fields: references/api.md.

Agent Usage Patterns

Pattern 1 — "Where is everyone?"

GET /api/snapshot                     → who is currently active and where

Pattern 2 — "What did person 1 do today?"

GET /api/person/1                     → timeline overview
POST /api/analyze {"global_id": 1, "include_vlm": true}
                                      → trajectory image (+ VLM labels if configured)
Present the image + summary to the user

Pattern 3 — Suspicious activity

GET /api/status (periodic)            → notice a person appeared at 3am
POST /api/analyze {"global_id": 2, "since": "03:00:00"}
                                      → analyze the suspicious segment
Alert the user with the trajectory image

Pattern 4 — "What happened while I was away?"

GET /api/events?since=09:00:00&until=18:00:00
                                      → list what happened
POST /api/analyze per person of interest → per-person summary

Pattern 5 — "Show me everyone's trajectories"

POST /api/analyze {"global_id": "all", "since": "08:00:00"}
                                      → one combined image, each person in its own color

Pattern 6 — "Where did they go after they disappeared?"

GET /api/person/1                     → when the timeline ends with `vanished`,
                                        `inferred_destination` suggests the most likely
                                        adjacent camera (topology transit-window heuristic),
                                        or explains "still in transit / blind zone / left coverage"

Event Types

The service writes these events to the event store (memory + daily JSONL file
under scripts/trajectories/events/):

Note: the data model additionally defines reappeared and track_update; the
current tracking loop does not write them — they remain valid as query filter
values (get_event_log?event_type=...).

Configuration

Key sections of scripts/settings.json (full field reference:
references/configuration.md):

• mode — tracking (default) or spatial
• cameras[] — camera id / name / RTSP source / description
• topology.links[] — which cameras are adjacent, transit-time window
  [min, max] seconds, overlap flag for shared fields of view
• tracking.tracker — botsort (default) or bytetrack
• tracking.track_classes — classes to track (default ["person"])
• tracking.reid_threshold — OSNet cosine similarity threshold (default 0.45)
• tracking.gallery_max_age_sec — feature-cache TTL (default 120)
• trajectory — floor plan image, camera poses on the plan, linger threshold
• vlm — optional VLM semantic labeling (OpenAI-compatible vision endpoint; API key from an env var, never stored in the file)
• api_server — host/port for the tool API

Tuning guidance (ReID thresholds, topology calibration, tracker choice):
references/tuning.md.

Modes

mode: "tracking" (default) runs multi-camera person tracking as described
above. mode: "spatial" falls back to the original single-camera spatial
detection (the code path shared with the rtsp-spatial-detector skill) and
expects scenarios[] + rtsp_url in settings — the shipped settings.json
contains neither, so prefer the dedicated sibling skill for spatial use cases.

Model Weights

Model weight files are not distributed with this skill (publishing
constraint). The tracker needs yolo26n.pt; obtain and verify it once:

python scripts/setup_model.py          # fetch if missing, then verify SHA-256
python scripts/setup_model.py --check  # verify only (no network access)
python scripts/setup_model.py --from /path/to/yolo26n.pt   # import a local copy, then verify

setup_model.py embeds no URLs — the download delegates to the installed
ultralytics official-asset mechanism, then the file is verified against the
SHA-256 fingerprint this skill was validated with. On slow/blocked networks,
--from copies an existing local file (another machine, a cache, offline
media) into scripts/ and runs the same fingerprint verification. Offline
machines can also place an official yolo26n.pt (matching the installed
ultralytics version) into scripts/ manually, then run
setup_model.py --check. The OSNet ReID weights are fetched by the torchreid
library on first use (no URLs in this skill either).

Known Limitations

• VLM semantic labeling is opt-in but config-ready. Fill the vlm section
  in settings.json (enabled / endpoint / model — any OpenAI-compatible
  chat/completions vision API) and set the API key env var (default
  VLM_API_KEY); the startup log states exactly why it is off otherwise.
  Without it, analyze_trajectory still renders the trajectory image and the
  Agent can read the image itself.
• ReID backend: torchreid OSNet preferred; if torchreid is unavailable the
  extractor falls back to a torchvision resnet18 head (lower accuracy —
  expect more fragmented IDs and retune reid_threshold).
• Track-loss threshold is fixed at 3 s in main.py (not exposed in
  settings).
• Floor-plan rendering requires Pillow; if missing, the renderer is
  disabled with a warning and trajectory_image comes back empty.
• Camera poses must be calibrated (trajectory.camera_positions) for
  meaningful floor-plan positions. Helper: python scripts/calibrate_positions.py — interactive click-position-then-heading
  calibration (ESC skip / R redo) with settings write-back, or --no-gui to
  emit a grid-annotated reference image. When the floor-plan file is missing a
  grid placeholder is generated automatically (no more blank canvas).
• Anonymous IDs only — no face recognition; "person 1" is whoever the
  system first saw, not a named identity.

References

• references/api.md — complete request/response reference for all 6 tools
• references/configuration.md — cameras / topology / tracking / trajectory / api_server fields
• references/tuning.md — ReID & topology tuning, tracker choice, pose calibration, RTSP performance
• references/troubleshooting.md — FAQ: config errors, port conflicts, offline model install, stream/connection diagnosis, anti-patterns