feat: monitor package — production decision replay + shared viewer

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        package: [core, train, eval, api]
+        package: [core, train, eval, api, monitor]
     defaults:
       run:
         working-directory: ${{ matrix.package }}

.gitignore

@@ -28,3 +28,6 @@ api/models/
 
 # brainstorming visual companion
 .superpowers/
+
+# direnv credentials (per-package .envrc.example documents the variables)
+.envrc

Makefile

@@ -1,4 +1,4 @@
-PACKAGES := core train eval api benchmark
+PACKAGES := core train eval api benchmark monitor
 
 # Released model.zip version fetched from HuggingFace by `fetch-model`.
 # Pinned in api/MODEL_VERSION — the repo version and the model version are

README.md

@@ -13,7 +13,7 @@ through the full pipeline, with figures generated from real sequences.
 
 ## Packages
 
-Five independent packages, each with its own `pyproject.toml` and `tests/`.
+Six independent packages, each with its own `pyproject.toml` and `tests/`.
 
 | Path | Import | Purpose |
 |------|--------|---------|
@@ -22,6 +22,7 @@ Five independent packages, each with its own `pyproject.toml` and `tests/`.
 | `eval/`  | `temporal_model.eval`  | DVC evaluation pipeline (packaged-model protocol metrics). Depends on `core`. |
 | `api/`   | `temporal_model.api`   | FastAPI serving layer, shipped as a Docker service. Depends on `core`. |
 | `benchmark/` | `temporal_model.benchmark` | Latency/throughput/resource benchmark with a per-stage `predict()` breakdown, runnable across VMs. Depends on `core`. |
+| `monitor/` | `temporal_model.monitor` | Production decision replay: import scored sequences from alert-api, re-run them through the pinned api+model release, view tubes in the eval viewer. |
 
 `train`/`eval`/`api`/`benchmark` depend on `core` via a `uv` path source
 (`temporal-model-core = { path = "../core", editable = true }`). `core` and
@@ -31,9 +32,9 @@ Five independent packages, each with its own `pyproject.toml` and `tests/`.
 
 ```bash
 make                # list all available targets (same as `make help`)
-make install        # uv sync across all five packages
-make test           # pytest across all five packages
-make lint           # ruff check across all five packages + docs/assets/scripts
+make install        # uv sync across all six packages
+make test           # pytest across all six packages
+make lint           # ruff check across all six packages + docs/assets/scripts
 ```
 
 Per package, `cd <pkg> && make install|lint|format|test`.

monitor/.dvc/.gitignore

@@ -0,0 +1 @@
+/config.local

monitor/.dvc/config

@@ -0,0 +1,5 @@
+[core]
+    remote = s3remote
+    analytics = false
+['remote "s3remote"']
+    url = s3://pyro-vision-rd/dvc/temporal-model/monitor/

monitor/.dvcignore

@@ -0,0 +1,3 @@
+# Add patterns of files dvc should ignore, which could improve
+# the performance. Learn more at
+# https://dvc.org/doc/user-guide/dvcignore

monitor/.envrc.example

@@ -0,0 +1,5 @@
+# Copy to monitor/.envrc (untracked) and fill in; direnv loads it automatically.
+# Credentials for the alert-api (the platform that calls /predict in production).
+export ALERT_API_URL=https://alertapi.pyronear.org
+export ALERT_API_LOGIN=changeme
+export ALERT_API_PASSWORD=changeme

monitor/Makefile

@@ -0,0 +1,18 @@
+.PHONY: install lint format test import
+
+install: ## uv sync
+	uv sync
+
+lint: ## ruff check
+	uv run ruff check .
+
+format: ## ruff format
+	uv run ruff format .
+
+test: ## pytest
+	uv run pytest tests/ -v
+
+import: ## import new sequences from alert-api, then dvc add + push the store
+	uv run temporal-monitor import $(ARGS)
+	uv run dvc add data/01_raw/sequences
+	uv run dvc push

monitor/README.md

@@ -0,0 +1,120 @@
+# 🔎🔥 monitor — production decision replay
+
+Answers "what did the temporal model decide in production, and why?".
+Imports sequences scored by the deployed API (alert-api records the
+probability + `version` provenance per sequence), replays them through the
+**exact** pinned release image with `verbose=true&compute_trigger=true`, and
+writes the eval-viewer contract so tubes/boxes/crops are explorable in
+[`viewer/`](../viewer).
+
+Design: [`docs/specs/2026-06-12-monitor-design.md`](../docs/specs/2026-06-12-monitor-design.md)
+
+## Just browsing? Pull the shared data (no credentials, no Docker)
+
+The store and the replay artifacts are pushed to the DVC remote, so viewing
+what production decided requires neither alert-api credentials nor Docker:
+
+```bash
+make install                                       # uv sync (brings dvc[s3])
+uv run dvc pull                                    # fetch store + reports from S3
+cd ../viewer && DATA_ROOT=../monitor npm run dev   # browse at localhost:3000
+```
+
+Everything below is only for IMPORTING new sequences and RE-RUNNING the
+model — the producer workflow.
+
+## Importing new data (alert-api credentials required)
+
+```bash
+make install
+cp .envrc.example .envrc     # fill in alert-api credentials (direnv loads it)
+```
+
+```bash
+make import                                  # 1. fetch new sequences (incremental),
+                                             #    dvc add + push the store
+make import ARGS="--date-from 2026-06-01 --date-to 2026-06-10"  # backfill a range
+make import ARGS="--all-orgs --exclude-org pyroadmins"  # admin token: every org, skipping CI org
+```
+
+The alert-api listing (`/sequences/all/fromdate`) is limited to the authenticated account's own
+organization; `--all-orgs` scans the global sequence-id space instead (admin token required), so
+every organization (sdis-07, sdis-77, ...) lands in the single `alert-api`
+source, distinguishable via the organization column/filter in the viewer.
+`--exclude-org` skips an organization (accepts a slug or raw name, e.g. `--exclude-org pyroadmins`
+for the CI camera org); pass it multiple times to exclude several. Exclusion affects future imports
+only — already-imported directories must be removed manually.
+
+## Replaying (Docker required)
+
+```bash
+uv run dvc repro                             # 2. replay through pinned releases
+uv run dvc push                              # 3. share store + artifacts so
+                                             #    teammates can just `dvc pull`
+cd ../viewer && DATA_ROOT=../monitor npm run dev   # 4. browse at localhost:3000
+```
+
+`replay` groups sequences by their recorded `temporal_api_version`, runs
+`pyronear/temporal-model-api:<tag>` (model.zip baked in) + a throwaway MinIO
+on ports 18000/19000 (offset from `api/`'s 8000/9000), uploads each
+sequence's frames under their original S3 keys, and reconstructs the exact
+production call (last ≤10 distinct frames oldest-first, ROI = envelope of
+the detections' primary bboxes — mirrors pyro-api's validation worker).
+Docker must be running; each version group costs one image pull.
+
+## Outputs (`data/08_reporting/alert-api/vit_dinov2_finetune/`)
+
+All organizations land in one reporting tree; organizations are a table column in the viewer.
+
+- `results.json` — eval columns carrying PRODUCTION's verdict (`decision`,
+  `probability` = the recorded score) + monitor extras: `replayed_probability`
+  / `replayed_decision` (the local re-run), `replay_matches` (|Δ| ≤ 1e-5),
+  `matched_window_frames` (when a mismatch is window drift: the sequence
+  length at which production's recorded score is reproduced; null = no window
+  matched, genuine drift), `temporal_model_version`, `temporal_api_version`.
+- `details/<key>.json` — tubes in the eval shape; `stabilized_window` is
+  recomputed client-side; trigger fields appear for releases shipping
+  `compute_trigger` (older images ignore the flag → "no trigger" in the
+  viewer).
+- `sequences/<key>.json`, `model_config.json`, `dropped.json` (skip reasons:
+  `no_temporal_version`, `no_recorded_score`, `invalid_api_version`,
+  `image_pull_failed`, `stack_unhealthy`, `model_version_mismatch`,
+  `too_few_frames`, `no_images`, `predict_failed`).
+
+A `replay_matches: false` row means the reconstruction diverged from the
+recorded score. When this happens, the replay automatically probes candidate
+windows (ascending distinct-frame counts from MIN_FRAMES) to check whether
+the mismatch is window drift — production scores a sequence early and stops
+once it validates, while detections keep arriving. If a shorter prefix of the
+sequence reproduces the recorded score, `matched_window_frames` is set to
+that count (window drift, not model drift); if no prefix matches, it remains
+null (genuine drift — see the spec).
+
+## Trigger frames
+
+Sequences scored by api v0.3.1 and earlier predate the `compute_trigger`
+feature, so their replays produce no `trigger_frame_index` or
+`first_crossing_frame` data. The enrichment pass fills these fields using a
+newer serving image (the same `model.zip` baked in) while keeping the pinned
+replay authoritative for tubes and probability.
+
+Pass `--trigger-image <image>` to `replay` (or leave the default in dvc.yaml).
+The enrichment probability must reproduce the pinned replay's within
+`SCORE_TOLERANCE` (1e-5) — a disagreement means the scoring path changed and
+the trigger fields are left empty for that sequence.
+
+Build the dev image from the repo root:
+
+```bash
+make fetch-model MODEL_VERSION=0.2.0   # repo root: same model production runs
+docker build -f api/Dockerfile --build-arg VERSION=dev -t temporal-model-api:dev .
+```
+
+Teammates who only need the artifacts can skip the build and use `dvc pull`
+instead of `dvc repro`.
+
+## Tests
+
+```bash
+make test    # offline: mocked HTTP, fake docker stack — no Docker needed
+```

monitor/data/01_raw/sequences.dvc

@@ -0,0 +1,6 @@
+outs:
+- md5: 8b9d7b77dd2a2dacd4f67fe623c7f323.dir
+  size: 448152089
+  nfiles: 3214
+  hash: md5
+  path: sequences

monitor/docker-compose.yml

@@ -0,0 +1,44 @@
+# Replay stack: the pinned release image (model.zip baked in) + throwaway MinIO.
+# Image tag comes from MONITOR_API_IMAGE (set by `temporal-monitor replay` per
+# api-version group). Ports are offset from api/docker-compose.yml so both
+# stacks can run side by side.
+services:
+  api:
+    image: ${MONITOR_API_IMAGE:?set by temporal-monitor replay}
+    ports:
+      - "18000:8000"
+    environment:
+      - TEMPORAL_API_S3_BUCKET=frames
+      - TEMPORAL_API_S3_ENDPOINT_URL=http://minio:9000
+      - TEMPORAL_API_S3_REGION=us-east-1
+      # local replay only — never reused for shared environments
+      - AWS_ACCESS_KEY_ID=minioadmin
+      - AWS_SECRET_ACCESS_KEY=minioadmin
+    depends_on:
+      minio:
+        condition: service_started
+      createbuckets:
+        condition: service_completed_successfully
+
+  minio:
+    image: minio/minio:latest
+    command: server /data
+    ports:
+      - "19000:9000"
+    environment:
+      - MINIO_ROOT_USER=minioadmin
+      - MINIO_ROOT_PASSWORD=minioadmin
+
+  # One-shot: wait for MinIO, then create the bucket the API reads from.
+  createbuckets:
+    image: minio/mc:latest
+    depends_on:
+      - minio
+    entrypoint: >
+      /bin/sh -c "
+      until mc alias set local http://minio:9000 minioadmin minioadmin; do
+        echo 'waiting for minio...'; sleep 1;
+      done;
+      mc mb --ignore-existing local/frames;
+      echo 'bucket frames ready';
+      "

monitor/dvc.lock

@@ -0,0 +1,50 @@
+schema: '2.0'
+stages:
+  replay:
+    cmd: uv run python -m temporal_model.monitor.cli replay --store 
+      data/01_raw/sequences --output-dir data/08_reporting --trigger-image 
+      temporal-model-api:dev
+    deps:
+    - path: data/01_raw/sequences
+      hash: md5
+      md5: 8b9d7b77dd2a2dacd4f67fe623c7f323.dir
+      size: 448152089
+      nfiles: 3214
+    - path: docker-compose.yml
+      hash: md5
+      md5: 985531d975ed5a5d3f2daf2ea5083d4a
+      size: 1403
+    - path: src/temporal_model/monitor/cli.py
+      hash: md5
+      md5: 1715ab3cbe191376fb8493d7928f7e63
+      size: 4196
+    - path: src/temporal_model/monitor/geometry.py
+      hash: md5
+      md5: 7f811828367c67fd9115ad5e6b69956d
+      size: 1485
+    - path: src/temporal_model/monitor/reconstruct.py
+      hash: md5
+      md5: fa83924a8b64a59e2e9ba7ade4c601b4
+      size: 3351
+    - path: src/temporal_model/monitor/replay.py
+      hash: md5
+      md5: 087c2ed380aff23a6032b45574a7edfd
+      size: 15782
+    - path: src/temporal_model/monitor/report.py
+      hash: md5
+      md5: 3726aeaa208d275c6e1f8100d329861e
+      size: 6579
+    - path: src/temporal_model/monitor/stack.py
+      hash: md5
+      md5: a46cd818f3301d3470e8cd3e3f9f8864
+      size: 2955
+    - path: src/temporal_model/monitor/store.py
+      hash: md5
+      md5: 8ebb0731d13c1b9903e80b3cfb08b546
+      size: 3363
+    outs:
+    - path: data/08_reporting
+      hash: md5
+      md5: 7b461fc059991bf2eb18fd7dff356982.dir
+      size: 342568
+      nfiles: 185

monitor/dvc.yaml

@@ -0,0 +1,19 @@
+stages:
+  replay:
+    cmd: >-
+      uv run python -m temporal_model.monitor.cli replay
+      --store data/01_raw/sequences
+      --output-dir data/08_reporting
+      --trigger-image temporal-model-api:dev
+    deps:
+      - src/temporal_model/monitor/cli.py
+      - src/temporal_model/monitor/store.py
+      - src/temporal_model/monitor/reconstruct.py
+      - src/temporal_model/monitor/stack.py
+      - src/temporal_model/monitor/replay.py
+      - src/temporal_model/monitor/geometry.py
+      - src/temporal_model/monitor/report.py
+      - docker-compose.yml
+      - data/01_raw/sequences
+    outs:
+      - data/08_reporting        # cached → dvc push shares it

monitor/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "temporal-model-monitor"
+version = "0.1.0"
+description = "Production decision replay for the temporal smoke classifier: import scored sequences from alert-api, re-run them with verbose details through the exact pinned api+model release, view tubes in the eval viewer"
+requires-python = ">=3.11"
+dependencies = [
+    "pydantic>=2.7",
+    "requests>=2.31",
+    "boto3>=1.34",
+]
+
+[project.scripts]
+temporal-monitor = "temporal_model.monitor.cli:main"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/temporal_model"]
+
+[dependency-groups]
+dev = [
+    "dvc[s3]>=3.56",
+    "pytest>=8.0",
+    "ruff>=0.9",
+]
+
+[tool.ruff]
+target-version = "py311"
+line-length = 88
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W", "UP", "B", "SIM", "PLC0415"]
+
+[tool.ruff.lint.isort]
+known-first-party = ["temporal_model"]
+
+[tool.ruff.format]
+quote-style = "double"

monitor/src/temporal_model/monitor/__init__.py

@@ -0,0 +1 @@
+"""Production decision replay: alert-api import + pinned-release re-run."""