feat(sleep): per-night evidence chain + live prompt registry - #151
Merged
Yif-Yang merged 2 commits intoJul 21, 2026
Conversation
* skillopt_sleep/evidence.py — append-only, thread-safe, redacted evidence.jsonl per night: harvest sessions -> miner exchanges (verbatim prompt/reply) -> mined tasks with checks -> split assignment -> every replay attempt (phase-tagged, cache hits marked) -> per-task scores with failing checks named -> reflect exchanges + parsed edits -> gate trials and the final decision with its score arithmetic -> staged artifacts. Config-gated (evidence_log, default on; evidence_max_chars cap). * skillopt_sleep/prompts.py — central registry of the four LLM prompt templates (miner/attempt/judge/reflect), byte-identical defaults to the previously inlined strings; user overrides in prompts.json take effect on the next model call (mtime-checked), no restart needed. * cycle.py builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report; staging.new_staging_dir() de-collides same-second runs; latest_staging() now skips non-adoptable (evidence-only) folders. * tests/test_sleep_evidence.py — 8 no-network stdlib tests: chain completeness, redaction/truncation/ordering, disable flag, prompt override round-trip + live effect, no-tasks-night adoption guard.
Yif-Yang
force-pushed
the
feat/sleep-evidence-chain
branch
from
July 21, 2026 17:26
7829552 to
b12b54c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a per-night, append-only evidentiary chain (evidence.jsonl) to make SkillOpt-Sleep nights fully auditable, and introduces a centralized prompt registry with live, file-based overrides so miner/attempt/judge/reflect templates become versionable and tunable without code changes.
Changes:
- Introduce
EvidenceLogand wire evidence events across harvest → mine → replay → reflect/gate → staging. - Add
skillopt_sleep.promptsprompt registry and refactor backend/miner to render prompts from the registry (with live overrides). - Improve staging lifecycle (pre-created staging dir, de-collide naming, and make
latest_staging()skip non-adoptable evidence-only nights).
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sleep_evidence.py | Adds unit tests for evidence logging and prompt override behavior. |
| tests/test_sleep_engine.py | Updates a mock patch target to the new build_backend entrypoint. |
| skillopt_sleep/staging.py | Adds unique staging-dir helper and prevents adopting evidence-only folders. |
| skillopt_sleep/replay.py | Logs per-task replay results into evidence chain (phase-tagged). |
| skillopt_sleep/prompts.py | New live prompt registry with defaults + override file (mtime-checked). |
| skillopt_sleep/llm_miner.py | Moves miner prompt into registry and logs miner exchanges/tasks into evidence. |
| skillopt_sleep/evidence.py | New thread-safe JSONL evidence logger + reader + backend attachment helpers. |
| skillopt_sleep/cycle.py | Creates evidence log per night, pre-creates staging dir, dual-backend support wiring. |
| skillopt_sleep/consolidate.py | Adds evidence phase tagging and logs gate baseline/trials/decision formula. |
| skillopt_sleep/config.py | Adds config knobs for evidence logging and dual-backend split. |
| skillopt_sleep/backend.py | Uses prompt registry for attempt/judge/reflect and logs cached/model calls + reflect exchanges. |
| docs/sleep/README.md | Documents evidence logging behavior and retention/disable guidance. |
| CHANGELOG.md | Notes new evidence chains and prompt registry in Unreleased additions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+83
to
+90
| def new_staging_dir(project: str) -> str: | ||
| """A staging path that is unique even for two runs in the same second.""" | ||
| base = os.path.join(staging_root(project), _ts_dir()) | ||
| out, i = base, 2 | ||
| while os.path.exists(out): | ||
| out = f"{base}-{i}" | ||
| i += 1 | ||
| return out |
Comment on lines
+166
to
+170
| ev = EvidenceLog( | ||
| ev_path, | ||
| max_chars=int(cfg.get("evidence_max_chars", 4000) or 4000), | ||
| redact=bool(cfg.get("redact_secrets", True)), | ||
| ) |
Comment on lines
+161
to
+162
| ev_path = os.path.join( | ||
| cfg.state_dir, "evidence", f"dryrun-{_ts_dir()}.jsonl") |
Comment on lines
+117
to
+123
| def _run(self, **cfg_extra): | ||
| proj = tempfile.mkdtemp() | ||
| home = tempfile.mkdtemp() | ||
| cfg = load_config( | ||
| invoked_project=proj, projects="invoked", backend="mock", | ||
| claude_home=os.path.join(home, ".claude"), auto_adopt=False, | ||
| **cfg_extra) |
Comment on lines
+196
to
+200
| proj = tempfile.mkdtemp() | ||
| home = tempfile.mkdtemp() | ||
| cfg = load_config( | ||
| invoked_project=proj, projects="invoked", backend="mock", | ||
| claude_home=os.path.join(home, ".claude")) |
Comment on lines
+194
to
+196
| os.makedirs(os.path.dirname(path) or ".", exist_ok=True) | ||
| with open(path, "w", encoding="utf-8") as f: | ||
| json.dump(current, f, ensure_ascii=False, indent=2) |
Alphaxalchemy
pushed a commit
to Alphaxalchemy/SkillOpt
that referenced
this pull request
Aug 1, 2026
A zero-dependency (stdlib http.server) web UI over one project's sleep
pipeline, arranged to mirror the data flow: per-stage role/model/prompt
display with live prompt editing, a per-night evidentiary chain browser
over evidence.jsonl, the gate arithmetic spelled out, a config editor, a
run/dry-run launcher, and explicit adopt (nothing auto-adopts from the UI).
python -m skillopt_sleep dashboard [--project DIR] [--port N]
Addresses the review on microsoft#152:
* Rebased onto current main; the evidence-chain commit landed as microsoft#151 and
is dropped here. This commit is only the dashboard and its tests.
* Request authorization. Binding to 127.0.0.1 is not an authorization
boundary — any page in the user's browser can POST cross-origin to
loopback, and a hostile name resolving to loopback (DNS rebinding) makes
those requests look local. Every request now passes through _authorize:
- loopback Host is required on every request, GET included, with the
port matching the bound port. This is what defeats rebinding: the
browser sends the attacker's name, not ours.
- state-changing POSTs additionally require a trusted Origin (missing
and foreign are both rejected), an application/json content type, and
an unguessable per-process capability token in a custom header.
Each is independently sufficient to stop a cross-origin page: a form
cannot set a custom header or a JSON content type without a preflight,
which this server deliberately does not answer. No CORS headers are
emitted anywhere, so foreign origins cannot read responses either.
The token is minted per server process and substituted into the served
HTML, so it never travels in a URL and restarting the dashboard
invalidates every previously served page.
* Request bodies are validated, not coerced. An absent, empty, malformed,
non-object or oversized body is now a specific 4xx instead of silently
becoming {}. That coercion is what let a contentless cross-origin POST
to /api/run start a real run. Oversized bodies are drained within a
bound before the 413 so well-behaved clients read the status rather than
a connection reset.
tests/test_dashboard_security.py adds 25 tests. Every negative case
asserts twice — that the request was refused, and that the side effect did
not happen, since a 403 that still started a run would pass a status-only
test. Covers foreign Host (all seven endpoints), wrong/absent port,
rebinding names, foreign/missing/null/lookalike Origins, form,
text/plain, multipart and absent content types, missing/wrong/
cross-process tokens, empty, malformed, non-object and oversized bodies,
absence of CORS headers, unanswered preflight, and the reported vector
verbatim (empty cross-origin form POST to /api/run). Same-origin JSON
happy paths for config, prompts, run and adopt are retained.
Alphaxalchemy
added a commit
to Alphaxalchemy/SkillOpt
that referenced
this pull request
Aug 1, 2026
A zero-dependency (stdlib http.server) web UI over one project's sleep
pipeline, arranged to mirror the data flow: per-stage role/model/prompt
display with live prompt editing, a per-night evidentiary chain browser
over evidence.jsonl, the gate arithmetic spelled out, a config editor, a
run/dry-run launcher, and explicit adopt (nothing auto-adopts from the UI).
python -m skillopt_sleep dashboard [--project DIR] [--port N]
Addresses the review on microsoft#152:
* Rebased onto current main; the evidence-chain commit landed as microsoft#151 and
is dropped here. This commit is only the dashboard and its tests.
* Request authorization. Binding to 127.0.0.1 is not an authorization
boundary — any page in the user's browser can POST cross-origin to
loopback, and a hostile name resolving to loopback (DNS rebinding) makes
those requests look local. Every request now passes through _authorize:
- loopback Host is required on every request, GET included, with the
port matching the bound port. This is what defeats rebinding: the
browser sends the attacker's name, not ours.
- state-changing POSTs additionally require a trusted Origin (missing
and foreign are both rejected), an application/json content type, and
an unguessable per-process capability token in a custom header.
Each is independently sufficient to stop a cross-origin page: a form
cannot set a custom header or a JSON content type without a preflight,
which this server deliberately does not answer. No CORS headers are
emitted anywhere, so foreign origins cannot read responses either.
The token is minted per server process and substituted into the served
HTML, so it never travels in a URL and restarting the dashboard
invalidates every previously served page.
* Request bodies are validated, not coerced. An absent, empty, malformed,
non-object or oversized body is now a specific 4xx instead of silently
becoming {}. That coercion is what let a contentless cross-origin POST
to /api/run start a real run. Oversized bodies are drained within a
bound before the 413 so well-behaved clients read the status rather than
a connection reset.
tests/test_dashboard_security.py adds 25 tests. Every negative case
asserts twice — that the request was refused, and that the side effect did
not happen, since a 403 that still started a run would pass a status-only
test. Covers foreign Host (all seven endpoints), wrong/absent port,
rebinding names, foreign/missing/null/lookalike Origins, form,
text/plain, multipart and absent content types, missing/wrong/
cross-process tokens, empty, malformed, non-object and oversized bodies,
absence of CORS headers, unanswered preflight, and the reported vector
verbatim (empty cross-origin form POST to /api/run). Same-origin JSON
happy paths for config, prompts, run and adopt are retained.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-night evidence chain + live prompt registry for SkillOpt-Sleep
Why (origin of this change)
This started from a user-side audit of what a completed sleep night actually lets you verify. Running real nights (DeepSeek via the
openai_compatiblebackend) against real transcripts, the report tells you that the miner processed N conversations and what rules the reflect step added — but the evidential chain in between is invisible:None of the intermediate artifacts (the miner's verbatim exchange, which checks each task carried, which replay attempt failed which check, the reflect model's raw reply before parsing, the gate's arithmetic) were recorded anywhere. Given how strict/fragile the acceptance gate is by design (strict improvement or nothing), a rejected night was essentially unexplainable after the fact.
The motivation goes beyond auditability. Once every miner/reflect exchange is persisted per night, three things become possible that aren't today:
What
skillopt_sleep/evidence.py— append-only, thread-safe, secret-redactedevidence.jsonlwritten into each night's staging dir. The full chain is logged: harvested sessions → miner exchanges (verbatim prompt/reply) → mined tasks with their checks → train/val split assignment → every replay model call (phase-tagged; cache hits marked key-only) → per-task scores with failing checks named → reflect exchanges + parsed edits → gate baseline/trials and the final decision with its score arithmetic spelled out → staged artifacts. Config-gated:evidence_log(default on),evidence_max_charstruncation cap.skillopt_sleep/prompts.py— central registry of the four LLM prompt templates (miner / attempt / judge / reflect). Defaults are byte-identical to the previously inlined strings (covered by a test). User overrides inprompts.jsontake effect on the next model call (mtime-checked) — no restart. This is what makes the prompts first-class, versionable, and eventually gate-optimizable objects.cycle.py— builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report;staging.new_staging_dir()de-collides same-second runs;latest_staging()skips non-adoptable (evidence-only) folders so a no-tasks night can never be accidentally adopted.Tests
tests/test_sleep_evidence.py— 8 pure-stdlib, deterministic, no-network tests: chain completeness end-to-end on the mock backend, redaction/truncation/ordering, corrupt-line tolerance,evidence_log=Falsedisable path, prompt-default byte-parity, override round-trip + live effect without restart, and the no-tasks-night adoption guard.Full suite: no new failures vs pristine
main(same 11 pre-existing research-suite loader errors on both; this branch adds 8 tests, all passing).Live smoke: a real DeepSeek night through the
openai_compatiblebackend produced a 95-event chain in which the gate's rejection is fully reconstructable from the log, including its formula (baseline = 0.5·0.000 + 0.5·0.333 = 0.167; candidate = 0.167 → no strict improvement → reject).Series
This is part 1 of a 3-PR series: this PR (evidence chain + prompt registry, no UI) → #152 (a stdlib-only local dashboard that renders the chain and edits the prompts) → #153 (a new harvest source, per the "more sources" direction of #97).