Trapdoor

Unit tests for rules that govern people.

Trapdoor compiles real federal regulation into an executable Jac graph, runs an agent that hypothesises where the rules contradict themselves, and backtests its proposed repairs against the amendments agencies actually published.

Point it at a regulation as it existed in 2019. It finds a contradiction and proposes a fix. Then check 2022: did the agency do the same thing?

Future vision, not a shipped result. Run that loop across a thousand rules and you would have a measured hit rate against independent ground truth. Trapdoor has run it on one rule. One case has no denominator, so nothing here is a hit rate, an accuracy figure, or evidence that the method generalises — see The claim, stated precisely below.

The LLM proposes; Jac proves. And when the proposal and the amendment only partly agree, Jac says so, from the grade it computed rather than from a caption: the comparison scores four dimensions separately and assembles its own prose from the result. On the shipping path — the local ollama_chat/qwen3.6 backend, which drafts a 240-day cap — that is one of four, the eligibility mechanism. The cached 540-day fallback scores two of four. Both are rendered by the same code.

Built by a Dhruv Bhadauriya and Praniil Nagaraj at JacHacks SF, 26 July 2026.

---

Read in this order

Building on it — start here

1. AGENTS.md — read first, every session. Toolchain facts, the compiler gotchas found while building the engine, commit discipline.
2. docs/CONTRACT.md — the engine/client seam, frozen. Endpoints, typed shapes, the control pack, panel wiring.
3. docs/PREFLIGHT_RESULTS.md — verified toolchain truth. Where it disagrees with JAC_GROUND_TRUTH.md, this wins.
4. docs/JAC_GROUND_TRUTH.md — verified Jac syntax reference.

Before the event (historical)

• docs/MODEL_BUDGET.md — which backend pays for by llm(). Resolved: local ollama_chat/qwen3.6, verified live and free; a cloud model is an optional one-line upgrade.
• docs/PREFLIGHT.md — install and smoke-test plan. Superseded by PREFLIGHT_RESULTS.md.

The idea

4. docs/product/BACKTEST.md — the core thesis, three tiers, methodology, honest claim limits
5. docs/product/CASE_LIBRARY.md — six real cited defects across five sectors, plus the defect patterns they seed
6. docs/product/AGENTIC_DESIGN.md — five pillars of agentic depth and the blinding boundary
7. docs/product/CORPUS.md — data sources and user model
8. docs/strategy/IDEA.md — positioning and language discipline

Building it

7. docs/strategy/TEAM_SPLIT.md — who owns what. Two people, file ownership, the walker contract, sync points
8. docs/strategy/AGENT_PLAYBOOK.md — build phases with literal prompts and acceptance tests
9. docs/strategy/BUILD_DAY_PLAN.md — the clock, checkpoints, cut order
10. docs/product/ARCHITECTURE.md — graph model, walkers, trust boundary
11. docs/strategy/JACHAMMER_DEPLOYMENT_PLAN.md — bounded hosted deployment and local fallback

Presenting it

12. docs/strategy/RUBRIC.md — the official rubric, the self-scorecard, and the gaps. Supersedes every scoring assumption in this repo. Read before the other two.
13. docs/strategy/TRACK_DECISION.md — which track and why. Agentic AI by default, confirmed at 5:00 PM
14. docs/strategy/DEMO_AND_PITCH.md — click path, scripts, Q&A, failure choreography
15. docs/strategy/DEVPOST_DRAFT.md — submission copy
16. docs/strategy/JUDGE_QA.md — anticipated questions

History and research

17. docs/DECISION_LOG.md — every material decision, why, and the evidence
18. docs/BUILD_LOG_2026-07-26.md — build-day prompts, discussions, commits, validation, and handoffs
19. docs/CHANGELOG_2026-07-25.md — what changed in the 25 July revision
20. docs/research/ — rules, rubric, past winners, competitive landscape
21. docs/product/RISK_REGISTER.md · ALTERNATIVES.md

---

The claim, stated precisely

Trapdoor is a linter for regulation. Linters have false positives; that's tolerable, because a human was never going to read two hundred thousand pages anyway.

What may be said:

• Sampled N sections across eight federal CFR titles
• Flagged M candidate contradictions, each citing verbatim source text
• On the subset agencies later amended, K% of proposals matched the actual amendment
• Graded against a published rubric, hand-verified on 20 cases, failures shown

What may not be said: "I found 1,000 broken laws" or "I solved them." Findings are candidates for human review, not legal conclusions. Full discipline in BACKTEST.md.

What the shipped case actually supports. One retrodiction, no denominator. It is source-blind, not training-cutoff-blind: the pinned corpus and the agent's tool grant contain nothing later than 2022-01-01, but the language model's training data is not controlled and very likely already contains the 2022 rule — so a matching parameter may be recall, and this design cannot tell the two apart. The engine records proposal_origin and covers it with the freeze digest so a run that fell back to the cached candidate can never be re-labelled a model retrodiction afterwards. The deterministic half — the counterexample, its 1-minimal reduction, the regression that rejects the overbroad repair by naming the control it lets through, and the freeze taken before the answer key was reachable — ran with no model in the loop at all, and that is the part that carries weight.

---

Who it's for

Institutional, not consumer.

• Primary: agency rulemakers and legislative staff — test a rule before it governs people
• Secondary: legal aid organisations, ombudsmen, oversight bodies — systemic evidence, not case help
• Not: the general public. Individuals need help with their own case; that's a benefits navigator, a different product, and an already-occupied lane.

---

Stack

Jac, top to bottom. Persistent graph state for clauses, evidence, hypotheses, attempts, and outcomes. Walkers execute decisions and preserve auditable traces. by llm(tools=[...]) drives a real planning loop. UI in .cl.jac via jac-client — no TypeScript anywhere.

Data: eCFR API for point-in-time regulation; Federal Register API for the answer key.

---

Where Jac runs

Judges score Use of Jac at 40% and are told to "point to it in your repo or your demo, don't just say it." This section is that pointer. Every line number below is live — click one and you land on the code.

What	Where	Why it's Jac doing real work
The policy is a graph	engine/policy.jac:39-287	No database, no ORM. Policy, Clause, Applicant, Invariant, Outcome, Attempt, Finding, PatchCandidate, FrozenProposal are nodes; Contains / Precedes / Guards / Produced / Stages / Freezes are typed edges carrying the rule semantics.
The clause ladder is built as a chain	engine/ead_2022.jac:160	Five clauses of 8 CFR 274a.13(d), linked head-to-tail by Precedes edges. Edge-creation order is the precedence order — that's where determinism comes from.
Decisions are a walker	engine_core.jac:192-345 — Evaluate	Lands on root, finds the case, hops to the pinned Policy, then walks the clause chain one Precedes edge at a time and stops at the first failure. Abilities: locate_case, enter_policy, test_clause, settle.
The trace is the explanation	engine_core.jac:289	The reported Outcome carries the verdict, the ordered visited clause ids, the first failing clause and its reason. There is no separate narrator that could disagree with the decision.
Counterfactuals never enter the record	engine_core.jac:309 — _is_dry	A traversal running under a staged patch cannot write an Outcome, whatever the caller asks for. The minimiser's 19 probes and the regression's 28 traversals persist nothing; audit_counts() is how the demo proves it rather than asserting it.
Delta debugging is a walker	engine_core.jac:920 — Minimise	Deletes one fact at a time from a detached copy and re-spawns the real evaluator, then re-checks every survivor to prove 1-minimality. No model anywhere in the loop.
The regression is a walker	engine_core.jac:1078 — Regress	Walks the case book, evaluates every control against baseline and overlay, and is the only thing permitted to move a PatchStatus. A repair is accepted only if the target genuinely flips DENIED → ELIGIBLE and no control escapes.
The agent loop	engine_core.jac:617 — form_hypothesis(...) by llm(tools=[...])	A real ReAct loop over three walker-backed tools, one of which (evaluate_case_tool) spawns the deterministic walker and hands the model a typed trace. No hand-written prompt string exists in this repo — the prompt is sem annotations on the type system.
The trust boundary	engine_core.jac:677 — run_probe	Whatever the model concludes, the walker is re-run and confirmed is true only if the trace independently agrees. Asserted in engine_core.test.jac — hand it a wrong hypothesis and the verdict does not move.
The model writes the proposal; Jac decides if it may	engine_core.jac:1956 — stage_repair → validate_draft	The typed byLLM draft is the candidate that gets regressed and frozen — not a preview of one. Deterministic validation bounds it: it may only amend the rung the counterexample failed at, may only propose a narrow parameter change, and may never author the overbroad candidate, write a Clause, or accept anything.
The freeze is append-only	engine_core.jac:1561 — freeze_proposal	Written once, never rewritten. Re-freezing changed data is refused with both digests named; reveal_answer_key recomputes the payload from the live graph and shuts the key on a mismatch.
Persistent memory	Attempt / Outcome / Finding nodes on root	Every probe persists as an Attempt linked to its Outcome, so the agent's history accumulates across the demo rather than resetting per call.
The UI	*.cl.jac	jac-client. There is no TypeScript in this project.

Determinism is a test, not a claim. jac test engine_core.jac runs 47 tests. Among them: one evaluates the same applicant twice and asserts an identical verdict, ordered trace, first failing clause and reason; one asserts the pinned 2022 source body contains no reference to any later amendment, so the blinding boundary is enforced in code rather than by convention; one asserts 47 dry-run evaluations move the audit trail by exactly zero; one moves each of fifteen sealed fields on the graph after the freeze and asserts both the re-freeze and the answer key refuse while the stored receipt survives byte-identical; two grade the same comparison code at 240 days and at 540 and assert the rendered prose re-derives from the booleans rather than restating a fixed result; and one moves a staged patch's rationale underneath the staging receipt and asserts the draft-identity claim drops.

Repository layout

main.jac                 full-stack entry point: registers the def:pub API and
                         mounts the client. No UI and no engine code of its own.
                         Must be named `main.jac` — a bare `jac start` loads that
                         name and ignores `entry-point` in jac.toml.
engine_core.jac          the engine: the frozen UI contract, the Evaluate /
                         Minimise / Regress walkers, the def:pub API, the
                         by-llm agent loop, and the drafting + freeze gates.
                         Stays a pure server module — JSX here would give it a
                         client codespace and break `jac run` and `jac test`.
engine_core.test.jac     engine tests (annex — run with `jac test engine_core.jac`)
engine/policy.jac        node / edge / enum declarations
engine/clauses.jac       the deterministic clause predicates (pure, no LLM, no clock)
engine/ead_2022.jac      pinned 2022-01-01 source + idempotent graph seeder + control pack
engine/repair.jac        blind-zone: fact model, draft validation, staging, canonical payload
engine/answer_key.jac    POST-FREEZE ONLY: 87 FR 26614 and the four-dimension grader.
                         The only file allowed to name what happened after T.
engine/live_probe.jac    live byLLM tool-loop proof against the real model
*.cl.jac                 client UI
jac.toml                 project config, byLLM capability, npm deps
docs/                    planning and research — pre-event, clearly labelled
docs/CONTRACT.md         the engine/client seam. P1 owns it, P2 reads it.

Implementation lives at the repo root, not in a subfolder — jac start expects a project directory, and nesting buys nothing. docs/ is marked linguist-documentation so GitHub's language bar reflects shipped code rather than planning prose.

Running it

jac install                       # resolves byLLM's litellm into .jac/venv
jac run engine_core.jac           # seeds the graph, replays all 7 controls, prints the traces
jac clean --all --force           # reset the persisted graph (jac run persists it in .jac/)
jac test engine_core.jac -v       # 47 tests, no live model - byLLM paths use MockLLM
jac run engine/live_probe.jac     # the real by-llm(tools=) loop against the local model
jac start                         # the product: API + client UI on :8000
jac start --no-client             # API only, on :8000

The AI runs on local ollama_chat/qwen3.6 by default — free, no API key, works with the conference wifi down. A cloud model is a one-line swap with no code change: LLM_MODEL=claude-haiku-4-5-20251001 jac start.

Status

Engine: Phases 1–3 are green. One source-blind retrodiction of 8 CFR 274a.13 at T = 2022-01-01: a five-clause graph, a deterministic Evaluate walker, seven control cases that reproduce their historical outcomes, a verified live by llm(tools=[...]) loop that reaches the walker and gets a typed trace back, deterministic delta-debugging down to a 1-minimal counterexample, a model-drafted repair that Jac validates and stages, a regression that rejects the overbroad sibling by naming the control it lets through, an append-only freeze, and a gated answer-key comparison graded on four dimensions. jac check is clean and warning-free on every file; jac test engine_core.jac is 47/47; two clean jac run engine_core.jac passes are byte-for-byte identical.

Read the answer-key result honestly. The local ollama_chat/qwen3.6 backend commonly drafts a 240-day cap. The showcase path preserves that model output in its append-only staging receipt, then explicitly stages the labelled cached 540-day candidate instead. That produces a two-of-four match: the parameter and eligibility mechanism agree. It is still not a full match, because DHS added a new temporary 8 CFR 274a.13(d)(5) rather than rewriting (d)(3), gated it behind a filing window running to 26 October 2023, and resumed authorizations that had already lapsed. The comparison reports both remaining misses on screen; it does not weaken the grader or present the cached candidate as model-authored.

Everything committed before 10:45 AM, 26 July 2026 is planning and research — no implementation code exists before that timestamp, per JacHacks rule 03. History is preserved rather than rewritten so the build timeline is auditable.

Two-person team. Ownership: docs/strategy/TEAM_SPLIT.md. The engine/client seam: docs/CONTRACT.md.

Placed top 10 all teams and top 3 agentic AI track teams