Skip to content

Commit cc31f89

Browse files
feat(conformance): cut contract-v2 with the 18-scenario corpus (#97)
The vendored corpus has been frozen at 15 scenarios since 0.5.0 (ADR-0012) while the toolkit's grew to 18: firmware-update-success and firmware-update-failure in toolkit 0.4.3/0.4.4, refused-authorization in toolkit 0.4.5. Those three are exactly the blind spots that let #92 hide. No pinned fixture exercised a FirmwareStatusNotification status, a non-Invalid idTagInfo, or any of the 22 connector-status transitions the old matrix flagged wrongly, so the harness sat at 15/15 while three rules disagreed with the specification, first in the same way as the toolkit and then in a different way once the toolkit was fixed. Verifying #93 took an ad-hoc probe of 97 traces across both engines, which proves a point in time and guards nothing afterwards. Widening the corpus is the part that keeps holding. ADR-0012 names a new scenario as a trigger for a new contract version, so this is contract-v2: 18 fixtures and goldens regenerated from the published toolkit 0.4.5, gate now 18/18 under both native test and studio ci. Recorded in ADR-0013, which refines ADR-0012's versioning half rather than superseding it. contract-v2 is a strict superset of v1. Regenerating all 18 reproduces v1's 15 fixtures and goldens byte for byte, so every scenario v1 pinned, v2 pins identically, with three added. v1 is not vendored a second time: the v0.5.4 tag's tree carries it, so every released Studio stays pinned to a reproducible reference, which is the consequence ADR-0012 named. harness.zig derives everything from scenario_names and @embedfile, so three names are the only code change. docs/CONTRACT.md, the conformance README, the top-level README and docs/cli-parity.md are updated; ADRs and CHANGELOG history are left as written, since they record what was true when written. The contract docs now also state that the corpus is a sample rather than a proof. 18 scenarios still do not cover the 81 ordered connector-status pairs or most of the status cross-products, and #92 is the standing evidence that a green harness is evidence rather than proof. This does not settle #94, which asks whether a rule correction that leaves every golden matching needs a bump at all. That case stays open; this one was never ambiguous.
1 parent e6cc8c4 commit cc31f89

15 files changed

Lines changed: 550 additions & 23 deletions

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,17 @@ reference are all pre-1.0, minor (0.x) releases may include breaking changes.
88

99
## [Unreleased]
1010

11+
### Changed
12+
13+
- The conformance contract is now **`contract-v2`**: 18 shared scenarios, up from
14+
15, regenerated against toolkit `0.4.5`. `studio ci` and the build-time harness
15+
gate on 18/18. The three added scenarios (`firmware-update-success`,
16+
`firmware-update-failure`, `refused-authorization`) cover the firmware statuses
17+
and the non-`Invalid` authorization refusals the frozen corpus never reached,
18+
which is how the rule defects fixed in 0.5.4 stayed green at 15/15. `contract-v2`
19+
is a strict superset of v1: the 15 existing fixtures and goldens are unchanged,
20+
byte for byte. See [ADR-0013](docs/adr/0013-contract-v2-corpus.md).
21+
1122
## [0.5.4] - 2026-07-30
1223

1324
### Fixed

‎CURRENT_STATE.md‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,35 @@ workflow, which builds and attaches the macOS + Linux packages. Next themes
4242

4343
## What's done
4444

45+
### contract-v2: the 18-scenario corpus (Issue #96)
46+
47+
The vendored corpus was frozen at 15 scenarios for 0.5.0 (ADR-0012) while the
48+
toolkit's grew to 18. The three missing ones were exactly the blind spots that let
49+
#92 hide: no pinned fixture exercised a `FirmwareStatusNotification` status, a
50+
non-`Invalid` `idTagInfo`, or the connector-status transitions the old matrix got
51+
wrong, so the harness sat at 15/15 while three rules disagreed with the spec.
52+
53+
- ✅ **`contract-v2`** cut per ADR-0012's rule, recorded in
54+
[ADR-0013](docs/adr/0013-contract-v2-corpus.md): 18 fixtures + goldens
55+
regenerated from the published toolkit `0.4.5`, harness gate now **18/18** under
56+
both `native test` and `studio ci`
57+
- ✅ Added `firmware-update-success` (`[]`), `firmware-update-failure`
58+
(`["FIRMWARE_UPDATE_FAILURE"]`, covering the `InstallationFailed` the old rule
59+
missed) and `refused-authorization` (`["FAILED_AUTHORIZATION"]`, covering
60+
`Blocked` / `Expired` / `ConcurrentTx`)
61+
- ✅ **Strict superset of v1:** regenerating all 18 reproduces v1's 15 fixtures and
62+
goldens byte for byte, so nothing previously asserted changed. `contract-v1`
63+
stays reachable through the `v0.5.4` tag's tree rather than being vendored twice
64+
- ✅ `harness.zig` derives everything from `scenario_names`, so the three names are
65+
the only code change; `docs/CONTRACT.md`, the conformance README, the top-level
66+
README and `docs/cli-parity.md` updated. ADRs and CHANGELOG history left as
67+
written
68+
- The contract docs now say plainly that the corpus is a sample rather than a
69+
proof: 18 scenarios still do not cover the 81 ordered connector-status pairs or
70+
most of the status cross-products, so a green harness is evidence, not proof
71+
- Does not settle #94 (whether a rule fix that moves no golden needs a bump); "a
72+
new scenario" is named explicitly by ADR-0012, so this cut is the clear case
73+
4574
### OCPP 1.6 spec-conformance parity (Issue #92)
4675

4776
Three rules read the 1.6 spec wrongly in the same way the toolkit's did, fixed
@@ -240,4 +269,4 @@ live inspector surface with notifications (#54–#60) — is landed.
240269
| `ui` (native views) | ✅ S3 inspector (#27–#32) + replay transport (#44) + live-capture view (#59) + live notifications (#60) |
241270
| `capture` (live proxy) | ✅ S5: WS transport (#54) + frame decode (#55) + MITM proxy (#56) + live notifications (#60) |
242271
| `cli` (headless) | ✅ inspect/report/diff/anonymize/ci/scenario (#45) + live `capture` (#57) |
243-
| `conformance` | ✅ S2 (15/15) + frozen `contract-v1` (#72, docs/CONTRACT.md) |
272+
| `conformance` | ✅ S2 (15/15) + frozen `contract-v1` (#72, docs/CONTRACT.md); widened to `contract-v2` (18/18) in #96 |

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Studio is one of two independent products under the OCPP DebugKit umbrella:
2020
| [`@ocpp-debugkit/toolkit`](https://github.com/ocpp-debugkit/toolkit) | TypeScript | npm library · CLI · web app | The library and CI brain — parse, analyze, and report OCPP traces anywhere |
2121
| **`ocpp-debugkit/studio`** (this repo) | Zig | native desktop app | The instrument on the bench — live capture, native performance, OS integration |
2222

23-
The two share **no code**. They meet only at a *conformance contract*: the same trace format, the same normalized event model, the same failure taxonomy, and the same scenario fixtures. A trace captured in Studio opens in the toolkit's web inspector, and vice versa — two independent implementations, one format, checked in CI on every change. See **[the conformance contract](docs/CONTRACT.md)** (`contract-v1`) and [ADR-0001](docs/adr/0001-independent-implementation.md).
23+
The two share **no code**. They meet only at a *conformance contract*: the same trace format, the same normalized event model, the same failure taxonomy, and the same scenario fixtures. A trace captured in Studio opens in the toolkit's web inspector, and vice versa — two independent implementations, one format, checked in CI on every change. See **[the conformance contract](docs/CONTRACT.md)** (`contract-v2`) and [ADR-0001](docs/adr/0001-independent-implementation.md).
2424

2525
## What it does
2626

‎docs/CONTRACT.md‎

Lines changed: 32 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# The conformance contract (`contract-v1`)
1+
# The conformance contract (`contract-v2`)
22

33
OCPP DebugKit Studio and the TypeScript [toolkit](https://github.com/ocpp-debugkit/toolkit)
44
are **two independent implementations** of the same OCPP analysis behavior — one
@@ -28,35 +28,56 @@ and it is enforced in CI on every change, not asserted in prose.
2828
2 info**), and the same thresholds as the reference (heartbeat 60 s, slow
2929
response 10 s, ±50 % heartbeat deviation, session bounds, the 5-minute boot
3030
window). The rules live in [`src/ocpp/detection.zig`](../src/ocpp/detection.zig).
31-
- **Scenarios.** **15 shared scenario traces** exercising the rules end to end,
31+
- **Scenarios.** **18 shared scenario traces** exercising the rules end to end,
3232
each pinned to the exact de-duplicated, sorted `FailureCode` set the reference
3333
implementation detects.
3434

3535
## How conformance is enforced
3636

37-
The 15 scenarios and their expected failure-code sets are **vendored from the
37+
The 18 scenarios and their expected failure-code sets are **vendored from the
3838
toolkit — the source of truth — never hand-authored**, so a golden can't drift
3939
from what the reference actually detects. On every change, `native test` runs
4040
Studio's full engine (`parseTrace → buildSessionTimeline → detectFailures`) over
4141
each fixture and asserts its detected code set equals the golden — the same
42-
comparison the toolkit's `evaluateScenario` makes. **15/15 must match; a release
42+
comparison the toolkit's `evaluateScenario` makes. **18/18 must match; a release
4343
is blocked otherwise.** `studio ci` runs the same check from the command line.
4444

45+
The corpus is a **sample, not a proof**: two implementations can both match every
46+
golden and still disagree on inputs no fixture reaches. That is not theoretical.
47+
Three rules disagreed with the specification here for as long as they did in the
48+
toolkit while the harness stayed green, because no fixture exercised a firmware
49+
status, a non-`Invalid` `idTagInfo`, or the connector-status transitions involved
50+
(#93). When a rule changes, widen the corpus rather than trusting the count.
51+
4552
The vendored fixtures, goldens, the harness, and the exact regeneration recipe
4653
live in [`src/ocpp/conformance/`](../src/ocpp/conformance/README.md).
4754

4855
## Freeze & versioning
4956

50-
For the 0.5.0 release the contract is **frozen as `contract-v1`** (OCPP 1.6J):
51-
the vendored fixtures and goldens are pinned, and the harness gates every build
52-
against them. The freeze rationale is recorded in
53-
[ADR-0012](adr/0012-freeze-contract-v1.md).
57+
The current contract is **`contract-v2`** (OCPP 1.6J), 18 scenarios, regenerated
58+
against toolkit `0.4.5`. The vendored fixtures and goldens are pinned, and the
59+
harness gates every build against them. The cut is recorded in
60+
[ADR-0013](adr/0013-contract-v2-corpus.md); `contract-v1` (15 scenarios, frozen
61+
for 0.5.0) is [ADR-0012](adr/0012-freeze-contract-v1.md).
62+
63+
`contract-v2` is a **strict superset** of v1: regenerating all 18 against toolkit
64+
0.4.5 reproduces v1's 15 fixtures and goldens byte for byte, so every scenario v1
65+
pinned, v2 pins identically.
5466

5567
The contract is **immutable within a version.** A change that alters detected
5668
output — a new or changed rule, a new scenario, a threshold change, or OCPP
57-
2.0.1 — is a **new contract version** (`contract-v2`, …), regenerated against the
58-
matching toolkit release and re-tagged, never an in-place edit of `contract-v1`.
59-
This keeps "two implementations, one format" a checkable claim rather than a hope.
69+
2.0.1 — is a **new contract version** (`contract-v3`, …), regenerated against the
70+
matching toolkit release and re-tagged, never an in-place edit of a released
71+
version. Earlier versions stay reachable through the release tags: `v0.5.4`'s tree
72+
carries `contract-v1`, so every released Studio remains pinned to a reproducible
73+
reference. This keeps "two implementations, one format" a checkable claim rather
74+
than a hope.
75+
76+
One question this policy leaves open, tracked in #94: whether a rule correction
77+
that leaves **every** golden matching needs a version bump at all, given that the
78+
contract's artifacts are the fixtures and the goldens. The corpus expansion above
79+
is not that case, since "a new scenario" is named explicitly, so it does not
80+
settle it.
6081

6182
See also [ADR-0001](adr/0001-independent-implementation.md) (independent
6283
implementation + shared contract) and [docs/cli-parity.md](cli-parity.md)
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# ADR-0013 — Cut `contract-v2`: the 18-scenario corpus
2+
3+
- **Status:** Accepted
4+
- **Date:** 2026-07-30
5+
6+
## Context
7+
8+
[ADR-0012](0012-freeze-contract-v1.md) froze the vendored conformance corpus as
9+
`contract-v1` (15 scenarios) for the 0.5.0 release, and set the rule that a new
10+
scenario means a new contract version rather than an in-place edit.
11+
12+
The toolkit's corpus has since grown to 18. `firmware-update-success` and
13+
`firmware-update-failure` landed in toolkit 0.4.3/0.4.4, and
14+
`refused-authorization` in toolkit 0.4.5. None of the three was in the contract.
15+
16+
That gap turned out to matter. Three detection rules disagreed with the OCPP 1.6
17+
specification here for as long as they did in the toolkit (#92, fixed in #93,
18+
released in 0.5.4), and the harness stayed at 15/15 the whole time, because no
19+
pinned fixture exercised a `FirmwareStatusNotification` status, a non-`Invalid`
20+
`idTagInfo`, or any of the 22 connector-status transitions the old matrix flagged
21+
wrongly. The contract was green while both implementations were wrong in the same
22+
way, and later while they were wrong in *different* ways. Verifying #93 needed an
23+
ad-hoc probe of 97 traces across both engines, which proves a point in time but
24+
guards nothing afterwards.
25+
26+
The three missing scenarios land precisely on those blind spots.
27+
28+
## Decision
29+
30+
**Cut `contract-v2` (OCPP 1.6J): 18 scenarios, regenerated against toolkit
31+
`0.4.5`.** The harness gate becomes 18/18, and `studio ci` reports 18 scenarios.
32+
33+
`contract-v2` is a **strict superset** of `contract-v1`. Regenerating all 18 from
34+
the published toolkit 0.4.5 reproduces v1's 15 fixtures and goldens byte for byte;
35+
every scenario v1 pinned, v2 pins identically, with three added. So this is an
36+
additive widening of coverage, not a revision of anything previously asserted.
37+
38+
`contract-v1` is not kept side by side in the tree. ADR-0012's "never edited in
39+
place" is satisfied by history: the `v0.5.4` tag's tree carries the v1 corpus, so
40+
every released Studio stays pinned to a reproducible reference, which is the
41+
consequence ADR-0012 named. Vendoring both versions at once would double the
42+
embedded test data for no reachable consumer.
43+
44+
## Consequences
45+
46+
- The gate now covers the firmware statuses and the non-`Invalid` authorization
47+
refusals it previously never reached, so the class of divergence #92 describes
48+
is caught by CI rather than by someone reading the spec against two codebases.
49+
- The corpus is still a **sample**, and the contract docs now say so. 18 scenarios
50+
do not cover the 81 ordered connector-status pairs, or most of the
51+
`AuthorizationStatus` and `FirmwareStatus` cross-products. The lesson from #92 is
52+
that a green harness is evidence, not proof; when a rule changes, widen the
53+
corpus.
54+
- Regeneration is now expected to be a byte-identical no-op for existing files,
55+
which makes an unintended fixture change a visible signal rather than noise.
56+
- This ADR refines ADR-0012's versioning half by cutting the next version under
57+
its rule; it does not supersede it, and ADR-0012 stands as the record of the v1
58+
freeze.
59+
- It does not answer #94, which asks whether a rule correction that leaves every
60+
golden matching needs a version bump at all. "A new scenario" is named
61+
explicitly by ADR-0012, so this cut is the unambiguous case; the ambiguous one
62+
stays open.

‎docs/adr/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,4 @@ supersedes the old one rather than editing history.
2222
| [0010](0010-live-proxy-transport-concurrency.md) | Live-proxy transport & concurrency | Accepted |
2323
| [0011](0011-live-notification-services.md) | Live-capture notifications via the effects-bound platform services | Accepted |
2424
| [0012](0012-freeze-contract-v1.md) | Freeze `contract-v1` for the 0.5.0 release | Accepted |
25+
| [0013](0013-contract-v2-corpus.md) | Cut `contract-v2`: the 18-scenario corpus | Accepted |

‎docs/cli-parity.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ intentional difference.
1313
| `report <file> -f <fmt>` | `studio report <file> [-f markdown\|html]` | ✅ Full — Markdown + self-contained HTML. |
1414
| `diff <a> <b> --format <fmt>` | `studio diff <a> <b> [--format text\|json]` | ✅ Full — text and JSON. |
1515
| `anonymize <file>` | `studio anonymize <file>` | ✅ Full — same field rules (see the anonymize module). |
16-
| `ci [dir]` | `studio ci` | ⚠️ Runs the **vendored** `contract-v1` scenarios (Studio embeds the contract); the `[dir]` argument is not accepted. Exits `0` (all pass) / `1` (any fail). |
16+
| `ci [dir]` | `studio ci` | ⚠️ Runs the **vendored** `contract-v2` scenarios (Studio embeds the contract); the `[dir]` argument is not accepted. Exits `0` (all pass) / `1` (any fail). |
1717
| `scenario list` | `studio scenario list` | ✅ Full. |
1818
| `scenario run <name>` | `studio scenario run <name>` | ✅ Built-in scenarios. |
1919
| `scenario run --file <path>` | — | ⚠️ Deferred — Studio runs only the built-in contract scenarios. |
@@ -26,7 +26,7 @@ intentional difference.
2626
convenience is a deferred follow-up (it needs the sandboxed `init.io` write
2727
path); redirection covers the same need today.
2828
- **`ci` runs the embedded contract**, not a user directory. Studio vendors the
29-
15 `contract-v1` fixtures + goldens under `src/ocpp/conformance/` (ADR-0004),
29+
18 `contract-v2` fixtures + goldens under `src/ocpp/conformance/` (ADR-0004),
3030
so `ci` is a self-contained conformance gate — the same logic the
3131
`native test` conformance harness runs. See [the conformance
3232
contract](CONTRACT.md).

‎src/ocpp/conformance/README.md‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
# Conformance contract (`contract-v1`)
1+
# Conformance contract (`contract-v2`)
22

33
Studio and the TypeScript toolkit are **independent implementations** of the same
44
OCPP analysis behavior (ADR-0001). This directory is the machine-checked contract
55
between them: the shared scenarios and the failure codes each must detect.
66

7-
- **`fixtures/<name>.json`** — the 15 shared scenario **traces** (the input).
7+
- **`fixtures/<name>.json`** — the 18 shared scenario **traces** (the input).
88
- **`goldens/<name>.json`** — each scenario's expected **de-duplicated, sorted
99
`FailureCode` set** (the output to match), as a JSON array of wire codes.
1010
- **`harness.zig`** — runs, under `native test`, the full engine
@@ -39,6 +39,16 @@ lets the harness embed it directly — no runtime file I/O, fully hermetic.
3939

4040
## Versioning
4141

42-
Tagged **`contract-v1`** (OCPP 1.6J). When the contract changes — new rules,
43-
new scenarios, or OCPP 2.0.1 — regenerate against the matching toolkit release
44-
and bump the tag.
42+
Tagged **`contract-v2`** (OCPP 1.6J), 18 scenarios, regenerated against toolkit
43+
`0.4.5`. When the contract changes — new rules, new scenarios, or OCPP 2.0.1 —
44+
regenerate against the matching toolkit release and bump the tag. Earlier versions
45+
stay reachable through the release tags rather than being kept side by side here:
46+
`v0.5.4`'s tree carries `contract-v1` (15 scenarios).
47+
48+
Regenerating is expected to reproduce the existing files byte for byte, and that
49+
is worth checking when you do it: if a fixture you did not mean to touch changes,
50+
either the toolkit version is wrong or the change is bigger than you thought.
51+
`contract-v2` was cut that way, so it is a strict superset of v1.
52+
53+
See [ADR-0013](../../../docs/adr/0013-contract-v2-corpus.md) for the v2 cut and
54+
[docs/CONTRACT.md](../../../docs/CONTRACT.md) for the public contract.
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
{
2+
"traceId": "scenario-firmware-update-failure",
3+
"metadata": {
4+
"stationId": "CS-SYNTHETIC-019",
5+
"ocppVersion": "1.6",
6+
"source": "synthetic-scenario",
7+
"description": "Station boots, reports Downloading firmware, then InstallationFailed within seconds, followed by a heartbeat to confirm station is still online."
8+
},
9+
"events": [
10+
{
11+
"timestamp": "2026-01-15T07:00:00.000Z",
12+
"direction": "CS_TO_CSMS",
13+
"message": [
14+
2,
15+
"msg-001",
16+
"BootNotification",
17+
{
18+
"chargePointVendor": "SyntheticVendor",
19+
"chargePointModel": "SM-100",
20+
"chargePointSerialNumber": "CS-SYNTHETIC-019",
21+
"firmwareVersion": "1.0.0"
22+
}
23+
]
24+
},
25+
{
26+
"timestamp": "2026-01-15T07:00:00.500Z",
27+
"direction": "CSMS_TO_CS",
28+
"message": [
29+
3,
30+
"msg-001",
31+
{
32+
"currentTime": "2026-01-15T07:00:00.500Z",
33+
"interval": 300,
34+
"status": "Accepted"
35+
}
36+
]
37+
},
38+
{
39+
"timestamp": "2026-01-15T07:00:30.000Z",
40+
"direction": "CS_TO_CSMS",
41+
"message": [
42+
2,
43+
"msg-002",
44+
"FirmwareStatusNotification",
45+
{
46+
"status": "Downloading"
47+
}
48+
]
49+
},
50+
{
51+
"timestamp": "2026-01-15T07:00:30.500Z",
52+
"direction": "CSMS_TO_CS",
53+
"message": [
54+
3,
55+
"msg-002",
56+
{}
57+
]
58+
},
59+
{
60+
"timestamp": "2026-01-15T07:01:00.000Z",
61+
"direction": "CS_TO_CSMS",
62+
"message": [
63+
2,
64+
"msg-003",
65+
"FirmwareStatusNotification",
66+
{
67+
"status": "InstallationFailed"
68+
}
69+
]
70+
},
71+
{
72+
"timestamp": "2026-01-15T07:01:00.500Z",
73+
"direction": "CSMS_TO_CS",
74+
"message": [
75+
3,
76+
"msg-003",
77+
{}
78+
]
79+
},
80+
{
81+
"timestamp": "2026-01-15T07:02:00.000Z",
82+
"direction": "CS_TO_CSMS",
83+
"message": [
84+
2,
85+
"msg-hb-1",
86+
"Heartbeat",
87+
{}
88+
]
89+
},
90+
{
91+
"timestamp": "2026-01-15T07:02:00.500Z",
92+
"direction": "CSMS_TO_CS",
93+
"message": [
94+
3,
95+
"msg-hb-1",
96+
{
97+
"currentTime": "2026-01-15T07:02:00.500Z"
98+
}
99+
]
100+
}
101+
]
102+
}

0 commit comments

Comments
 (0)