feat(conformance): ADPP conformance harness — foundation (#25) - #26
Conversation
Adds a conformance/ package (shipped in the anolis-protocol wheel) that drives any provider binary through the ADPP v1 lifecycle and asserts compliance: - client.py: provider-agnostic stdio+uint32_le AdppClient (promoted from sim). - profiles.py: per-provider profiles + known divergences as non-strict xfails. - spec.py: status codes resolved from the proto enum (never hardcoded). - plugin.py / cli.py: pytest plugin + anolis-adpp-conformance console script. - test_conformance.py: 7 assertion groups + CLI checks. - ADPP-CONFORMANCE.md: the executable spec. Validated against the released sim/ezo/bread binaries (mock mode): green-as- baseline. ezo is fully conformant; tracked divergences (xfail): sim T0.1 + no --version; bread T0.4 + wait_ready missing init_time_ms. Also: pyproject ships the package + [conformance] extra + entry points; CI gains a conformance(sim) lane; handshake.proto comment fixed; Python .gitignore entries. Refs #25.
|
The latest Buf updates on your PR. Results from workflow CI / lint (pull_request).
|
…e plugin, harden verifier Reworks the harness per review on PR #26: Normative correctness (the harness must not contradict the spec): - Unknown signal id: accept either fail-NOT_FOUND OR partial-results (semantics 7.1); test consistency, not a forced NOT_FOUND. Removes the false sim waiver. - function_id/function_name conflict: spec says MUST prefer function_id (6.2), so bread is conformant; removed the reversed assertion + false bread waiver. - Unsupported version: accept FAILED_PRECONDITION or UNIMPLEMENTED (3). - Drop the "one in-flight" claim (concurrency/out-of-order is allowed; it is a runtime profile, not ADPP). Plugin isolation: removed the global pytest11 entry point (it forced Anolis options + an autouse fixture onto every unrelated pytest run); the console script loads the plugin via -p. Missing provider options now skip, not fail. CI asserts an unrelated pytest suite still passes with the wheel installed. Robustness: malformed-input checks are now crash-aware (inspect exit codes; fail on signal death / undocumented exit / hang / over-cap response). Real monotonic read deadline (a byte-drip can't stretch it). Response frames bounded by the 1 MiB cap. request_id correlation enforced. Verifier self-tests: fake faulty providers (hang, signal-crash, over-cap, drip, mid-frame close, wrong request_id, missing status) prove the harness rejects bad providers — hermetic, run in CI. Structure/docs: split into ADPP-core / framed-stdio / executable-profile; health is genuinely non-gating (experimental marker, excluded by default); added fragmentation/coalescing, include_health, typed-value, unknown-device coverage; metadata values asserted; readiness derived from advertised metadata; ready_client asserts wait_ready OK. ADPP-CONFORMANCE.md defers to semantics.md and is scoped as a foundation (not "all of #25"); Linux-only declared. CI pins the sim release. Refs #25.
CameronBrooks11
changed the title
|
Thanks for the rigorous review — it caught real problems. Pushed Blocking 1 — normative contradictions (verified against
Blocking 2 — plugin isolation. Removed the global Blocking 3 — crash-aware malformed input. Blocking 4 — deadline + response size. Blocking 5 — doc accuracy. Rewrote Blocking 6 — verifier self-tests. New Blocking 7 — rescope. Retitled to "foundation"; the doc states this is not all of #25. Provider CI lanes, full assertions, version-pin alignment, Windows, and strict provider-owned waivers are explicit follow-ups (#25 stays open). Also: request_id correlation enforced; metadata values asserted (not just keys); readiness derived from advertised metadata; Still not merging — please re-review. Remaining items I'd propose as follow-ups (not this PR): the per-provider |
e0c5686 to
6e099f2
Compare
The protocol repo must not encode knowledge of its implementers. Invert the dependency: ship a generic, versioned verifier that providers pull and parametrize, rather than baking sim/ezo/bread into the contract. - profiles.py: delete the hardcoded PROFILES registry; keep the ProviderProfile schema and add load_profile() for a provider-supplied TOML manifest (provider_name + has_mock_devices + [waivers]). - plugin.py: replace --profile <name> (in-repo lookup) with --provider-profile <file> (provider-owned manifest); validate on load. - ci.yml: drop the provider-sim release-binary canary (a contract -> implementer reverse edge); gate only the hermetic verifier self-tests. Cross-provider verification moves to each provider's CI lane. - self-tests: cover load_profile (valid/defaults/invalid) hermetically. - docs: describe the provider-owned manifest; reduce implementer mentions to a single "currently implemented by" line. - pyproject: tomli fallback for Python < 3.11.
6e099f2 to
f852cef
Compare
Review hardening: ensure the verifier cannot report green without meaningfully verifying. All findings checked against docs/semantics.md (authoritative) before changing. - modes: provider runs require all of --provider-bin/--provider-config/ --provider-profile; a bare or partial invocation is a usage error. Add --self-test for the hermetic verifier lane. CI asserts each misconfiguration exits nonzero. - waivers: scope to executable_profile-marked tests only (rejected at collection otherwise, so a waiver can't mask a core/transport/verifier failure); apply as strict xfail (a fixed gap fails as XPASS). - malformed input: factor validation into checks.assert_controlled_malformed; reject CODE_OK/CODE_UNSPECIFIED on garbage and respond-then-crash. Self-tests exercise the real helper against those fake modes. - unknown identifiers: require CODE_NOT_FOUND for unknown function_id and function_name (semantics.md §8.3). - reads: unknown-signal test now asserts the known signal survives and the policy is consistent across mixed/repeat/unknown-only (§7.4); default read restricted to OK|UNAVAILABLE; include_health requires per-device coverage when has_mock_devices (inventory.proto). - contract boundary: move Hello transport-metadata assertions from ADPP core into the framed-stdio profile (§8). Tighten --version, fragmentation, coalescing, and add a trailing-garbage case; reject unknown profile keys. - CI: add a Python 3.10 lane to exercise the tomli fallback.
) Second-round review hardening. Each finding checked against semantics.md / proto / release workflow before changing. - malformed-input: a response-then-exit is validated against ALLOWED_MALFORMED_EXIT_CODES (undocumented positive exits now rejected, not just signals); the error status must be a DEFINED enum code, not any nonzero integer. New respond_error_then_exit_bad self-test. - waiver scope: move test_multiple_roundtrips_stay_framed into the framed-stdio (non-waivable) suite — a transport-integrity test must not be waivable. - coalescing: drive two post-handshake ListDevices instead of a repeated Hello (semantics.md §3 does not define a second handshake as must-succeed). - reads: partial unknown-signal results must be a subset of requested ids (no unrelated inventory). Drop the non-empty CapabilitySet assertion from core (§6.1 requires complete, not non-empty). - status presence: factor checks.assert_status_present (§10), wire it into the suite, and make the missing-status self-test exercise the real validator. - guards prove they fired: the waiver-scope CI regression now uses real bin/config paths and asserts the specific diagnostic. - normative profiles: add docs/profiles/framed-stdio-v1.md (uint32_le, 1 MiB, Hello metadata, malformed/exit-code behavior) and anolis-executable-profile-v1.md (organizational, waivable); each suite cites its source; semantics.md §2 points at the binding. - docs: fix the shell example, the direct-pytest invocation (-p), stale strict=False comments, the §7.4 reference, and the README (no PyPI publish — install the GitHub-release wheel; note the conformance package + profiles).
v1.1.4/v1.2.0 predate the conformance harness, so their wheels have neither the [conformance] extra nor anolis_conformance. Install from a checkout until the first release containing it; placeholder for the release-wheel form.
1 participant
Implements Wave 0 of the provider convergence → SDK plan: a cross-provider ADPP conformance harness that drives any provider binary through the ADPP v1 wire lifecycle and asserts compliance against
docs/semantics.md(the normative spec). Scope is the foundation — a generic, versioned verifier; provider lanes and the full assertion set are staged follow-ups under #25.Architecture: the contract owns the verifier, not its implementers
anolis-protocolis the abstract contract; it must not encode knowledge of its implementers. So the harness ships as a generic, versioned artifact in theanolis-protocolwheel that providers pull (pinned, like the C++ FetchContent of the proto) and parametrize with their own identity/config/waivers.client.py(stdio +uint32_leAdppClient),spec.py(ADPP v1 constants; status codes resolved from the proto enum at runtime — the enum isn't sequential),checks.py(reusable validators), the three contract suites, the hermetictest_selftest.py, packaging ([conformance]extra +anolis-adpp-conformanceconsole script).conformance.tomlmanifest —provider_name,has_mock_devices,[waivers]— plus the mock config and aprovider.conformanceCI lane. The protocol package never re-releases for a provider-specific exception.The plugin is not a global
pytest11entry point (it loads explicitly via-p), so installing the wheel never affects unrelated pytest runs.Three separate contracts, each with a normative source
Each suite cites its authority — so framing/executable conventions can't silently become protocol law through tests:
test_adpp_core.py) →docs/semantics.md— messages, status,request_idcorrelation, capabilities, read/call semantics.test_framed_stdio.py) →docs/profiles/framed-stdio-v1.md(new, normative) —uint32_leframing, the 1 MiB cap, fragmentation/coalescing, Hello metadata, controlled malformed-stream behavior (incl. respond-then-exit codes).test_executable_profile.py) →docs/profiles/anolis-executable-profile-v1.md(new, organizational) — CLI surface, WaitReady diagnostics, process hygiene. Not ADPP — the only tests a provider waiver may xfail (enforced at collection; transport/core tests are non-waivable).The verifier cannot report green without verifying
A foundational conformance tool must fail loudly when misused:
--provider-*args (bare/partial invocations are usage errors);--self-testruns only the hermetic lane. CI asserts each misconfiguration exits nonzero.executable_profiletest (rejected at collection otherwise, so it can never mask a core/transport/verifier failure); strictxfailmeans a fixed gap fails asXPASS.checks.assert_controlled_malformed):CODE_OK/CODE_UNSPECIFIEDon garbage, hangs, crashes, and respond-then-crash are rejected; only a framed error response or a clean documented exit passes.CI (hermetic — no downstream coupling)
buf lint/breaking + the verifier self-tests + the plugin-isolation regression + the misconfiguration gate, on Python 3.10 and 3.12 (3.10 exercises the
tomlifallback). The protocol repo proves the verifier is sound; cross-provider verification runs in each provider's lane (it pulls this pinned wheel) — the contract's CI never downloads a provider release.Validation
--versionwaiver applies as a strict xfail; the unknown-signal consistency check surfaces a real sim divergence (§7.4: partial for mixed butNOT_FOUNDfor unknown-only) — to be fixed in sim's lane, and correctly not gating this repo's hermetic CI.Follow-ups (staged under #25, not in this PR)
Per-provider
provider.conformancelanes +conformance.tomlinanolis-provider-{sim,ezo,bread}; the remaining normative assertion coverage (positive calls, arg/bounds validation, selector precedence, deadlines, typed-value/quality/timestamp); an org-level released-version compatibility matrix; the sim Python wheel pin alignment (1.1.3 → 1.2.0); Windows portability.🔬 Please review — do not merge until you approve.