docs(uts): spec-first handling of UTS spec errors (fail-fast test pattern)#498
Open
sacOO7 wants to merge 1 commit into
Open
docs(uts): spec-first handling of UTS spec errors (fail-fast test pattern)#498sacOO7 wants to merge 1 commit into
sacOO7 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the UTS “writing derived tests” guide to treat UTS spec mistakes as spec bugs (to be fixed at the source) and to enforce a fail-fast test pattern that keeps spec errors visible during evaluation, while keeping translation a purely mechanical process.
Changes:
- Clarifies that cross-validating UTS specs against features specs is evaluation-only, not part of translation.
- Adds fetchable links to the relevant feature/spec authority documents and explains two spec-error modes (contradiction vs internal inconsistency).
- Renames and expands the test-patterns section to include a spec-error fail-fast placeholder pattern, and updates the decision tree + deviations categorization accordingly.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sacOO7
force-pushed
the
docs/uts-derived-tests-spec-first-fail-fast
branch
from
74359f6 to
b611f8c
Compare
sacOO7
added a commit
to ably/ably-java
that referenced
this pull request
Jun 30, 2026
Align the skill's Step 6 with the spec manual (ably/specification#498) so a diagnosed UTS spec error is handled spec-first instead of being quietly adapted to green: - Step 6 intro now names three acceptable end-states (spec-correct pass, SDK deviation stays green, UTS spec error fails fast) — removes the earlier "a red test is never acceptable" contradiction. - Decision tree NO-branch: fix the spec at source + record under deviations.md "UTS Spec Errors" + emit a fail-fast test. - Add the "spec-error fail-fast" Kotlin pattern (mirrors the manual's JS example) and retitle the section to "Test patterns for a diagnosed failure" since fail-fast is not a deviation. - Defer the deviations-file entry format to the manual's "Recording deviations" sections (removes the divergent inline field list).
sacOO7
force-pushed
the
docs/uts-derived-tests-spec-first-fail-fast
branch
from
b611f8c to
cb7e8a4
Compare
sacOO7
force-pushed
the
docs/uts-derived-tests-spec-first-fail-fast
branch
from
cb7e8a4 to
162e42e
Compare
sacOO7
force-pushed
the
docs/uts-derived-tests-spec-first-fail-fast
branch
from
162e42e to
1781ebd
Compare
…st pattern When a derived test fails because the UTS spec itself is wrong, the manual previously said to "fix the test to match the features spec" — which quietly buries the spec bug and leaves it for every other SDK to re-hit. Make the guidance spec-first: the UTS spec is the source of truth, so a spec error must be fixed there, surfaced by a loud, self-describing test rather than papered over in the translation. - 2a (Is the UTS spec wrong?): mark this an evaluation-only step (translation never validates the spec); link the features specs as fetchable raw URLs (all live under specifications/); cover both failure modes (contradicts the features spec, or internally inconsistent); and require recording in UTS Spec Errors + emitting a fail-fast test instead of rewriting the test. - 3 (renamed "Test patterns for a diagnosed failure"): add the third pattern, Spec-error fail-fast — a placeholder that fails immediately, names the spec id and points at deviations.md — alongside the two SDK-deviation patterns, plus a Resolution note (fix the spec, then re-translate; don't resolve by editing the test). - Decision tree + Recording deviations: NO-branch is now a spec-first fail-fast; add the UTS Spec Errors section (four sections total).
sacOO7
force-pushed
the
docs/uts-derived-tests-spec-first-fail-fast
branch
from
1781ebd to
f035a80
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
uts/docs/writing-derived-tests.mdtreat a wrong UTS spec as a spec problem, not a test problem.Today, when a derived test fails because the UTS spec itself is wrong, the manual says to "fix the test to match the features spec." That quietly buries the spec bug: the test goes green, the spec stays wrong, and every other SDK re-hits it — defeating the point of a single source of truth. This PR flips that to spec-first: the spec is the source of truth, so a spec error is fixed there, surfaced by a loud, self-describing test rather than papered over in translation.
It also closes a practical gap — the manual told you to "compare against the features spec" but never gave you a fetchable link to one.
What changed
section 2a — Is the UTS spec wrong?
specifications/directory; added the raw, fetchable base URL plus per-module links (features.md,objects-features.md,chat-features.md) and the related authorities (protocol.md,encryption.md).section 3 — renamed Deviation test patterns → Test patterns for a diagnosed failure
The section now honestly covers all three diagnosed-failure outcomes:
deviations.md. There are no spec-correct assertions to write (the spec is the thing that's wrong), so failing loud is the forcing function.section 4 Decision tree + Recording deviations
Why it matters
Scope
Documentation only — no spec rule (
RTL*/RTN*/RTO*/…) is changed. Affects how SDK teams write derived tests from the UTS specs.