Skip to content

test(formats): port the Open VSX gateway probe as test-vscode-remote (release gate) - #376

Open
cazlo wants to merge 1 commit into
artifact-keeper:mainfrom
cazlo:test/vscode-remote-release-gate
Open

cazlo wants to merge 1 commit into
artifact-keeper:mainfrom
cazlo:test/vscode-remote-release-gate

Conversation

@cazlo

@cazlo cazlo commented Aug 15, 2026

Copy link
Copy Markdown

fixes #375

Summary

artifact-keeper/artifact-keeper#3253 merged 2026-08-15, shipping AK as a governed VS Code extension gateway (composed Open VSX gallery pull-through, every asset URL rewritten onto AK). The PR's own dogfood harness carried a note that its probe subcommand should move to the release gate; it never did. This adds tests/formats/test-vscode-remote.sh, ported from scripts/dogfood-vscode-openvsx.sh's probe subcommand, and registers it in both format-tests.yml and release-gate.yml (node batch in each).

What it covers

  • Gallery manifest names AK's own extensionquery/latest endpoints, not open-vsx.org's.
  • extensionquery POST with the exact VS Code client request-body shape resolves a reviewed extension (redhat.vscode-yaml) and every returned assetUri/fallbackAssetUri/files[].source is AK-owned.
  • {publisher}/{name}/latest resolves the same extension, all asset URLs AK-owned.
  • The code-server compatibility alias (/gallery/vscode/{publisher}/{name}/latest) resolves identically to the primary latest lookup, checked as its own direct probe so a regression in the alias specifically can't hide behind the primary path passing.

Not ported: the dogfood script's vscodium-ui, code-server-ui, install-, update-, and TRACE_NETWORK subcommands drive a real VS Code-family client against an isolated profile. That doesn't fit a headless, 120s-per-script release-gate suite, and the gate runners have no VSCodium/code-server binary on PATH.

I also fixed a real bug while porting rather than copying it forward: the dogfood script's inline jq asset-URL assertions build [.assetUri?, .fallbackAssetUri?, (.files[]?.source?)] without filtering nulls before all(startswith($prefix)). The ? suffix suppresses type errors, not missing keys, so a version object lacking fallbackAssetUri still yields null for that slot and null | startswith(_) is a jq runtime error, which aborts the whole jq -e and inverts the diagnosis (reads as "retains a non-AK asset URL" when the truth is "field absent"). The new _assert_asset_urls_ak_owned helper null-filters first.

Validation against the shipped composed gateway

artifact-keeper/artifact-keeper#3253 landed a gallery skeleton+synthesis rework on 2026-08-15, after I first wrote this port, so I re-validated every assertion against a live composed-gateway backend rather than trusting the pre-rework version of the script.

Bounded extensionquery responses (I measured 30 stable + 29 prerelease versions returned for the probe extension, against the newest-30-per-platform-x-channel cap, instead of raw upstream counts), the /latest age-gate walk-back (200 with the newest allowed version instead of 404), and the flags-conditional presence of files/assetUri on synthesized entries are all confirmed compatible with this script as written. None of the assertions check an exact or all-versions count, so the bounding doesn't reach them, and the script always queries with flags=511, which requests full asset detail, so it's never exposed to the omitted-field case. No assertion needed changing.

I then ran the script end to end against that backend: 9 passed, 0 failed, 0 skipped in 3s, against the gate's 120s-per-script budget. The run also confirms the cleanup path — the throwaway repo it creates is gone afterwards — and that the backend accepts format: vscode with an Open VSX gallery-adapter upstream_url through the ordinary repository-creation API.

Red-proof

This is new-feature coverage, not a regression test for an existing defect, so there's no prior "broken behavior" to construct a red state against. The natural red state is that these routes didn't exist at all before artifact-keeper/artifact-keeper#3253 merged (404, not a wrong-answer 200) -- I'm treating route-absence as sufficient rather than adding a synthetic failure-injection leg, consistent with how this repo's other remote-proxy format tests (npm-remote, go-remote, etc.) were added as straight coverage additions without a constructed red leg.

Also found while here

Having to register this script by hand sent me looking at what else isn't registered. tests/formats is the only suite without auto-discovery — fourteen other suites run through scripts/run-suite.sh, which globs tests/<suite>/**/test-*.sh, while formats enumerates scripts by name in the workflow YAML. That gap leaves 59 of the 119 tests/formats/test-*.sh scripts named in no workflow at all, including a regression gate written for artifact-keeper/artifact-keeper#2477 that has never executed. Filed separately as #374 rather than folding a repo-wide fix into this PR.

Test plan

  • Manual run against a live composed-gateway backend: 9 passed, 0 failed, 0 skipped, 3s.
  • format-tests.yml node batch, release-gate.yml node batch: both green in this repo's own CI.

…(release gate)

Source: artifact-keeper PR #3253 scripts/dogfood-vscode-openvsx.sh, the
`probe` subcommand (manifest fetch, gallery extensionquery POST with the
exact VS Code client body shape, {publisher}/{name}/latest lookup, the
code-server compatibility alias /gallery/vscode/{p}/{n}/latest, and the
"every assetUri/fallbackAssetUri/files[].source is AK-owned" assertions).
The dogfood script's other subcommands (vscodium-ui, code-server-ui,
install-*, update-*, TRACE_NETWORK) drive a real VS Code-family client
against an isolated profile and don't fit a headless, 120s-per-script
release-gate suite, so they do not survive the port.

Fixes a real bug while porting rather than copying it: the dogfood script's
inline jq asset-URL assertions (its lines ~167-170 and ~183-186) built
`[.assetUri?, .fallbackAssetUri?, (.files[]?.source?)]` without filtering
nulls before `all(startswith($prefix))`. `X?` on an ABSENT key still
produces `null`, and `null | startswith(_)` is a jq runtime error, which
aborts the whole `jq -e` and inverts the diagnosis (reads as "retains a
non-AK asset URL" when the real story is "field absent"). The new
_assert_asset_urls_ak_owned helper null-filters first, matching the shape
of the dogfood script's own already-fixed assert_ak_urls helper (~lines
105-116) instead of its buggy inline duplicates.

Registered in .github/workflows/release-gate.yml (node batch,
matrix.batch.scripts) and .github/workflows/format-tests.yml (node batch)
so the suite actually runs; tests/formats is not auto-discovered.
@cazlo
cazlo marked this pull request as ready for review August 15, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release gate has no coverage for the vscode Open VSX gallery gateway

1 participant