Config-dependent fixes can ship inert: audit env-gated behaviour for boot-time validation

[JSONbored]

Summary

Several production-behaviour fixes are gated on server env vars that were never set, so the shipped code path stayed inert and the deployment silently behaved exactly like the pre-fix build. No unit test can catch this class: tests verify the mechanism works when enabled, never that an operator set the variable.

This is a process/observability gap, not a single bug. It has already caused at least one "fixed" bug to stay live for weeks.

Confirmed instance: bare-score public-comment filter

LOOPOVER_PUBLIC_SCORE_TERMS_ALLOWED_REPOS was unset on edge-nl-01. It fails closed, so BARE_SCORE_TERM_PATTERN = /\bscore\w*\b/i was enforced for every repo.

sanitizePublicComment throws on a match and toPublicSafe catches → returns null. It is all-or-nothing, so any AI review narrative containing score / scores / scoring / scored had its entire summary discarded and replaced by the generic placeholder. Nits survived because they are filtered individually.

The composeAdvisoryNotes doc comment already documents this exact failure:

"Metagraphed#8038-class bug: without this, ANY review of code with a legitimately-public score-named field ... had its entire narrative assessment silently discarded in favor of the generic 'did not include a separate narrative summary' fallback -- observed live, recurring."

The allowBareScoreTerm mechanism was built to fix it and is correctly wired at ai-review.ts (allowBareScoreTerm: isPublicScoreTermSafeForRepo(env, input.repoFullName)) — but with the env var unset it never engaged. The fix shipped and was inert.

Mitigated by setting the allowlist to the three gated repos. A boot-time warning (selfhost_public_score_terms_allowlist_unset) is added in the linked PR so this specific one can't silently recur.

The general problem

Other config-dependent safety/behaviour paths have the same shape and no equivalent signal. Worth auditing for:

• fail-closed defaults where "unset" silently disables a fix rather than failing loudly
• env vars whose absence changes review output or gate disposition (not just performance)
• anything whose doc comment says a bug was "observed live, recurring"

Proposed work

1. Audit src/env.d.ts for vars whose unset state silently disables a shipped fix, and classify each: hard-fail at boot / warn at boot / genuinely optional.
2. Extend assertSelfHostPreflight (or the boot warning block in server.ts) to cover the ones that change output correctness.
3. Consider surfacing "inert config" state on /health or the ops dashboard so it is visible without reading boot logs.

Precedent to follow

shouldWarnRagEmbedUnavailable (src/selfhost/ai.ts → shouted in src/server.ts) is the existing pattern: a pure predicate plus a single structured boot-time console.error, warn-only so a legitimate configuration is not blocked. The new shouldWarnPublicScoreTermsAllowlistUnset mirrors it exactly.

[linear-code]

JSO-6521

[JSONbored]

First instance addressed by #9437 — adds shouldWarnPublicScoreTermsAllowlistUnset plus a boot-time shout, so the bare-score exemption can no longer sit inert unnoticed. The broader audit described in this issue (classifying every env-gated behaviour as hard-fail / warn / genuinely optional) is still open.

[JSONbored]

Status: confirmed instance shipped; broader audit (item 1) not yet started

The one confirmed instance — LOOPOVER_PUBLIC_SCORE_TERMS_ALLOWED_REPOS unset silently disabling allowBareScoreTerm — has its boot-time warning merged (#9437, shouldWarnPublicScoreTermsAllowlistUnset, mirroring shouldWarnRagEmbedUnavailable exactly as proposed). The allowlist itself is set for all three repos in production.

Also worth folding in here: this session found the deeper version of the same failure shape underneath this one. The bare-score allowlist only ever addressed the numeric \bscore\w*\b pattern — but FORBIDDEN_PUBLIC_COMMENT_WORDS's explicit-phrase list (reward, ranking, cohort, ...) had the identical all-or-nothing discard behavior with no allowlist escape at all, and was the actual dominant cause of #9432's missing-summary reports. Fixed generally in #9445 by filtering per sentence instead of per config flag — which is arguably the more robust fix for this issue's underlying concern too: a config-gated escape hatch (allowlist) can itself be forgotten or misconfigured per repo, whereas sentence-level filtering doesn't depend on an operator having set anything correctly in the first place.

The broader audit this issue proposes (item 1: sweep src/env.d.ts for other vars whose unset state silently disables a shipped fix) has not been started. Leaving open, scoped to that remaining work.