Skip to content

Stop staging after the end to end check and a configurable hold window, so it does not run permanently - #1728

Merged
atriaybagur merged 2 commits into
developfrom
1596-staging-hold-dispatch
Nov 3, 2026
Merged

atriaybagur merged 2 commits into
developfrom
1596-staging-hold-dispatch

Conversation

@atriaybagur

@atriaybagur atriaybagur commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #1621 (#1596): the two review threads left open when it merged.

The hold no longer blocks the next merge

hold-then-stop-staging called staging_lifecycle.yml as a reusable workflow, so the hold was a job of the apply run. The workflow-level tf-apply-<ref> group (cancel-in-progress: false) holds a run until its last job ends. A develop merge during the window therefore queued for the whole hold (120 min by default) and then started an estate that was still stopping. GitHub keeps only one pending run per group, so a third merge cancelled the second.

  • hold-then-stop-staging now dispatches Staging Lifecycle with hold-then-stop, at the same ref and with the same hold_minutes. A workflow_dispatch made with GITHUB_TOKEN starts a run, and the job needs only actions: write. The apply run ends when the dispatch is made.
  • The stag-hold claim moves off start-staging onto a no-op supersede-hold job. A newer deploy still cancels a sleeping hold, and the lifecycle's stop declines a cancelled hold. A hold-then-stop dispatched while start-staging waits on RDS (up to 50 min) now cancels that no-op instead of the start, which used to skip the apply and leave the run grey.

Drift plans staging, stopped or not

The drift stand-down assumed a stopped RDS fails the refresh. Nothing in the root does more than describe the instance (only the aws and null providers; no database connection), and DescribeDBInstances answers for a stopped instance. Desired counts are ignore_changes, so a stopped estate plans clean. With the stand-down, drift would almost never check staging, because staging is down at 03:00. This PR drops the staging-state job and corrects the comments that gave the refresh as the reason to start first. The real reason is that a stopped instance cannot be modified, so start-staging still runs before the apply.

Verification

  • test_staging_lifecycle_workflow.py and test_terraform_stag_pause.py: 23 passed. The AWS deploy tests and .github/tests pass; the only failures are the three in test_verify_deploy_readiness, which also fail on develop because they need a local env file. Each new assertion was mutation-checked (start-staging claiming the group again, no dispatch, drift waiting on a probe): each mutation fails exactly one test.
  • staging-power.sh suite 162/162; ruff 0.14.7 check and format clean.
  • Empirical, before ready: re-ran this PR's aws-stag plan with staging stopped (RDS stopped since 23:27 on 2 Nov, no start event since, still stopped after the plan). The instance refreshed at 08:22:13 on 3 Nov and the plan reported "No changes. Your infrastructure matches the configuration." (run 41485947616, attempt 3). The 18:59 attempt on 2 Nov does not count: a queued develop apply had started RDS 52 s before the refresh, the queueing this PR removes.
  • The first develop merge after this lands: Stop staging after the end to end check and a configurable hold window, so it does not run permanently #1728's own merge. Its apply (run 41602898322) started staging, applied, passed the end to end check and ended at 08:33 on 3 Nov once it had dispatched the hold; Staging Lifecycle run 41604199198 holds the window.

… staging in drift (#1596)

- terraform_apply.yml: hold-then-stop-staging dispatches staging_lifecycle.yml
  instead of calling it. The tf-apply-<ref> group held the run for the whole
  window, so the next merge queued behind it (and a third cancelled the second).
- terraform_apply.yml: a no-op supersede-hold job claims stag-hold in place of
  start-staging, so a hold dispatched mid-start no longer cancels the apply.
- terraform_drift.yml: drop the stopped-staging stand-down. A refresh only
  describes RDS, which works on a stopped instance, and desired_count is
  ignore_changes, so a stopped estate plans clean.
- Comments, README and tests: start before the apply because a stopped
  instance cannot be modified, not because it fails the refresh.

Signed-off-by: Alex Triay Bagur <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur atriaybagur self-assigned this Oct 30, 2026
@github-actions github-actions Bot changed the title Staging hold runs outside the apply run, and drift plans stopped staging (#1596) Stop staging after the end to end check and a configurable hold window, so it does not run permanently Oct 30, 2026
@atriaybagur atriaybagur added ci CI / build pipeline changes (release-notes category) bug Something isn't working labels Oct 30, 2026
…dispatch

Signed-off-by: Alex Triay Bagur <alexandre.triay_bagur@kcl.ac.uk>
@atriaybagur
atriaybagur marked this pull request as ready for review November 3, 2026 08:23
@atriaybagur
atriaybagur merged commit ea4d579 into develop Nov 3, 2026
66 checks passed
@atriaybagur
atriaybagur deleted the 1596-staging-hold-dispatch branch November 3, 2026 08:23

This branch was previously deployed

2 inactive deployments
aws-stag — 5a78881b Deployed Nov 3, 2026 by atriaybagur via Plan (stag) #474
flip — 5a78881b Deployed Nov 2, 2026 by atriaybagur via FLARE heavy unit tests #810
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci CI / build pipeline changes (release-notes category)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant