Repository navigation
Stop staging after the end to end check and a configurable hold window, so it does not run permanently - #1728
Merged
Merged
Conversation
… staging in drift (#1596) - terraform_apply.yml: hold-then-stop-staging dispatches staging_lifecycle.yml instead of calling it. The tf-apply-<ref> group held the run for the whole window, so the next merge queued behind it (and a third cancelled the second). - terraform_apply.yml: a no-op supersede-hold job claims stag-hold in place of start-staging, so a hold dispatched mid-start no longer cancels the apply. - terraform_drift.yml: drop the stopped-staging stand-down. A refresh only describes RDS, which works on a stopped instance, and desired_count is ignore_changes, so a stopped estate plans clean. - Comments, README and tests: start before the apply because a stopped instance cannot be modified, not because it fails the refresh. Signed-off-by: Alex Triay Bagur <alexandre.triay_bagur@kcl.ac.uk>
This was referenced Nov 2, 2026
…dispatch Signed-off-by: Alex Triay Bagur <alexandre.triay_bagur@kcl.ac.uk>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1621 (#1596): the two review threads left open when it merged.
The hold no longer blocks the next merge
hold-then-stop-stagingcalledstaging_lifecycle.ymlas a reusable workflow, so the hold was a job of the apply run. The workflow-leveltf-apply-<ref>group (cancel-in-progress: false) holds a run until its last job ends. A develop merge during the window therefore queued for the whole hold (120 min by default) and then started an estate that was still stopping. GitHub keeps only one pending run per group, so a third merge cancelled the second.hold-then-stop-stagingnow dispatchesStaging Lifecyclewithhold-then-stop, at the same ref and with the samehold_minutes. Aworkflow_dispatchmade withGITHUB_TOKENstarts a run, and the job needs onlyactions: write. The apply run ends when the dispatch is made.stag-holdclaim moves offstart-stagingonto a no-opsupersede-holdjob. A newer deploy still cancels a sleeping hold, and the lifecycle's stop declines a cancelled hold. Ahold-then-stopdispatched whilestart-stagingwaits on RDS (up to 50 min) now cancels that no-op instead of the start, which used to skip the apply and leave the run grey.Drift plans staging, stopped or not
The drift stand-down assumed a stopped RDS fails the refresh. Nothing in the root does more than describe the instance (only the
awsandnullproviders; no database connection), andDescribeDBInstancesanswers for a stopped instance. Desired counts areignore_changes, so a stopped estate plans clean. With the stand-down, drift would almost never check staging, because staging is down at 03:00. This PR drops thestaging-statejob and corrects the comments that gave the refresh as the reason to start first. The real reason is that a stopped instance cannot be modified, sostart-stagingstill runs before the apply.Verification
test_staging_lifecycle_workflow.pyandtest_terraform_stag_pause.py: 23 passed. The AWS deploy tests and.github/testspass; the only failures are the three intest_verify_deploy_readiness, which also fail on develop because they need a local env file. Each new assertion was mutation-checked (start-staging claiming the group again, no dispatch, drift waiting on a probe): each mutation fails exactly one test.staging-power.shsuite 162/162; ruff 0.14.7 check and format clean.stoppedsince 23:27 on 2 Nov, no start event since, stillstoppedafter the plan). The instance refreshed at 08:22:13 on 3 Nov and the plan reported "No changes. Your infrastructure matches the configuration." (run 41485947616, attempt 3). The 18:59 attempt on 2 Nov does not count: a queued develop apply had started RDS 52 s before the refresh, the queueing this PR removes.