Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 58 additions & 8 deletions .github/workflows/production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,21 @@ on:
operation:
description: Production action
type: choice
options: [deploy, rollback]
options: [deploy, deploy_existing, rollback]
default: deploy
required: true
publication_run_id:
description: Successful main publication run for deploy_existing
type: string
required: false
source_commit:
description: Full source commit SHA for deploy_existing
type: string
required: false
image_digest:
description: Published sha256 digest for deploy_existing (digest only)
type: string
required: false
rollback_run_id:
description: Optional saved deployment id for rollback; blank selects latest
type: string
Expand All @@ -26,7 +38,7 @@ concurrency:

jobs:
test:
if: github.event_name != 'workflow_dispatch' || inputs.operation == 'deploy'
if: github.event_name != 'workflow_dispatch' || inputs.operation == 'deploy' || inputs.operation == 'deploy_existing'
runs-on: ubuntu-latest
permissions:
contents: read
Expand Down Expand Up @@ -83,7 +95,10 @@ jobs:
docker run --rm --network none
--volume "$GITHUB_WORKSPACE/tests:/tmp/codelens-tests:ro"
--volume "$GITHUB_WORKSPACE/scripts:/tmp/scripts:ro"
--volume "$GITHUB_WORKSPACE/.github/workflows/production.yml:/tmp/production.yml:ro"
--volume "$GITHUB_WORKSPACE/pytest.ini:/tmp/pytest.ini:ro"
--env CODELENS_WORKFLOW_FILE=/tmp/production.yml
--env PYTHONPATH=/tmp
--env DEBUG=false
--env DATABASE_URL=postgresql+asyncpg://test:test@127.0.0.1/test
--env REDIS_URL=redis://127.0.0.1:6379/0
Expand Down Expand Up @@ -134,7 +149,10 @@ jobs:
docker run --rm --network none
--volume "$GITHUB_WORKSPACE/tests:/tmp/codelens-tests:ro"
--volume "$GITHUB_WORKSPACE/scripts:/tmp/scripts:ro"
--volume "$GITHUB_WORKSPACE/.github/workflows/production.yml:/tmp/production.yml:ro"
--volume "$GITHUB_WORKSPACE/pytest.ini:/tmp/pytest.ini:ro"
--env CODELENS_WORKFLOW_FILE=/tmp/production.yml
--env PYTHONPATH=/tmp
--env DEBUG=false
--env DATABASE_URL=postgresql+asyncpg://test:test@127.0.0.1/test
--env REDIS_URL=redis://127.0.0.1:6379/0
Expand Down Expand Up @@ -171,12 +189,44 @@ jobs:
[[ "$digest" =~ ^sha256:[a-f0-9]{64}$ ]]
echo "digest=$digest" >> "$GITHUB_OUTPUT"

verify_existing:
needs: test
if: github.event_name == 'workflow_dispatch' && inputs.operation == 'deploy_existing' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
packages: read
outputs:
image: ${{ steps.verify.outputs.image }}
source_commit: ${{ steps.verify.outputs.source_commit }}
steps:
- uses: actions/checkout@v7.0.1
with:
persist-credentials: false
- name: Verify prior tested publication and immutable digest
id: verify
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_ACTOR: ${{ github.actor }}
PUBLICATION_RUN_ID: ${{ inputs.publication_run_id }}
SOURCE_COMMIT: ${{ inputs.source_commit }}
IMAGE_DIGEST: ${{ inputs.image_digest }}
shell: bash
run: |
set -euo pipefail
python3 scripts/verify_ghcr_publication.py \
--publication-run-id "$PUBLICATION_RUN_ID" \
--source-commit "$SOURCE_COMMIT" \
--image-digest "$IMAGE_DIGEST"

deploy:
needs: publish
needs: [publish, verify_existing]
if: >-
github.ref == 'refs/heads/main' && needs.publish.result == 'success' &&
((github.event_name == 'workflow_dispatch' && inputs.operation == 'deploy') ||
(github.event_name == 'push' && vars.CODELENS_AUTO_DEPLOY == 'true'))
always() && github.ref == 'refs/heads/main' &&
((github.event_name == 'workflow_dispatch' && inputs.operation == 'deploy' && needs.publish.result == 'success') ||
(github.event_name == 'workflow_dispatch' && inputs.operation == 'deploy_existing' && needs.verify_existing.result == 'success') ||
(github.event_name == 'push' && needs.publish.result == 'success' && vars.CODELENS_AUTO_DEPLOY == 'true'))
runs-on: ubuntu-latest
environment: production
permissions:
Expand Down Expand Up @@ -227,8 +277,8 @@ jobs:
echo "ssh_known_hosts=$SSH_DIR/known_hosts" >> "$GITHUB_ENV"
- name: Deploy exact published image digest
env:
IMAGE: ${{ needs.publish.outputs.image }}@${{ needs.publish.outputs.digest }}
EXPECTED_SHA: ${{ github.sha }}
IMAGE: ${{ inputs.operation == 'deploy_existing' && needs.verify_existing.outputs.image || format('{0}@{1}', needs.publish.outputs.image, needs.publish.outputs.digest) }}
EXPECTED_SHA: ${{ inputs.operation == 'deploy_existing' && needs.verify_existing.outputs.source_commit || github.sha }}
GHCR_USERNAME: ${{ github.actor }}
shell: bash
run: |
Expand Down
34 changes: 28 additions & 6 deletions docs/ghcr-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,17 +49,39 @@ deployment helper.
credentials revoked. Keep the replacement values only in their protected
provider settings and production `.env`; never paste or print them in Actions
logs or deployment diagnostics.
3. On the `main` branch, run Actions → **Test, publish, and deploy CodeLens** →
Run workflow → `deploy`. The image is published only after both test runs
pass. Approve the `production` environment job only after reviewing the
commit and published image digest.
4. The EC2 helper verifies the digest's embedded source commit and `linux/amd64`
3. For the already-published first-release image, open Actions → **Test, publish,
and deploy CodeLens** → **Run workflow**, select branch `main`, and set:
- `operation`: `deploy_existing`
- `publication_run_id`: the numeric run ID from the successful publication
run's URL (not the UI run number). For the reported run #9, copy the ID from
its `/actions/runs/<id>` URL.
- `source_commit`:
`b3c7a3cc74336272b907c10eb32ca874aaa908cb`
- `image_digest`:
`sha256:c2da031dca76bccab86e95b05a469048dc876d926c5ba978a0e0e41fba7b2e3a`
The verification job checks that this is a successful main publication from
this workflow, that its test and image-publish steps succeeded, that the
commit tag resolves to the supplied immutable digest, and that the image
config identifies the same source commit and `linux/amd64`. It reads registry
manifests/config only; it does not pull, rebuild, or republish the image.
4. The `deploy_existing` mode runs the current workflow's tests and then the
verifier. It skips the publisher, so it cannot collide with or overwrite the
source commit tag. The existing `deploy` mode remains for a newly unpublished
main commit: it tests, builds, publishes, and deploys that same run's digest.
5. Confirm the `production` environment has a required reviewer. Approve the
deployment job only after checking the source commit and digest in the run.
Keep `CODELENS_AUTO_DEPLOY` absent or `false`.
6. Immediately before approval, confirm the deployment's Celery gate is clear:
active, reserved, scheduled, unacknowledged, and unacknowledged-index counts
must all be zero. If inspection is unavailable or any count is nonzero, the
helper aborts before changing services.
7. The EC2 helper verifies the digest's embedded source commit and `linux/amd64`
platform, checks free space and inodes, and requires the worker to report no
active, reserved, scheduled, or unacknowledged Celery deliveries. If any
inspection is unavailable or work remains in flight, it stops without
changing services. Ready messages may remain queued in Redis; the worker
consumes them after restart.
5. The helper tags each current service image by its exact image ID, pulls the
8. The helper tags each current service image by its exact image ID, pulls the
GHCR digest once, checks post-pull headroom, and updates only `web`, `worker`,
and `flower` in project `codelens`. It checks local and public FastAPI health,
Celery worker health and ping, and Flower reachability. Failure triggers an
Expand Down
Loading
Loading