Skip to content

feat: the umbrella's worker serves the default network zone; pin the zone releases - #27

Merged
leandrolerena merged 32 commits into
mainfrom
feature/network-zones-01-central-worker-zone
Nov 7, 2026
Merged

leandrolerena merged 32 commits into
mainfrom
feature/network-zones-01-central-worker-zone

Conversation

@leandrolerena

@leandrolerena leandrolerena commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Description

  • Release step: 11 (Phase C)
  • Waits on: core v2.4.0, engine v0.46.0, SDK v0.5.0 and web client v6.1.0; each tag pushes its chart to oci://quay.io/zebbra/charts
  • Merge only after: every final exists
  • Stacked on: main

Existing installs: before upgrading to these pins, hand the engine's tables to neops with the script from the engine's upgrade docs, and set the engine's POSTGRES_PASSWORD to the neops password. From 0.45.0, not 0.45.0-beta.1, the engine connects as POSTGRES_USER (zebbra/neops-workflow-engine#506); without the hand-over it refuses to start and its log names the script; without the password it fails authentication as neops. The README's upgrade section gives the commands.

  • The umbrella's values set NEOPS_WORKER_NETWORK_ZONES: default for its central worker (charts/neops/values.yaml:115), never the neops-worker chart's defaults, so a per-zone release that misses its own setting is refused instead of serving default.
  • charts/neops/Chart.yaml pins core 2.4.0, engine and monitor 0.46.0, worker 0.5.0 and web client 6.1.0, and bumps the umbrella to 0.3.0.
  • The README's The superuser's role section: core 2.4.0's migrate hook gives the superuser an admin role with the workflow grants and write on network-zones (core 05 zebbra/neops-core#2469), re-applied on every install and upgrade; other users need network-zones write in a role of their own; and never start a site's worker with default to work around a missing grant.
  • The README's Going back to engine 0.44.1: let every run finish; scale each zone's worker release and the central worker to 0, since an engine without zones would hand a zone worker any zone's jobs and a 0.5 worker cannot read 0.44's; with the engine stopped, revert from a one-off kubectl run pod on the 0.46 image (the README gives the command), whose down to the first zones migration also reverts 0.45's Migration20261102104651 and, since engine develop's zebbra/neops-workflow-engine#507, Migration20261102113953, whose revert deletes the rows of refused workers (the revert runs in name order, and 0.44.1 applies neither again); set the engine's password back to postgres's (and to neops's again going forward); from the still-running 2.4.0 core, drain the log outbox and migrate core 0204_merge_20261029_logging_develop, since main pins core 2.3.0; upgrade with main's chart, so the central worker goes back to 0.3.0 with engine 0.44.1. Going forward again, restart the central worker and scale the zone releases back up, so every worker re-registers its zones.
  • The README's Network zones section covers one neops-worker release per zone, two workers per zone (beyond the developer-preview engine), escaping commas in --set, creating a zone on the web client's Network zones page before its workers start, and the order for retiring a site.

Look at first

  • charts/neops/values.yaml:115: the setting lives in the umbrella's values only.
  • charts/neops/Chart.lock is gitignored, so the PR carries no lock. Once the charts are on Quay, helm dependency update charts/neops resolves the new pins locally.

Deviations from the design

None.

Testing

Manual lab integration run zones-it, twice on 2026-10-31:

  • round 1, before the review fixes: core 75cf9bf57, engine 029e791e, SDK 725b28b, web a26da57;
  • round 2, after every review fix: core 839a61400, engine 50dc36d9, SDK cc98ac9, web ebf30ad, lab 2636d6a. The final heads differ only by one engine code comment and doc line, and the SDK's unit-tested formatter for list messages.

No lab round exercised this PR: helm was not part of them. Live on KIND (namespace neops-zones-test, since deleted), with the charts packaged locally over zones images: the umbrella installed; core's hook seeded the superuser's role, and the engine answered /workers and /network-zones for it; the README's zone-b release with --set nodeSelector.network-zone=zone-b ran two workers on the labelled node, both online in zone-b; a selector naming a label no node has left its pod Pending. Not verified: helm dependency update and the README's Quay-chart form, which need the charts on Quay; the published images. Since the KIND control-plane node is tainted, the positive scheduling alone doesn't prove the selector; the Pending case does. The README's "Going back to engine 0.44.1" recipe was not rehearsed on a cluster: neither the one-off kubectl run engine down nor migrate core 0204_merge_20261029_logging_develop from the 2.4.0 pod. The lab proved the engine down under compose only.

Network Zones train, in merge order
# PR Branch Step Then
1 neops-workflow-engine, no PR 0 done: the user tagged engine v0.45.0 from develop
2 zebbra/neops-worker-sdk-py#318 neops-worker-sdk-py fix/pin-engine-client-0.45.0 0 merged 2026-11-04 (f7cdf41, client ==0.45.0); SDK v0.4.0 is tagged on it and released (2026-11-06); Claude has merged it into SDK 01–03
3 zebbra/neops-core#2467 neops-core fix/pyasn1-0.6.5 1 merged 2026-11-02
4 zebbra/neops-core#2464 neops-core feature/network-zones-01-model 1 merged
5 zebbra/neops-core#2465 neops-core feature/network-zones-02-run-writes 1 merged
6 zebbra/neops-core#2466 neops-core feature/network-zones-03-search-and-lock 1 merged
7 zebbra/neops-core#2468 neops-core feature/network-zones-04-zone-paging-and-sort 1 merged
8 zebbra/neops-core#2469 neops-core feature/network-zones-05-chart-admin-role 1 merged 2026-11-05; v2.4.0-beta.1 is tagged on its tip (d2b030b19)
9 zebbra/neops-core#2473 neops-core feature/network-zones-06-unlock-write-refused 1 merged 2026-11-05; v2.4.0-beta.2 is tagged on its tip (da4974bb6): the unlock answers WRITE_REFUSED, which engine 04 needs
10 zebbra/neops-workflow-engine#497 neops-workflow-engine feature/network-zones-01-nest-client-2.3.0 2 merged 2026-11-02
11 zebbra/neops-workflow-engine#506 neops-workflow-engine fix/postgres-user 2 merged 2026-11-02, after v0.45.0-beta.1; v0.45.0 is the first release that holds it
12 zebbra/neops-workflow-engine#498 neops-workflow-engine feature/network-zones-02-definitions 2 merged 2026-11-04
13 zebbra/neops-workflow-engine#499 neops-workflow-engine feature/network-zones-03-routing 2 merged 2026-11-04
14 zebbra/neops-workflow-engine#500 neops-workflow-engine feature/network-zones-04-device-writes 2 after core 06, the user merges 04–10 back to back
15 zebbra/neops-workflow-engine#501 neops-workflow-engine feature/network-zones-05-prerelease-versions 2
16 zebbra/neops-workflow-engine#502 neops-workflow-engine feature/network-zones-06-refuse-invalid-pushes 2
17 zebbra/neops-workflow-engine#503 neops-workflow-engine feature/network-zones-07-cms-request-timeout 2
18 zebbra/neops-workflow-engine#504 neops-workflow-engine feature/network-zones-08-never-polled-and-log-context 2
19 zebbra/neops-workflow-engine#505 neops-workflow-engine feature/network-zones-09-monitor-context 2
20 zebbra/neops-workflow-engine#510 neops-workflow-engine feature/network-zones-10-cms-at-lock-and-write-back 2 v0.46.0-beta.3 is tagged on its tip (4c4f2f7c) and published client 0.46.0b3; the engine accepts the SDK's 0.46.0b2
21 zebbra/neops-worker-sdk-py#313 neops-worker-sdk-py feature/network-zones-01-job-meta 3 SDK v0.4.0 is released; the user merges 01–03 back to back
22 zebbra/neops-worker-sdk-py#314 neops-worker-sdk-py feature/network-zones-02-zones 3
23 zebbra/neops-worker-sdk-py#315 neops-worker-sdk-py feature/network-zones-03-strict-compatibility 3 v0.5.0-beta.2 is tagged on its tip (907d443), on client 0.46.0b2
24 zebbra/neops-web-client#183 neops-web-client feature/network-zones-01-angular-client-2.3.0 4 merged 2026-11-02
25 zebbra/neops-web-client#184 neops-web-client feature/network-zones-02-zones 4 merged 2026-11-05; v6.1.0-beta.1 is tagged on its tip (953db30)
26 zebbra/neops-worker-boilerplate#5 neops-worker-boilerplate feature/network-zones-01-sdk-0.4 5 merged 2026-11-03 at its earlier head: develop pins SDK ==0.4.0b1
27 zebbra/neops-worker-boilerplate#6 neops-worker-boilerplate fix/network-zones-sdk-0.5.0b1-pin 5 SDK ==0.5.0b2, uv lock; the user merges it
28 neops-core, no PR 6 after the beta soak, the user tags core v2.4.0
29 not opened yet neops-workflow-engine re-pin PR, not opened yet 7 nest 2.4.0, MINIMUM_CORE_VERSION 2.4.0, generated paged networkZones, then tag engine v0.46.0
30 not opened yet neops-worker-sdk-py re-pin PR, not opened yet 8 client ==0.46.0, then tag SDK v0.5.0
31 not opened yet neops-worker-boilerplate re-pin PR, not opened yet 9 SDK ==0.5.0
32 not opened yet neops-web-client re-pin PR, not opened yet 10 angular 2.4.0 exactly, library peer floor back to >=2.4.0 <3, plain npm install, re-render docs/dashboard-cards/table.md against core v2.4.0, then tag web v6.1.0
33 zebbra/neops-lab#50 neops-lab feature/network-zones-01-scenario 11 once every final exists
34 zebbra/neops-docker-compose#42 neops-docker-compose feature/network-zones-01-central-worker-zone 11 once every final exists
35 zebbra/neops-docker-compose#43 neops-docker-compose feature/network-zones-02-pins 11
36 this PR neops-helm feature/network-zones-01-central-worker-zone 11 once every final exists

Phase A (steps 1–5) ships the zones betas: Claude tags each on the tip of its stack once CI is green, and nothing merges into develop for a beta. Claude keeps every zones branch current by merging develop in. Merges into develop or main, step 0 (closing the 0.45 line from develop before engine 02 or SDK 01 merges) and every final tag are the user's. Phase B (6–10) ships the finals, and Phase C (11) the deployments that pin them.

Checklist:

  • PR label has been set
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings

…mp the umbrella to 0.3.0

Network zones need all four together: core stores the zones, the engine
routes jobs by zone, the monitor ships with the engine, and worker 0.4.0
reads NEOPS_WORKER_NETWORK_ZONES. These charts appear on Quay with the
core v2.4.0, engine v0.45.0 and SDK v0.4.0 tags.
The central worker lists default in NEOPS_WORKER_NETWORK_ZONES in the
umbrella's values, never in the neops-worker chart's defaults, so a
per-zone release that misses its own setting is refused instead of
serving default. The README documents one neops-worker release per zone,
two workers per zone, escaping commas in --set, and creating a zone
before its workers start.
Its Network zones page is where the README tells operators to create a
zone. Like the other pins in the umbrella's 0.3.0, the chart appears on
Quay only with its tag, so this waits on web client v6.1.0 next to core
v2.4.0, engine v0.45.0 and SDK v0.4.0.
Web client 6.1.0 ships the page, so it leads the options; the mutation
and Django admin remain for those without it.
Devices leave the zone while its workers still run, or by an override
step or Django admin once the site is gone; then the zone's worker release
goes, and the zone last. Otherwise the move waits up to an hour for a
worker that never comes.
It admits at most 2 online workers, which the default zone alone fills.
The chart defaults to quay.io/zebbra/neops-worker-sdk at its appVersion,
which a KIND loop built from the working tree does not have; the
umbrella's KIND values use neops-worker-sdk:cutting-edge.
From core 2.4.0, superuser.enabled also seeds the admin role, the
workflow admin profile and write on network-zones, as compose's cms-init
does plus the zone grant. Say so, and warn against starting a site's
worker with default when nobody can create its zone.
…45.0

Engine 0.45.0 connects as POSTGRES_USER, which this chart sets to neops,
while existing installs' tables belong to postgres. Without the hand-over
the engine fails at startup with relation "mikro_orm_migrations" already
exists. The README gives the kubectl exec form that runs the engine's
owner-to-postgres-user.sql in the chart's Postgres pod, and the password
change.

The same commit says how the KIND zone release takes the two image flags.
Engine 0.44 always connects as postgres, so a rollback needs the engine's
POSTGRES_PASSWORD set back to it, after the engine's own down migration.
The tables handed to neops need nothing, as postgres is a superuser.
…d, and take the script from docs.neops.io

The engine repository is private, so the upgrade script and the engine docs
are linked on docs.neops.io.
Engine 0.45.0 and SDK 0.4.0 are released from develop without network
zones, so the charts that carry zones come with the engine v0.46.0 and
SDK v0.5.0 tags.
The POSTGRES_USER fix first ships in engine 0.45.0, not in the beta tagged
before it merged.
…ations

The zones migrations now ship in 0.46, and 0.45 adds one migration of its
own, so a rollback to main's engine pin reverts both, newest first.
An install that jumped from 0.44.1 to 0.46 ran the rename after the zones
migrations, so reverting those from the 0.46 image already undid it; a
second down from the 0.45 image would target a migration no longer
executed.
The down reverts every executed migration whose name sorts at or after
its target, so 0.45's rename goes with the zones migrations whatever the
install's history.
An engine without zones ignores a worker's zones, so a zone's worker would
take any zone's jobs, and a 0.5 worker cannot read 0.44's jobs. Going
forward again, the engine counts workers that registered before it in
default until they restart.
The engine is stopped for the down, so there is no pod to exec into; the
pod takes the 0.46 image, the Quay pull secret, the engine's database
settings and its Secret.
…e the engine to 0.45.0 or later

main pins core 2.3.0, which cannot run on 2.4.0's migrations, so the way
back drains the log outbox and migrates core to 2.3.0's last migration
first. The chart now jumps from engine 0.44.1 to 0.46.0, past 0.45.0.
Engine develop's refused-worker records (zebbra/neops-workflow-engine#507)
add Migration20261102113953; it sorts after the zones migrations too, so
the down reverts it and deletes the refused-worker rows.
…44.1

Give the engine page's SQL a command, and say to restore the neops
password when going forward again.
Once this merges, main pins the zones release, so upgrading from main re-installs it.
The umbrella pairs engine 0.46 with worker 0.5; the gate compares the client the
worker pins, not the worker's own version.
… migrations

rollout status reports a Deployment scaled to 0 as done while its pod still
terminates, so the SQL and the down could overlap a live engine.
Undoing 0206_network_zone deletes every zone and device zone, so zone releases
scaled back up are refused until the zones exist again. Link core's 2.4.0 notes,
which hold the rule, and recreate the zones and move the devices back first.
…uthorizes nothing

The engine checks a worker's zones only at registration, so a running worker
that lists none keeps claiming default jobs. Any pod or host that reaches the
worker routes can register for any zone, so restrict and encrypt the path.
Link the engine's Network Zones page for the rules.
core 2.4.0's bootstrap_admin_role gives the role it seeds the network-zones
write also when other users hold it (zebbra/neops-core#2474); the hook still
skips the workflow profile on its exit 3.
Every holder of the shared role, so under OIDC every user the identity provider
puts in it, may write every zone; the seed re-applies the write on every upgrade.
@leandrolerena
leandrolerena merged commit cc57cce into main Nov 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-new-feature PR introduces new feature/s

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant