Umbrella Helm chart for the NeOps platform plus a local KIND loop.
charts/neops composes the NeOps services as OCI subcharts from
oci://quay.io/zebbra/charts (each service repo owns and publishes its own chart under
chart/<name>/ on every tag) together with Bitnami postgresql, redis,
elasticsearch and minio. values.yaml holds in-cluster wiring and pins the Bitnami
charts' images to bitnamilegacy/* (Bitnami no longer publishes the tags its charts name);
every credential comes from an environment values file. values-kind.yaml is the KIND
environment: all credentials are unsafe, Neops images are the locally built cutting-edge tags, and every
service defaults to http://*.neops.local. Setting KIND_DOMAIN and KIND_SCHEME in
.env (e.g. KIND_DOMAIN=neops.example.com, KIND_SCHEME=https) publishes the release
under both *.${KIND_DOMAIN} and *.neops.local.
Quote numeric values under the workflow engine's env:, e.g. BLACKBOARD_PENDING_JOB_TIMEOUT: "3600000":
Helm renders an unquoted large integer as 3.6e+06, which engine 0.44.1 and later refuse at boot.
cp .env.dist .env # QUAY_USER / QUAY_TOKEN (private images + charts)
make local-deploy # cluster-up → install → mint CMS token → install again
make config-hosts # points the release's *.neops.local hosts at 127.0.0.1Working tree instead of Quay: from the workspace root run make deploy-cutting-edge
(builds every image, kind loads it, packages the sibling charts locally, then runs
make local-deploy here). No registry is involved, so that flow needs no .env — the
credentials above are only for pulling the images and charts from Quay.
The default KIND values publish neops.local and the cms, engine, workflows and
gateway subdomains over HTTP without a tls: section. When KIND_DOMAIN is set in
.env, make local-upgrade and make local-deploy automatically generate overrides so
each service serves both its ${KIND_DOMAIN} host (for access through an external reverse
proxy terminating TLS) and its .neops.local host (for make config-hosts). storage. is
declared alongside them but is not served until neops-storage.enabled is turned on.
The KIND cluster kind is shared with other projects; NeOps installs into namespace
neops and make local-clean only removes that namespace. Every target lives in
.make-scripts/ and is listed in the Makefile.
The engine and gateway authenticate against core with an API key that only exists once
core is running: make cms-token mints it into the gitignored cms_api_key.env, and
make local-upgrade passes it along whenever that file exists. That same step mints a dev
JWT keypair into the gitignored jwt/, hands both PEMs to core with --set-file and the
public one to the engine, which verifies core's tokens with it; a real environment supplies
its own PEMs the same way.
With neops-core.superuser.enabled (on in values-kind.yaml), core's migrate hook creates
the Django superuser. From core 2.4.0 the same hook gives it the Neops role
neops-core.superuser.role (default admin) on every install and upgrade:
- write on the scope
Globaland on thenetwork-zonespermission element, so the superuser can create zones (core'smanage.py bootstrap_admin_role); - the workflow
adminprofile (core'smanage.py grant_workflow_permissions).
neops-docker-compose's cms-init runs the same two commands. Without the role, the superuser
can log in and then read and write nothing. The seed re-applies every grant on every install and
upgrade, so a grant removed from that role by hand comes back. If other users already hold a role
of this name, the seed widens it only by the network-zones write and warns, skipping the scope
grant and the workflow profile. Every holder of that role, under OIDC whoever the identity provider
puts in it, then may write every zone (zones are not per tenant). Set neops-core.superuser.role to
a role only the superuser holds; the seed re-applies the write on every upgrade, so remove it from
the shared role by hand afterwards if it should not have it.
Other users who create zones need write on network-zones in a role of their own: grant it in
Django admin under Role permission elements. Never work around a missing grant by starting a
site's workers with default: a site's worker that serves default reaches the default
network's addresses, with their devices' credentials, from inside the site's network.
A worker claims only the jobs of the network zones it lists in NEOPS_WORKER_NETWORK_ZONES;
the engine's Network Zones page has the rules.
Zones need core 2.4.0, engine 0.46.0 and worker 0.5.0 or later; older workers ignore the
variable. The umbrella's own worker is the central one: values.yaml sets
neops-worker.env.NEOPS_WORKER_NETWORK_ZONES: default. Keep it out of the neops-worker
chart's own defaults and out of any values file a remote site copies, so a zone release that
misses its own setting is refused rather than serving default.
Each other zone gets one more release of the neops-worker chart, pinned to nodes inside that zone's network. Install it at the umbrella's worker version, since on 0.x the engine accepts only workers whose engine client is of its own minor, each worker release pins one client, and every zone's release upgrades with the umbrella:
kubectl label node <node-in-network-b> network-zone=zone-b
WORKER_VERSION=$(helm dependency list charts/neops | awk '$1 == "neops-worker" {print $2}')
helm install neops-worker-b oci://quay.io/zebbra/charts/neops-worker --namespace neops \
--version "$WORKER_VERSION" \
--set replicaCount=2 \
--set 'imagePullSecrets[0].name=neops-quay-registry' \
--set nodeSelector.network-zone=zone-b \
--set env.URL_BLACKBOARD=http://neops-neops-workflow-engine:3030 \
--set env.WORKER_NAME=neops-worker-b \
--set env.NEOPS_WORKER_NETWORK_ZONES=zone-bThe namespace and the neops-neops-workflow-engine host follow from the umbrella's release
name and namespace, both neops here; adjust them if yours differ.
On KIND, run the zone's release on the worker image the umbrella's KIND values use, the
locally built one, instead of the Quay image the chart defaults to. Add these two flags to the
zone release's helm install above:
--set image.repository=neops-worker-sdk \
--set image.tag=cutting-edge- Several zones.
--setsplits on commas, so escape them and single-quote the argument, or the shell drops the backslash:--set 'env.NEOPS_WORKER_NETWORK_ZONES=default\,zone-b'. List several zones on one worker only if their addresses don't overlap from that node. - Two workers per zone. A rollback goes back to the forward job's zone, and the worker it
needs is often the one that just died. A zone with one worker also holds up its runs during
every restart of that worker. Hence
replicaCount=2above; give the central worker the same with--set neops-worker.replicaCount=2on the umbrella. The developer-preview engine caps an installation at 2 workers, so a zones setup with more workers needs the licensed engine, the image the neops-workflow-engine chart defaults to. - Order. Create the zone first, on the web client's Network zones page, which needs web
client 6.1.0 and write on the
network-zonespermission element. ThenetworkZoneUpsertmutation needs the same write, and Django admin works too. No worker needs to be up for that. A zone's workers are refused at registration while the zone doesn't exist. Once any zone besidesdefaultexists, a worker that lists no zones is refused at its next start, but only then: one already running keeps claimingdefaultjobs. After creating the first zone, setNEOPS_WORKER_NETWORK_ZONESon every worker that lists none and restart it; the Monitor App's Workers page flags the workers that registered no zone list. - Retiring a site. First move the zone's devices to another zone with
device_update_device, or delete them. If the site is already unreachable, give that stepnetworkZone: { name: <a live zone>, override: true }, or delete the devices in Django admin, which needs no worker; otherwise the step waits up to an hour for a worker that never comes and holds other runs meanwhile. Thenhelm uninstallthe zone's worker release, and last delete the zone, on the Network zones page or in Django admin, once no workflow run names it. - Networking. A zone's management ranges must not overlap the cluster's pod and service
ranges as seen from that zone's nodes. Giving a remote site's nodes a network path to the
engine's worker routes, which are unauthenticated, is the operator's job; this chart adds none.
A zone routes jobs and authorizes nothing: every pod or host that reaches those routes can
register for any zone,
defaultincluded, and receive its devices' passwords. Restrict who reaches the engine, for example with a NetworkPolicy, and encrypt the path between sites, with the CNI's encryption or a VPN.
Since core chart 2.3.0, core's post-upgrade migrate Job rebuilds the search indices this
core or this Elasticsearch version did not build. The first upgrade to such a core rebuilds all of
them, and search is down until the Job finishes. Helm's default --timeout (5m) is shorter than the
Job's default deadline (neops-core.migrate.activeDeadlineSeconds, 900 s), so on a real
inventory raise both, the timeout a few minutes above the deadline, and leave --atomic off for
this upgrade, for example
helm upgrade ... --timeout 35m --set neops-core.migrate.activeDeadlineSeconds=1800. Core's
release notes ("The rebuild runs on upgrade" in
docs/releases/v2.3.0.md)
cover sizing and the Job's log; going back is under "Going back to an
earlier release" there.
Engines before 0.45.0, 0.45.0-beta.1 included, ignore POSTGRES_USER and connect as postgres,
so an existing install's engine tables are owned by postgres. From 0.45.0 the engine connects as
neops-workflow-engine.env.POSTGRES_USER, which this chart sets to neops. Before
helm upgrade, on every install that ran an older engine:
-
Set
neops-workflow-engine.secrets.POSTGRES_PASSWORDto theneopspassword,postgresql.auth.password, notpostgresql.auth.postgresPassword. -
Hand the engine's tables to
neops: save the script from the engine's upgrade docs asowner-to-postgres-user.sqland run it aspostgresin theneops-workflowdatabase:kubectl -n neops exec -i neops-postgresql-0 -c postgresql -- sh -c \ 'PGPASSWORD="$(cat /opt/bitnami/postgresql/secrets/postgres-password)" psql -h 127.0.0.1 -U postgres -d neops-workflow -v ON_ERROR_STOP=1 -v role=neops -f -' \ < owner-to-postgres-user.sql
The pod is
<release>-postgresql-0in the umbrella's namespace. The old engine can keep running meanwhile, and running the script twice changes nothing.
Skip it and the upgraded engine refuses to start: its log says that it connects as neops while
postgres owns its tables, and names the script. It changes nothing: run the script, then restart
the engine. Skip step 1 instead, and it stops earlier, with
password authentication failed for user "neops". The engine's
upgrade docs have the rest.
Going back to the engine this chart pinned before, 0.44.1, crosses two engine minors. An engine without zones ignores a worker's zones, so a zone's worker left polling it takes any zone's jobs, and a 0.5 worker cannot read the jobs 0.44 hands out.
-
Let every run finish, and start no new ones. A job claimed by a worker you stop next fails as worker-offline after about 6 minutes, and its rollback finds no worker in that zone.
-
Scale every zone's worker release to 0, one command per zone release, and the umbrella's central worker too: it is a 0.5 worker, which cannot read the jobs 0.44 hands out. Step 6 brings it back at 0.3.0.
kubectl -n neops scale deployment neops-worker-b --replicas=0 kubectl -n neops scale deployment neops-neops-worker --replicas=0
-
Revert the 0.46 migrations from the 0.46 image, as the engine's Going back to 0.45 describes, with this chart's engine settings (
POSTGRES_USERneopsand its password), which own the tables since the upgrade. This also reverts 0.45's two migrations,Migration20261102104651andMigration20261102113953, whose revert deletes the rows of refused workers: the revert runs in name order, and both names sort after the zones migrations. 0.44.1 applies neither again.The engine must be stopped, so there is no pod to
kubectl execinto. Stop it, save that page's SQL asgoing-back-0.45.sqland run it, then run the down in a one-off pod from the 0.46 image, with the engine's pull secret, database settings and Secret (<release>-neops-workflow-engine, which holdsPOSTGRES_PASSWORD):kubectl -n neops scale deployment neops-neops-workflow-engine --replicas=0 kubectl -n neops wait --for=delete pod --timeout=5m \ -l app.kubernetes.io/name=neops-workflow-engine,app.kubernetes.io/instance=neops kubectl -n neops exec -i neops-postgresql-0 -c postgresql -- sh -c \ 'PGPASSWORD="$(cat /opt/bitnami/postgresql/secrets/postgres-password)" psql -h 127.0.0.1 -U postgres -d neops-workflow -v ON_ERROR_STOP=1 -f -' \ < going-back-0.45.sql kubectl -n neops run engine-migrate-down --rm -i --restart=Never \ --image=quay.io/zebbra/neops-workflow-engine:0.46.0 \ --env=POSTGRES_HOST=neops-postgresql --env=POSTGRES_PORT=5432 --env=POSTGRES_USER=neops \ --override-type=strategic --overrides='{"apiVersion": "v1", "spec": { "imagePullSecrets": [{"name": "neops-quay-registry"}], "containers": [{"name": "engine-migrate-down", "envFrom": [{"secretRef": {"name": "neops-neops-workflow-engine"}}]}]}}' \ -- node -e 'const { MikroORM } = require("@mikro-orm/core"); const config = require("./dist/neops-workflow-engine/src/mikro-orm.config").default; MikroORM.init(config).then(async orm => { await orm.getMigrator().down({ to: "Migration20261030184705" }); await orm.close(true); });'
-
Set
neops-workflow-engine.secrets.POSTGRES_PASSWORDback to thepostgrespassword before step 6, since 0.44 always connects aspostgres:kubectl -n neops get secret neops-postgresql -o jsonpath='{.data.postgres-password}' | base64 -d
The secret is
<release>-postgresql. The tables handed toneopsneed nothing:postgresis a superuser and reaches them as they are. -
Take core's database back to 2.3.0 while the 2.4.0 core still runs, since step 6's chart pins core 2.3.0, which cannot run on 2.4.0's migrations. Stop core's worker and beat first, as step 1 of the Helm way back in core's 2.3.0 notes does. Then drain the log outbox, as "Going back" under "Log lines survive a log store outage" in core's 2.4.0 notes says: run
log_store_statusuntilrowsis 0. Last, undo0206_network_zone, which deletes every zone and every device's zone, as Going back below 2.4.0 in the same notes warns (note which devices are in which zone first), and0205_log_outbox:kubectl -n neops exec deploy/neops-neops-core -- /docker-entrypoint.sh ./manage.py log_store_status kubectl -n neops exec deploy/neops-neops-core -- /docker-entrypoint.sh ./manage.py migrate core 0204_merge_20261029_logging_develop
-
Upgrade the umbrella with this repo's chart at commit
6f93b5f(umbrella 0.2.1), which pins core 2.3.0, engine 0.44.1, worker 0.3.0 and web client 6.0.1 together, so the central worker goes back with the engine. This repo has no tags andmainpins the zones release, so check out that commit and upgrade fromcharts/neopswith the values you installed with. That commit's README has no going-back section, so keep this one open on GitHub:git switch --detach 6f93b5f helm dependency update charts/neops helm upgrade neops charts/neops --namespace neops -f <your values file> # plus any --set-file you installed with
Going forward again, switch this checkout back to the chart you go forward to (git switch main,
then helm dependency update charts/neops), and set neops-workflow-engine.secrets.POSTGRES_PASSWORD
back to the neops password first (undoing step 4), or the 0.46 engine fails with
password authentication failed for user "neops". The zones are gone and every device is in
default, and the engine counts every worker that registered before the upgrade in default.
After the umbrella upgrade, restart the central worker, recreate the zones on the web client's
Network zones page, and move their devices back with runs, as core's
Going back below 2.4.0
says. Then scale each zone release back up; a zone's workers are refused while their zone does not
exist. A zone release already on the umbrella's worker version needs no upgrade; otherwise upgrade
it with --reuse-values, since a plain helm upgrade resets its NEOPS_WORKER_NETWORK_ZONES and
the worker is refused at start:
kubectl -n neops rollout restart deployment neops-neops-worker
kubectl -n neops scale deployment neops-worker-b --replicas=2The deployment names follow from the release names used above: <release>-neops-worker for the
umbrella's worker, and the release name for a zone's.