Skip to content

Host mode: Traefik, direct ports, durable data, bare compose - #29

Open
obisimon wants to merge 20 commits into
developfrom
feature/host-mode
Open

obisimon wants to merge 20 commits into
developfrom
feature/host-mode

Conversation

@obisimon

Copy link
Copy Markdown

Summary

  • Add host mode behind bundled Traefik (path prefixes /cms, /engine, /monitor, /djstatic, dashboard at /traefik) with cms-proxy for Django redirects and cross-origin monitor iframe auth (:8443 / :3031).
  • Add host-direct mode (laptop-style published ports on LAB_HOST, no reverse proxy) and parallel host-* / host-direct-* make targets that leave local-* unchanged.
  • Persist host Postgres + CMS /tmp under ./data/ (docker-compose.host-data.yml); Elasticsearch/Redis stay ephemeral. make host-env-init writes COMPOSE_FILE into .env so bare docker compose up/down works on the host.

Test plan

  • make test / make py39-check / make shell-syntax
  • On a host: set LAB_HOST + LAB_SCHEME, run make host-env-init, confirm .env has COMPOSE_FILE, then docker compose ps / up -d / down without make
  • Confirm ./data/postgres and ./data/cms_tmp are created and survive compose down
  • HTTPS: web client at https://LAB_HOST/, accept cert once on https://LAB_HOST:8443/, open monitor via web-client menu (iframe auth)
  • make host-check-cms keeps /cms on admin redirects; /djstatic reaches CMS
  • Optional: make host-env-down then make host-direct-env-init and hit :8080 / :8001 / :3030 / :3031

Simon Obi added 18 commits September 10, 2026 23:25
Parallel host-* targets keep local-* unchanged while publishing the lab on a single LAB_HOST; rename the ES volume to avoid host collisions.
Resolve conflicts by keeping host-mode Traefik targets alongside spie authz
(lab_token, lab-grant, personas docs) and aligning monitor/CORS for LAB_HOST.
Host modes bind-mount Postgres and CMS /tmp under ./data, and host-env-init
writes COMPOSE_FILE so docker compose up/down works without make.
@leandrolerena

leandrolerena commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Merge recommendation: ⛔ don't merge — recommend closing

Maintainer decision: the lab will not have its own host mode. Host deployments of the control plane go through https://github.com/zebbra/neops-docker-compose (./neops up with compose.traefik.yaml / compose.tls-acme.yaml, durable data, COMPOSE_FILE in .env). So I recommend closing this PR rather than rebasing it. If something here is still needed (e.g. the cms-proxy handling of Django prefix redirects, or the cross-origin monitor iframe auth), it belongs in neops-docker-compose as its own PR. The review below is kept for the record.

This branch is 56 commits behind develop and conflicts in 9 files: Makefile, README.md, .env.example, pyproject.toml, tools/import_host_scripts.py, tests/test_host_invariants.py, mkdocs_custom.yml, docs/20-operations/index.md, plus cms/oidc-config.host.json.template, which lands in a directory that develop moved to scenarios/_base/cms/. Most of these conflicts are not mechanical. The new host-* and host-direct-* targets are copies of the local-env-init / local-lab-up recipes as they were in September. Since then, develop has changed those recipes: scenario overlays (scenario-resolve, generated/<scenario>/scenario/), engine-token minting for lab_bootstrap, DISCOVER_WORKFLOW 1.3.0, the CMS image override, and more. A merge that only resolves the text would leave four stale copies of the lab bring-up that fail at runtime: no resolved scenario, no engine token, wrong discovery version. So I have not pushed a merge commit. The fix is a redesign of how host mode reuses the laptop recipes, which is the author's call. A plain conflict resolution won't do it. The branch also fails its own make lint as it stands.

Blocking:

  • Conflicts with develop in the 9 files above. Merge develop in and re-run make check.
  • Makefile:261, :300, :375, :405: host-env-init, host-lab-up, host-direct-env-init and host-direct-lab-up duplicate the local-* recipes: API-key minting three times, the lab bring-up three times. Host mode is a different COMPOSE_FILE and a different OIDC render, so drive the existing local-* recipes with an overlay variable (e.g. LAB_MODE=host|host-direct selecting COMPOSE_FILE). That way host mode picks up every future change to the bring-up, and none of this duplicate code has to be maintained by hand. Makefile:325 and :430 already show the drift: they hardcode simple_lab_discovery:1.2.0, while develop uses $(DISCOVER_WORKFLOW), currently 1.3.0.
  • make lint fails at the head: ruff format --check wants to reformat gen_host_oidc:54, and ruff check reports 3 errors: gen_compose_env:32 (PIE810), gen_compose_env:47 (PLR2004), tests/test_host_invariants.py:57 (PLC0415).
  • docker-compose.traefik.yml:74 / docker-compose.host-direct.yml:22 mount ./cms/oidc-config.host.json. On develop, CMS assets come from the resolved scenario tree, so gen_host_oidc and both overlays need to follow the overlay rule in AGENTS.md ("no consumer implements the fallback itself").

Notes:

  • Host mode is intended for a reachable host (it supports Let's Encrypt), yet it serves the Traefik dashboard without auth (docker-compose.traefik.yml:34, /traefik), the CMS admin with the lab's default neops / neops (Makefile:252), and direct ports :8001/:3030/:3031/:8080 on all interfaces. At minimum, document that this is for a trusted network only. Better: put the dashboard behind basic auth, and bind the debug ports to 127.0.0.1 unless explicitly asked.
  • The monitor runs npm install && npm run dev (a Vite dev server, NODE_ENV=development) at container start (docker-compose.traefik.yml:102). The engine now publishes a containerized monitor image (quay.io/zebbra/neops-monitor-app). Using it removes the runtime npm fetch and the __VITE_ADDITIONAL_SERVER_ALLOWED_HOSTS workaround.
  • The commit history (fix, monitor auth ×2, fix override, …) would benefit from a squash or a cleanup before merge.
  • docs/20-operations/50-host-proxy.md collides in number with develop's 50-kubernetes-lab.md. The nav is explicit, so this is cosmetic.

Verified locally: worktree at the PR head 7b52bb4.

  • make lint → fails: ruff format (1 file) and ruff check (3 errors), listed above.
  • make typeCheck test py39-check shell-syntax → pyrefly 0 errors, 51 passed, 3.9 import check and bash parse green.
  • git merge --no-commit origin/develop → conflicts in the 9 files listed (merge aborted, nothing pushed).

Not verified: host mode itself, on a host with LAB_HOST/TLS. Bringing up a Traefik stack on :80/:443 here would clash with the stacks other sessions are running.

# Conflicts:
#	.env.example
#	Makefile
#	README.md
#	docs/10-concepts/10-architecture.md
#	docs/20-operations/20-images.md
#	docs/20-operations/index.md
#	mkdocs_custom.yml
#	pyproject.toml
#	scenarios/_base/cms/oidc-config.host.json.template
#	tools/import_host_scripts.py
Since the merge of develop, docker-compose.yml and docker-compose.worker.yml
refuse to interpolate without SCENARIO and the LAB_* ports, networks and
subnets, so the bare `docker compose` that host mode promises failed with
"required variable ... is missing a value". gen_compose_env now copies every
${NAME:?} the selected compose files name from make's environment into .env,
and refuses to run when one is unset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants