Repository navigation
[Feature]: SPCS service specs and Makefile for the Snowflake trust (trust/deploy/snowflake/) - #1600
Open
garciadias wants to merge 23 commits into
Open
garciadias wants to merge 23 commits into
garciadias wants to merge 23 commits into
Conversation
…ake trust Adds trust/deploy/snowflake/, the Snowflake (Snowpark Container Services) shape of a FLIP trust node, beside helm/ and the compose files. Infrastructure stays with the Terraform root trust/deploy/terraform/snowflake (#1581); this tree owns images, secrets, specifications and the services. One specification template per service in the naming contract, rendered from the trust's kit file: trust-api, data-access-api, imaging-api, fl-client-net-<N>, xnat-web, xnat-db, xnat-nginx, orthanc and the dcm2niix job service. No endpoint is public; the only inbound ports are the two DICOM TCP endpoints (orthanc 4242, xnat-web XNAT_PORT) that the DQR return leg needs. Credentials reach containers only as Snowflake SECRET objects, mapped per the contract: the fl-client maps the trust-internal key alone and no AES key, so researcher training code cannot pass the cohort-admin gate, and it mounts only its own net-<N> images prefix and a read-only FL kit. Block volumes for xnat-db and orthanc, stage volumes for every shared tree. Inter-service URLs are derived from SPCS DNS rather than copied from compose aliases. sync_snowflake_kit.py writes the SECRET objects from the kit and pushes the FL kit and governance document to their stages. Values go to snow sql on stdin, never as arguments, and never into output; CREATE SECRET IF NOT EXISTS + ALTER SECRET keeps the grants a CREATE OR REPLACE would drop. mirror_images.py copies GHCR images into the account repository as linux/amd64 and records digests the renderer pins. The Makefile adds render, mirror-images, sync-kit, up, down, suspend, status and a placeholder selftest (real assertions in #1585), with PROD mapped through deploy/env_mode.mk and DRY_RUN=true printing the SQL instead of running it, which is what makes the deploy path testable with no account. 51 pytest guards over rendered specs, the deploy SQL and a fake snow binary, run by the new path-gated workflow test_snowflake_specs.yml. Open spikes #1577, #1578 and #1579 may change details; the assumptions are tabulated in trust/deploy/snowflake/README.md. Refs #1582, #1589 Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
github-actions
Bot
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…utput; allowlist detect-secrets false positives (#1582) Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…(CodeQL py/clear-text-logging) (#1582) Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…euristic (#1582) Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
garciadias
marked this pull request as ready for review
October 20, 2026 19:39
…ired admin role (#1582) FLIP_<CODE>_ADMIN is retired by the two-tier decision in epic #1589: the account's security objects belong to the break-glass Terraform root, and everything this directory does now runs as the least-privilege workload role. - naming.py drops the admin_role property. - sync-kit creates the secrets as FLIP_<CODE>_DEPLOY, which owns them, and grants READ to FLIP_<CODE>_SVC. An owner grants on its own objects without MANAGE GRANTS, which the deploy role deliberately does not hold. - The rendered manifest carries the role, and the deploy SQL (up, down, suspend) opens with USE ROLE FLIP_<CODE>_DEPLOY, so services are created and owned by the same role that owns everything they reference. - README gains a Roles section: the four account privileges the deploy role holds, the ownership model, and that FLIP_<CODE>_HUB_EAI is deliberately not attached here (tracked by #1586). New guards: every deploy statement runs as the deploy role and names no admin role; the secrets are owned by deploy and read by svc; neither the deploy SQL nor any rendered spec references the hub EAI. The CodeQL-safe structure in sync_snowflake_kit.py is unchanged: the printed SQL is still built from placeholders by a function that never receives kit values. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
15 of 18 tasks
…; never emit STOP ALL (#1582) The live XNAT spike (#1577) showed snow sql rewriting '&&' to '&' in a service spec: by default it resolves its own &name / &{name} syntax before sending. sync-kit and every make target pipe SQL through it, so a secret value holding '&' would have been stored silently altered. Pass --enable-templating NONE on both paths and test that sync-kit does. The same spike showed ALTER COMPUTE POOL ... STOP ALL deletes every service in the pool, block volumes included, past snapshotOnDelete. The tooling never used it; a guard now keeps it that way. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com> # Conflicts: # .gitignore # AGENTS.md
…ERVICE only for the deploy (#1578) A service's OAuth token carries its owner's role and SPCS refuses GRANT OWNERSHIP ON SERVICE, so services created by the deploy role gave every container, the fl-client among them, a token able to drop the trust database. make up now runs spcs_sql.py apply-up: grant CREATE SERVICE on APP to FLIP_<CODE>_SVC, CREATE OR ALTER each service as that role, revoke in a finally. Specs are rendered before the grant, so a broken one grants nothing. Tests pin the order, the revoke on failure and on Ctrl-C (proven to fail with the finally removed), and that make up uses the runner. The dcm2niix job is now started by the Terraform-owned APP.CONVERT_SCAN; the README role table and account privilege list are brought up to date. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…FLIP#1479) kitfile.py reads HUB, and every usage line, error hint and README example says HUB=<env>. PROD now fails at parse time in deploy/env_mode.mk. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
7 of 10 tasks
…pushes local builds (#1582) The node pulled fl-client-nvflare / fl-client-flower, repositories nobody publishes: the FL workflows push flare-fl-client and flower-supernode, the names deploy/fl_backend.mk carries. naming.fl_client_image() is now the one map, used by the renderer and the mirror, and a test pins it to fl_backend.mk. Found on the live test account. mirror-images MIRROR_PULL=false pushes the local daemon's image as is, so an unpublished change (a local build tagged with its GHCR name) can be tested on Snowflake; it refuses an image that is not built. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
FLARE: the client chmods /app/startup and writes /app/local, but the spec
mounted the kit read-only at /app/fl-kit. It now gets local volumes at
/app/{startup,local,transfer}, seeded from the read-only kit stage before
the image's own entrypoint runs.
Flower: the stage volume is FL_KIT/net-N itself, so the kit sits at the
mount root; the args named /app/fl-kit/net-N/... and pointed at nothing.
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
The registry token is short-lived, so a mirror that relied on an earlier docker login failed at its first push (found on the live test account). snow spcs image-registry login hands the token to docker directly. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…er existing (#1582) The live account rejected the first make up with "syntax error ... unexpected 'OR'". apply-up now reads SHOW SERVICES first: a new service is CREATEd, an existing one gets ALTER SERVICE ... FROM SPECIFICATION (an in-place upgrade that keeps its block volumes) plus SET for the settings CREATE would have given it. Never OR REPLACE. The same run proved the finally: the failed deploy still revoked CREATE SERVICE from the services role. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…session (#1582) The live account rejected data-access-api's spec: "'path' overlaps a location reserved by Snowflake". SPCS mounts the OAuth token at /snowflake/session/token by default, which is where every reader looks, so the setting goes; a guard keeps every rendered spec off that path. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Snowflake refuses MIN/MAX_INSTANCES on a service with a block volume as scaling, even to the same value (xnat-db on the live account). Instance counts are fixed at CREATE. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
… require; no xnat-nginx on SPCS (#1582) Found on the live account, where all three crash-looped: - xnat-db's guard needs the superuser password (POSTGRES_PASSWORD, from the kit's XNAT_DATASOURCE_ADMIN_PASSWORD) and the xnat role's (XNAT_DATASOURCE_PASSWORD) as two different values; the spec passed one. - Orthanc refuses to start without ORTHANC__REGISTERED_USERS, a JSON map. A spec injects a secret whole, so sync-kit composes the map as its own secret. - xnat-nginx proxies to Compose host names and binds port 80 as non-root, which SPCS does not allow; nothing in the trust calls it. Dropped. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
MIRROR_IMAGES=<subset> rewrote images.lock.json with only that subset, so the next render silently fell back to tags for everything else. The lock now merges onto the previous one for the same trust, env and registry. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
POSTGRES_USER=xnat made the stock entrypoint create xnat as the superuser with the admin password, so XNAT's datasource password was refused on the live account. Same rule the Helm chart already follows. Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
…s-specs Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com> # Conflicts: # AGENTS.md # trust/deploy/README.md
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Compose sets ORTHANC__DICOM_MODALITIES inline; the SPCS spec did not, so every C-STORE to XNAT was a 404 "No modality with symbolic name: XNAT" (live account). Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Orthanc rejects a modality host of the full 58-character SPCS name as too long. Every trust service is in the APP schema and the container's DNS search list holds app.<db>.snowflakecomputing.internal, so the bare name resolves to the same address (both checked on the live account). Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
trust/deploy/snowflake/, the Snowpark Container Services node shape for epic #1589: one SPCS service per container, rendered from the trust's kit file. Object names come fromnaming.py, the same contract as the Terraform root (#1581).public: false; internal TCP endpoints for DICOM; inter-service URLs derived from SPCS DNS.snowflakeSecretonly (fl-client gets no AES key); fl-client mounts only its net-N image prefix and the FL kit; block volumes only for xnat-db and Orthanc.sync_snowflake_kit.py: kit to SECRET objects viasnow sql --stdin(never argv;CREATE IF NOT EXISTS+ALTER SECRET, keeping grants), FL kit and governance doc to their stages.mirror_images.py: GHCR to the account repository, linux/amd64, digests locked.CREATE OR ALTER SERVICE, neverOR REPLACE, which would drop block volumes), down, suspend (services and pools), status, selftest;DRY_RUN=trueprints SQL.test_snowflake_specs.yml; README with the assumptions pending spikes [Spike]: Run XNAT on Snowpark Container Services without Docker #1577/[Spike]: dcm2niix as an SPCS job service triggered by imaging-api #1578/[Spike]: SPCS runtime compatibility of the trust images #1579.Linked Issues
Fixes #1582
Checklist
Type of Change
make -C docs/ docs.Testing
make -C trust/deploy/snowflake test: 51 passed.scripts/tests/test_pr_paths_changed.py: PASS=16 FAIL=0.ruff@0.14.7 checkandformat --checkclean.make render,up DRY_RUN=true,suspend DRY_RUN=true,sync-kit DRY_RUN=trueagainst a throwaway kit produce the expected SQL.Additional Notes
Not run against a live account. Details depending on the XNAT, dcm2niix and runtime spikes are marked as assumptions in the README. Known gap for #1586: the hub EAI is not yet attached to trust-api / fl-client.