Repository navigation
feat: stack snapshots for fast dev iteration - #18
Merged
Merged
Conversation
alanshaw
approved these changes
Apr 22, 2026
alanshaw
left a comment
alanshaw
left a comment
Member
There was a problem hiding this comment.
I think in general I'm ok with this. As discussed, I'd ideally like the snapshots to live outside this repo and to be able to pull them in as requested but I understand that is a bunch more work and I can live with the current setup for the time being.
| // Incompatible with WithSnapshot, WithPiriCount, WithPiriNodes. The | ||
| // SMELT_TEST_NO_SNAPSHOT env-var skip pattern applies the same as with | ||
| // WithSnapshot. | ||
| func WithEmbeddedSnapshot(name string) Option { |
Adds `smelt snapshot save/load/list/rm` plus `make up SNAPSHOT=<name-or-path>` so developers can capture a running stack (chain state, volumes, keys, proofs, manifest) and restore it later to skip the ~45s cold-boot cost of contract deploy + piri registration. Restored boots reach the same state in ~10s. Core mechanics: - On save, the stack is stopped gracefully. The blockchain container's SIGTERM trap dumps anvil state to generated/snapshot-scratch/; smelt then archives every named docker volume via a busybox tar helper, copies keys/proofs/smelt.yml, and writes a manifest.json descriptor. - On load, the snapshot's smelt.yml becomes a *session manifest* at generated/snapshot-scratch/smelt.yml. While present, smelt generate and smelt snapshot save read from it; project-root smelt.yml is never modified. make clean / make nuke end the session. - Chain state bind-mount now points at generated/snapshot-scratch/ so make down's SIGTERM dump is what make up reads next — no more revert-to-baseline on restart. Portability / shareability: - compose.yml pins `name: smelt` so docker volume names are stable across checkouts regardless of directory basename. - Snapshot descriptor captures per-service image tag AND content digest via `docker image inspect`, so load reports tag drift (teammate's .env differs) and digest drift at same tag (rolling tag re-pulled) as separate warnings. - Makefile's new check-docker target fails early on engine <25 (start_interval and top-level `name:` require 25+). Required adjacent changes: - systems/common/compose.yml: dynamodb-local gets -dbPath + a volume so delegator's allow-list and upload's provider registry survive restarts (were previously in-memory only). - systems/upload/post_start.sh: idempotent — tolerates "already registered" from sprue when booting a snapshot that captured upload's provider registry. - Healthchecks across all services gain start_interval: 1s so service_healthy dependencies resolve ~1s after readiness instead of waiting for the next 5-10s interval tick. - systems/blockchain/compose.yml: healthcheck switched to `test -f /tmp/ready` sentinel (set by the entrypoint after contracts deploy and mockrpc is up) — more accurate than a port probe, and properly distinguishes init-mode-deploy-in-progress from ready. Docs: - New docs/SNAPSHOTS.md covers everything above end-to-end. - docs/GETTING_STARTED.md updated: docker engine 25+ prerequisite, Linux/macOS only, correct upload command flow (space generate + source add + upload), piri-N naming, make clean/nuke semantics. - docs/TROUBLESHOOTING.md: new Prerequisites Failing section, piri → piri-0 in examples, cross-ref SNAPSHOTS.md. - docs/ARCHITECTURE.md: describes filecoin-services MODE=init|load and how smelt wires the SIGTERM-dump resume loop. - docs/EXTENDING.md: healthcheck template gains start_interval. - README.md + CLAUDE.md updated with snapshot commands and semantics.
Previously every service published on a port identical to the default of some common dev tool (3000, 5432, 6379, 8000, 8080, 8545, 9000, ...), causing collisions on any machine that also runs those tools. Moves every host-side port binding into a dedicated 15XXX block, sized and positioned to avoid both common dev-tool defaults and the Linux ephemeral port range. Container-internal ports and in-network DNS URLs (http://indexer:80, ws://blockchain:8545) are unchanged. Ports: blockchain 15000, dynamodb 15010, redis 15020, signing-service 15030, delegator 15040, indexer 15050, upload/sprue 15060, minio 15070/15071, piri-minio 15072/15073, smtp/smtp4dev 15080/15081, ipni 15090/15091/15092, piri-N 15100+N. Does not address parallel SDK test collisions (issue #5) — two concurrent stacks still collide on the same 15XXX ports. That needs dynamic/ephemeral host ports or an SDK-only ports override and is a separate change.
* feat: ephemeral host ports for SDK stack tests (#5) Two `pkg/stack.NewStack` calls in parallel previously collided on fixed host ports (8545, 8080, 4000+N, ...). This makes every SDK-provisioned stack use ephemeral host ports so tests can run concurrently. Changes in pkg/stack, applied to the extracted compose tree per test: - Rewrite every `- "HOST:CONTAINER"` ports entry to `- "CONTAINER"` so Docker assigns a random host port. `PiriEndpointN`/`EmailEndpoint` already use `MappedPort` and pick up the ephemeral port automatically. - Drop `external: true` from the root compose's storacha-network declaration so each stack gets a project-scoped network (compose namespaces it by project name). Parallel stacks otherwise share the one bridge and collide on service DNS names. - Point sprue's `public_url` at `http://upload:80` so validation emails embed an in-network URL instead of a now-random host:port. With the sprue URL in-network, the host can't click validation links directly. Adds `ExecDoer` in pkg/clients/guppy that issues HTTP requests via curl inside a stack container. `SMTP4DevLoginValidator` gains an optional `Clicker` for the validation POST (defaults to `http.DefaultClient`); `ContainerClient` wires in `ExecDoer{guppy}` so the default in-container flow keeps working. Closes #5. * Addresses @alanshaw's review on #21 (with the direction @Peeja suggested in the follow-up xref to #20). Every host-side port mapping, the root network's `external:` flag, and sprue's `public_url` move from hand-maintained fixed values (previously rewritten at test time by pkg/stack/ports.go) to `${VAR:-default}` interpolation in the compose files. `make up` leaves the vars unset and gets the fixed 15XXX ports + external storacha-network. pkg/stack's SDK test path passes an env map to composeStack.WithEnv, flipping every port to ephemeral (Docker picks the host port, MappedPort resolves it), SMELT_NETWORK_EXTERNAL=false (each test gets a project-scoped network), and SPRUE_SERVER_PUBLIC_URL=http://upload:80 (validation emails embed an in-network URL the ExecDoer-based clicker can reach). - pkg/stack/ports.go + ports_test.go: deleted - pkg/stack/testmode_env.go: new, defines the SMELT_* env map - pkg/generate/compose.go: piri-N, piri-postgres, piri-minio all emit the interpolated form - systems/*/compose.yml, compose.yml: every port + the network external flag wrapped; upload.compose.yml adds \`- SPRUE_SERVER_PUBLIC_URL\` pass-through
frrist
force-pushed
the
frrist/snapshots
branch
from
April 22, 2026 20:42
d50e838 to
833d200
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requires storacha/filecoin-services#16
Adds
smelt snapshot save/load/list/rmplusmake up SNAPSHOT=<name-or-path>so developers can capture a running stack (chain state, volumes, keys, proofs, manifest) and restore it later to skip the ~45s cold-boot cost of contract deploy + piri registration. Restored boots reach the same state in ~10s.Core mechanics:
Portability / shareability:
name: smeltso docker volume names are stable across checkouts regardless of directory basename.docker image inspect, so load reports tag drift (teammate's .env differs) and digest drift at same tag (rolling tag re-pulled) as separate warnings.name:require 25+).Required adjacent changes:
test -f /tmp/readysentinel (set by the entrypoint after contracts deploy and mockrpc is up) — more accurate than a port probe, and properly distinguishes init-mode-deploy-in-progress from ready.Docs:
Depends on filecoin-services load-mode SIGTERM dump changes (separate PR to storacha/filecoin-services).