Skip to content

feat(archive): add local retention policy storage and lifecycle - #5719

Open
wpfleger96 wants to merge 3 commits into
mainfrom
wpfleger/archive-retention-phase1
Open

feat(archive): add local retention policy storage and lifecycle#5719
wpfleger96 wants to merge 3 commits into
mainfrom
wpfleger/archive-retention-phase1

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 13, 2026

Copy link
Copy Markdown
Member

Phase 1 of local-archive retention: the schema and policy-lifecycle backend that lets the desktop app expire archived observer frames instead of growing ~/.buzz/archive/archive.db without bound. This is backend only — no Settings UI (Phase 3) and no prune worker or space reclamation (Phases 2 and 4) yet, so nothing is deleted or vacuumed on this branch.

What this adds

  • retention_policies table keyed (identity_pubkey, relay_url, scope_type, scope_value, kind) with a nullable days column (NULL = keep forever). The policy lifecycle is deliberately independent of save_subscriptions: disabling a kind or deleting a subscription leaves its policy behind so already-archived data keeps expiring. A policy is removed only by the explicit delete_retention_policy command.
  • archive_meta k/v table for the Phase 2 prune lease/state.
  • Covering scope-age index idx_archived_event_scopes_age on archived_event_scopes (identity_pubkey, relay_url, scope_type, scope_value, archived_at, id). The retention age basis is the scope row's archived_at ("kept locally for N days"), so the index seeks by age; the scope PK leads with id and cannot range-seek. No event created_at index is created — that column is not the age basis.
  • Migration M4 creates the two tables plus the index and seeds a default policy for every existing (subscription, kind) pair — kind 24200 (observer frames) gets days=30, every other kind gets an explicit NULL (Forever) row. Because a fresh DB can be opened by two connections concurrently (the observer- and metric-archive seed hooks), M4 takes BEGIN IMMEDIATE up front, rechecks the marker inside the lock, then writes the marker last in the same transaction. A cheap pre-lock guard keeps steady-state opens off the write lock. Seeding fails closed on malformed kinds JSON (marker not written, next open retries).
  • Full schema-shape validation in M4. Because the DDL is CREATE ... IF NOT EXISTS, an object that already carries the expected name but the wrong shape is silently preserved — name presence alone cannot certify the schema. Inside the BEGIN IMMEDIATE transaction, before seeding or writing the marker, M4 validates the complete expected shape of both tables (every column's name, declared type, nullability, and primary-key position via PRAGMA table_info) and the scope-age index's exact key-column order (via pragma_index_info) and non-partiality (via pragma_index_list's partial flag — a partial index with the right columns cannot serve the unrestricted prune-age range scan). A wrong-shaped named index (wrong key order or partial) is dropped and rebuilt (an index carries no data, so a rebuild is safe); a wrong-shaped named table is rejected and the whole transaction rolls back with no marker, so the next open re-runs M4 once the schema is corrected. This guarantees the marker never certifies a mis-shaped schema Phase 2 would inherit.
  • Gated DB adapter (ArchiveDb in archive_db.rs) now owns every production archive open. A tokio::sync::OnceCell init barrier runs schema + all migrations (including M4's index build over the existing 1.3M-row archive) exactly once; every command awaits it before opening an ordinary connection. A tokio::sync::RwLock read guard is held across the full lifetime of each connection (acquired before blocking dispatch, released only after the closure returns and its connection drops) — the guard-lifetime contract the Phase 4 VACUUM write path depends on. A failed init is not cached, so a transient error does not wedge the archive for the process lifetime.
  • create_save_subscription moved off its direct open_db() call onto the adapter (also fixing its prior blocking-thread violation). All archive commands now route through ArchiveDb::with_conn.
  • Startup warm-init (spawn_warm_init, wired in lib.rs setup) pays M4's one-time index-build cost at startup rather than on a user's first archive command. Non-fatal — the first real caller retries and surfaces any error.
  • Policy mutation helpers (retention.rs): one transactional path for seed/set/delete, fail-closed validate_days (rejects 0, negatives, and values over ~100 years so no policy encodes a non-expiring "0 days" or an overflowing cutoff). create_save_subscription and merge_owner_p_kinds inject default policies through the helper so kinds and policy rows mutate together under the existing BEGIN IMMEDIATE.
  • Three Tauri commands: set_save_subscription_retention (identity/relay derived server-side, days bounded positive or null), list_retention_policies (independent of live subscription rows; each policy tagged active or orphaned), delete_retention_policy.

Notes

  • ArchiveDb::warm_init covers workspace/identity timing: the archive DB is a single per-nest file (identity is a row column, not part of the path) resolved from nest_dir(), which is fixed early in setup() before the async workspace relay override settles — so the globally-mounted observer producer cannot open a connection ahead of the barrier.
  • M4 keeps its own BEGIN IMMEDIATE + in-lock recheck alongside the in-process barrier: the barrier is startup orchestration for this process, the immediate-lock is durability against a second OS process or a direct test open.

Tests

retention_tests.rs (27) — default/validation rules, seed idempotency (an existing explicit choice always wins over a re-seed), set/delete/list behavior, active-vs-orphaned tagging scoped to identity+relay, the full policy lifecycle walked on one evolving state (active → kind-disabled → subscription-deleted → orphan edited → orphan deleted), M4 idempotency and seeding, malformed-kinds abort-and-recover, partial-schema/marker-absent recovery, a wrong-shaped named archive_meta that M4 rejects with no marker, a wrong-key-order scope-age index that M4 drops and rebuilds, a partial scope-age index (correct columns, WHERE predicate) that M4 rebuilds non-partial before the marker commits, the two-connection WAL first-open race on a populated legacy DB (neither contender times out; marker written once), concurrent merge, and a deterministic concurrent merge/remove/set interleaving proving the subscription stays valid, the explicit choice survives, and no policy row is deleted by a kinds mutation.

archive_db_tests.rs (2) — pin the ArchiveDb orchestration directly against the real OnceCell/RwLock (via a #[cfg(test)] path/hook seam, not raw SQLite contention): the first-open barrier serializes init so exactly one initialization runs while every production-shaped with_conn caller awaits it and no ordinary connection opens until init completes; and the maintenance read guard lives for the full connection lifetime, blocking a write-lock contender until the closure returns and its connection drops.

store_tests.rs in_memory() now runs apply_schema_migrations so its subscription mutators match production open_archive_db.

Introduce per-subscription/per-kind retention policies for the local
archive, governing how long archived events of each kind are kept before
a future prune pass expires them. Observer frames (kind 24200) default to
a 30-day rolling window; every other kind defaults to Forever.

Policies live in a normalized `retention_policies` table created by a new
crash- and race-safe migration (M4). Because the archive DB is opened by
two connections on first use, M4 cannot use the M1-M3 DEFERRED pattern; it
takes the write lock up front with BEGIN IMMEDIATE, rechecks the marker and
schema shape inside the lock, seeds defaults from existing subscriptions,
and writes its marker last so a crash before COMMIT rolls back cleanly.
Malformed subscription `kinds` JSON fails the migration closed rather than
leaving already-archived data ungoverned.

All production archive DB access now routes through a gated `ArchiveDb`
adapter: a process-wide init barrier runs every migration exactly once
before any command opens a connection, and a maintenance RwLock reserves a
home for the Phase-4 VACUUM path. The policy lifecycle is deliberately
independent of `save_subscriptions` — disabling a kind or deleting a
subscription orphans its policy so historical data keeps expiring; only the
explicit delete command removes one.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 13, 2026 00:26
Duncan and others added 2 commits August 12, 2026 23:58
Resolve the three review findings on the retention Phase-1 branch.

M4 now validates the COMPLETE expected shape of its objects inside the
BEGIN IMMEDIATE transaction — every column's name, type, nullability and
primary-key position via PRAGMA table_info, and the scope-age index's key
order via pragma_index_info — instead of counting object names. CREATE ...
IF NOT EXISTS preserves a wrong-shaped object that already carries the
expected name, so name presence alone could mark a wrong-shaped named
table valid and hand Phase 2 an unusable table or missing access path. A
wrong-shaped named index is dropped and rebuilt (an index carries no data);
an incompatible named table is rejected and the whole transaction rolls
back with no marker.

The init-barrier and guard-lifetime contracts are now pinned directly
against the ArchiveDb OnceCell/RwLock orchestration through a cfg(test)
path/hook seam, rather than via raw SQLite contention: production-shaped
with_conn callers race a held init and prove none opens a connection until
initialization completes and exactly one initialization runs; a separate
test proves a write-lock contender cannot enter until a with_conn closure
returns and its connection drops.

Adds the full policy lifecycle on one evolving state (active -> kind
disabled -> subscription deleted -> orphan edited -> orphan deleted) and a
deterministic concurrent merge/remove/set interleaving asserting the
subscription stays valid, the explicit choice survives, and no policy row
is deleted by a kinds mutation.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
`pragma_index_info` reports an index's key columns but not whether it
carries a `WHERE` predicate, so a partial index with the exact expected
columns passed `scope_age_index_is_correct` and M4 wrote its marker over
it. That index cannot serve the unrestricted scope-age range scan the
Phase-2 prune query needs — SQLite falls back to the primary-key
autoindex — so the marker would certify a missing access path.

Probe `pragma_index_list`'s `partial` flag inside the same
`BEGIN IMMEDIATE` transaction and treat a partial named age index like
any other wrong shape: drop and rebuild it non-partial (an index carries
no data, so a rebuild is safe). A mutation-sensitive test precreates a
partial index with correct table and ordered columns and asserts M4
replaces it with a non-partial index before committing the marker.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant