feat(archive): add local retention policy storage and lifecycle - #5719
Open
wpfleger96 wants to merge 3 commits into
Open
feat(archive): add local retention policy storage and lifecycle#5719wpfleger96 wants to merge 3 commits into
wpfleger96 wants to merge 3 commits into
Conversation
Introduce per-subscription/per-kind retention policies for the local archive, governing how long archived events of each kind are kept before a future prune pass expires them. Observer frames (kind 24200) default to a 30-day rolling window; every other kind defaults to Forever. Policies live in a normalized `retention_policies` table created by a new crash- and race-safe migration (M4). Because the archive DB is opened by two connections on first use, M4 cannot use the M1-M3 DEFERRED pattern; it takes the write lock up front with BEGIN IMMEDIATE, rechecks the marker and schema shape inside the lock, seeds defaults from existing subscriptions, and writes its marker last so a crash before COMMIT rolls back cleanly. Malformed subscription `kinds` JSON fails the migration closed rather than leaving already-archived data ungoverned. All production archive DB access now routes through a gated `ArchiveDb` adapter: a process-wide init barrier runs every migration exactly once before any command opens a connection, and a maintenance RwLock reserves a home for the Phase-4 VACUUM path. The policy lifecycle is deliberately independent of `save_subscriptions` — disabling a kind or deleting a subscription orphans its policy so historical data keeps expiring; only the explicit delete command removes one. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Resolve the three review findings on the retention Phase-1 branch. M4 now validates the COMPLETE expected shape of its objects inside the BEGIN IMMEDIATE transaction — every column's name, type, nullability and primary-key position via PRAGMA table_info, and the scope-age index's key order via pragma_index_info — instead of counting object names. CREATE ... IF NOT EXISTS preserves a wrong-shaped object that already carries the expected name, so name presence alone could mark a wrong-shaped named table valid and hand Phase 2 an unusable table or missing access path. A wrong-shaped named index is dropped and rebuilt (an index carries no data); an incompatible named table is rejected and the whole transaction rolls back with no marker. The init-barrier and guard-lifetime contracts are now pinned directly against the ArchiveDb OnceCell/RwLock orchestration through a cfg(test) path/hook seam, rather than via raw SQLite contention: production-shaped with_conn callers race a held init and prove none opens a connection until initialization completes and exactly one initialization runs; a separate test proves a write-lock contender cannot enter until a with_conn closure returns and its connection drops. Adds the full policy lifecycle on one evolving state (active -> kind disabled -> subscription deleted -> orphan edited -> orphan deleted) and a deterministic concurrent merge/remove/set interleaving asserting the subscription stays valid, the explicit choice survives, and no policy row is deleted by a kinds mutation. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
`pragma_index_info` reports an index's key columns but not whether it carries a `WHERE` predicate, so a partial index with the exact expected columns passed `scope_age_index_is_correct` and M4 wrote its marker over it. That index cannot serve the unrestricted scope-age range scan the Phase-2 prune query needs — SQLite falls back to the primary-key autoindex — so the marker would certify a missing access path. Probe `pragma_index_list`'s `partial` flag inside the same `BEGIN IMMEDIATE` transaction and treat a partial named age index like any other wrong shape: drop and rebuild it non-partial (an index carries no data, so a rebuild is safe). A mutation-sensitive test precreates a partial index with correct table and ordered columns and asserts M4 replaces it with a non-partial index before committing the marker. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of local-archive retention: the schema and policy-lifecycle backend that lets the desktop app expire archived observer frames instead of growing
~/.buzz/archive/archive.dbwithout bound. This is backend only — no Settings UI (Phase 3) and no prune worker or space reclamation (Phases 2 and 4) yet, so nothing is deleted or vacuumed on this branch.What this adds
retention_policiestable keyed(identity_pubkey, relay_url, scope_type, scope_value, kind)with a nullabledayscolumn (NULL= keep forever). The policy lifecycle is deliberately independent ofsave_subscriptions: disabling a kind or deleting a subscription leaves its policy behind so already-archived data keeps expiring. A policy is removed only by the explicitdelete_retention_policycommand.archive_metak/v table for the Phase 2 prune lease/state.idx_archived_event_scopes_ageonarchived_event_scopes (identity_pubkey, relay_url, scope_type, scope_value, archived_at, id). The retention age basis is the scope row'sarchived_at("kept locally for N days"), so the index seeks by age; the scope PK leads withidand cannot range-seek. No eventcreated_atindex is created — that column is not the age basis.(subscription, kind)pair — kind24200(observer frames) getsdays=30, every other kind gets an explicitNULL(Forever) row. Because a fresh DB can be opened by two connections concurrently (the observer- and metric-archive seed hooks), M4 takesBEGIN IMMEDIATEup front, rechecks the marker inside the lock, then writes the marker last in the same transaction. A cheap pre-lock guard keeps steady-state opens off the write lock. Seeding fails closed on malformedkindsJSON (marker not written, next open retries).CREATE ... IF NOT EXISTS, an object that already carries the expected name but the wrong shape is silently preserved — name presence alone cannot certify the schema. Inside theBEGIN IMMEDIATEtransaction, before seeding or writing the marker, M4 validates the complete expected shape of both tables (every column's name, declared type, nullability, and primary-key position viaPRAGMA table_info) and the scope-age index's exact key-column order (viapragma_index_info) and non-partiality (viapragma_index_list'spartialflag — a partial index with the right columns cannot serve the unrestricted prune-age range scan). A wrong-shaped named index (wrong key order or partial) is dropped and rebuilt (an index carries no data, so a rebuild is safe); a wrong-shaped named table is rejected and the whole transaction rolls back with no marker, so the next open re-runs M4 once the schema is corrected. This guarantees the marker never certifies a mis-shaped schema Phase 2 would inherit.ArchiveDbinarchive_db.rs) now owns every production archive open. Atokio::sync::OnceCellinit barrier runs schema + all migrations (including M4's index build over the existing 1.3M-row archive) exactly once; every commandawaits it before opening an ordinary connection. Atokio::sync::RwLockread guard is held across the full lifetime of each connection (acquired before blocking dispatch, released only after the closure returns and its connection drops) — the guard-lifetime contract the Phase 4 VACUUM write path depends on. A failed init is not cached, so a transient error does not wedge the archive for the process lifetime.create_save_subscriptionmoved off its directopen_db()call onto the adapter (also fixing its prior blocking-thread violation). All archive commands now route throughArchiveDb::with_conn.spawn_warm_init, wired inlib.rssetup) pays M4's one-time index-build cost at startup rather than on a user's first archive command. Non-fatal — the first real caller retries and surfaces any error.retention.rs): one transactional path for seed/set/delete, fail-closedvalidate_days(rejects0, negatives, and values over ~100 years so no policy encodes a non-expiring "0 days" or an overflowing cutoff).create_save_subscriptionandmerge_owner_p_kindsinject default policies through the helper sokindsand policy rows mutate together under the existingBEGIN IMMEDIATE.set_save_subscription_retention(identity/relay derived server-side,daysbounded positive or null),list_retention_policies(independent of live subscription rows; each policy taggedactiveor orphaned),delete_retention_policy.Notes
ArchiveDb::warm_initcovers workspace/identity timing: the archive DB is a single per-nest file (identity is a row column, not part of the path) resolved fromnest_dir(), which is fixed early insetup()before the async workspace relay override settles — so the globally-mounted observer producer cannot open a connection ahead of the barrier.BEGIN IMMEDIATE+ in-lock recheck alongside the in-process barrier: the barrier is startup orchestration for this process, the immediate-lock is durability against a second OS process or a direct test open.Tests
retention_tests.rs(27) — default/validation rules, seed idempotency (an existing explicit choice always wins over a re-seed), set/delete/list behavior, active-vs-orphaned tagging scoped to identity+relay, the full policy lifecycle walked on one evolving state (active → kind-disabled → subscription-deleted → orphan edited → orphan deleted), M4 idempotency and seeding, malformed-kindsabort-and-recover, partial-schema/marker-absent recovery, a wrong-shaped namedarchive_metathat M4 rejects with no marker, a wrong-key-order scope-age index that M4 drops and rebuilds, a partial scope-age index (correct columns,WHEREpredicate) that M4 rebuilds non-partial before the marker commits, the two-connection WAL first-open race on a populated legacy DB (neither contender times out; marker written once), concurrent merge, and a deterministic concurrent merge/remove/set interleaving proving the subscription stays valid, the explicit choice survives, and no policy row is deleted by akindsmutation.archive_db_tests.rs(2) — pin theArchiveDborchestration directly against the realOnceCell/RwLock(via a#[cfg(test)]path/hook seam, not raw SQLite contention): the first-open barrier serializes init so exactly one initialization runs while every production-shapedwith_conncaller awaits it and no ordinary connection opens until init completes; and the maintenance read guard lives for the full connection lifetime, blocking a write-lock contender until the closure returns and its connection drops.store_tests.rsin_memory()now runsapply_schema_migrationsso its subscription mutators match productionopen_archive_db.