fix(storage): bound history reclamation and fence checkpoints

.changeset/bounded-mvcc-maintenance.md

@@ -0,0 +1,12 @@
+---
+"@reddb-io/cli": patch
+---
+
+Reclaim obsolete table history in bounded background batches for writable
+standalone embedded runtimes. Avoid copying row payloads during VACUUM and
+planner statistics collection, and bound cardinality and histogram working sets.
+Rebalance deleted B-tree pages iteratively so cascading merges do not retain
+page buffers from every level on a maintenance worker's stack.
+Reuse trees after completed physical delete batches instead of re-encoding
+unchanged survivors at the next checkpoint, with revision checks that retain
+full validation after untracked edits.

.changeset/checkpoint-bulk-row-publication.md

@@ -0,0 +1,5 @@
+---
+"@reddb-io/cli": patch
+---
+
+Fence bulk row insertion through its WAL capture or append so a checkpoint cannot publish an unfinished batch that starts during serialization. Add forced-crash regressions for explicit transactions and acknowledged autocommit batches.

.changeset/checkpoint-concurrent-row-publication.md

@@ -0,0 +1,7 @@
+---
+"@reddb-io/cli": patch
+---
+
+Fence row mutations and delete stamps through their WAL append against checkpoint
+publication. Transactions starting during a checkpoint cannot leak uncommitted
+rows, and concurrent acknowledged autocommit updates retain their recovery WAL.

.changeset/checkpoint-unfinished-transactions.md

@@ -0,0 +1,8 @@
+---
+"@reddb-io/cli": patch
+---
+
+Exclude unfinished row versions from crash-recoverable checkpoints and retain
+their committed preimages. Invalidate checkpoint reuse when transaction outcomes
+change, and fence commit WAL publication with visibility so a concurrent
+checkpoint cannot discard a durable commit's recovery records.

.changeset/reopened-positional-row-updates.md

@@ -0,0 +1,7 @@
+---
+"@reddb-io/cli": patch
+---
+
+Restore shared column names on owned reads of reopened positional rows. Preserve
+untouched values during SQL updates and include positional rows when rebuilding
+secondary indexes without changing the file format or copying payloads again.

.changeset/transaction-pager-flush.md

@@ -0,0 +1,5 @@
+---
+"@reddb-io/cli": patch
+---
+
+Keep captured transaction inserts and row upserts out of published B-tree pages until their committed WAL is durable. This prevents incidental pager flushes from recovering unfinished inserts or delete stamps after a crash. Add crash regressions for INSERT, bulk INSERT, UPDATE and DELETE with open, rolled-back and committed transactions.

.red/adr/0014-mvcc-history-store-and-transaction-recovery.md

@@ -49,8 +49,8 @@ This ADR ratifies the implementation contract for the first rollout:
   paths explicitly adopt the resolver.
 - RedDB implements snapshot isolation with first-committer-wins write conflict
   detection. It does not implement serializable isolation or SSI.
-- The first rollout includes manual `VACUUM` / GC only. It does not include an
-  autovacuum daemon.
+- The original rollout includes manual `VACUUM` / GC only. The bounded
+  standalone maintenance evolution below adds automatic table-history cleanup.
 - Current secondary indexes plus MVCC recheck and fallback are the first
   correctness target. Historical secondary indexes are deferred.
 - Full multi-model adoption is deferred behind table-row correctness.
@@ -431,3 +431,40 @@ duplicating rules.
   (the baseline); the new non-table PATCH path is flag-gated. For a non-versioned
   document the two verbs therefore differ in history accrual (invisible without
   AS OF, which is rejected for non-versioned). Align if uniform behavior is wanted.
+
+## Evolution: bounded standalone maintenance (2026-10-20)
+
+Writable standalone embedded runtimes reclaim obsolete table versions through
+an existing 200 ms maintenance tick. Each tick inspects at most 4,096 physical
+positions and reclaims at most 256 versions. These limits include empty or
+retired scan positions. Source cursors use weak segment references so a slow
+sweep does not retain retired payloads across consolidation. A subsequent sweep
+visits replacement segments and writes appended after cursor capture.
+
+Reclamation uses the same transaction, statement and external-pin horizons as
+manual VACUUM. Aborted creators can be removed, while aborted deletions never
+make a live version eligible. Transaction outcomes remain retained until a
+separate global reference proof permits pruning them. Physical deletes and
+WAL publication share the checkpoint fence; a busy checkpoint defers a tick.
+Collection replacement invalidates its old cursor. Failed batches retry with
+backoff and rate-limited diagnostics.
+
+`runtime.mvcc.autovacuum.enabled` defaults to `true` in this supported scope.
+`runtime.mvcc.autovacuum.scan_rows_per_tick` and
+`runtime.mvcc.autovacuum.reclaim_rows_per_tick` can reduce their respective
+budgets; values are clamped to 1..4096 and 1..256. Read-only runtimes and the
+serverless, primary-replica and cluster deploy profiles do not enable this local collector: they require their
+own write-authority and distributed history-retention contract first.
+
+Manual VACUUM borrows headers instead of cloning a collection's payloads.
+Statistics collection also borrows rows, caps exact fingerprint cardinality at
+30,000 entries before using HyperLogLog, and caps each column's histogram
+sample at 1 MiB. Values over 4 KiB leave min/max bounds unknown. Statistics are
+planner estimates and must not be used as proof that a data range is empty.
+
+Resource sketch: a collector tick buffers at most 256 source IDs and 256
+reclamation IDs, plus bounded lookup scratch (about 6 KiB total), rather than
+N rows multiplied by payload width. Cursor metadata scales with segment count.
+Statistics use bounded cardinality and sample buffers per column; neither path
+makes a second full collection of row payloads. Explicit VACUUM still traverses
+the target collection; the automatic collector resumes between ticks.

.red/context/persistence.md

@@ -27,11 +27,29 @@ Part of the [glossary map](../CONTEXT-MAP.md). The storage engine shared by **ev
 
 ## Checkpoint reuse
 
+Checkpoints exclude row creators whose transactions are active or aborted and
+clear their delete stamps on committed preimages. Live transaction state stays
+unchanged. Transaction visibility revisions invalidate reuse after begin, commit
+or rollback; equivalent exclusion sets can retain the existing proof. Commit
+WAL publication and visibility changes share the checkpoint publication fence.
+While store WAL actions are captured for a transaction, row inserts and
+upserts stay in the live segment and deferred WAL, leaving published B-tree
+pages unchanged. Incidental pager flushes cannot publish unfinished inserts
+or delete stamps; a successful commit makes the captured WAL batch durable.
+Bulk inserts hold the checkpoint publication fence before installing their
+segment rows through WAL capture or append, including transactions that begin
+after checkpoint visibility has already been frozen.
+
+Owned row reads restore missing positional column names from the collection's
+shared schema before mutation or index backfill. Explicit row schemas and named
+fields remain authoritative. This changes neither the stored record format nor
+the raw segment revisions used by checkpoint reuse.
+
 Paged checkpoints reuse unchanged collection trees using volatile segment mutation
-revisions and manager identity. Aborted transaction visibility, pending vault
+revisions and manager identity. Active and aborted transaction visibility, pending vault
 versions and record format also participate in reuse admission.
 Visibility changes outside a collection are checked against borrowed row headers
-without serializing its payloads. Each checkpoint shares one frozen aborted-XID
+without serializing its payloads. Each checkpoint shares one frozen excluded-XID
 set across collections for both reuse admission and row normalization.
 WAL replay and every entity or metadata mutation invalidate the corresponding
 generation. New mutation paths must advance the segment revision before exposing mutable data.
@@ -64,6 +82,11 @@ visibility changes and versioned rows retain conservative validation. Durable
 replay reconciliation seeds the current raw tree generation before runtime
 visibility normalization is attached.
 
+Completed WAL-backed physical delete batches may advance that proof when every
+tree removal succeeds and the segment revision delta accounts for exactly the
+deleted rows. Earlier untracked edits, changed segment topology, pending vault
+versions and format changes retain full checkpoint validation.
+
 Changed checkpoints encode ordered borrowed rows in batches capped at 1 MiB or
 512 records (a single larger value keeps the existing value-size contract).
 Comparison with a published tree is streamed; a replacement copies the already
@@ -131,3 +154,9 @@ The runtime rejects incomplete remote snapshots before advertising success.
 - **LSN-pinned analytical scan** — read protocol for the columnar analytics projection. An analytical read is an ordinary MVCC read: it pins the current snapshot/LSN, scans columnar segments up to the last materialized LSN, and concatenates the un-materialized tail (rows past the last checkpoint boundary) through the normal row read path. There is a single consistency class — analytical reads are never a separate stale tier — and `AS OF` time-travel composes on top by pinning a historical LSN instead of the current one.
 - **Store fork** — metadata-level fork of an entire store: a new operational manifest generation pinned to a fork LSN, sharing the parent's immutable artifacts and writing its own private WAL from that point forward. Creation is an O(metadata) operation, never a physical copy of the dataset. Deliberately distinct from the VCS data model's *branch* (#1567 vocabulary): a store fork is storage/deploy mechanics for experiment-and-discard workflows (e.g. testing a risky migration or letting an agent run against production-shaped data), not version-control semantics over collection contents. Mechanically, a fork is a lazy in-place restore-to-LSN riding the operational backup/restore boundary (ADR 0043): immutable artifacts (append-only segments, checkpoint artifacts, columnar projection segments) are shared by manifest reference; mutable collection files are hydrated lazily (replay/CoW) the first time the fork touches them, so creation stays O(metadata) and cost is deferred to what the fork actually writes. Lifecycle is diverge-and-discard: a fork ends by being discarded, kept as an independent store, or promoted to new primary (the recovery/PITR path) — there is no merge-back of divergent stores at the storage layer; merge semantics belong to the VCS data model, which has vocabulary for conflicts.
 - **WAL spool** — versioned (v2) write-ahead log records used for replication streaming. Includes magic, version byte, lsn, timestamp, payload-len, payload, crc32. `sync_all` after every append. (The replica-side consumer is the **logical change applier** — see [Primary-replica](primary-replica.md) / [Clustering](clustering.md).)
+
+Row mutation batches and tombstone batches take the publication fence before
+changing live segments and retain it through WAL capture or append. Fencing only
+COMMIT is insufficient: a transaction may begin after checkpoint visibility was
+frozen, and an autocommit's recovery records must not be truncated before its
+mutations belong to the published generation.

crates/reddb-server/src/application/ports_impls_entity.rs

@@ -1368,6 +1368,10 @@ impl RedDBRuntime {
         }
 
         let store = self.db().store();
+        // Fence the in-memory change and its WAL append together. Otherwise a
+        // checkpoint can copy a new transaction's rows or truncate a concurrent
+        // autocommit's WAL before those rows belong to its frozen generation.
+        let _publication = store.vault_publication_lock.lock();
         let collection = &applied[0].collection;
         let Some(manager) = store.get_collection(collection) else {
             return Err(crate::RedDBError::NotFound(format!(

crates/reddb-server/src/runtime.rs

@@ -1571,6 +1571,7 @@ mod mutation_index_topology_tests;
 #[cfg(test)]
 mod mutation_memory_tests;
 pub(crate) mod mvcc_lifecycle;
+pub(crate) mod mvcc_maintenance;
 pub(crate) mod node_load_telemetry;
 pub(crate) mod occupancy_sampler;
 pub(crate) mod primary_queue_store;

crates/reddb-server/src/runtime/impl_core.rs

@@ -1457,9 +1457,9 @@ impl RedDBRuntime {
                             .read()
                             .get(&conn_id)
                             .is_some_and(|writes| !writes.is_empty());
-                        let vault_store = self.inner.db.store();
+                        let store = self.inner.db.store();
                         let _vault_guard =
-                            has_vault_writes.then(|| vault_store.vault_publication_lock.lock());
+                            has_vault_writes.then(|| store.vault_publication_lock.lock());
                         let context = self.inner.transaction_state.commit(conn_id, |ctx| {
                             let mut own_xids = std::collections::HashSet::new();
                             own_xids.insert(ctx.xid);
@@ -1510,8 +1510,14 @@ impl RedDBRuntime {
                                 undo?;
                                 return Err(err);
                             }
+                            // Validate before taking the publication fence:
+                            // conflict rollback can acquire index topology.
+                            // Retain the fence through WAL and visibility so a
+                            // checkpoint cannot discard a durable commit's WAL.
+                            let publication = store.vault_publication_lock.lock();
                             self.restore_pending_write_stamps(conn_id);
                             if let Err(err) = self.flush_pending_store_wal_actions(conn_id) {
+                                drop(publication);
                                 let undo = self.revive_pending_versioned_updates(conn_id);
                                 self.revive_pending_tombstones(conn_id);
                                 self.discard_pending_queue_dedup(conn_id);
@@ -1521,7 +1527,7 @@ impl RedDBRuntime {
                                 undo?;
                                 return Err(err);
                             }
-                            Ok(())
+                            Ok(publication)
                         });
                         if context.is_err() {
                             self.discard_pending_vault_writes(conn_id)?;

crates/reddb-server/src/runtime/impl_dml_batch.rs

@@ -26,6 +26,9 @@ impl RedDBRuntime {
         }
 
         let store = self.db().store();
+        // Keep tombstone stamps and their WAL in one publication interval,
+        // including writers that start after checkpoint visibility is frozen.
+        let _publication = store.vault_publication_lock.lock();
         let Some(manager) = store.get_collection(collection) else {
             return Ok((0, vec![]));
         };

crates/reddb-server/src/runtime/impl_kv/transactions.rs

@@ -383,17 +383,21 @@ mod tests {
                 .unwrap();
             let vault_store = runtime.inner.db.store();
             let _publication = vault_store.vault_publication_lock.lock();
-            let _: RedDBResult<_> = runtime.inner.transaction_state.commit(121, |context| {
-                runtime.prepare_pending_vault_writes(121, context)?;
-                // Exercise both an incidental pager flush and a full checkpoint
-                // exactly after materialization, before COMMIT's WAL append.
-                runtime.inner.db.store().pager().unwrap().flush().unwrap();
-                runtime.checkpoint()?;
-                if !operational {
-                    runtime.publish_serverless_generation()?;
-                }
-                std::process::exit(0);
-            });
+            let _: RedDBResult<_> =
+                runtime
+                    .inner
+                    .transaction_state
+                    .commit(121, |context| -> RedDBResult<()> {
+                        runtime.prepare_pending_vault_writes(121, context)?;
+                        // Exercise both an incidental pager flush and a full checkpoint
+                        // exactly after materialization, before COMMIT's WAL append.
+                        runtime.inner.db.store().pager().unwrap().flush().unwrap();
+                        runtime.checkpoint()?;
+                        if !operational {
+                            runtime.publish_serverless_generation()?;
+                        }
+                        std::process::exit(0);
+                    });
             panic!("crash injection did not exit");
         }
         let directory = tempfile::tempdir().unwrap();

crates/reddb-server/src/runtime/impl_lifecycle.rs

@@ -915,6 +915,8 @@ impl RedDBRuntime {
                         let tick = std::time::Duration::from_millis(200);
                         let work_interval = std::time::Duration::from_secs(60);
                         let mut last_work = std::time::Instant::now();
+                        let mut mvcc_maintenance =
+                            super::mvcc_maintenance::MvccMaintenance::default();
                         loop {
                             std::thread::park_timeout(tick);
                             if stopped.load(AtomicOrdering::Acquire) {
@@ -929,6 +931,11 @@ impl RedDBRuntime {
                                 let _stats = inner.db.store().context_index().stats();
                                 last_work = std::time::Instant::now();
                             }
+                            let rt = RedDBRuntime {
+                                background_tasks: None,
+                                inner,
+                            };
+                            rt.reclaim_history_tick(&mut mvcc_maintenance);
                         }
                     },
                 );