Repository navigation
fix(storage): bound history reclamation and fence checkpoints - #2434
Merged
Merged
Conversation
filipeforattini
marked this pull request as draft
October 21, 2026 11:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated updates could retain obsolete physical table history indefinitely, while reclamation and checkpoints copied large payloads or rewrote unchanged trees. Reclaim old and aborted versions in bounded ticks, borrow rows during VACUUM and statistics collection, release B-tree merge frames iteratively, and reuse completed physical-delete generations only when their revision proof remains valid. Reopened positional rows recover their shared column names before updates and index backfill.
Crash recovery now excludes unfinished and aborted checkpoint versions while preserving committed preimages. Row mutations, commit publication and WAL pruning share the publication fence, including transactions that begin during checkpoint serialization. Captured transaction inserts and upserts stay out of published pager pages, preventing incidental flushes from persisting an unfinished INSERT or DELETE. No wire or persistent format changes.
Regression coverage includes active snapshots, rollback, indexes, reopen, physical page reuse, concurrent checkpoint writers, and forced termination before and after COMMIT. The incidental-flush tests failed for single INSERT, bulk INSERT and DELETE before the fix; all four mutation cases now pass across open, rolled-back and committed outcomes (12 forced recoveries).
Bulk INSERT also holds the publication fence before installing segment rows through WAL capture or append. A forced-crash regression reproduces an unfinished batch leaking when its transaction begins during checkpoint serialization; the companion autocommit case checks acknowledged batches remain durable.
The current revision incorporates the latest WAL and query-budget changes. Validation so far:
Canonical CI: 7,251 unit and 3,477 integration tests pass, including the 13 checkpoint/concurrent-writer/incidental-flush cases and the vault-preparation crash case. One existing timing-based page-cache scaling test passes on its automatic retry; this run is not a performance comparison.
Local integration coverage: 32 cases pass; formatting, cargo check and Clippy with warnings denied pass.
A persisted fixture retains its visible contents and indexed reads through bounded history reclamation, concurrent checkpoints and reopening. Checkpoints finish in 7,773 ms before reopening and 51 ms after reopening, under a 4 GiB/no-swap/one-CPU limit; concurrent reads, writes, login and catalog requests remain available.
The final controlled qualification passes 120 five-second mutation/reclamation/checkpoint cycles. The last 60 cycles show zero growth in both the main file and the complete persistence directory. Maximum checkpoint time is 61 ms. Three forced process crashes preserve acknowledged revisions and discard unfinished indexed edits. Complete backup member hashes, restored rows and restored indexes match. The duration check excludes dependency compilation.
All qualification above is tied to revision 1a3aaf8. No wire or persistent format migration is introduced.