The compactor screens candidates by the catalog's own file sizes, so a file at or above the threshold is never opened (T-463) - #330
Merged
Conversation
chasers
added this pull request to stack #332
September 11, 2026 17:07
chasers
force-pushed
the
t-463-size-from-catalog
branch
from
September 11, 2026 19:54
a80bb9b to
d4bdcc9
Compare
…a file at or above the threshold is never opened (T-463) Every sweep read every owned segment's footer to find the undersized ones, and once the engines stopped caching file reads (T-461) a table with nothing to compact cost two store requests per file per sweep, forever — tens of thousands of GETs per sweep on a 10k-file S3 table, and a sweep that spent its interval sizing. A table is now listed through Catalog.segment_files/3, which carries the whole-file size DuckLake recorded at registration, and a file at or above compact_below_bytes by that measure is never a candidate: its compressed-data sum is smaller still, so the screen only tightens the threshold by a footer's width. A size the catalog does not know keeps the file a candidate. Footers are still read for the candidates, since sizing is where a corrupt file first fails and the quarantine keys on that. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014K63Kxq4pf7eD9K1416Eju
…he files it was handed The compactor fetched the current snapshot and then the file list, two statements on the serialized compaction connection where one had been, and the same pair was copied in the merge. Catalog.segment_files/3 now accepts :current and DuckLake resolves the snapshot in one place; the compactor and the merge ask for :current. The screen re-joined sizes onto paths it had stripped one line earlier through a map with a default and a sentinel clause that could not fire; it filters the listed files by the plannable paths instead, and the docs name plan/2 and segment_files/3 again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014K63Kxq4pf7eD9K1416Eju
chasers
force-pushed
the
t-463-size-from-catalog
branch
from
September 11, 2026 21:15
d4bdcc9 to
c45ce19
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracker: T-463. Stacked on #329 (T-464).
The compactor read every owned segment's Parquet footer on every sweep to find the undersized ones. Once the engines stopped caching file reads (T-461) that was two store requests per file per sweep on tables with nothing to compact — tens of thousands of GETs per sweep on a 10k-file S3 table, and a sweep spending its interval sizing (raised by the review of #325/#326).
A table is now listed through
Catalog.segment_files/3, which carries the whole-file size DuckLake recorded at registration, and a file at or abovecompact_below_bytesby that measure is never a candidate. Its compressed-data sum is smaller still, so the screen only tightens the threshold by a footer's width. A size the catalog does not know keeps the file a candidate. Footers are still read for the candidates, since sizing is where a corrupt file first fails and the quarantine keys on that.Test: with every file above the threshold, a corrupted one costs nothing and the sweep reports no failure. Deployment note added.
🤖 Generated with Claude Code
https://claude.ai/code/session_014K63Kxq4pf7eD9K1416Eju