Compact superseded database build records

[peter-jerry-ye]

Why

The append-only build database records every successful build, so repeated builds make replay progressively more expensive and let the database grow without bound.

What changed

• Treat a complete build record as superseded when a later record shares any output, invalidating the older record as a whole.
• Compact databases larger than 2 MiB when the retained log fits in one third of the original size.
• Preserve every path record and its existing ID; path garbage collection and ID remapping remain a separate follow-up.
• Stream retained records into a same-directory temporary file, synchronize it, replace the old log, and return a usable append handle.
• Keep compaction best-effort and allow it to normalize an incomplete record-header tail already tolerated by replay.

Why this is correct

Record liveness is derived from the complete database log rather than the current graph, so opening the same database with a partial graph cannot decide which shared history is retained. Build state applies to the whole build, so any newer output ownership conservatively invalidates the complete older record. Preserving path records and IDs avoids changing record references in this first compaction step.

Replacement requires exclusive external coordination for callers that share the same database path, as documented on db::open.

Scope

This PR intentionally does not remove unused path records or remap packed path IDs. That optimization can be reviewed independently after build-record compaction is established.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a2dfb50bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lynzrand]

Stumbled upon this while reviewing other stuff. IMHO you can use https://lib.rs/crates/redb and wont need to worry about this any more

[peter-jerry-ye]

Stumbled upon this while reviewing other stuff. IMHO you can use https://lib.rs/crates/redb and wont need to worry about this any more

Thanks for the suggestion. redb looks relevant, but it would not directly replace the compaction implemented here. Our compaction is application-level: a newer build record sharing any output supersedes the entire older record, while records belonging to other partial graphs must remain (this is specific to Moon build system requirement and even different from ninja, which has the whole graph every time since everything is defined in one file). We would still need to encode those semantics ourselves or redesign the database around materialized indexed state.

We also want to preserve the current hot path: one sequential append per completed build. Moving to transactional B-tree updates may add meaningful overhead, so we would want workload-specific benchmarks before making that trade-off. For now, keeping the append-only log with infrequent domain-specific compaction seems like the better fit, but redb is worth revisiting if measurements show that replay time or database growth remains a problem.

[lynzrand]

a newer build record sharing any output supersedes the entire older record, while records belonging to other partial graphs must remain

Fair. I was thinking about using some kind of order-independent canonical output hash as the key to the build record, and there could be a output file -> output record ID -> output record lookup in the invalidation lookup. A change to the output ID associated with a specific file would invalidate the file, while records that share the same output ID would supersede the previous record.

Moving to transactional B-tree updates may add meaningful overhead

redb is a mmap'd database, and build systems are best-effort systems, so maybe it's okay to use non-durable commits. I am not an expert of this kind of systems, but this should be able to give some quite promising numbers being all in-memory actions.

(Interestingly, I have explored an alternative build system and DB design at https://github.com/lynzrand/n2o5. Feel free to build upon it, borrow its ideas, copy from it, or make use of the code or any part of it however you see fit. For clarity, I explicitly grant the MoonBit project permission to use my contributions to n2o5 under the terms of the MIT License, in addition to the repository’s existing MPL-2.0 license.)

Anyway, it's just my two cents while accidentally revisiting this projects. Please take it easy. Good luck and happy hacking!