perf(drive-abci): stop rewriting the whole platform state every block - #4554
Closed
PastaPastaPasta wants to merge 2 commits into
Closed
perf(drive-abci): stop rewriting the whole platform state every block#4554PastaPastaPasta wants to merge 2 commits into
PastaPastaPasta wants to merge 2 commits into
Conversation
…story Protocol version 11 turns checkpoints on, with a policy of one every 600 seconds of chain time keeping the last 3. Mainnet blocks are about 2.5 minutes apart, so that is a checkpoint every four blocks — which at replay speed is roughly 37 a second, each one a RocksDB checkpoint over the whole database plus a copy of the platform state, and all but the last three deleted again immediately. Measured at 15.1 ms a block from height 318,704, against about 7 ms for everything else in a block put together. A finished mainnet sync had four checkpoint directories left on disk out of some 26,000 created. Checkpoints are restore points for a running node, so skip them for blocks more than ten minutes old and write the first real one on reaching the tip. They live outside the tree, so no app hash changes.
The saved platform state is 1.28 MB on mainnet, almost all of it masternode lists, validator sets and quorum sets, and it was serialized and written to GroveDB aux storage on every block. Serialization also cloned the entire state first, so a block paid two full copies of some 4,000 masternodes. Serialization now builds the saving form from a borrowed state, and a test asserts the bytes are identical to the owned path. The heavy fields carry a dirty flag set by the accessors that can change them, and while replaying history the full record is rewritten only when it is set, with a small companion record holding the per-block fields written every block; both land in the block's transaction, and a database without the companion record reads exactly as before. Once at the tip the full record is written every block again, so an up-to-date node always has a complete record on disk. The two Core-driven update paths now decide read-only whether anything actually moved before taking a mutable borrow, because Core reports the same masternodes and quorums on most blocks and the borrow alone would force the rewrite.
Contributor
|
Warning Review limit reachedNext included review available in 28 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (14)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
|
🕓 Ready for review — 16 ahead in queue (commit a319483) |
This was referenced Aug 31, 2026
Member
Author
|
Superseded by #4571, which is the same change from a branch in 🤖 Posted autonomously by Claude on behalf of pasta. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
The saved platform state is written to GroveDB aux storage on every block, and on mainnet it is 1.28 MB — almost all of it masternode lists, validator sets and the chain-lock and instant-lock quorum sets.
PlatformSerializable::serialize_to_bytesalso didself.clone()before converting to the saving form, so a block paid two full deep copies of some 4,000 masternodes plus the validator sets, then serialized 1.28 MB, then wrote it.Measured replaying mainnet, per block: 1.08 ms serializing, 0.23 ms for the extra clone, and 1.28 MB into the block's transaction — 545 GB of writes over a full sync.
Stacked on #4553, which introduces the
utils::is_historical_blockpredicate this uses.What was done?
Three things.
Serialize from a borrowed state. A
TryFrom<&PlatformState>for the saving form clones each field once instead of cloning the whole state first. A test asserts the bytes are byte-identical to the owned path.Rewrite the full record only when it changed. The heavy fields carry a dirty flag, set by the accessors that can change them. While replaying history the full record is written only when the flag is set; a small companion record under a second aux key carries the per-block fields — block info, quorum hashes, protocol versions — every block. Both are written in the block's transaction, so a reader never sees them disagree, and a database without the companion record reads exactly as before.
Once the node is at the tip the full record is written every block again, so an up-to-date node always has a complete record on disk and an older drive-abci can still read it. Skipping is confined to a node that is catching up, where the remedy for any format trouble is the resync it is already doing.
Don't take a mutable borrow when nothing moved. Core reports the same quorums on most blocks and an empty masternode diff often, but
update_quorum_infoandupdate_state_masternode_listreached for_mut()accessors regardless — and the borrow alone marks the state dirty. Both now decide read-only whether anything actually changed first.How Has This Been Tested?
Full mainnet replay, genesis to 424,981. Every committed app hash matched a reference sync across all 424,971 heights, and the final app hash matched exactly.
Crash recovery specifically: two mid-sync restarts, at heights 40,200 and 80,188. Both resumed at the right height from the companion record and continued with no app-hash mismatch — which requires the heavy fields to be correct too, since they feed the app hash through masternode identity updates and rewards.
cargo test -p drive-abci --lib— 2,776 passed, including the new byte-identity test.Breaking Changes
A node interrupted mid-initial-sync writes a full record that lags the database, and an older drive-abci reading that database panics in the
Infohandler, which compares the saved state's app hash against GroveDB's root hash. This is why the skip is confined to historical blocks: a node that has caught up is always downgrade-safe. A node interrupted mid-sync is not, and the remedy there is the resync it was already doing.Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code