exec3: parallel execution performance improvements #19826

Overview

Track performance improvements for the parallel execution pipeline (exec3). The goal is to get steady-state block processing as close to flat GC allocation as possible — block processing is cyclical, so with no RPC traffic the memory profile should be flat.

Current observations (branch: exec3-parallel-direct-finalize)

• Parallel alloc ~13.5GB vs serial ~7.5GB — nearly 2x memory overhead from version maps, read/write sets, and TxResult collector buffers
• RPC tip staleness — eth_blockNumber can lag 60+ blocks behind actual committed tip during batch processing. Root cause: shared domains not shared with RPC provider, and RPC waits for DB transaction rotation instead of reading from in-memory cache
• Batch buffer overflow — buf=510MB/500MB seen in logs, suggesting batch size accounting isn't tight enough

Improvement areas

Memory / GC

• Profile heap allocations per block — identify top allocation sources (VersionedWrite, VersionedRead, ReadSet maps, etc.)
• Pool/reuse TxResult and collector write slices across transactions instead of fresh allocation each TX
• Reduce version map overhead — explore more compact representations
• Evaluate read/write set data structure alternatives (sorted slices vs maps)
• Track alloc baseline between GC cycles — should be flat sawtooth, not ratcheting

RPC tip latency

• RPC layer should read from committed in-memory state, not wait for MDBX transaction rotation
• Ensure forkchoice update notifies RPC provider immediately when canonical head advances
• Investigate notification path: execution commit → domain update → RPC visibility

Batch sizing

• Tighten batch buffer accounting to avoid >500MB overflows
• Consider adaptive batch sizing based on block gas usage

Finalize path (follow-on from #19824)

• Data structure optimization for VersionedWrites/ReadSet (currently append-only slices with linear scans)
• Reduce allocations in finalizeTx direct path

Measurement

• pprof_monitor.sh captures periodic heap profiles + metrics CSV for parallel vs serial comparison
• go tool pprof -diff_base=early.pb.gz late.pb.gz to identify growing allocations
• Monitor: block number, heap alloc, sys memory, tip age — every 60s

Related

• Extract EEST edge cases into unit tests + finalize data structure optimization #19824 — Extract EEST edge cases into unit tests + finalize data structure optimization
• exec3_parallel: eliminate IBS round-trip in finalize path #19814 — Direct finalize path (exec3-parallel-direct-finalize branch)

🤖 Generated with Claude Code