[bloatnet] explore initial sync using snapshots for --chain=perf-devnet-2

The EF team doing the state bloat research is planning on doing "initial sync tests" using Syncoor -
https://syncoor.perf-devnet-2.ethpandaops.io/#/?directory=perf-devnet-2&network=perf-devnet-2

Currently, Erigon starts off syncing from an old block (the block at which the shadowfork branched off of mainnet at https://etherscan.io/block/22758966 according to Pari) and then has to re-exec all blocks until the tip of perf-devnet-2. This currently takes a very long time since the initial blocks of bloatnet were full 500MGas blocks which create a lot of new state and we are currently inefficient for such a heavy load. We are going to work on optimising this but even once we have it optimised, we will still be re-execing a lot of blocks on initial sync tests for Syncoor. This won't be representative numbers for our sync speed in an environment where the state is x2/x3/x4 the size of mainnet since we aren't leveraging our snapshot sync and are re-execing the blocks.

One thing we can consider doing for this is to setup a snapshotter for perf-devnet-2 and support for running Erigon with --chain=perf-devnet-2 so that initial sync can use bloatnet snapshots.

yperbasis added Imp2 as Importance

update 01/12/25: we are syncing a new snapshotter for bloatnet based on commit 07e7965c48254d4378316cdc380ff27d36364d6e from performance branch.

No changes on step size were made, goal here is to reach the tip and make easier sync available. Waiting for chain tip.

update 08/12: at block 22.89M using --sync.block.loop.limit == 128. Will try to double it to 256 and observe the effects.

update 10/12: back to block 22.7M bc deleted chaindata 2 days ago; trying --sync.block.loop.limit == 1024 in order to observe effects on monitoring.

it looks like my hypothesis was correct, bigger --sync.block.loop.limit is actually better.

going from 256 -> 1024 increased blocks by 4x, but time taken to exec+commitment increased by only ~2x.

256: 30min exec + 30min commitment
1024: 1h exec + 1h commitment

the tradeoff is losing more uncommited work in case of stop erigon, but here our goal is to reach chaintip ASAP.

let's then increase --sync.block.loop.limit more 4x, 1024 -> 4096 (which is closer to the default of 5000, but let's keep using multiples of previous tests)

--sync.block.loop.limit=4096 does not work well with 64GB machine, OOO

[1322923.224586] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/erigon.service,task=erigon,pid=348048,uid=1000
[1322923.225486] Out of memory: Killed process 348048 (erigon) total-vm:4428816168kB, anon-rss:56179076kB, file-rss:7892kB, shmem-rss:0kB, UID:1000 pgtables:1402632kB oom_score_adj:200
[1322926.093216] oom_reaper: reaped process 348048 (erigon), now anon-rss:0kB, file-rss:340kB, shmem-rss:0kB

let's reduce and experiment with 2048

more about previous OOO:

at block ~23.73M now (end of 11/2025), it passed the first bloat, there is a second bloat post-devconnect.

so far my conclusions:

step size reduction didn't matter afterall. the experiment showed not much reduction in chaindata size because data is already big. The snapshotter is running with no step size changes.
the assumption of reducing --sync.loop.block.limit is wrong as well. on the contrary, reducing it make sync times worse as explained above.
increasing --sync.loop.block.limit revealed another bottleneck. The maximum I was able to increase it was 1024 for a 64GB mem machine. More than that results in OOM due to (probably, as far as I could track it with pprof) batch of commitments being loaded into memory by TemporalMemBatch.

"deleting chaindata" is not effective, as there are few transactions with huge amount of data, so everytime we delete it, the next restart has to recompute a lot of data taking a lot of time again, hence step size reduction being ineffective.
best approach was to just "let chaindata growth" and "accept it will be slow, but it'll finish" and "tune --sync.loop.block.limit to avoid OOM"
so far, chaindata is > 500GB, but interestingly most of it is reclaimable space. not sure if that says something.

@wmitsuda

at block ~23.73M now (end of 11/2025), it passed the first bloat, there is a second bloat post-devconnect.
so far my conclusions:

step size reduction didn't matter afterall. the experiment showed not much reduction in chaindata size because data is already big. The snapshotter is running with no step size changes.

the assumption of reducing --sync.loop.block.limit is wrong as well. on the contrary, reducing it make sync times worse as explained above.

increasing --sync.loop.block.limit revealed another bottleneck. The maximum I was able to increase it was 1024 for a 64GB mem machine. More than that results in OOM due to (probably, as far as I could track it with pprof) batch of commitments being loaded into memory by TemporalMemBatch.

* "deleting chaindata" is not effective, as there are few transactions with huge amount of data, so everytime we delete it, the next restart has to recompute a lot of data taking a lot of time again, hence step size reduction being ineffective. * best approach was to just "let chaindata growth" and "accept it will be slow, but it'll finish" and "tune `--sync.loop.block.limit` to avoid OOM" * so far, chaindata is > 500GB, but interestingly most of it is reclaimable space. not sure if that says something.

@wmitsuda you've misunderstood the reason behind reducing --sync.loop.block.limit. It was not to increase sync speed (nobody said that). It was to gather more fine granularity of metrics about table sizes (chart we have in Grafana).

@wmitsuda you've misunderstood the reason behind reducing --sync.loop.block.limit. It was not to increase sync speed (nobody said that). It was to gather more fine granularity of metrics about table sizes (chart we have in Grafana).

alright, probably was my assumption then

for the record, the bootnode ethpandaops is running is showing similar behavior, also 64GB mem machine, stuck around same block, OOM forever.

I recommended --sync.loop.block.limit=1024 to constrain mem usage.

update: both ethpandaops and our own snapshotter reached the tip during the holidays.

I'm now properly configuring lighthouse into our snapshotter (was manually issuing FCU in order to isolate and have a more controlled progress during first sync).

next steps:

properly setup and automate (if possible) lighthouse into our snapshotter.
solve pending snapshot publishing requirements (e.g., I've turned off persisted receipts during first sync in order to remove possible pain points on bloatnet, but snapshotter may require it for integrity checks)
setup publishing workflow

@sudeepdino008

@sudeepdino008 added flag to disable Receipts integrity check.

@sudeepdino008 added flag to disable Receipts integrity check.

https://github.com/erigontech/devops-metarepo-for-issues/issues/186#issuecomment-3714054702

I'm going to close this one, you should be able to sync bloatnet by:

checkout and build erigon from performance branch
run it with --chain=bloatnet --prune-mode=minimal, it has all bloatnet settings already embedded (networkid, bootnodes, etc).
note: we didn't generate persisted receipts or commitment history on bloatnet, you need to manually deactivate those settings if it gets turned on depending on your pruning mode.
since txs on bloatnet are more dense, even after the first evergreen sync, it'll take a lot of time to complete execution up to chaintip.

there are a few smaller polishing issues I'm tracking down separately, but this one can be considered complete.

note: I wanted the chain name to be perfdevnet-2, but bloatnet was already configured for some assets and the chain name is used as a convention to build a lot of names, so I left it this way... if we do a perfdevnet-3, then we can fix the naming.

Navigation Menu

Repository navigation

[bloatnet] explore initial sync using snapshots for --chain=perf-devnet-2 #17280

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[bloatnet] explore initial sync using snapshots for --chain=perf-devnet-2 #17280

Description

Activity

VBulikov commented on Oct 3, 2025

wmitsuda commented on Dec 1, 2025

wmitsuda commented on Dec 8, 2025

wmitsuda commented on Dec 10, 2025

wmitsuda commented on Dec 11, 2025

wmitsuda commented on Dec 13, 2025

wmitsuda commented on Dec 13, 2025

wmitsuda commented on Dec 18, 2025

taratorio commented on Dec 19, 2025

wmitsuda commented on Dec 19, 2025

wmitsuda commented on Dec 19, 2025

wmitsuda commented on Jan 4, 2026

wmitsuda commented on Jan 5, 2026

AskAlexSharov commented on Jan 5, 2026

sudeepdino008 commented on Jan 6, 2026

wmitsuda commented on Jan 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions