[bloatnet] experiment: change StepSize

tool to allow us to change our snapshot files from 1 step size to another - can be achieved just by renaming the files (as long as step sizes are multiples of 2) and deleting chaindata
can be fully automated ie on startup we can have a flag for step size -> detect what our current step size is in existing files -> apply the necessary renames -> delete chaindata -> continue
this should make it easy to experiment with currentStepSize/2, currentStepSize/4, currentStepSize/8, currentStepSize/16, etc. and gathering metrics to assess performance differences

taratorio added Imp2 as Importance

some status update here:

did the initial script, that was the easy part.
besides renaming, it requires some small adjustments to constants in config3.go.

weird stuff that needs to be done:

"delete chaindata" seems to be broken somehow in recent main: "delete chaindata" is broken on recent main #17383
- unrelated to this one, but made me waste some time. need to track it down anyway bc rebasing steps requires cleaning up chaindata
running against old code the execution seems to restart from 0, which means there may be additional hardcoded assumptions in code about step geometry that I'm going to debug.

there is actually a limitation to this approach with the current default step size because it is defined as 1_562_500; it can be divided by 4, but not by 8, etc...

we can use the rename approach to, let's say, divide the step by 20, so 1 becomes 20 in the filenames, but our "piramid-merging" algorithm may go crazy.

I actually want to divide by 20 on bloatnet experiment since the gas limit there went 30M -> 500M, let me see if we can ignore background merging eventual errors, but we may want to revisit the "piramid-merging" algorithm or make a "not-just-rename-but-total-rewrite" step rebasing implementation.

erigon seg step-rebase tool (#17394)

@AskAlexSharov

I wrote my conclusions for the first round of tests: https://hackmd.io/@wmitsuda/BkFmxXL6ge

TLDR; I didn't notice improvement on execution time; aggressive step size reduction was effective in limiting chaindata growth though.

happy to hear about improvements in the methodology if we want to do another round of tests, or we consider this done for bloatnet purposes.

cc: @AskAlexSharov

I wrote my conclusions for the first round of tests: https://hackmd.io/@wmitsuda/BkFmxXL6ge

TLDR; I didn't notice improvement on execution time; aggressive step size reduction was effective in limiting chaindata growth though.

happy to hear about improvements in the methodology if we want to do another round of tests, or we consider this done for bloatnet purposes.

cc: @AskAlexSharov

I see in the experiments you only run for ~1700 blocks? For me that was not enough to see the bloat and drastic degradation. The bloat used to happen after letting the node run for ~1 day which is a lot more than 1700 blocks. Also it really really depends which blocks you executed from the bloated ones. I suggest to execute all from 22600001 (the fork point or earlier doesn't have to be exact) to 22877864 (slot https://dora.perf-devnet-2.ethpandaops.io/slot/80112) which is where the bloating stopped (as per the screenshot below) - if you go on Dora you will see that all blocks before it are pretty much at 500MGas. Also curious what --loop.sync.block.limit did you use in your experiment?

oh, nice.

worst case of chaindata size get visible after pruning couple steps from db (after couple new files production)

my head expected 2x reduction on stepSize/2 :-) plz add to hackmd (to any case):

./build/bin/mdbx_stat -efa /erigon-data/chaindata/   | awk '
    BEGIN { pagesize = 4096 }
    /^  Pagesize:/ { pagesize = $2 }
    /^Status of/ { table = $3 }
    /Branch pages:/ { branch = $3 }
    /Leaf pages:/ { leaf = $3 }
    /Overflow pages:/ { overflow = $3 }
    /Entries:/ {
      total_pages = branch + leaf + overflow
      size_gb = (total_pages * pagesize) / (1024^3)
      printf "%-30s %.3fG\n", table, size_gb
    }
  ' | grep -v '0.000G'

About overflow pages: dupsort table can't have overflow pages. Because it has hard-limit on value-length: 2kb (enough for Storage table).

2KB == sum of all dup values of same key? or 2KB each val of same key?

in StorageVals table - yes. But i think value length there is longer than keys length.

it is not:

    - Processing table StorageHistoryVals...
         key-value kind: usual-key => multi-value, flags: dupsort (0x04)
         key length density: 52=49407065
         value length density: 8=49334039, 10=1507, 11=6703, 12=31313, 14=427, 15=2243, 24=2, 39=299, 40=12097
       = summary: 49407065 records, 40995 dups, 2569167380 key's bytes, 396013041 data's bytes, 0 problem(s)
    - Processing table StorageVals...
         key-value kind: usual-key => multi-value, flags: dupsort (0x04)
         key length density: 52=49368633
         value length density: 10=32, 11=547, 12=4910435, 13=2405, 24=2, 37=69, 38=598, 39=145687, 40=36934317
       = summary: 49368633 records, 2563 dups, 2567168916 key's bytes, 1657549382 data's bytes, 0 problem(s)

About overflow pages: dupsort table can't have overflow pages. Because it has hard-limit on value-length: 2kb (enough for Storage table).

2KB == sum of all dup values of same key? or 2KB each val of same key?

2KB each val of same key

I remember 1 problem of using dupsort for Storage - we can use it for StorageDomain, but can’t do same for CommitmentDomain - because CommitmentDomain has different key lengths (sort order will break if dupsort them).

if you run mdbx_chk -vvv i guess it can show you how much free space is in each DB Table.

I assume it is the following section:
 - Page allocation:
      backed by file: 20023296 pages (29.8%), 47085568 left to boundary (70.2%)
      used: 7377420 page(s), 36.8% of backed, 11.0% of boundary
      remained: 12045332 page(s) (60.2%) of backed, 59130900 to boundary (88.1% of boundary)
      reclaimable: 600120 (3.0% of backed, 0.9% of boundary), GC 600544 (3.0% of backed, 0.9% of boundary)
      detained by reader(s): 424 (0.0% of backed, 0.0% of boundary), 1 reader(s), lag 0
      allocated: 7977964 page(s), 39.8% of backed, 11.9% of boundary
      available: 12645452 page(s) (63.2%) of backed, 59731020 to boundary (89.0% of boundary)
    = upto 67108864 pages, backed 20023296 (29.8%), allocated 7977964 (11.9%), available 59731020 (89.0%)
I ran a "much more aggressive step size" just to see what happens: I used "step size / 100", then in < 12 hours I got a chaindata of 320GB, but examining the DB, it is only using ~120GB of data.

in the mdb_chk output I see backed matches the 320GB mdbx.dat, the allocated matches the ~120GB "real" usage.

and the ~200GB extra allocated space seems to be the "available", but where does that come from? i.e. something mdbx or some erigon behavior lead to that?

I think mdbx_chk can print statistic for each DBI too - and i think can see there. mdbx_chk is not mdbx_stat.

I know “reclaimable” is FreeList.

Domain statistics support tool (#17795)

@wmitsuda

@wmitsuda look - maybe this can reduce db size on bloatnet - try: #17962

Giulio2002 changed Start date from 18/10/2025 to 29/10/2025

Giulio2002 changed Due date from 14/12/2025 to 25/12/2025

just dumping some personal notes about bloatnet so I don't lose those info:

shadow fork started at 22_600_000
between 22_600_000 and 22_811_745: offline bloat through custom scripts
after 22_811_745: new PoS chain started; bloat via spamoor scripts

Merge v3.3.1 (#19)

let's closing this one as:

the step size experiment for bloatnet has become inconclusive as there were other bottlenecks
we'll continue the experiment for regular mainnet
tooling for managing step size from an existing node was 99% done

Navigation Menu

Repository navigation

[bloatnet] experiment: change StepSize #16765

46 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[bloatnet] experiment: change StepSize #16765

Description

Activity

VBulikov commented on Aug 21, 2025

wmitsuda commented on Oct 8, 2025

wmitsuda commented on Oct 9, 2025

wmitsuda commented on Oct 10, 2025

taratorio commented on Oct 10, 2025

AskAlexSharov commented on Oct 10, 2025

46 remaining items

wmitsuda commented on Nov 6, 2025

wmitsuda commented on Nov 6, 2025

AskAlexSharov commented on Nov 7, 2025

AskAlexSharov commented on Nov 7, 2025

AskAlexSharov commented on Nov 7, 2025

AskAlexSharov commented on Nov 20, 2025

VBulikov commented on Nov 25, 2025

VBulikov commented on Nov 25, 2025

wmitsuda commented on Dec 4, 2025

wmitsuda commented on Jan 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions