Skip to content

Record the cgroup memory breakdown on every result#1045

Open
MDA2AV wants to merge 1 commit into
mainfrom
record-memory-breakdown
Open

Record the cgroup memory breakdown on every result#1045
MDA2AV wants to merge 1 commit into
mainfrom
record-memory-breakdown

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Groundwork for #1015. The issue argues the published memory figures include the Linux page cache and so overstate what applications actually use. I tried to reproduce that first, and it changed the picture — so this records the evidence instead of acting on the assumption.

What the measurements say

docker stats, which the harness already uses, reports memory.current minus inactive_file — it does subtract page cache:

container memory.current page cache docker stats
holding 512 MiB of file cache 527 MiB 512 MiB 15 MiB
re-reading those files continuously 530 MiB 512 MiB 18 MiB

The second row matters: the kernel keeps sequential reads on the inactive list, so nothing gets promoted to active_file and nothing extra is counted. I also confirmed the sampler polls only the framework's own container — no Postgres, Redis or load generator is summed in.

So cache doesn't explain the gigabyte figures, and two things remain unmeasured:

  1. Memory the application really does allocate at 4096 concurrent connections.
  2. Kernel socket buffers — cgroup v2 charges sock to the container, so docker stats includes them. At 4096 high-throughput connections that is not a rounding error.

Worth noting too: the issue's comparison column came from a local 8-core wrk run, well below the 512/4096 connections used on the bench box, so some of that gap is load rather than accounting. And STATS_PEAK_MEM takes the peak across the run, so a single spike sets the published number.

What this PR does

Redefining "memory used" on a guess would move every published number and the composite's memory factor. So instead it records what the number is made of:

"memory_detail": {"current":264.4,"anon":120.2,"file":56.0,"active_file":1.2,
                  "inactive_file":54.8,"sock":41.0,"slab":7.3,"kernel":7.5}

MiB, read from the container's cgroup at the snapshot where peak memory was observed — so it describes the same instant the memory field reports, rather than a max-per-field mixture of different moments.

Nothing changes yet

  • memory is untouched
  • memory_detail is not in the generator's BASE_FIELDS, and data.js regenerates byte-identical — the leaderboard and every score are unaffected
  • it rides into site/data/results/<framework>.json because rebuild_site_data.py stores result rows verbatim, which is what makes it analysable later

Degrades quietly: cgroup layout varies by driver and version, so the path is probed and skipped if unreadable. Verified the emitted JSON parses both with the field and without it.

After a round runs with this

We'll be able to say whether those GiB are anon, sock or active_file — and then decide what the metric should mean, with data instead of inference.

🤖 Generated with Claude Code

Groundwork for #1015. The issue argues the published memory figures include
the Linux page cache and so overstate what applications use. Checking that
first changed the picture, so this records the evidence rather than acting on
the assumption.

What the measurements show:

`docker stats`, which the harness already uses, reports memory.current minus
inactive_file - so it does subtract page cache. A container holding 512 MiB of
cache reports 15 MiB. Re-reading those files continuously does not change it:
the kernel keeps sequential reads on the inactive list, so nothing is
promoted and nothing extra is counted. The sampler also polls only the
framework's own container, so no sidecar or load generator is summed in.

That leaves the gigabyte figures unexplained by cache, and two candidates
unmeasured: memory the application really does allocate at 4096 concurrent
connections, and kernel socket buffers, which cgroup v2 charges to the
container and `docker stats` therefore includes. The comparison numbers in
the issue also come from a local 8-core wrk run, far below the 512/4096
connections used on the bench box, so part of the gap is load rather than
accounting.

Rather than redefine "memory used" on a guess - a change that would move
every published number and the composite's memory factor - this records what
the number is made of:

  "memory_detail": {"current":264.4,"anon":120.2,"file":56.0,"active_file":1.2,
                    "inactive_file":54.8,"sock":41.0,"slab":7.3,"kernel":7.5}

MiB, read from the container's cgroup at the snapshot where peak memory was
observed, so it describes the same instant the `memory` field reports rather
than a max-per-field mixture of different moments.

Nothing consumes it yet. `memory` is unchanged, memory_detail is not in the
generator's BASE_FIELDS, and data.js regenerates byte-identical - so the
leaderboard and every score are untouched. It rides along into
site/data/results/<framework>.json because rebuild_site_data.py stores result
rows verbatim, which is what makes it available to analyse later.

Degrades quietly: cgroup layout varies with driver and version, so the path
is probed and skipped if unreadable. Verified the emitted JSON parses both
with the field and without it.
@MDA2AV MDA2AV linked an issue Jul 24, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory usage calculation is incorrect

1 participant