Record the cgroup memory breakdown on every result#1045
Open
MDA2AV wants to merge 1 commit into
Open
Conversation
Groundwork for #1015. The issue argues the published memory figures include the Linux page cache and so overstate what applications use. Checking that first changed the picture, so this records the evidence rather than acting on the assumption. What the measurements show: `docker stats`, which the harness already uses, reports memory.current minus inactive_file - so it does subtract page cache. A container holding 512 MiB of cache reports 15 MiB. Re-reading those files continuously does not change it: the kernel keeps sequential reads on the inactive list, so nothing is promoted and nothing extra is counted. The sampler also polls only the framework's own container, so no sidecar or load generator is summed in. That leaves the gigabyte figures unexplained by cache, and two candidates unmeasured: memory the application really does allocate at 4096 concurrent connections, and kernel socket buffers, which cgroup v2 charges to the container and `docker stats` therefore includes. The comparison numbers in the issue also come from a local 8-core wrk run, far below the 512/4096 connections used on the bench box, so part of the gap is load rather than accounting. Rather than redefine "memory used" on a guess - a change that would move every published number and the composite's memory factor - this records what the number is made of: "memory_detail": {"current":264.4,"anon":120.2,"file":56.0,"active_file":1.2, "inactive_file":54.8,"sock":41.0,"slab":7.3,"kernel":7.5} MiB, read from the container's cgroup at the snapshot where peak memory was observed, so it describes the same instant the `memory` field reports rather than a max-per-field mixture of different moments. Nothing consumes it yet. `memory` is unchanged, memory_detail is not in the generator's BASE_FIELDS, and data.js regenerates byte-identical - so the leaderboard and every score are untouched. It rides along into site/data/results/<framework>.json because rebuild_site_data.py stores result rows verbatim, which is what makes it available to analyse later. Degrades quietly: cgroup layout varies with driver and version, so the path is probed and skipped if unreadable. Verified the emitted JSON parses both with the field and without it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #1015. The issue argues the published memory figures include the Linux page cache and so overstate what applications actually use. I tried to reproduce that first, and it changed the picture — so this records the evidence instead of acting on the assumption.
What the measurements say
docker stats, which the harness already uses, reportsmemory.currentminusinactive_file— it does subtract page cache:memory.currentdocker statsThe second row matters: the kernel keeps sequential reads on the inactive list, so nothing gets promoted to
active_fileand nothing extra is counted. I also confirmed the sampler polls only the framework's own container — no Postgres, Redis or load generator is summed in.So cache doesn't explain the gigabyte figures, and two things remain unmeasured:
sockto the container, sodocker statsincludes them. At 4096 high-throughput connections that is not a rounding error.Worth noting too: the issue's comparison column came from a local 8-core
wrkrun, well below the 512/4096 connections used on the bench box, so some of that gap is load rather than accounting. AndSTATS_PEAK_MEMtakes the peak across the run, so a single spike sets the published number.What this PR does
Redefining "memory used" on a guess would move every published number and the composite's memory factor. So instead it records what the number is made of:
MiB, read from the container's cgroup at the snapshot where peak memory was observed — so it describes the same instant the
memoryfield reports, rather than a max-per-field mixture of different moments.Nothing changes yet
memoryis untouchedmemory_detailis not in the generator'sBASE_FIELDS, anddata.jsregenerates byte-identical — the leaderboard and every score are unaffectedsite/data/results/<framework>.jsonbecauserebuild_site_data.pystores result rows verbatim, which is what makes it analysable laterDegrades quietly: cgroup layout varies by driver and version, so the path is probed and skipped if unreadable. Verified the emitted JSON parses both with the field and without it.
After a round runs with this
We'll be able to say whether those GiB are
anon,sockoractive_file— and then decide what the metric should mean, with data instead of inference.🤖 Generated with Claude Code