Skip to content

Report errors, memory and cost alongside the latency numbers - #20

Merged
lgoyal6 merged 1 commit into
mainfrom
capability/bench-errors-memory-cost
Sep 6, 2026
Merged

lgoyal6 merged 1 commit into
mainfrom
capability/bench-errors-memory-cost

Conversation

@lgoyal6

@lgoyal6 lgoyal6 commented Sep 6, 2026

Copy link
Copy Markdown
Owner

The C02 columns from .agent-work/wt-c02c08. bench/ was untouched by #19, so this applied without conflict.

Why

A latency table never says three things on its own: what failed while the numbers were being produced, how much memory the process held, and what the work cost. A p999 next to an unreported delivery failure is not a measurement of the transport, it is a measurement of the messages that happened to arrive.

Errors, per role, counted rather than dropped:

  • The receiver silently returned on a payload too short to hold the header, so a framing bug would have surfaced as a slightly lower delivery ratio and nothing else.
  • A sequence that does not advance is a duplicate or a reorder. The receiver cannot tell which from one number, so it reports the count and does not claim to know.
  • Offered minus received is the error a latency table hides best: a message that never arrived contributes no sample, so dropping it improves every percentile printed above it.
  • The request-reply client is closed loop with one outstanding, so the reply to request N must be N. A stale or duplicate reply still set got_reply and still pushed a sample, timed from the wrong request. That mismatch is counted, and so is the wait cap firing, which the old loop passed over silently: a wedged transport produced a short run and a clean-looking percentile table.
  • The sender's error is offered work it never handed to the transport before the wall cap; reporting only sent hides the shortfall.

Memory is peak and steady RSS together. ru_maxrss is bytes on Darwin and kilobytes on Linux, and getting that wrong is silently wrong by three orders of magnitude in the flattering direction on exactly one of the two platforms, so the trap is named in a comment instead of guessed at.

Cost is CPU seconds, the axis that matters for a poll/tick transport with no event-loop driver: both roles spin, so wall time says how long the run lasted and CPU time says what it burned to last that long. Involuntary context switches sit next to it so a tail from a busy machine can be told apart from a tail from the transport. The USD figure is arithmetic on cpu_s, so the rate is printed with it.

resource.h is deliberately separate from common.h: common.h is the wire format and schedule shared by the taut, TCP and ENet binaries and documented in docs/BENCHMARKS.md, while this is process accounting and writes its own CSV beside the latency one, so no documented schema changes.

Verification, and its limits

bench/ is Linux-only in CMake and no CI job builds it, so neither the normal local build nor CI compiles this file. Rather than land it unchecked:

  • latency_bench.cc compiles clean under the project's full strict set (-Werror -Wpedantic -Wshadow -Wconversion -Wsign-conversion -Wold-style-cast -Wcast-align -Wdouble-promotion) on this machine.
  • Both platform branches of resource.h were compiled and run, not just the local one. The Darwin branch reads a real RSS and emits a well-formed CSV row; the Linux branch was compiled by forcing its arm of the conditional and exercised for header, row and path behaviour. resource_path rewrites the extension as documented.
  • The library suite is unaffected: 61/61.

What is not verified is an end-to-end benchmark run over real sockets, which needs Linux and netem.

🤖 Generated with Claude Code

A latency table never says three things on its own: what failed while the
numbers were being produced, how much memory the process held to produce
them, and what the work cost. A p999 next to an unreported delivery failure
is not a measurement of the transport, it is a measurement of the messages
that happened to arrive.

Errors, per role, counted rather than dropped:

- The receiver silently `return`ed on a payload too short to carry the
  header, so a framing bug would have shown up as a slightly lower delivery
  ratio and nothing else. Now counted.
- A sequence number that does not advance is a duplicate or a reorder. The
  receiver cannot tell which from one number, so it reports the count and
  does not claim to know which.
- Offered minus received is the error a latency table hides best: a message
  that never arrived contributes no sample, so dropping it *improves* every
  percentile in the row above it.
- The request-reply client is closed loop with one outstanding, so the
  reply to request N must be N. A stale or duplicate reply still set
  got_reply and still pushed a sample, timed from the wrong request. The
  mismatch is counted, and so is the wait cap firing, which the old loop
  passed over silently: a wedged transport produced a short run and a
  clean-looking percentile table.
- The sender's error is offered work it never handed to the transport
  before the wall cap. In open-loop mode the schedule says how many
  arrivals the run should have produced, and reporting only `sent` hides
  the shortfall.

Memory is peak and steady RSS together, so a transient spike can be told
apart from a working set that stays resident. ru_maxrss is bytes on Darwin
and kilobytes on Linux; that trap is named in a comment rather than
papered over with a guessed factor of 1024, because getting it wrong is
silently wrong by three orders of magnitude in the flattering direction on
exactly one of the two platforms.

Cost is CPU seconds, which is the axis that matters for a poll/tick
transport with no event-loop driver: both roles spin, so wall time says how
long the run lasted and CPU time says what it burned to last that long.
Involuntary context switches are reported next to it so a tail that came
from a busy machine can be told apart from a tail that came from the
transport. The USD figure is arithmetic on cpu_s rather than a measurement,
so the rate is printed with it.

resource.h is deliberately separate from common.h. common.h is the wire
format and the schedule, shared by the taut, TCP and ENet binaries and
documented in docs/BENCHMARKS.md; this is process accounting, and it writes
to its own CSV beside the latency CSV so no documented schema changes.

Verification, and its limits. bench/ is Linux-only in CMake and no CI job
builds it, so neither the normal local build nor CI compiles this file.
Instead: latency_bench.cc compiles clean under the project's full strict
set, -Werror -Wpedantic -Wshadow -Wconversion -Wsign-conversion
-Wold-style-cast -Wcast-align -Wdouble-promotion, on this machine. Both
platform branches of resource.h were compiled and run, not just the local
one: the Darwin branch reads a real RSS and emits a well-formed CSV row,
and the Linux branch was compiled by forcing its arm of the conditional and
exercised for header, row and path behaviour. resource_path rewrites the
extension as documented. The library suite is unaffected at 61/61.
@lgoyal6
lgoyal6 merged commit 33be691 into main Sep 6, 2026
15 checks passed
@lgoyal6
lgoyal6 deleted the capability/bench-errors-memory-cost branch September 6, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant