Skip to content

feat(trace): per-LLM-call attribution, and the reports that read it - #1450

Open
rodosingh wants to merge 21 commits into
mainfrom
feat/per-call-llm-attribution-and-reports
Open

rodosingh wants to merge 21 commits into
mainfrom
feat/per-call-llm-attribution-and-reports

Conversation

@rodosingh

@rodosingh rodosingh commented Sep 8, 2026

Copy link
Copy Markdown
Member

What this is

Right now, when a Hyperloom run finishes, we can tell you what the whole run cost —
but not what any individual piece of it cost. The ledger records one row per turn,
and a turn can be many API calls. So questions like "which task is eating our tokens?"
or "is this subtask cheap enough to hand to a smaller model?" have no answer in the data.

This PR adds that missing layer: one row per LLM API call, with the time, input
tokens (ISL), output tokens (OSL), thinking tokens, cache reads, tool calls and a
derived dollar cost — all hung off the task path that produced it. Plus two report
generators that turn those rows into a readable phase → task → subtask → tool tree.

Why it matters

Two things came out of this the moment it started recording:

  • Sub-agent spend was invisible. Specialist calls carried no call_id and no
    model name, so they got double-counted against their parent and priced at $0.
    Both bugs are fixed here. On a real session that correction moved the bill by ~5x.
  • We could finally see where the money goes. On the Qwen3-14B-FP8 run, the GEAK
    phase alone is 74% of the session's LLM bill — which is not something anyone would
    have guessed from the turn-level numbers.

What's in it

Piece What it does
trace/call_detail.py Writes the per-call rows beside the existing turn rows
trace/pricing.py Prices a call from a rate card shipped in-repo, so costs are reproducible and not hardcoded
trace/geak_harvest.py Pulls GEAK's own spend out of its Claude Code transcripts
tools/dump_llm_call_report.py Renders the Hyperloom-side report
tools/dump_geak_call_report.py Renders the GEAK-side report
tools/render_geak_html_report.py The GEAK run as a self-contained HTML page
tools/render_hyperloom_html_report.py The whole session as an HTML page — every phase, not just GEAK
tools/_report_html.py Formatting and stylesheet shared by both pages
tools/_report_agents.py The per-phase deep dive shared by both pages
skills/{hyperloom,geak}-llm-report/SKILL.md So you don't have to remember the flags

Nothing existing changes shape: the turn rows are untouched, and the per-call rows are
additive. Totals still reconcile — that invariant is covered by the tests.

The session-wide page

The GEAK HTML report answers where one phase's time and money went. A session also
spends in FRAMEWORK_AGENT, PRELUDE, SWEEP and WARM_REPLAY, and none of that
had a page. render_hyperloom_html_report covers all of them:

PYTHONPATH=src python3 -m hyperloom.inference_optimizer.tools.render_hyperloom_html_report \
    --session-dir <SESSION_DIR> --output <SESSION_DIR>/reports/hyperloom_report.html

It reads the same two ledgers as dump_llm_call_report — reusing its tree and its
counting rule, so every derived figure agrees with the tree totals — and joins them
against session_breakdown.json to show what the spend bought: the outcome ladder
from baseline to final with every KEEP/REJECT decision, dollars per +1% gain, the ISL
growth curve across a conversation, and the model mix.

The join is only as good as the map, so the page refuses to guess:

  • Spend reaches a gain source only through a declared phase map. A phase outside it
    renders not attributed rather than being credited to whichever source is nearest.
  • A session with no session_breakdown.json renders not recorded, never +0.00%.
    "Not measured" and "gained nothing" are different claims.
  • Unpriced calls stay out of the USD total, which makes that total a floor — the
    coverage banner says so, with counts.
  • Growth-curve buckets with fewer than three calls are dropped. A median of one call
    is not a median.

Inside a phase, on both pages

The session page told you which phase cost the most and then stopped. The GEAK page,
on the same data one level down, told you where inside a phase the money went. Two
pages, one of them useful.

tools/_report_agents.py now holds that machinery and both pages render through it,
so a column means the same thing on either: cost by position in the conversation, how
few agents carry the phase, what tools the work consisted of, an agent roster where
every row drills into its own API calls, and the delegation-signals table.

Identity is the one thing the two ledgers genuinely disagree on, so it is injected
rather than assumed. GEAK's ledger carries prompt text and no label; Hyperloom's
carries a role field that some producers write and others do not — on a real
gpt-oss-120b session, 27 of 845 turn rows. So a Hyperloom agent is its task path
(specialist/<instance>, populated on 830 of those rows) and its name is the role
where one was recorded. An agent with neither is shown unlabelled, never named from
its id, and the coverage banner counts them.

KERNEL_AGENT is the phase this was worth doing for: 55.6% of that session's bill and,
seen from the session ledger alone, a single lumped agent. Where the run left a
geak_calls.jsonl behind, GEAK's own per-call ledger is grafted in beneath it — eval
dir read from geak/result.json, then handoff.json, then state.json, never guessed
from the directory layout — and the harvested ext/geak-* rows are dropped in the same
step, because both describe the same API calls and keeping both would bill GEAK twice.

A GEAK build without the trace mirror leaves no ledger to graft. The banner then says
so and names the tool that rebuilds it, rather than showing a thin phase and letting it
read as a cheap one.

A note on reading the output

Every report leads with a Coverage line. If some calls couldn't be priced or matched
to a task, that line says so, and you should read it before quoting any number from the
report. A missing call is reported as missing, never as zero.

Testing

  • 63 new unit tests across call detail, pricing, GEAK harvest and both report tools —
    all passing.

  • ruff check and ruff format --check clean on the new and touched modules.

  • Verified end-to-end against a real finished run (Qwen3-14B-FP8): 5,140 calls,
    109 agents, ISL 579,743,865, OSL 2,818,841, $602.81, 0 unpriced.

  • The shared-chrome refactor is provably a no-op: re-rendering the shipped
    Qwen3-14B-FP8 GEAK report from it is byte-identical apart from the generation
    timestamp. The deep-dive extraction was held to the same bar and passes it — the
    same 1.8 MB page, byte for byte, before and after.

  • The session page rendered against the finished gpt-oss-120b run: 441 KB, and its
    per-phase spend sums to $184.36, which reconciles with that run's independently
    computed cost anatomy. The graft's absent-ledger branch is exercised by that same
    run, whose GEAK cycle predates the trace mirror.

  • The session-wide page cross-checked against dump_llm_call_report's JSON on a real
    gpt-oss-120b session — 248 calls, 203 turns, ISL 15,232,743, OSL 167,140, $3.669404,
    and the same per-phase shares — and independently recomputed straight from the JSONL
    by a script that imports neither tool, which agrees on every figure. A second session
    with no session_breakdown.json exercises the "not recorded" branch.

One bug this shook out

Building the second reader against the first is what exposed it. The turn-to-detail
join used the raw call_id; where a producer wrote none — the specialist and critic
runners, on sessions recorded before they learned to emit one — the id collapsed to
"" on both sides and matched nothing, so the detail row was booked as an orphan
and its turn was booked as undetailed and counted a second time.

On that gpt-oss-120b session, 195 of 247 detail rows had no call id and every one
matched a turn row on (session_id, task_path, turn) with byte-identical token
fields. Calls 443 → 248, ISL 27,131,576 → 15,232,743, OSL 332,731 → 167,140, agent
time 1:54:43 → 1:00:20, unpriced 390 → 195. USD is unaffected — the duplicated
rows were all unpriced, so the cost total and the per-phase shares were right all
along. call_identity() now makes the fallback key explicit and both readers go
through it.

🤖 Generated with Claude Code

rodosingh and others added 7 commits September 8, 2026 20:11
llm_calls.jsonl records one row per agentic turn, which answers "what did this
task cost" but not "which call inside it cost that". A turn that ran fifteen
tool round-trips was a single opaque number, so there was no way to tell an
expensive reasoning call from fourteen cheap tool-result continuations -- which
is exactly the question you have to answer before deciding what can be
delegated to a smaller model.

Write llm_calls_detail.jsonl beside it: one row per API call, joined to its
turn on call_id. The turn rows are unchanged, and the detail rows are their
expansion, never an addition -- readers must not sum the two files.

Sub-agent spend was the sharp edge here. Specialist calls carried neither a
call_id nor a model, so they were attributed to their caller's turn and priced
at nothing. Threading task_scope through the specialist runner, the dispatcher,
the sub-agent runner and the critic gives every call an owner.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Not every backend returns a provider cost, and a run that mixes ones that do
with ones that do not cannot be totalled without a fallback. Derive the missing
figures from a rate card shipped as package data.

The two sources stay labelled: provider is the vendor's own charge, derived is
this table applied to token counts, and a total spanning both is an estimate
that says so. An unpriced model is excluded from the total and counted in
coverage -- never silently priced at zero, which would read as free rather than
as unknown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GEAK runs out of process and keeps no ledger of its own: its calls are recorded
only in the Claude Code transcripts of the workflow that drove it. Left
unharvested, the majority of a session's LLM bill is simply absent from the
session's own trace, and KERNEL_AGENT reads as nearly free.

Harvest those transcripts into an ext/geak-*.jsonl shard. Out-of-process
producers write their own shard rather than appending to llm_calls.jsonl,
because that append is not atomic across processes.

Two accounting inversions in the transcripts have to be respected or the
numbers come out wrong in opposite directions: thinking is a subset of output
there (the opposite of this ledger, where it sits beside output), and messages
repeat across streamed blocks, so rows must be deduplicated by message id --
without which the bill overstates by roughly 60%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Turn the ledgers into an answerable document: a phase -> task -> subtask tree
for a Hyperloom session, and a phase -> agent -> API call -> tool call tree for
a GEAK run, each carrying ISL/OSL, tokens, wall-clock, cost and tool calls.
--join-hyperloom nests the GEAK tree under the session's KERNEL_AGENT phase so
one document spans both hierarchies, dropping rows the harvester already wrote
into that session's shard so the run is not billed twice.

Both lead with a Coverage section rather than burying the caveats, because
every number in these reports has a qualifier -- unpriced calls, derived versus
provider cost, apportioned timing, turns with no detail rows -- and a total
quoted without them is wrong in a way that is invisible.

--include-text writes a sidecar with the prompt, thinking and output text
beside every metric. The tree says where the money went; the sidecar says what
was being asked, which is what you actually need to judge whether a task is
cheap enough for a smaller model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tools are only half the capability; the other half is knowing which numbers
in their output can be quoted and which cannot. Both skills lead with reading
Coverage first, and spell out the traps that make a confident wrong answer easy
here: ISL is dominated by cache reads and is not comparable to a "tokens"
figure from anywhere else; thinking is inside output in the GEAK hierarchy and
beside it in Hyperloom's; summed agent durations exceed wall-clock wherever the
script fanned out; and a report with no geak subtree is understating the
session, not describing a cheap one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The invariants that keep the totals honest are the ones under test: detail rows
expand turn rows rather than adding to them; an unpriced model is excluded and
counted, not zeroed; the GEAK harvest deduplicates by message id and treats
thinking as a subset of output; and a specialist call carries its own call_id
and model so it is neither double-counted nor free.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The new per-call fields are useless in Langfuse if only the local reports can
see them. Project the USD breakdown onto cost_details and put task_path,
task_depth, api_calls, tool_call_count, the timing split and cost_source into
generation metadata.

An unpriced row contributes no cost keys at all, so Langfuse shows a gap rather
than a confident zero -- the same rule the reports follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rodosingh
rodosingh requested a review from a team as a code owner September 8, 2026 20:12
@rodosingh rodosingh added domain:inference Related to inference optimization type:feature New capability or improvement labels Sep 8, 2026
Comment on lines +782 to +783
"| Cost | Derived from the shipped rate card. Claude Code records no "
"provider cost, so no figure here is provider-reported. |",
Comment on lines +787 to +789
"| Per-call time | Inferred from consecutive message timestamps; an "
"agent's first call absorbs its queue wait. Agent wall-clock is the "
"runtime's own `durationMs`. |",
Comment on lines +790 to +791
"| Thinking time | Not recorded anywhere. Thinking *tokens* are exact; "
"thinking *seconds* are 0 by construction. |",
Comment on lines +792 to +793
"| Thinking tokens | A subset of `output_tokens` in these transcripts, "
"so OSL is output alone, not output + thinking. |",
Comment on lines +797 to +799
"| Agents in parallel | The summed agent durations exceed the run's "
"wall-clock wherever the script fanned out; the record's `durationMs` "
"is the wall-clock. |",
Comment on lines +814 to +818
"Tool calls, agent count and per-agent duration reconcile exactly. The "
"record's token figure does not, and is not expected to: it sums the "
"`tokens` field of the progress entries, a streamed snapshot that lands "
"on either side of the transcript sums agent by agent. The transcripts "
"are authoritative for tokens.",
Comment on lines +560 to +562
"Thinking tokens sit beside the visible output tokens, not inside them: "
"`OSL` is output + thinking, and the cost total is input + output + "
"thinking + cache.",
Comment on lines +566 to +567
"| Node | Calls | $ share | ISL | OSL | Tok in | Tok think | Tok out "
"| Time s | Think s | Out s | $ total | $ think | $ out | Tools |",
log.warning("llm_pricing: cannot read %s (%s); costs will be unavailable", path, exc)
except Exception as exc: # noqa: BLE001 -- yaml raises its own error tree
log.warning("llm_pricing: %s is not valid YAML (%s)", path, exc)
_rates_cache = rates
Comment thread src/hyperloom/orchestrator/trace/pricing.py Fixed
Comment on lines +782 to +783
"| Cost | Derived from the shipped rate card. Claude Code records no "
"provider cost, so no figure here is provider-reported. |",
Comment on lines +787 to +789
"| Per-call time | Inferred from consecutive message timestamps; an "
"agent's first call absorbs its queue wait. Agent wall-clock is the "
"runtime's own `durationMs`. |",
Comment on lines +790 to +791
"| Thinking time | Not recorded anywhere. Thinking *tokens* are exact; "
"thinking *seconds* are 0 by construction. |",
Comment on lines +792 to +793
"| Thinking tokens | A subset of `output_tokens` in these transcripts, "
"so OSL is output alone, not output + thinking. |",
Comment on lines +797 to +799
"| Agents in parallel | The summed agent durations exceed the run's "
"wall-clock wherever the script fanned out; the record's `durationMs` "
"is the wall-clock. |",
Comment on lines +814 to +818
"Tool calls, agent count and per-agent duration reconcile exactly. The "
"record's token figure does not, and is not expected to: it sums the "
"`tokens` field of the progress entries, a streamed snapshot that lands "
"on either side of the transcript sums agent by agent. The transcripts "
"are authoritative for tokens.",
Comment on lines +560 to +562
"Thinking tokens sit beside the visible output tokens, not inside them: "
"`OSL` is output + thinking, and the cost total is input + output + "
"thinking + cache.",
Comment on lines +566 to +567
"| Node | Calls | $ share | ISL | OSL | Tok in | Tok think | Tok out "
"| Time s | Think s | Out s | $ total | $ think | $ out | Tools |",
log.warning("llm_pricing: cannot read %s (%s); costs will be unavailable", path, exc)
except Exception as exc: # noqa: BLE001 -- yaml raises its own error tree
log.warning("llm_pricing: %s is not valid YAML (%s)", path, exc)
_rates_cache = rates
except Exception as exc: # noqa: BLE001 -- yaml raises its own error tree
log.warning("llm_pricing: %s is not valid YAML (%s)", path, exc)
_rates_cache = rates
_rates_cache_key = key
dump_geak_call_report produces a call tree, which is the right shape for
"walk me through this run" and the wrong shape for "where is the money going,
and is any of it wasted". On a real run that tree is thousands of rows deep and
nobody reads it.

This renders the same ledger as a page organised around the questions people
actually ask: what did the run cost, what did each phase buy for its share of
that, and -- inside the expensive phases -- where within a conversation the cost
lands. That last part is the interesting one: every agent's conversation is
stretched onto the same 0-9 position scale, which shows that calls get steadily
more expensive the longer a conversation runs, because each call re-sends the
conversation so far. On the Qwen3-14B-FP8 run the median input length grows
3.2x from a conversation's first tenth to its last.

Two rules the whole file is built around:

* nothing is modelled or extrapolated -- every number is computed from rows in
  the ledger, and a coverage banner states what the ledger does not contain;
* a quantity that was not recorded renders as "not measured", never as zero,
  because "we did not measure it" and "it was nothing" are different claims.

Output is self-contained: no external scripts, styles or fonts, so the page
still works after being copied to shared storage.
…pure ASCII

Two things the report was getting wrong.

It answered "what did this cost" well and "what did it buy" only in passing --
the gains were buried in a column of a cost-ranked table. A run exists to make
something faster, so the page now opens with a throughput ladder in run order:
baseline, then each phase's from/to tok/s, its gain, and its share of the
summed measured gain. On the Qwen3-14B-FP8 run that reads P4 ConfigSweep 41.5%
and P6 TuningSkillset 58.5% of +3,238 tok/s, with the kernel phases stated
plainly as a measured zero rather than left blank -- they ran, they were
benchmarked, and they did not beat what was there. The share column is
arithmetic on recorded numbers, not an attribution model, and the page says so
and names the handoff seam that stops it reconciling with the end-to-end
+114.64%.

Second, the page is written UTF-8 and declares it, but it gets opened from
shared storage, out of archives, and through viewers that ignore the
declaration -- where an em dash arrives as mojibake and gives the reader no
clue why. Non-ASCII now travels as numeric entities, so the bytes are pure
ASCII and cannot be mis-decoded. A test pins it.
Comment thread src/hyperloom/inference_optimizer/tools/render_geak_html_report.py Fixed
Comment thread src/hyperloom/inference_optimizer/tools/render_geak_html_report.py Fixed
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

Hyperloom Forge E2E — ❌ Timeout

item value
result ❌ Timeout
example triton-softmax-forge-loop (max_hours=1.0, max_iters=100)
resources 1× GPU
workspace control-plan-hyperloom-ci
PR branch feat/per-call-llm-attribution-and-reports
commit a74875b7221e05a98a198c506da182389e68f12a
session_id 7bdecc9b-742f-4aa4-8d07-e53dad2fe29d
queue → dispatch
run time
total 1m 9s

Actions run

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

CI E2E report — ❌ Timeout

item value
result ❌ Timeout
model Qwen/Qwen3-0.6B (dense)
resources 1× GPU, TP=1
PR branch feat/per-call-llm-attribution-and-reports
commit a74875b7221e05a98a198c506da182389e68f12a
session_id 4d45144a-cf92-4665-861f-6a1b5cde934c
queue → dispatch 24m 0s
run time 156m 55s
total 180m 55s
reason Timed out — the run never reached a terminal state in time (task stuck, or the GPU stayed queued too long).
detail not terminal after 13200s

details

A HeadKernel phase that validated its own kernel end to end has a real
measured number: integrate_result.json's e2e_delta_pct. The opbench
Amdahl ceiling is only the upper bound that applies when nothing reached
validation, and it reads 0.00% whenever an incumbent library backend
stays fastest -- which made a phase that actually shipped a +3.54% kernel
render as having bought nothing.

The cost table now prefers the A/B and shows cost per +1%, and the
throughput section separates kernels that were validated from kernels
that were not, carrying the harness's own verdict text verbatim rather
than restating it.
… invoked it

The three blocks in 'Inside each phase' sit in a CSS grid, and a grid item
defaults to min-width:auto -- so a table of nowrap cells refuses to shrink
below its min-content width and paints over the column beside it. Let the
items shrink, and give each table its own scroll container so a genuinely
wide table scrolls inside its column instead of across its neighbour.

Separately, call_geak now exports GEAK_INVOKED_BY=hyperloom. GEAK also runs
standalone, the two modes are not comparable, and GEAK uses this to name its
report hl_run_report_<model>.html rather than geak_run_report_<model>.html.
A GEAK build that predates the marker simply ignores it.
Claude Code writes its ledger into a config home the run does not own, so
a run can finish, measure a real result, and have nothing left to read --
which is exactly what happened to the first gpt-oss-120b run when its
container was rebuilt. Refusing to render that run buries a measured
outcome that survived intact.

With only geak_outcome.json present the renderer now warns and produces
the outcome half: the throughput ladder and what each phase measured.
Every spend figure is marked absent rather than shown as zero, the three
sections that would be empty are omitted rather than rendered headed and
rowless, and the header credits only the file actually read. A $0.00 card
beside a real throughput number would say the run was free, which is the
one thing it certainly was not. Both files missing is still an error.
"P8 HeadKernel h0" names the head, not the operator, so a reader could not
tell which kernel the run spent $359 on. The task name is the only place the
operator is identified; it now travels with the phase everywhere it appears:
the spend table, the per-phase deep dive, the cost-vs-result row and the
end-to-end verdict.

Kernel tasks that no ledger phase is named after used to vanish from the page
entirely. They are now listed too, with no cost attributed to them.
Extract the number formatting, bar/sparkline helpers and stylesheet the GEAK
HTML report had inline into _report_html.py, so a second report spells a
dollar, a percentage and a duration exactly the same way. num() keeps its
strict semantics on purpose: a bool is not a token count and NaN is not a
cost, so both are rejected rather than summed.

Rendering the shipped Qwen3-14B-FP8 report from this refactor is
byte-identical to the version on disk apart from the generation timestamp.
The GEAK run report answers where one phase's time and money went. A
Hyperloom session spends in FRAMEWORK_AGENT, PRELUDE, SWEEP and WARM_REPLAY
as well, and until now none of that had a page.

render_hyperloom_html_report reads the same two trace ledgers as
dump_llm_call_report -- and reuses its tree and counting rule, so every
derived figure agrees with the tree totals -- then joins them against
session_breakdown.json to show what the spend bought: the outcome ladder,
dollars per +1% gain, the ISL growth curve across a conversation, and the
model mix.

Honesty rules it enforces, because the join is only as good as the map:

- Spend is attributed to a gain source only through ATTRIBUTION_PHASES. A
  phase outside that map renders 'not attributed' rather than being
  quietly credited to whichever source is nearest.
- A session with no session_breakdown.json renders 'not recorded'. It never
  renders '+0.00%' -- 'not measured' and 'gained nothing' are different
  claims.
- Unpriced calls are excluded from the USD total, which makes that total a
  floor; the coverage banner says so, with the counts.
- Growth-curve buckets holding fewer than three calls are dropped. A median
  of one call is not a median.

Cross-checked against dump_llm_call_report's JSON on a real session: 443
calls, 203 turns, ISL 27,131,576, OSL 332,731, $3.669404, and the same
per-phase shares.
Eleven cases over a fixture session with one detailed turn and one bare
turn. The ones that matter are the refusals: an unmapped phase must render
unattributed rather than credited, a missing breakdown must say 'not
recorded' and never '+0.00%', unpriced calls must be declared a floor, and
main() must exit 2 on a session with no ledger rather than emit an empty
page.
The skill documented only the markdown/JSON tree. Adds the render step and,
more importantly, the caveats a reader has to carry out of the page: the
declared phase map, 'not attributed' vs credited, and 'not recorded' vs
+0.00%.
"""
if isinstance(value, bool) or not isinstance(value, (int, float)):
return 0.0
if value != value or value in (float("inf"), float("-inf")): # NaN / inf
"""
if isinstance(value, bool) or not isinstance(value, (int, float)):
return 0.0
if value != value or value in (float("inf"), float("-inf")): # NaN / inf
…l id

The turn-to-detail join used the raw call_id field. When a producer wrote
none -- the specialist and critic runners, on sessions recorded before they
learned to emit one -- the id fell back to the empty string on both sides,
which matches nothing. So the detail row was booked as an orphan AND its
turn row was booked as undetailed and counted again. The same API call
landed in the totals twice.

Not a small effect. On a real gpt-oss-120b session, 195 of 247 detail rows
had no call id, and every one of them matched a turn row on session, task
path and turn number with byte-identical token fields:

  calls        443 -> 248
  ISL   27,131,576 -> 15,232,743   (11,898,833 counted twice)
  OSL      332,731 -> 167,140
  agent time 1:54:43 -> 1:00:20
  unpriced     390 -> 195
  orphan rows  195 -> 0

USD is unaffected: the duplicated rows were all unpriced, so the cost total
and the per-phase shares were already right.

call_identity() makes the fallback explicit -- session, task path and turn
number identify a turn uniquely and are present on both sides. Both the
tree builder and the HTML report's flat row list go through it, so they
cannot drift apart again.

Also drops the coverage note's guess that orphan rows are 'typically
out-of-process children writing through the ext shards'. That session had
no ext shards at all, so the sentence was asserting a cause the data did
not support.
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

Pre-release E2E — Running (0/10 legs done)

CI_VERSION 1.1.0.dev202609091416+ci · target_gain 100% · commit a74875b7221e05a98a198c506da182389e68f12a

leg verdict detail
baremetal-sglang-12h ⏳ pending running
baremetal-sglang-3h ⏳ pending running
baremetal-vllm-12h ⏳ pending running
baremetal-vllm-3h ⏳ pending running
docker-sglang-12h ⏳ pending running
docker-sglang-3h ⏳ pending running
docker-sglang-forge-12h ⏳ pending running
docker-vllm-12h ⏳ pending running
docker-vllm-3h ⏳ pending running
docker-vllm-forge-12h ⏳ pending running

run details

rodosingh and others added 2 commits September 10, 2026 18:09
The page had a renderer and a skill telling you to run it, which meant a
run that nobody thought to post-process left only JSONL behind. A
gpt-oss-120b session finished with a complete 1,956-call ledger and no
readable report; rebuilding it afterwards was a one-liner, but nothing
said so at the time.

Render it in the CLI teardown instead. Both branches converge there, so
the sequencer path and the safety-net path both get the page. It sits
before the artifact package so the zip carries it, and after the
session_breakdown.json write so the outcome ladder it joins is present.

A session with no ledger is a skip with a printed reason, not a file:
rendering one publishes a page whose every figure is blank, and a blank
figure reads as a measured zero. Failures are logged and swallowed --
a report must never mask the run's stop_reason. The output name comes
from the renderer's own DEFAULT_OUTPUT so there is one convention, and
HYPERLOOM_SKIP_HTML_REPORT=1 opts out.

GEAK already does this from claude_trace_mirror.render_html_report;
this closes the same gap on the Hyperloom side.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The session page told you which phase cost the most and then stopped. The
GEAK page, on the same data one level down, told you where inside a phase
the money went -- how cost grows with the conversation, how few agents
carry the total, what the work consisted of, and every agent drillable to
its own calls. Two pages, one of them useful.

Move that machinery into _report_agents and render both pages through it,
so a column means the same thing on either. The GEAK page is unchanged:
byte-identical output on a real 1.8 MB run report, timestamp aside.

Identity is the one thing the ledgers genuinely disagree on, so it is
injected rather than assumed. GEAK has prompt text and no label; Hyperloom
has a role field written by some producers and not others -- on the
gpt-oss-120b run, 27 of 845 rows. So a Hyperloom agent is its task path
(specialist/<instance>, populated on 830 rows) and its name is the role
where one was recorded. An agent with neither is unlabelled, never named
from its id, and the coverage banner counts them.

KERNEL_AGENT is the phase this was worth doing for: 55.6% of that run's
bill and, seen from outside, a single lumped agent. Graft GEAK's own
per-call ledger in beneath it when the run left one, dropping the
harvested ext/geak rows in the same step -- both describe the same API
calls, so keeping both bills twice. A build without GEAK's trace mirror
leaves nothing to graft; the banner then says so and names the tool that
rebuilds it, rather than showing a thin phase and letting it read as a
cheap one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:inference Related to inference optimization type:feature New capability or improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants