fix(instrumentation): handle MessageDeltaUsage in Anthropic streaming token extraction

[Aftabbs]

Summary

Fixes #754 — Anthropic streaming calls recorded incorrect output token counts because _extract_anthropic_tokens raised AttributeError on MessageDeltaUsage, which the dont_throw wrapper silently swallowed.

Root cause

Anthropic's streaming protocol emits two usage objects across the stream lifecycle:

Event	Type	Fields
message_start	Usage	input_tokens, output_tokens (early estimate), cache_*
message_delta	MessageDeltaUsage	output_tokens (authoritative final count) only

_extract_anthropic_tokens accessed usage.input_tokens directly. On MessageDeltaUsage this raised AttributeError, caught silently by dont_throw. Result: the final output token count from message_delta was never written to the span — every streaming call showed the early estimate from message_start instead.

Fix design

Three targeted changes, zero behaviour change to non-streaming paths:

1. _extract_anthropic_tokens — use getattr(usage, field, None) for all four fields; return int | None so callers can distinguish "field absent" from "field is zero".

2. _apply_usage_to_span (new helper) — writes only the non-None fields to the span. This prevents a message_delta chunk from zeroing out input_tokens that message_start already recorded.

3. Non-streaming post_hook — full Usage is always present; callers apply or 0 so the existing integer-attribute contract is preserved.

Before / after for a streaming call with input=15, final output=42

Span attribute	Before fix	After fix
JUDGMENT_USAGE_NON_CACHED_INPUT_TOKENS	15 ✓	15 ✓
JUDGMENT_USAGE_OUTPUT_TOKENS	5 (early estimate) ✗	42 (final count) ✓

Tests

Adds TestSyncStreaming and TestAsyncStreaming — the streaming code paths had no test coverage at all before this PR. New tests verify:

• A span is created for streaming calls

• input_tokens is recorded from message_start and not overwritten by message_delta

• The final output_tokens from message_delta is what the span ultimately reflects

• Existing non-streaming tests still pass (12/12 green)

• No dependency changes

---

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 Cache token counts go missing on streamed calls made through the stream helper

The token-count reader now returns "missing" instead of zero for absent cache fields (getattr(usage, ..., None) at src/judgeval/instrumentation/llm/llm_anthropic/messages.py:55-58), but the stream-helper path that reuses this reader hands those values straight to the span without converting missing to zero, so the cache token counts that used to be recorded as 0 are now dropped entirely.
Impact: Spans produced by the .stream() helper stop recording cache-read and cache-creation token counts whenever the model returns no caching info (the common case), losing data that was previously stored as 0.

Return-contract change not propagated to messages_stream.py callers

Before this PR, _extract_anthropic_tokens coerced None fields to 0, so its four return values were always int. The PR changed it to return int | None (src/judgeval/instrumentation/llm/llm_anthropic/messages.py:48,55-58). The callers inside messages.py were updated accordingly (or 0 in the non-streaming post_hook, and the new _apply_usage_to_span which skips None).

However, src/judgeval/instrumentation/llm/llm_anthropic/messages_stream.py:122-138 and 260-276 also call _extract_anthropic_tokens(final_message.usage) and pass cache_read / cache_creation directly into span.set_attribute(...) with no or 0. For a full Usage object input_tokens/output_tokens are always ints, but cache_read_input_tokens/cache_creation_input_tokens are commonly None. OpenTelemetry's set_attribute rejects a None value (logs a warning and skips it), so those attributes are no longer set to 0 as before — a behavior regression for the .stream() context-manager path. The whole block is wrapped in try/except Exception: pass, so if any version raises instead of warning, the model-name attribute set afterward would also be skipped.

Prompt for agents

The return contract of _extract_anthropic_tokens in src/judgeval/instrumentation/llm/llm_anthropic/messages.py changed from always-int to int | None. The callers in src/judgeval/instrumentation/llm/llm_anthropic/messages_stream.py (around lines 122-138 and 260-276) still pass the unpacked prompt_tokens/completion_tokens/cache_read/cache_creation directly into span.set_attribute(...) without an `or 0` guard. Since cache_read/cache_creation are commonly None for a full Usage object, these attributes will no longer be recorded (previously recorded as 0). Update those call sites to apply `or 0` (mirroring the non-streaming post_hook in messages.py) so the integer-attribute contract is preserved, or use the new _apply_usage_to_span helper if appropriate.

---

Was this helpful? React with 👍 or 👎 to provide feedback.