fix(anthropic): surface cache_read/write input tokens in metadata chunk#2302
Open
jehoon-shin-mathflat wants to merge 1 commit into
Open
Conversation
Anthropic returns `input_tokens` as the NON-CACHED portion only when prompt caching is in use. `cache_read_input_tokens` and `cache_creation_input_tokens` were dropped during metadata chunk formatting, so downstream consumers (`Agent.metrics.accumulated_usage` and anything that derives cost from it) saw only the uncached delta and under-reported real usage / cost — sometimes by an order of magnitude on image+text workloads where the image dominates the cached prefix. This change: - Maps `cache_read_input_tokens` → `cacheReadInputTokens` and `cache_creation_input_tokens` → `cacheWriteInputTokens` on the metadata chunk, both already defined as optional members of `types.event_loop.Usage`. - Recomputes `totalTokens` as `uncached_input + cache_read + cache_write + output_tokens` so it reflects the actual billed input. - Omits the cache fields when both are zero/absent, preserving the existing chunk shape for non-cached responses (no consumer change required). Added tests covering the cached and non-cached metadata shapes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Anthropic returns
input_tokensas the non-cached portion only when prompt caching is in use. The metadata chunk formatter inmodels/anthropic.pyreadsinput_tokens/output_tokensfrom the API response but drops thecache_read_input_tokensandcache_creation_input_tokensfields. As a result, anything that derives cost fromAgent.metrics.accumulated_usageunder-reports real usage — sometimes by an order of magnitude on image+text workloads where the cached prefix carries most of the prompt.Repro
A vision classifier I run sends
[{"text": ...}, {"cachePoint": {}}, {"image": ...}]so that the text+image prefix is cached. After warmup, the Anthropic API returns roughly:Strands currently surfaces only
inputTokens=3, so accumulated cost looks ~10× lower than what Anthropic actually bills. TheUsageTypedDict intypes/event_loop.pyalready declares the cache fields as optional members — they just weren't being populated by the Anthropic adapter.Change
models/anthropic.py— in themetadatacase offormat_chunk:cache_read_input_tokensandcache_creation_input_tokensfrom the usage dict (defaulting to 0 when absent).cacheReadInputTokens/cacheWriteInputTokenson the usage chunk, matching the existing field names already defined intypes.event_loop.Usage.totalTokensasuncached_input + cache_read + cache_write + output_tokensso it reflects the full billed input.Tests
tests/strands/models/test_anthropic.py:test_format_chunk_metadata_with_cache_tokens— both cache fields present.test_format_chunk_metadata_omits_zero_cache_tokens— fields absent/zero, legacy shape preserved.All 59 tests in
tests/strands/models/test_anthropic.pypass.Related Issues
n/a — surfaced while debugging cost accounting in a downstream POC.
Documentation PR
n/a — no public API surface change, only adds optional fields that
Usagealready declares.Type of change
Testing
Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.