Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions docs/configure-plugins/adaptive/response-cache.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -329,10 +329,15 @@ an equivalent provider-native stream. Available saved-token and estimated-cost
values are reported on the `response_cache` mark, never by editing a buffered
response body.

## Cache Keys
## Key Strategies

`key_strategy` selects what counts as "the same request". Strategies never
share cache entries: changing the strategy starts from an empty keyspace.

### `exact_request` (default)

Two requests hit the same entry when they are the same request after
normalization, under the default `key_strategy = "exact_request"`:
normalization:

- The request is decoded to its normalized form and fingerprinted with
SHA-256, so provider-shaped differences that mean the same thing collapse to
Expand Down Expand Up @@ -363,6 +368,23 @@ normalization, under the default `key_strategy = "exact_request"`:
- Requests containing integers outside the exactly representable RFC 8785
range (less than `-2^53` or greater than `2^53`) bypass the cache.

### `logical`

Everything keys exactly as `exact_request` except the `tools` array: each tool
is keyed on its full definition with human-readable `description` text
removed, and the array is sorted. Rewording a tool description or reordering
the `tools` array no longer busts the cache; any other change to a tool
definition — its name, its parameter schema (including constraints such as
`enum` and `required`), its type, or its settings — still does.

```toml
[components.config.response_cache]
key_strategy = "logical"
```

Use `logical` when prompt-engineering iterations on tool descriptions keep
invalidating entries whose behavior did not change.

## Tool-Result Cache

The same `response_cache` section can also cache results from
Expand Down Expand Up @@ -478,7 +500,7 @@ cacheable classes and cacheable overrides, plus the default policy.
| `priority` | `50` | LLM execution intercept priority. Lower values run earlier. |
| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. |
| `cache_nondeterministic` | `false` | Only requests with an explicit numeric `temperature = 0` are eligible. Set `true` to cache and reuse sampled responses. |
| `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. |
| `key_strategy` | `"exact_request"` | What counts as the same request: `"exact_request"` or `"logical"`. Refer to [Key Strategies](#key-strategies). |
| `header_allowlist` | `[]` | Trusted, non-secret response-affecting headers folded into the key (case-insensitive). A non-empty normalized allowlist policy also partitions the key. Known auth headers are rejected. |
| `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). |
| `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. |
Expand Down Expand Up @@ -518,7 +540,7 @@ Redis capacity and eviction in Redis itself.

- `namespace` is empty or whitespace-only, `ttl_seconds` is `0`, or
`bypass_rate` is outside `[0.0, 1.0]`.
- `key_strategy` is not `"exact_request"`.
- `key_strategy` is neither `"exact_request"` nor `"logical"`.
- `header_allowlist` names an auth header such as `authorization` or
`x-api-key`.
- `in_memory` `max_bytes` is zero or is not an integer.
Expand Down