From 02865082bc56b7d26af87e421d93fd6e48cd9280 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Fri, 10 Jul 2026 17:49:53 -0700 Subject: [PATCH] docs: document the logical key strategy Restructures the Response Cache page's key section into per-strategy subsections and documents logical: tool definitions key on a structural name/parameter fingerprint, so reworded or reordered tools still hit and only a changed tool interface misses. Signed-off-by: Zhongxuan Wang --- .../adaptive/response-cache.mdx | 30 ++++++++++++++++--- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index 0fa1fa945..598b5873e 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -329,10 +329,15 @@ an equivalent provider-native stream. Available saved-token and estimated-cost values are reported on the `response_cache` mark, never by editing a buffered response body. -## Cache Keys +## Key Strategies + +`key_strategy` selects what counts as "the same request". Strategies never +share cache entries: changing the strategy starts from an empty keyspace. + +### `exact_request` (default) Two requests hit the same entry when they are the same request after -normalization, under the default `key_strategy = "exact_request"`: +normalization: - The request is decoded to its normalized form and fingerprinted with SHA-256, so provider-shaped differences that mean the same thing collapse to @@ -363,6 +368,23 @@ normalization, under the default `key_strategy = "exact_request"`: - Requests containing integers outside the exactly representable RFC 8785 range (less than `-2^53` or greater than `2^53`) bypass the cache. +### `logical` + +Everything keys exactly as `exact_request` except the `tools` array: each tool +is keyed on its full definition with human-readable `description` text +removed, and the array is sorted. Rewording a tool description or reordering +the `tools` array no longer busts the cache; any other change to a tool +definition — its name, its parameter schema (including constraints such as +`enum` and `required`), its type, or its settings — still does. + +```toml +[components.config.response_cache] +key_strategy = "logical" +``` + +Use `logical` when prompt-engineering iterations on tool descriptions keep +invalidating entries whose behavior did not change. + ## Tool-Result Cache The same `response_cache` section can also cache results from @@ -478,7 +500,7 @@ cacheable classes and cacheable overrides, plus the default policy. | `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | | `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. | | `cache_nondeterministic` | `false` | Only requests with an explicit numeric `temperature = 0` are eligible. Set `true` to cache and reuse sampled responses. | -| `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | +| `key_strategy` | `"exact_request"` | What counts as the same request: `"exact_request"` or `"logical"`. Refer to [Key Strategies](#key-strategies). | | `header_allowlist` | `[]` | Trusted, non-secret response-affecting headers folded into the key (case-insensitive). A non-empty normalized allowlist policy also partitions the key. Known auth headers are rejected. | | `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). | | `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. | @@ -518,7 +540,7 @@ Redis capacity and eviction in Redis itself. - `namespace` is empty or whitespace-only, `ttl_seconds` is `0`, or `bypass_rate` is outside `[0.0, 1.0]`. -- `key_strategy` is not `"exact_request"`. +- `key_strategy` is neither `"exact_request"` nor `"logical"`. - `header_allowlist` names an auth header such as `authorization` or `x-api-key`. - `in_memory` `max_bytes` is zero or is not an integer.