[feature] Make Responses continuation state shareable across replicas

[cpakkamisaac-sae]

Problem

Responses continuation affinity is stored inside each ClientRouter in a process-local
Mutex<StateOwners>. A restart loses the mapping, and replicas do not share it.

This makes an otherwise valid deployment fail when sequential turns land on different replicas:

1. Replica A routes a stored Responses request to provider A and returns its opaque response ID.
2. The caller sends that ID in previous_response_id to replica B.
3. Replica B cannot find the owner, runs routing again, and can select provider B.
4. Provider B rejects provider A's ID with state_not_found.

I reproduced this on current main (c6e5958a) with two independently constructed server states
and one deterministic loopback upstream. The seed succeeded through provider A; the follow-up was
reclassified to provider B and returned HTTP 400. The result was stable across repeated runs.

This is distinct from #802/#803. That work fixes conversation-history indexing within one process;
it does not make ownership or canonical history available after a restart or on another replica.

Proposed solution

Introduce an asynchronous Responses-state storage port owned by switchyard-llm-client.

The contract should support:

• looking up an opaque response or conversation ID;
• atomically recording the response and conversation aliases for one turn;
• storing the selected model for provider-owned state;
• storing canonical message history for cross-format continuations;
• detecting conflicting owners without overwriting an existing provider-owned record; and
• distinguishing a missing record from a storage failure.

Keep the current bounded in-memory implementation as the default so existing embedders and
single-process deployments do not change behavior or gain an external dependency. Let hosts inject
another implementation. A later server-facing increment can configure a concrete shared backend,
including namespace, retention, credentials, and connection settings.

The first implementation PR should establish the storage boundary and prove it with two independent
routers sharing one test store. It should not select a production database or add deployment TOML
before the contract is reviewed.

Required semantics

• Store operations must be async so a networked implementation does not block request workers.
• Response and conversation aliases for one turn must be committed atomically.
• Keys must be scoped by deployment/route so identical provider IDs in unrelated routes cannot
  conflict.
• Provider-native records store only target ownership. Cross-format records also store the canonical
  history required to rebuild a request.
• Retention must be explicit and bounded. Expiration must not silently route known continuations to
  another provider.
• A store outage must not cause rerouting to a different provider. The error path should be explicit
  and observable, including after streaming headers have been sent.
• Stored history can contain user content, so the production adapter must document access control,
  encryption, retention, and deletion behavior.
• Metrics and structured logs must use bounded attributes and must not expose IDs or message content.

Alternatives considered

Sticky load balancing

Reduces cross-replica misses but does not survive process replacement, scaling events, or lost
affinity cookies. It also moves a correctness requirement into deployment-specific networking.

Encode the target into returned IDs

Provider IDs are opaque and appear in buffered responses, streams, and later operations. Rewriting
them is compatibility-sensitive, does not carry cross-format canonical history, and needs integrity
protection.

Require callers to resend full history

Safe, but removes the Responses continuation behavior Switchyard already supports and changes caller
contracts.

Add a database directly to the server

Would solve one deployment shape but couples the core behavior to a backend before lookup, conflict,
retention, privacy, and failure semantics have been agreed.

Scope notes

• Primary owner: switchyard-llm-client; server configuration belongs in a later increment.
• The injected storage port is a new public Rust API. Existing constructors keep the in-memory
  default for backward compatibility.
• No wire-format or Python API change is required for the first increment.
• A concrete distributed adapter and deployment TOML are intentionally out of scope until the
  contract is accepted.

Acceptance criteria

• A continuation created through one router is resolved by a separately constructed router sharing
  the same store, without another routing decision.
• Provider-owned and cross-format state both work for buffered and completed streaming responses.
• Concurrent conflicting writes are atomic and preserve the original owner.
• Missing, expired, unavailable, and capacity-limited state have explicit tested behavior.
• Existing constructors and single-process behavior remain unchanged.
• The full Rust workspace, formatting, and Clippy pass with warnings denied.

Additional context

PR #721 explicitly described the current map as a best-effort compatibility fix and documented that
restarts lose mappings and replicas do not share them. PR #781 added materialized cross-format
history, which makes the storage boundary responsible for both affinity records and, when needed,
conversation content.

[afourniernv]

@cpakkamisaac-sae could you give this issue a tighter editing pass? There is useful detail here, but the actual problem and proposed solution are hard to pick out.

I think it should clearly cover:

• The exact replica failure and a short reproduction.
• Why keeping state inside each ClientRouter causes it.
• The proposed boundary: what libsy-llm-client owns, what a host-provided store owns, and what stays out of scope.
• The public API and runtime behavior this would change.
• The decisions still open around key scope, caller isolation, missing versus expired state, store outages, capacity, retention, and the conversation behavior in fix(llm-client): store canonical history under the conversation id #803.
• A short interface sketch so the team can see the proposed shape.

That should give us enough to make a design call before settling on the implementation.