diff --git a/AGENTS.md b/AGENTS.md index c7c84ca..605bd74 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ go test -tags e2e -run 'TestE2E' -timeout 15m -v . # LIVE provider e2e (see be ``` - Always run tests with `-count=1` and an explicit `-timeout` (house rule: no unbounded runs). -- Coverage sits at ~97.7% of statements. The residual is documented unreachable defensive code — do not pad with fake tests to move the number. +- Coverage sits at ~99.0% of statements (unit suite). The residual is documented unreachable defensive code — do not pad with fake tests to move the number. ## Architecture (flat package) @@ -29,9 +29,11 @@ go test -tags e2e -run 'TestE2E' -timeout 15m -v . # LIVE provider e2e (see be | `openai.go` / `gemini.go` / `anthropic.go` | Per-format request builders, response/stream mappers, model listing | | `responses.go` | OpenAI Responses API (`/v1/responses`) for GPT-5.6+ tools+reasoning | | `tts.go` | Text-to-speech: `Speak`/`SpeakRequest`/`SpeakResult` — OpenAI-compat `/audio/speech` + Gemini AUDIO modality, no transcoding | +| `controls.go` | Request controls: `ToolChoice`, `ResponseFormat` (incl. Anthropic forced-tool JSON emulation + `foldAnthropicJSON`), `ParallelToolCalls`, per-format mapping and boundary validation | +| `embed.go` | Embeddings: `Embed`/`EmbedRequest`/`EmbedResult` — OpenAI-compat `/embeddings` + Gemini `batchEmbedContents` | | `stt.go` | Speech-to-text: `Transcribe`/`TranscribeRequest`/`TranscribeResult` — OpenAI-compat `/audio/transcriptions` (multipart), 25MB input cap | | `sse.go` | SSE parser (abort-safe via `done` channel) + idle-watchdog pump | -| `retry.go` | Backoff/jitter/`Retry-After`/`retrySleep` (8 attempts, cap 30s) | +| `retry.go` | `RetryPolicy` + the single buffered ladder `withRetry` (chat, tts, stt, embed); backoff/jitter/`Retry-After`/`retrySleep` (default 8 attempts, cap 30s) | | `provider.go` | Built-in registry, quirks flags, config validation | | `models.go` | `ListModels` orchestration (retries, caps, cache backing) | | `auth.go`, `transport.go`, `errors.go` | Env resolution, pooled HTTP clients, typed errors | @@ -40,19 +42,22 @@ The canonical type system is OpenAI-shaped; `gemini.go`/`anthropic.go` translate ## Invariants (breaking any of these is a defect) -1. **Canonical finish reasons** are `stop | length | tool_calls | content_filter | ""`. Unmapped provider stop reasons map to `""` on every format — provider-specific strings never leak. +1. **Canonical finish reasons** are `stop | length | tool_calls | content_filter | ""`. Unmapped provider stop reasons map to `""` on every format — provider-specific strings never leak. A turn with tool calls finishes `tool_calls` on every format (Gemini's `STOP` is mapped). 2. **API keys never appear** in error text, `String()`, or any typed error. -3. **Streaming**: retries only before the first emitted delta; a failure after partial output returns the partial `*ChatResult` + wrapped error and is **never retried**; a premature close (no completion signal) is an error, never a silent empty success; the parser goroutine is always released (abort-safe `done` protocol). +3. **Streaming**: retries only before the first emitted delta; a failure after partial output returns the partial `*ChatResult` + wrapped error and is **never retried**; a premature close (no completion signal — on every format, Gemini included; an unmapped finish reason still counts as one) is an error, never a silent empty or partial success; the parser goroutine is always released (abort-safe `done` protocol). A successful non-SSE response is parsed directly with the usual body cap and format mapper. It learns buffered mode for future requests, but never discards the current generation or makes a replacement request; parse/read errors remain terminal for that response. 4. **Unknown message roles are rejected** at the SDK boundary (`ConfigError`) — never dropped or reinterpreted per format. 5. **Unknown provider data stays unknown** (zero values) — no static guesses, no fallback model tables. 6. **Learn-once fallback state is per-`Provider`**, monotonic, atomic, shared by every `ChatClient` — never move it back to per-client. -7. Anthropic extended-thinking rounds trip via `Message.ThinkingSignature` / `ChatResult.ThinkingSignature` — replay is signature-gated and the thinking block goes **first**. +7. Anthropic extended-thinking rounds trip via `ThinkingBlocks` (every `thinking` / `redacted_thinking` block, each with its own signature, in order) or the legacy single `ReasoningContent` + `ThinkingSignature` pair — replay is signature-gated and the thinking blocks go **first**. Signatures are per block, never concatenated. Gemini `thoughtSignature`s round-trip per part (`ToolCall.Signature`, `ThinkingSignature`). +8. **`Delta.ToolIndex` is the call's position in `ChatResult.ToolCalls`** on every format — never a provider content-block or output-item index. +9. **`Usage` has one meaning everywhere**: `PromptTokens` uncached-only, cache reads/writes in their fields (a DeepSeek miss is uncached input, not a write), `CompletionTokens` includes reasoning, `CachedTokens` is a diagnostic subset never summed. +10. **The request timeout is a whole-call budget**, retries included, on every entry point (buffered, streaming, speech, transcription, embeddings). ## Testing conventions - **RED-first TDD**: failing test first, then the fix. Table-driven tests; `httptest` servers for hermetic coverage; `newTestClient` helper pins `backoffUnit` to 1ms — restore package vars in `t.Cleanup`. -- Two timing knobs are package vars for tests: `backoffUnit`, `streamIdleTimeout`. Operators override the idle watchdog via `SetStreamIdleTimeout` (positive values only). +- Timing knobs for tests: the package var `backoffUnit` (use `fastBackoff(t)`), and the process-wide idle default (atomic; use `setIdleForTest(t, d)`). Operators override the idle watchdog per SDK with `WithStreamIdleTimeout`, or process-wide with the race-safe `SetStreamIdleTimeout` (positive values only). Retry shape is per SDK via `WithRetryPolicy`. - The e2e suite (`e2e_test.go`) is behind a `e2e` build tag and hits **live APIs**. It must never lose that tag. Keys come from env or a gitignored `.env`; contents are never logged; tests skip when a key is absent. Adding a provider = one `e2eTarget` entry; models overridable via `_E2E_MODEL`. - Live-provider behavior (e.g. DeepSeek eliding `reasoning_content`) is **not an SDK contract** — probe softly, assert only what the SDK guarantees (call success, parsing, canonical finish). Model answer correctness is never an assertion. Elision is absorbed by an explicit provider quirk rather than hope: on a chat-completions request that carries tools, providers flagged `Quirks.EchoReasoningWithTools` always get the `reasoning_content` key, empty included — including on assistant turns that made no tool call themselves (a request diverted to `/responses` never reaches that builder). That flag is off for every provider except `deepseek`, and whether DeepSeek *accepts* a present-but-empty value is measured live by the tag-gated `TestE2EDeepSeekEmptyReasoningEcho` probe — not asserted by the unit suite. - Timer hygiene: since Go 1.23 no drain-before-`Reset` is needed for `time.Timer`. Note: on some dev machines `time.After` + select-default spin loops have hung — prefer deadline loops in tests. diff --git a/README.md b/README.md index 3f639b3..c3e3a96 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ res, err = chat.CallStream(ctx, req, func(d llm.Delta) error { switch d.Kind { case llm.DeltaReasoning: // thinking fragment case llm.DeltaContent: // text fragment - case llm.DeltaToolArgs: // tool-call argument fragment (d.ToolID, d.ToolName) + case llm.DeltaToolArgs: // tool-call argument fragment (d.ToolID, d.ToolName; d.ToolIndex = position in res.ToolCalls) } return nil // or an error to abort — partial result comes back with *StreamAbortedError }) @@ -89,7 +89,7 @@ res, err = chat.CallStream(ctx, req, func(d llm.Delta) error { | `kimi` | openai | `https://api.moonshot.ai/v1` | `KIMI_API_KEY` (`MOONSHOT_API_KEY`) | `KIMI_BASE_URL` | | `anthropic` | anthropic | `https://api.anthropic.com` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -Primary env var beats its alias. Explicit keys (`WithAPIKey`) beat env. Base-URL overrides accept any gateway speaking the provider's format. A bad URL passed to `WithBaseURL` is rejected at wiring time: the provider is marked invalid, `Providers()` omits it, and `Chat` returns a `*ConfigError` — no request is sent. +Primary env var beats its alias. Explicit keys (`WithAPIKey`) beat env. Base-URL overrides accept any gateway speaking the provider's format. A bad URL passed to `WithBaseURL` (wrong scheme or malformed) is rejected at wiring time: the provider is marked invalid, `Providers()` omits it, and `Chat` returns a `*ConfigError` — no request is sent. Custom gateways: @@ -101,14 +101,33 @@ sdk := llm.New(llm.WithProvider("my-gateway", )) ``` +Extra HTTP headers ride every request a provider makes — chat, streaming, model listing, speech, transcription, embeddings. They are applied last, so they can override the SDK's own; an empty value removes a header. Values are treated like keys: never logged, never in `String()` or errors. + +```go +llm.WithProvider("openrouter", llm.WithFormat(llm.FormatOpenAI), + llm.WithBaseURL("https://openrouter.ai/api/v1"), llm.WithEnvKeys("OPENROUTER_API_KEY"), + llm.WithHeaders(map[string]string{"HTTP-Referer": "https://myapp.example", "X-Title": "myapp"})) + +llm.WithProvider("anthropic", llm.WithHeaders(map[string]string{"anthropic-beta": "interleaved-thinking-2025-05-14"})) + +// api-key gateways (e.g. Azure OpenAI): drop the Bearer header, send api-key. +llm.WithProvider("azure", llm.WithFormat(llm.FormatOpenAI), llm.WithBaseURL(azureURL), llm.WithAPIKey(key), + llm.WithHeaders(map[string]string{"Authorization": "", "api-key": key})) +``` + +Redirects to another host are never followed (the redirect response surfaces as an `*APIError`): Go strips only `Authorization` across hosts, so `x-api-key`, `x-goog-api-key` and custom headers would otherwise leak to the target. `Provider` and `ProviderConfig` redact the key and header values under every `fmt` verb (`%v`, `%+v`, `%#v`). + +For request/response observability (logging, metrics, tracing), wrap the transport: `llm.WithTransport(myRoundTripper)` sees every HTTP exchange the SDK makes. + ## Canonical API Requests and results are provider-neutral. Unknown message roles are rejected at the SDK boundary (never silently dropped or reinterpreted). -User messages may contain ordered text and inline image parts. `Content` remains -the backwards-compatible plain-text form; use `TextPart` and `ImagePart` when -an image is needed. Parts are user-role only and cannot be combined with -`Content` on the same message; empty text parts are rejected. Accepted image +User messages and tool results may contain ordered text and inline image parts. +`Content` remains the backwards-compatible plain-text form; use `TextPart` and +`ImagePart` when an image is needed. Parts are user- and tool-role only and +cannot be combined with `Content` on the same message; empty text parts are +rejected. Accepted image types are png, jpeg, gif, and webp (the informal `image/jpg` alias is accepted and normalized to `image/jpeg` on the wire). Per-image (`MaxImageBytes`, 10 MiB) and per-request aggregate (`MaxRequestImageBytes`, 32 MiB) caps are enforced @@ -135,29 +154,75 @@ type ChatRequest struct { MaxTokens int // routed to max_completion_tokens on o-series/gpt-5 Temperature float64 // 0 = provider default; negative = explicit 0 TopP float64 // 0 = provider default; negative = explicit 0 - Stop []string // provider-native stop / stop_sequences / stopSequences + Stop []string // provider-native stop / stop_sequences / stopSequences (not sent on /responses) + ToolChoice *ToolChoice // nil = provider default; see "Request controls" + ParallelToolCalls *bool // with Tools: allow/forbid several calls per turn + ResponseFormat *ResponseFormat // JSON object / JSON Schema structured output + Seed *int // OpenAI chat completions, Gemini + Extra map[string]any // provider-specific top-level body fields (override the SDK's) } type ChatResult struct { Content string - ReasoningContent string // provider thinking text; replay rules per provider in "Extended thinking" - ThinkingSignature string // Anthropic: replay via Message.ThinkingSignature - ToolCalls []ToolCall // {ID, Name, Arguments} - FinishReason string // stop | length | tool_calls | content_filter | "" - Usage Usage // PromptTokens is uncached-only; cache volumes in CacheReadTokens / CacheCreationTokens / CachedTokens + ReasoningContent string // provider thinking text; replay rules per provider in "Extended thinking" + ThinkingSignature string // last signature (Anthropic / Responses / Gemini text part) + ThinkingBlocks []ThinkingBlock // every signed reasoning segment, in order (Anthropic, Responses) + ToolCalls []ToolCall // {ID, Name, Arguments, Signature (Gemini thoughtSignature)} + FinishReason string // stop | length | tool_calls | content_filter | "" + Usage Usage // see "Usage" below } ``` -Finish reasons are canonical: anything a provider reports outside that vocabulary maps to `""` (unknown) rather than leaking provider-specific strings. +Finish reasons are canonical: anything a provider reports outside that vocabulary maps to `""` (unknown) rather than leaking provider-specific strings. A turn that requested tools finishes as `tool_calls` on every format (Gemini reports `STOP` for it; the SDK maps it). + +Signatures (`ThinkingSignature`, `ThinkingBlocks`, `ToolCall.Signature`) are provider-specific: replay them only to the provider that produced them — never carry a conversation's signed turns across providers. + +`res.AssistantMessage()` returns the assistant turn to append before the next request, carrying every replay field (content, reasoning, signatures, thinking blocks, tool calls) — use it instead of copying fields by hand. + +### Usage + +One meaning on every format: + +- `PromptTokens` — uncached input only. Inclusive provider counts (OpenAI `cached_tokens`, DeepSeek `prompt_cache_hit_tokens`, Gemini `cachedContentTokenCount`) are moved to `CacheReadTokens`. +- `CacheReadTokens` / `CacheCreationTokens` — cache reads and writes. A DeepSeek cache *miss* is ordinary input, not a write, and stays in `PromptTokens`. +- `CompletionTokens` — every generated token, reasoning included (Gemini thoughts are added); `ReasoningTokens` is that subset. +- `CachedTokens` — the raw provider-reported cached count, for diagnostics; already inside `CacheReadTokens`, never add it. +- `InputTokens()` = prompt + cache reads + cache writes; `TotalTokens()` = input + completion. Use these instead of summing by hand. + +## Request controls + +```go +req := &llm.ChatRequest{ + Messages: msgs, + Tools: tools, + ToolChoice: llm.ToolChoiceNamed("lookup"), // or &llm.ToolChoice{Mode: llm.ToolChoiceRequired} + ParallelToolCalls: &no, + ResponseFormat: &llm.ResponseFormat{Type: llm.ResponseJSONSchema, Name: "answer", Schema: schema, Strict: true}, + Seed: &seed, + Extra: map[string]any{"service_tier": "flex"}, +} +``` + +| Control | OpenAI chat / Responses | Anthropic | Gemini | +|---|---|---|---| +| `ToolChoice` auto / none / required / tool | `tool_choice` | `tool_choice` auto / none / any / tool | `toolConfig.functionCallingConfig` AUTO / NONE / ANY (+ `allowedFunctionNames`) | +| `ParallelToolCalls` | `parallel_tool_calls` (only with tools) | `disable_parallel_tool_use` | — (ignored) | +| `ResponseFormat` json_object / json_schema | `response_format` / `text.format` | forced synthetic tool whose input is folded back into `Content` (buffered and streamed as `DeltaContent`) | `responseMimeType` + `responseJsonSchema` | +| `Seed` | `seed` (chat completions only) | — (ignored) | `generationConfig.seed` | +| `Extra` | merged into the body | merged into the body | merged into the body | + +Caveats: DeepSeek accepts `json_object` only (`Quirks.NoJSONSchema`; `json_schema` fails fast with a `ConfigError` — set the flag via `WithQuirks` for other gateways with the same limit). On Anthropic, JSON mode forces its own tool, so it cannot be combined with `Tools` and needs a top-level `"type": "object"` schema. Gemini 2.x models reject JSON mode together with function declarations; Gemini 3 accepts it. `Extra` is a shallow, top-level override: an object value replaces the SDK's whole object (e.g. `generationConfig`), and keys go to whichever endpoint the request lands on (including `/responses` for diverted OpenAI calls). It may not replace what the SDK validates (`model`, `messages`/`contents`/`input`, `system`/`systemInstruction`/`instructions`, `tools`) or `stream`. Gemini seeds must fit in 32 bits. + +Controls are validated before any network I/O (`*ConfigError`): `ToolChoice` needs `Tools` and a known tool name; `json_schema` needs a JSON object schema; names match `[A-Za-z0-9_-]{1,64}`; `Extra` may not set the reserved keys above. Anthropic rejects forced tool use under extended thinking, so `ToolChoice` required/tool and JSON `ResponseFormat` fail fast with `Thinking` on, and JSON mode cannot be combined with `ToolChoice` (it forces its own tool). ## Streaming semantics `CallStream` enforces four guarantees, each covered by regression tests: -1. **Idle watchdog** — a stream silent longer than `StreamIdleTimeout()` (120s default; override with `SetStreamIdleTimeout`, positive values only) fails with `ErrIdleTimeout`. Keepalive comments reset it. +1. **Idle watchdog** — a stream silent longer than the idle timeout (120s default; per SDK via `WithStreamIdleTimeout`, process-wide default via the race-safe `SetStreamIdleTimeout`; positive values only) fails with `ErrIdleTimeout`. Keepalive comments reset it. 2. **Hard wall-clock deadline** — the whole stream is bounded by the per-request timeout (`WithRequestTimeout`, default 120s; per-client via `SetRequestTimeout`). 3. **Retries never duplicate output** — retries happen only before the first emitted delta. A failure after partial output returns the partial `*ChatResult` plus a wrapped error and is never retried. -4. **No silent empty successes** — a provider that closes the stream before its completion signal (before `[DONE]` / `message_stop`) yields a retryable error, not an empty result. Gemini, whose streams legitimately end at EOF, is exempt. +4. **No silent truncation** — a provider that closes the stream before any completion signal (`[DONE]`, a `finish_reason`, `message_stop`, a Gemini `finishReason`, `response.completed`) yields an error: retryable before the first delta, the partial result plus a wrapped error after it. Gemini is no longer exempt. An unmapped provider finish reason still counts as completion. Aborting from the delta handler returns the partial result alongside `*StreamAbortedError` — the parser goroutine is always released, so aborted streams leak nothing. @@ -172,26 +237,26 @@ for { if len(res.ToolCalls) == 0 { return nil } - req.Messages = append(req.Messages, - llm.Message{Role: llm.RoleAssistant, Content: res.Content, - ReasoningContent: res.ReasoningContent, ThinkingSignature: res.ThinkingSignature, - ToolCalls: res.ToolCalls}) + req.Messages = append(req.Messages, res.AssistantMessage()) // every replay field for _, tc := range res.ToolCalls { + out, err := execute(tc) req.Messages = append(req.Messages, - llm.Message{Role: llm.RoleTool, ToolCallID: tc.ID, ToolName: tc.Name, Content: execute(tc)}) + llm.Message{Role: llm.RoleTool, ToolCallID: tc.ID, ToolName: tc.Name, Content: out, IsError: err != nil}) } } ``` +`IsError` marks a failed tool execution (Anthropic `is_error`, Gemini `{"error": …}`; OpenAI formats have no flag, so describe the failure in `Content`). Tool results may carry image `Parts`: Anthropic and the Responses API take them natively, Gemini sends them as `inlineData` in the same turn, and Chat Completions (whose tool messages are text-only) receives them in a user message right after the tool results. + On Gemini, a tool result's `ToolName` may be omitted — the SDK recovers the function name from the assistant `ToolCall` it answers, and errors loudly if it cannot. ## Extended thinking -- **OpenAI GPT-5.6+** — function tools plus reasoning cannot ride Chat Completions (`reasoning_effort` 400s). Those calls go to `POST /v1/responses` with `reasoning.effort` and `reasoning.summary=auto`; summaries land in `ReasoningContent` and encrypted reasoning replays via `ThinkingSignature`. `Thinking: disabled` stays on Chat Completions with `reasoning_effort: none`. Other OpenAI models keep `reasoning_effort` on Chat Completions. -- **Anthropic** — `thinking` blocks are parsed in both buffered and streaming modes. `ChatResult.ThinkingSignature` carries the provider signature; for tool loops, replay it on the assistant message (`Message.ReasoningContent` + `Message.ThinkingSignature`) — the SDK re-serializes it as the first block, as Anthropic's API requires. Unsigned thinking replay is rejected locally with `ConfigError`. +- **OpenAI GPT-5.6+** — function tools plus reasoning cannot ride Chat Completions (`reasoning_effort` 400s). Those calls go to `POST /v1/responses` with `reasoning.effort` and `reasoning.summary=auto`; summaries land in `ReasoningContent`, every reasoning item lands in `ThinkingBlocks` (encrypted content as `Signature`) and is replayed in order; the legacy `ThinkingSignature` carries the last one. `Thinking: disabled` stays on Chat Completions with `reasoning_effort: none`. Other OpenAI models keep `reasoning_effort` on Chat Completions. +- **Anthropic** — every `thinking` and `redacted_thinking` block is parsed in both buffered and streaming modes into `ChatResult.ThinkingBlocks`, each with its own signature (redacted blocks keep their opaque `data`). For tool loops, replay them via `Message.ThinkingBlocks` (or `res.AssistantMessage()`): the SDK re-serializes them first, in order, as Anthropic's API requires. The legacy single pair (`ReasoningContent` + `ThinkingSignature`) still works for one-block turns. Unsigned thinking replay is rejected locally with `ConfigError`. With thinking on, `max_tokens` always exceeds `budget_tokens` (unset `MaxTokens` → budget + 8192; a preset larger than half an explicit `MaxTokens` is clamped to half of it — never below 1024 — so the answer keeps room; an explicit `ThinkingBudget` that cannot fit is a `ConfigError`), and `temperature` / `top_p` are omitted (extended thinking rejects them). - **DeepSeek** — reasoning streams as `DeltaReasoning` fragments and lands in `ReasoningContent`. Assistant-turn replay echoes it as `reasoning_content`, which DeepSeek requires for tool loops. On a chat-completions request that carries tools, the key is echoed for **every** assistant turn, even where the provider returned no reasoning for that turn (empty value instead of a dropped key) — including turns that made no tool call themselves: DeepSeek documents that a tool-bearing request whose assistant turns omit `reasoning_content` returns 400 for *every* later request in the loop, so one elided turn would otherwise poison the rest of the session. Whether DeepSeek accepts a present-but-empty value is measured against the live API by `TestE2EDeepSeekEmptyReasoningEcho` (tag-gated), not assumed by the suite. - **zai / GLM** — same reasoning field shape and `DeltaReasoning` streaming; GLM maps thinking `medium` → `reasoning_effort` `high` (no medium level) and `max` → `max`. GLM does **not** get the empty echo: no z.ai documentation confirms the same replay requirement, so `Quirks.EchoReasoningWithTools` is off and `zai` bodies are byte-identical to the pre-fix shape. A custom provider registered under its own id inherits **no** quirks — set `WithQuirks` explicitly if it needs the echo. -- **Gemini** — `thought: true` parts map to reasoning deltas; `thinkingConfig` is derived from `Thinking` / `ThinkingBudget`. +- **Gemini** — a prompt blocked by safety (`promptFeedback.blockReason`, no candidates) completes with `content_filter` instead of erroring. `thought: true` parts map to reasoning deltas; `thinkingConfig` is derived from `Thinking` / `ThinkingBudget`. `thoughtSignature`s are captured per function call (`ToolCall.Signature`) and on text parts (`ThinkingSignature`) and replayed on the same parts — Gemini 3 rejects a function-call turn whose signature is missing. - **OpenAI-format gateways (OpenRouter, LiteLLM, vLLM)** — responses are accepted in every documented reasoning shape: `reasoning_content` (LiteLLM standardized), the `reasoning` string alias (OpenRouter), and the typed `reasoning_details` array (OpenRouter; `reasoning.text`/`reasoning.summary` fold in order, `reasoning.encrypted` entries are skipped). Precedence: `reasoning_content`, then `reasoning`, then the `reasoning_details` fold. Gateways that require an explicit opt-in get it via `Quirks.IncludeReasoning`, which sends the OpenRouter-documented legacy `include_reasoning: true` (equivalent to the canonical `reasoning: {}`) next to `reasoning_effort`. The flag is off by default — strict OpenAI-format endpoints reject unknown parameters. `ThinkingBudget`, when positive, overrides the selected non-disabled thinking preset (Anthropic enforces its 1024-token minimum). Canonical `max` selects the highest portable preset: OpenAI `high`, Gemini 24576, Anthropic 16384; GLM retains its native `max`. @@ -208,17 +273,17 @@ When a provider rejects a request pattern, the SDK learns the constraint **once | Rejects streaming itself | downgrade to buffered calls permanently | | Answers a streamed request with a non-SSE body | consume that JSON response directly, then use buffered calls permanently (no duplicate generation) | -Every engagement is observable: `SetLearnObserver(func(LearnEvent))` registers a process-wide callback fired once per engaged fallback, with the kind (`buffered`, `responses`, `none_effort`, `drop_stream_options`), provider id, triggering HTTP status, and the provider's error message. Nil (default) disables observation. The callback runs on the request goroutine — keep it fast and never call back into the SDK from it. +Every engagement is observable: `WithLearnObserver(func(LearnEvent))` registers a per-SDK callback (taking precedence over the process-wide `SetLearnObserver`) fired once per engaged fallback, with the kind (`buffered`, `responses`, `none_effort`, `drop_stream_options`), provider id, triggering HTTP status, and the provider's error message. Nil (default) disables observation. The callback runs on the request goroutine — keep it fast and never call back into the SDK from it. ## Retry policy -8 attempts, exponential backoff capped at 30s with ±20% jitter, `Retry-After` (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. A 429 whose body is billing exhaustion (`insufficient_quota`, `exceeded your current quota`, `insufficient balance`, `no resource package`) is not retryable and fails on the first attempt. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`. +8 attempts, exponential backoff capped at 30s with ±20% jitter (both configurable per SDK with `WithRetryPolicy(llm.RetryPolicy{MaxAttempts, MaxBackoff})`; `MaxAttempts: 1` disables retries), `Retry-After` (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. A 429 whose body is billing exhaustion (`insufficient_quota`, `exceeded your current quota`, `insufficient balance`, `no resource package`) is not retryable and fails on the first attempt. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`. ## Timeouts & cancellation - The pooled transport honors `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` via `http.ProxyFromEnvironment`. -- Buffered calls: per-request timeout on the HTTP client (`WithRequestTimeout`, per-client `SetRequestTimeout` — race-safe, swap is atomic). -- Streaming: the same timeout becomes the hard wall-clock deadline via context; per-attempt SSE reads are additionally bounded by the idle watchdog. +- The request timeout (`WithRequestTimeout`, per-client `SetRequestTimeout` — race-safe, swap is atomic) is the wall-clock budget of the **whole call, retries included**, for buffered chat, streaming, speech, transcription and embeddings alike — a failing provider can never hold a caller for timeout × attempts. +- Streaming: per-attempt SSE reads are additionally bounded by the idle watchdog. - Every wait (backoff, Retry-After, stream reads) selects on the caller's context — cancellation propagates everywhere, and a cancelled call never misreports as "retry exhausted". - Response bodies are capped (50 MB chat, 8 MB listings, 1 MiB SSE lines, 4 MiB SSE events) as an OOM bound. @@ -276,15 +341,29 @@ res, err := sdk.Transcribe(ctx, "openai", "whisper-1", llm.TranscribeRequest{ // res.Language, res.DurationSec — provider-reported, zero when omitted ``` -Requests carry the same retry ladder and error taxonomy as chat. Oversized audio (>25MB) and empty `Audio`/`Model` fail fast with a `ConfigError` before any network I/O. A 2xx body that is not JSON surfaces as a typed `*APIError`. Fields the provider does not report stay zero — the SDK never guesses. +Requests carry the same retry ladder and error taxonomy as chat. Oversized audio (>25MB), empty `Audio`/`Model`, and control characters in `Filename`/`MIMEType` (header injection) fail fast with a `ConfigError` before any network I/O. A 2xx body that is not JSON surfaces as a typed `*APIError`. Fields the provider does not report stay zero — the SDK never guesses. + +## Embeddings + +`Embed` turns texts into vectors via OpenAI-compatible `POST {base}/embeddings` or Gemini `batchEmbedContents`; Anthropic (no endpoint) returns a `ConfigError`. + +```go +res, err := sdk.Embed(ctx, "openai", "text-embedding-3-small", llm.EmbedRequest{ + Inputs: []string{"first", "second"}, // required, each non-empty + Dimensions: 256, // optional (0 = model default) +}) +// res.Embeddings[i] is the vector for Inputs[i]; res.Usage.PromptTokens when reported +``` + +Vectors come back in input order (the SDK sorts by the provider's `index` — or trusts response order when a gateway omits it — and rejects a count or index mismatch with a typed `*APIError`). Large inputs are split into provider-sized batches (Gemini 100, OpenAI 2048) under one whole-call budget; a `models/` prefix on Gemini model ids is accepted. Same retry ladder, budget, headers and error taxonomy as chat. ## Thread safety -`SDK` and `Provider` are safe for concurrent use. `ChatClient` is safe for concurrent `Call`/`CallStream`; `SetRequestTimeout` is race-safe (atomic swap) but should still be called before the first request so in-flight calls use one timeout. Learn-once state is shared per provider via atomics — monotonic, converging, race-free. +`SDK` and `Provider` are safe for concurrent use; `SetStreamIdleTimeout` and `SetLearnObserver` may be called while requests run. `ChatClient` is safe for concurrent `Call`/`CallStream`; `SetRequestTimeout` is race-safe (atomic swap) but should still be called before the first request so in-flight calls use one timeout. Learn-once state is shared per provider via atomics — monotonic, converging, race-free. ## Model discovery -`ListModels` hits each provider's models endpoint (Anthropic paginates with `after_id`, Gemini with `pageToken`), caches per SDK for 5 minutes (`WithModelCacheTTL(0)` disables, `ForceRefresh()` bypasses), and retries transient failures 3×. Fields the provider does not report stay zero — the SDK never guesses. +`ListModels` hits each provider's models endpoint (Anthropic paginates with `after_id`, Gemini with `pageToken`), caches per SDK for 5 minutes (`WithModelCacheTTL(0)` disables, `ForceRefresh()` bypasses), and retries transient failures 3× (a malformed body is not retried). Concurrent cache misses share one upstream fetch. A listing longer than 100 pages returns `ErrModelListTruncated` — never a silently truncated list. Fields the provider does not report stay zero — the SDK never guesses. ## Testing @@ -302,7 +381,7 @@ go test -tags e2e -run 'TestE2E' -timeout 15m -v . Credentials come from the environment or a repo-root `.env` file (`KEY=VALUE`); the file is gitignored and its contents are never logged. Override a target's model with `_E2E_MODEL` (e.g. `DEEPSEEK_E2E_MODEL`). Adding a provider is one `e2eTarget` entry in `e2e_test.go`. -Coverage sits at **97.7%** of statements, including the streaming failure-orchestration paths (deadline, 429, premature close, partial-output) that are usually the blind spot of SDK test suites. The residual ~2% is unreachable defensive code. +Coverage sits at **99.0%** of statements (unit suite, no e2e), including the streaming failure-orchestration paths (deadline, 429, premature close, partial-output) that are usually the blind spot of SDK test suites. The residual ~1% is unreachable defensive code (multipart writes into an in-memory buffer, `json.Marshal` of plain values, SSE shutdown races). ## Repo guidance diff --git a/anthropic.go b/anthropic.go index 0ec3df2..e7c9bfb 100644 --- a/anthropic.go +++ b/anthropic.go @@ -33,17 +33,20 @@ type anBlock struct { Source *anImageSource `json:"source,omitempty"` // CacheControl is set on user text blocks when Message.Cache is true. CacheControl *anCacheControl `json:"cache_control,omitempty"` - // thinking (replayed assistant turns; must be the FIRST block and - // carry the provider signature) - Thinking string `json:"thinking,omitempty"` - Signature string `json:"signature,omitempty"` + // thinking / redacted_thinking (replayed assistant turns; they go + // FIRST, in response order, and carry the provider signature or the + // opaque redacted data) + Thinking *string `json:"thinking,omitempty"` // pointer: an empty signed thinking text still sends the key + Signature string `json:"signature,omitempty"` + Data string `json:"data,omitempty"` // tool_use ID string `json:"id,omitempty"` Name string `json:"name,omitempty"` Input json.RawMessage `json:"input,omitempty"` - // tool_result (string content form) + // tool_result: string content, or text/image blocks ToolUseID string `json:"tool_use_id,omitempty"` - Result string `json:"content,omitempty"` + Result any `json:"content,omitempty"` + IsError bool `json:"is_error,omitempty"` } type anImageSource struct { @@ -70,20 +73,24 @@ type anThinking struct { } type anRequest struct { - Model string `json:"model"` - MaxTokens int `json:"max_tokens"` - Messages []anMessage `json:"messages"` - System []anSysBlock `json:"system,omitempty"` - Tools []anTool `json:"tools,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - TopP *float64 `json:"top_p,omitempty"` - Stop []string `json:"stop_sequences,omitempty"` - Stream bool `json:"stream,omitempty"` - Thinking *anThinking `json:"thinking,omitempty"` + Model string `json:"model"` + MaxTokens int `json:"max_tokens"` + Messages []anMessage `json:"messages"` + System []anSysBlock `json:"system,omitempty"` + Tools []anTool `json:"tools,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopP *float64 `json:"top_p,omitempty"` + Stop []string `json:"stop_sequences,omitempty"` + Stream bool `json:"stream,omitempty"` + Thinking *anThinking `json:"thinking,omitempty"` + ToolChoice *anToolChoice `json:"tool_choice,omitempty"` } const anthropicDefaultMaxTokens = 8192 +// anthropicMinThinkingBudget is Anthropic's minimum budget_tokens. +const anthropicMinThinkingBudget = 1024 + // anthropicThinkingBudget maps canonical thinking levels to budgets. // Anthropic requires budget_tokens >= 1024. func anthropicThinkingBudget(level string, explicit int) (int, bool) { @@ -101,11 +108,51 @@ func anthropicThinkingBudget(level string, explicit int) (int, bool) { return 0, false } if explicit > 0 { - budget = maxInt(explicit, 1024) + budget = maxInt(explicit, anthropicMinThinkingBudget) } return budget, true } +// anthropicPartBlocks renders ordered content parts as text/image blocks. +func anthropicPartBlocks(parts []ContentPart) []anBlock { + blocks := make([]anBlock, 0, len(parts)) + for _, p := range parts { + if p.Type == ContentPartImage { + blocks = append(blocks, anBlock{Type: "image", Source: &anImageSource{Type: "base64", MediaType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) + } else { + blocks = append(blocks, anBlock{Type: "text", Text: p.Text}) + } + } + return blocks +} + +// anthropicThinkingReplay renders the signed reasoning of a replayed +// assistant turn. ThinkingBlocks (every thinking and redacted_thinking block, +// in response order) supersede the legacy single ReasoningContent + +// ThinkingSignature pair, which is validated here and rendered by the caller. +func anthropicThinkingReplay(m Message, i int) ([]anBlock, error) { + if len(m.ThinkingBlocks) == 0 { + if m.ReasoningContent != "" && m.ThinkingSignature == "" { + return nil, &ConfigError{Msg: fmt.Sprintf("message %d: Anthropic thinking replay requires ThinkingSignature", i)} + } + return nil, nil + } + blocks := make([]anBlock, 0, len(m.ThinkingBlocks)) + for j, tb := range m.ThinkingBlocks { + switch { + case tb.Redacted != "": + blocks = append(blocks, anBlock{Type: "redacted_thinking", Data: tb.Redacted}) + case tb.Signature == "": + return nil, &ConfigError{Msg: fmt.Sprintf("message %d thinking block %d: Anthropic thinking replay requires a Signature", i, j)} + default: + blocks = append(blocks, anBlock{Type: "thinking", Thinking: strPtr(tb.Text), Signature: tb.Signature}) + } + } + return blocks, nil +} + +func strPtr(s string) *string { return &s } + func maxInt(a, b int) int { if a > b { return a @@ -146,11 +193,34 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, } out.TopP = &p } - if out.MaxTokens <= 0 { - out.MaxTokens = anthropicDefaultMaxTokens - } if budget, ok := anthropicThinkingBudget(req.Thinking, req.ThinkingBudget); ok { + // Anthropic requires max_tokens > budget_tokens, and the budget + // counts against max_tokens. An unset MaxTokens leaves the usual + // visible-output room on top of the budget; an explicit MaxTokens + // is a hard cap, so a preset level is clamped to half of it (the + // rest stays for the answer) and an explicit ThinkingBudget that + // cannot fit fails fast. + switch { + case out.MaxTokens <= 0: + out.MaxTokens = budget + anthropicDefaultMaxTokens + case budget >= out.MaxTokens && req.ThinkingBudget > 0: + return nil, &ConfigError{Msg: fmt.Sprintf("Anthropic ThinkingBudget %d must be below MaxTokens %d", budget, out.MaxTokens)} + case budget > out.MaxTokens/2 && req.ThinkingBudget <= 0: + // A preset larger than half an explicit cap would starve the + // visible answer: keep half for output (never below Anthropic's + // minimum budget, which must still fit under the cap). + budget = maxInt(out.MaxTokens/2, anthropicMinThinkingBudget) + if budget >= out.MaxTokens { + return nil, &ConfigError{Msg: fmt.Sprintf("Anthropic thinking needs MaxTokens above %d, got %d", anthropicMinThinkingBudget, out.MaxTokens)} + } + } out.Thinking = &anThinking{Type: "enabled", BudgetTokens: budget} + // Extended thinking rejects modified sampling; the provider default + // is the only accepted value. + out.Temperature, out.TopP = nil, nil + } + if out.MaxTokens <= 0 { + out.MaxTokens = anthropicDefaultMaxTokens } for _, t := range req.Tools { schema := t.Parameters @@ -167,6 +237,9 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, } out.Tools = append(out.Tools, tool) } + if err := anthropicToolControls(req, &out, out.Thinking != nil); err != nil { + return nil, err + } // Messages. Hard edges: // - consecutive tool messages merge into ONE user message carrying @@ -183,28 +256,22 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, if len(m.Parts) == 0 { blocks = append(blocks, anBlock{Type: "text", Text: m.Content}) } else { - for _, p := range m.Parts { - if p.Type == ContentPartImage { - blocks = append(blocks, anBlock{Type: "image", Source: &anImageSource{Type: "base64", MediaType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) - } else { - blocks = append(blocks, anBlock{Type: "text", Text: p.Text}) - } - } + blocks = append(blocks, anthropicPartBlocks(m.Parts)...) } if m.Cache && len(blocks) > 0 && blocks[0].Type == "text" { blocks[0].CacheControl = &anCacheControl{Type: "ephemeral"} } out.Messages = append(out.Messages, anMessage{Role: "user", Content: blocks}) case RoleAssistant: - if m.ReasoningContent != "" && m.ThinkingSignature == "" { - return nil, &ConfigError{Msg: fmt.Sprintf("message %d: Anthropic thinking replay requires ThinkingSignature", i)} + blocks, err := anthropicThinkingReplay(m, i) + if err != nil { + return nil, err } - var blocks []anBlock - if m.ReasoningContent != "" && m.ThinkingSignature != "" { + if len(m.ThinkingBlocks) == 0 && m.ReasoningContent != "" { // Anthropic requires a replayed thinking block to be the // FIRST block and to carry its signature; extended-thinking // tool loops are otherwise rejected mid-conversation. - blocks = append(blocks, anBlock{Type: "thinking", Thinking: m.ReasoningContent, Signature: m.ThinkingSignature}) + blocks = append(blocks, anBlock{Type: "thinking", Thinking: strPtr(m.ReasoningContent), Signature: m.ThinkingSignature}) } if m.Content != "" { blocks = append(blocks, anBlock{Type: "text", Text: m.Content}) @@ -230,11 +297,13 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, var results []anBlock for ; i < len(req.Messages) && req.Messages[i].Role == RoleTool; i++ { tm := req.Messages[i] - results = append(results, anBlock{ - Type: "tool_result", - ToolUseID: tm.ToolCallID, - Result: tm.Content, - }) + tr := anBlock{Type: "tool_result", ToolUseID: tm.ToolCallID, IsError: tm.IsError} + if len(tm.Parts) > 0 { + tr.Result = anthropicPartBlocks(tm.Parts) + } else if tm.Content != "" { + tr.Result = tm.Content + } + results = append(results, tr) } i-- // outer loop increment out.Messages = append(out.Messages, anMessage{Role: "user", Content: results}) @@ -251,6 +320,7 @@ type anRespBlock struct { Text string `json:"text"` Thinking string `json:"thinking"` Signature string `json:"signature"` + Data string `json:"data"` // redacted_thinking payload ID string `json:"id"` Name string `json:"name"` Input json.RawMessage `json:"input"` @@ -323,6 +393,15 @@ func parseAnthropicResponse(body []byte) (*ChatResult, error) { content = append(content, b.Text) case "thinking": thinking = append(thinking, b.Thinking) + res.ThinkingBlocks = append(res.ThinkingBlocks, ThinkingBlock{Text: b.Thinking, Signature: b.Signature}) + if b.Signature != "" { + // Legacy single-signature field: the last block's. + res.ThinkingSignature = b.Signature + } + case "redacted_thinking": + // Opaque, encrypted reasoning: never surfaced as text, but it + // must be replayed verbatim for tool loops to stay valid. + res.ThinkingBlocks = append(res.ThinkingBlocks, ThinkingBlock{Redacted: b.Data}) case "tool_use": args := string(b.Input) if args == "" { @@ -333,11 +412,6 @@ func parseAnthropicResponse(body []byte) (*ChatResult, error) { } res.Content = strings.Join(content, "") res.ReasoningContent = strings.Join(thinking, "") - for _, b := range r.Content { - if b.Type == "thinking" && b.Signature != "" { - res.ThinkingSignature = b.Signature - } - } res.Usage = usageFromAnthropic(r.Usage) return res, nil } @@ -365,9 +439,7 @@ type anStreamEvent struct { PartialJSON string `json:"partial_json"` } `json:"delta"` // message_delta usage (top-level sibling of delta) - Usage struct { - OutputTokens int `json:"output_tokens"` - } `json:"usage"` + Usage anUsage `json:"usage"` // message_delta: cumulative counts // error Error *struct { Type string `json:"type"` @@ -386,15 +458,25 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro case "message_start": acc.usage = usageFromAnthropic(ev.Message.Usage) case "content_block_start": - if ev.ContentBlock.Type == "tool_use" { + switch ev.ContentBlock.Type { + case "tool_use": + if acc.jsonTool != "" && acc.jsonBlock < 0 && ev.ContentBlock.Name == acc.jsonTool { + // JSON-mode tool: its input streams as content. + acc.jsonBlock = ev.Index + break + } c := acc.call(ev.Index) c.id, c.name = ev.ContentBlock.ID, ev.ContentBlock.Name deltas = append(deltas, Delta{ Kind: DeltaToolArgs, - ToolIndex: ev.Index, + ToolIndex: c.pos, ToolID: c.id, ToolName: c.name, }) + case "thinking": + acc.block(ev.Index) + case "redacted_thinking": + acc.block(ev.Index).redacted = ev.ContentBlock.Data } case "content_block_delta": switch ev.Delta.Type { @@ -403,16 +485,26 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro deltas = append(deltas, Delta{Kind: DeltaContent, Text: ev.Delta.Text}) case "thinking_delta": acc.reasoning.WriteString(ev.Delta.Thinking) + acc.block(ev.Index).text.WriteString(ev.Delta.Thinking) deltas = append(deltas, Delta{Kind: DeltaReasoning, Text: ev.Delta.Thinking}) case "signature_delta": - acc.thinkingSignature += ev.Delta.Signature + // Signatures belong to their own block; concatenating them + // across blocks would produce an invalid signature. + b := acc.block(ev.Index) + b.sig += ev.Delta.Signature + acc.thinkingSignature = b.sig case "input_json_delta": + if acc.jsonTool != "" && ev.Index == acc.jsonBlock { + acc.content.WriteString(ev.Delta.PartialJSON) + deltas = append(deltas, Delta{Kind: DeltaContent, Text: ev.Delta.PartialJSON}) + break + } c := acc.call(ev.Index) c.args.WriteString(ev.Delta.PartialJSON) deltas = append(deltas, Delta{ Kind: DeltaToolArgs, Text: ev.Delta.PartialJSON, - ToolIndex: ev.Index, + ToolIndex: c.pos, ToolID: c.id, ToolName: c.name, }) @@ -420,8 +512,28 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro case "message_delta": if ev.Delta.StopReason != "" { acc.finishReason = mapAnthropicStopReason(ev.Delta.StopReason) + acc.sawFinish = true + if acc.jsonBlock >= 0 && acc.finishReason == FinishToolCalls && len(acc.calls) == 0 { + acc.finishReason = FinishStop // the JSON tool is the answer, not a tool turn + } + } + // message_delta usage is cumulative; input and cache volumes may only + // arrive here. Non-zero values win, so a sparse delta never zeroes + // what message_start reported. + if ev.Usage.OutputTokens > 0 { + acc.usage.CompletionTokens = ev.Usage.OutputTokens + } + if ev.Usage.InputTokens > 0 { + acc.usage.PromptTokens = ev.Usage.InputTokens + } + if ev.Usage.CacheReadTokens > 0 { + acc.usage.CacheReadTokens = ev.Usage.CacheReadTokens + acc.usage.CacheReported = true + } + if ev.Usage.CacheCreationTokens > 0 { + acc.usage.CacheCreationTokens = ev.Usage.CacheCreationTokens + acc.usage.CacheReported = true } - acc.usage.CompletionTokens = ev.Usage.OutputTokens case "message_stop": return deltas, true, nil case "error": @@ -454,7 +566,7 @@ type anModelsPage struct { func listModelsAnthropic(ctx context.Context, pc *providerClient) ([]Model, error) { var out []Model pageID := "" - for page := 0; page < 10; page++ { + for page := 0; page < maxModelPages; page++ { url := pc.base + "/v1/models?limit=100" if pageID != "" { url += "&after_id=" + neturl.QueryEscape(pageID) @@ -465,7 +577,7 @@ func listModelsAnthropic(ctx context.Context, pc *providerClient) ([]Model, erro } var p anModelsPage if err := json.Unmarshal(data, &p); err != nil { - return nil, fmt.Errorf("llm: parse models response: %w", err) + return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) } for _, m := range p.Data { mm := Model{ID: m.ID, DisplayName: m.DisplayName} @@ -479,5 +591,5 @@ func listModelsAnthropic(ctx context.Context, pc *providerClient) ([]Model, erro } pageID = p.LastID } - return out, nil + return nil, fmt.Errorf("%w (%d pages)", ErrModelListTruncated, maxModelPages) } diff --git a/anthropic_test.go b/anthropic_test.go index 36a44b3..442a7e3 100644 --- a/anthropic_test.go +++ b/anthropic_test.go @@ -72,9 +72,10 @@ func TestBuildAnthropicRequest_Golden(t *testing.T) { } } - // max_tokens default + thinking budget. - if m["max_tokens"].(float64) != 8192 { - t.Errorf("max_tokens = %v, want default 8192", m["max_tokens"]) + // max_tokens default leaves the visible-output default on top of the + // thinking budget (Anthropic requires max_tokens > budget_tokens). + if m["max_tokens"].(float64) != 2000+8192 { + t.Errorf("max_tokens = %v, want budget+8192", m["max_tokens"]) } th := m["thinking"].(map[string]any) if th["type"] != "enabled" || th["budget_tokens"].(float64) != 2000 { diff --git a/anthropic_thinking_test.go b/anthropic_thinking_test.go new file mode 100644 index 0000000..a2b8ecc --- /dev/null +++ b/anthropic_thinking_test.go @@ -0,0 +1,224 @@ +package llm + +import ( + "encoding/json" + "errors" + "strings" + "testing" +) + +func decodeAnthropicBody(t *testing.T, req *ChatRequest) map[string]any { + t.Helper() + b, err := buildAnthropicRequest(req, "claude-test", false) + if err != nil { + t.Fatalf("build: %v", err) + } + var m map[string]any + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + return m +} + +// Anthropic requires max_tokens > thinking.budget_tokens. An unset +// MaxTokens must leave room for the budget plus visible output. +func TestBuildAnthropicRequest_ThinkingDefaultMaxTokensExceedsBudget(t *testing.T) { + for _, level := range []string{"enabled", "low", "medium", "high", "max"} { + m := decodeAnthropicBody(t, &ChatRequest{Thinking: level, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + budget := m["thinking"].(map[string]any)["budget_tokens"].(float64) + maxTok := m["max_tokens"].(float64) + if maxTok <= budget { + t.Errorf("%s: max_tokens %v must exceed budget_tokens %v", level, maxTok, budget) + } + if maxTok != budget+anthropicDefaultMaxTokens { + t.Errorf("%s: max_tokens = %v, want budget+%d", level, maxTok, anthropicDefaultMaxTokens) + } + } +} + +// A preset level larger than an explicit MaxTokens is clamped below it. +func TestBuildAnthropicRequest_ThinkingPresetClampedBelowMaxTokens(t *testing.T) { + m := decodeAnthropicBody(t, &ChatRequest{Thinking: "high", MaxTokens: 4000, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if m["max_tokens"].(float64) != 4000 { + t.Errorf("explicit max_tokens must be honored, got %v", m["max_tokens"]) + } + // Half the cap stays for visible output (never a one-token answer). + if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 2000 { + t.Errorf("budget_tokens = %v, want 2000 (half of max_tokens)", b) + } + m = decodeAnthropicBody(t, &ChatRequest{Thinking: "high", MaxTokens: 1500, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 1024 { + t.Errorf("small cap: budget_tokens = %v, want the 1024 minimum", b) + } + m = decodeAnthropicBody(t, &ChatRequest{Thinking: "low", MaxTokens: 1500, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 1024 { + t.Errorf("preset below cap is kept: budget_tokens = %v", b) + } +} + +// Contradictory explicit settings fail fast instead of a guaranteed 400. +func TestBuildAnthropicRequest_ThinkingBudgetConflicts(t *testing.T) { + cases := []ChatRequest{ + {Thinking: "enabled", ThinkingBudget: 8000, MaxTokens: 8000}, + {Thinking: "enabled", ThinkingBudget: 2000, MaxTokens: 1500}, + {Thinking: "low", MaxTokens: 1024}, // cannot clamp below the 1024 minimum + } + for i, req := range cases { + req.Messages = []Message{{Role: RoleUser, Content: "x"}} + _, err := buildAnthropicRequest(&req, "claude-test", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("case %d: err = %v, want ConfigError", i, err) + } + } +} + +// Extended thinking rejects modified temperature / top_p. +func TestBuildAnthropicRequest_ThinkingOmitsSampling(t *testing.T) { + m := decodeAnthropicBody(t, &ChatRequest{Thinking: "low", Temperature: 0.3, TopP: 0.5, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if _, ok := m["temperature"]; ok { + t.Error("temperature must be omitted with thinking") + } + if _, ok := m["top_p"]; ok { + t.Error("top_p must be omitted with thinking") + } + m = decodeAnthropicBody(t, &ChatRequest{Temperature: 0.3, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if m["temperature"].(float64) != 0.3 { + t.Error("temperature must survive without thinking") + } +} + +func TestParseAnthropicResponse_ThinkingBlocks(t *testing.T) { + res, err := parseAnthropicResponse([]byte(`{"content":[ + {"type":"thinking","thinking":"a","signature":"S1"}, + {"type":"redacted_thinking","data":"ENC"}, + {"type":"thinking","thinking":"b","signature":"S2"}, + {"type":"tool_use","id":"t","name":"f","input":{}}],"stop_reason":"tool_use"}`)) + if err != nil { + t.Fatal(err) + } + want := []ThinkingBlock{{Text: "a", Signature: "S1"}, {Redacted: "ENC"}, {Text: "b", Signature: "S2"}} + if len(res.ThinkingBlocks) != len(want) { + t.Fatalf("blocks = %+v", res.ThinkingBlocks) + } + for i := range want { + if res.ThinkingBlocks[i] != want[i] { + t.Errorf("block %d = %+v, want %+v", i, res.ThinkingBlocks[i], want[i]) + } + } + if res.ReasoningContent != "ab" || res.ThinkingSignature != "S2" { + t.Errorf("legacy fields = %q / %q", res.ReasoningContent, res.ThinkingSignature) + } +} + +func TestMapAnthropicStreamEvent_ThinkingBlocks(t *testing.T) { + acc := newStreamAccum() + for _, e := range []string{ + `{"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}}`, + `{"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"a"}}`, + `{"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"S1"}}`, + `{"type":"content_block_start","index":1,"content_block":{"type":"redacted_thinking","data":"ENC"}}`, + `{"type":"content_block_start","index":2,"content_block":{"type":"thinking","thinking":""}}`, + `{"type":"content_block_delta","index":2,"delta":{"type":"thinking_delta","thinking":"b"}}`, + `{"type":"content_block_delta","index":2,"delta":{"type":"signature_delta","signature":"S2"}}`, + } { + if _, _, err := mapAnthropicStreamEvent([]byte(e), acc); err != nil { + t.Fatal(err) + } + } + res := acc.result() + want := []ThinkingBlock{{Text: "a", Signature: "S1"}, {Redacted: "ENC"}, {Text: "b", Signature: "S2"}} + if len(res.ThinkingBlocks) != 3 { + t.Fatalf("blocks = %+v", res.ThinkingBlocks) + } + for i := range want { + if res.ThinkingBlocks[i] != want[i] { + t.Errorf("block %d = %+v, want %+v", i, res.ThinkingBlocks[i], want[i]) + } + } + // Signatures from different blocks are never concatenated. + if res.ThinkingSignature != "S2" { + t.Errorf("ThinkingSignature = %q, want last block's S2", res.ThinkingSignature) + } +} + +func TestBuildAnthropicRequest_ReplaysThinkingBlocksInOrder(t *testing.T) { + m := decodeAnthropicBody(t, &ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, Content: "ok", ReasoningContent: "ignored", ThinkingSignature: "ignored", + ThinkingBlocks: []ThinkingBlock{{Text: "a", Signature: "S1"}, {Redacted: "ENC"}}, + ToolCalls: []ToolCall{{ID: "t", Name: "f", Arguments: `{}`}}}, + {Role: RoleTool, ToolCallID: "t", Content: "r"}, + }}) + blocks := m["messages"].([]any)[1].(map[string]any)["content"].([]any) + types := []string{} + for _, b := range blocks { + types = append(types, b.(map[string]any)["type"].(string)) + } + want := []string{"thinking", "redacted_thinking", "text", "tool_use"} + if len(types) != len(want) { + t.Fatalf("types = %v, want %v", types, want) + } + for i := range want { + if types[i] != want[i] { + t.Fatalf("types = %v, want %v", types, want) + } + } + b0 := blocks[0].(map[string]any) + if b0["thinking"] != "a" || b0["signature"] != "S1" { + t.Errorf("thinking block = %v", b0) + } + if blocks[1].(map[string]any)["data"] != "ENC" { + t.Errorf("redacted block = %v", blocks[1]) + } +} + +func TestBuildAnthropicRequest_ThinkingBlockWithoutSignatureRejected(t *testing.T) { + _, err := buildAnthropicRequest(&ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, ThinkingBlocks: []ThinkingBlock{{Text: "a"}}}, + }}, "claude", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want ConfigError", err) + } +} + +func TestChatResult_AssistantMessage(t *testing.T) { + res := &ChatResult{ + Content: "c", ReasoningContent: "r", ThinkingSignature: "s", + ThinkingBlocks: []ThinkingBlock{{Text: "r", Signature: "s"}}, + ToolCalls: []ToolCall{{ID: "1", Name: "f", Arguments: "{}", Signature: "g"}}, + } + m := res.AssistantMessage() + if m.Role != RoleAssistant || m.Content != "c" || m.ReasoningContent != "r" || m.ThinkingSignature != "s" || + len(m.ThinkingBlocks) != 1 || len(m.ToolCalls) != 1 || m.ToolCalls[0].Signature != "g" { + t.Fatalf("message = %+v", m) + } + // Copies, not aliases. + m.ToolCalls[0].ID = "x" + m.ThinkingBlocks[0].Text = "x" + if res.ToolCalls[0].ID != "1" || res.ThinkingBlocks[0].Text != "r" { + t.Error("AssistantMessage must copy slices") + } + var nilRes *ChatResult + if got := nilRes.AssistantMessage(); got.Role != RoleAssistant { + t.Errorf("nil result → %+v", got) + } +} + +// A signed thinking block with empty text (summarized/omitted display) +// still carries the required "thinking" key on replay. +func TestBuildAnthropicRequest_EmptyThinkingTextKeepsKey(t *testing.T) { + for _, m := range []Message{ + {Role: RoleAssistant, ThinkingBlocks: []ThinkingBlock{{Signature: "sig"}}}, + } { + b, err := buildAnthropicRequest(&ChatRequest{Messages: []Message{{Role: RoleUser, Content: "q"}, m}}, "claude", false) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(b), `{"type":"thinking","thinking":"","signature":"sig"}`) { + t.Errorf("body = %s", b) + } + } +} diff --git a/chat.go b/chat.go index a1069eb..2707974 100644 --- a/chat.go +++ b/chat.go @@ -8,6 +8,7 @@ import ( "fmt" "io" "net/http" + neturl "net/url" "strings" "sync/atomic" "time" @@ -46,6 +47,23 @@ type providerClient struct { streamHTTP *http.Client // no deadline; SSE body reads base string // trimmed base URL learn *learnOnce // shared learn-once fallback state + opts clientOpts // SDK-level knobs (retry, idle, observer) +} + +// clientOpts carries SDK-level settings into a providerClient. +type clientOpts struct { + retry RetryPolicy + idle time.Duration // SSE idle watchdog; 0 = process default + observer func(LearnEvent) // learn-once observer; nil = process default +} + +// retryDelay picks Retry-After when present, else the policy's capped +// exponential backoff. +func (pc *providerClient) retryDelay(ra time.Duration, attempt int) time.Duration { + if ra > 0 { + return ra + } + return backoffDelayCapped(attempt+1, pc.opts.retry.backoffCap()) } // buffered returns the current buffered-path HTTP client. @@ -80,23 +98,35 @@ const ( maxErrorBodyPreview = 512 // bytes of error body kept in APIError ) -// streamIdleTimeout bounds the silence between SSE events. Thinking models -// can legitimately spend minutes before their first event, so the default -// is generous. Package var so tests can shorten it; operators override -// via SetStreamIdleTimeout. -var streamIdleTimeout = 120 * time.Second +// defaultStreamIdleNs bounds the silence between SSE events, process-wide, +// for SDKs without WithStreamIdleTimeout. Thinking models can legitimately +// spend minutes before their first event, so the default is generous. +// Atomic so SetStreamIdleTimeout is safe while streams are running. +var defaultStreamIdleNs atomic.Int64 + +func init() { defaultStreamIdleNs.Store(int64(120 * time.Second)) } -// SetStreamIdleTimeout overrides the SSE idle watchdog. Call at startup, -// before the first request; non-positive values are ignored. +// SetStreamIdleTimeout overrides the process-wide default SSE idle +// watchdog; non-positive values are ignored. Prefer the per-SDK +// WithStreamIdleTimeout option, which takes precedence. func SetStreamIdleTimeout(d time.Duration) { if d > 0 { - streamIdleTimeout = d + defaultStreamIdleNs.Store(int64(d)) } } -// StreamIdleTimeout reports the active idle watchdog (introspection/tests). +// StreamIdleTimeout reports the process-wide default idle watchdog. func StreamIdleTimeout() time.Duration { - return streamIdleTimeout + return time.Duration(defaultStreamIdleNs.Load()) +} + +// idleTimeout resolves this client's SSE idle watchdog: the SDK option when +// set, else the process-wide default. +func (pc *providerClient) idleTimeout() time.Duration { + if pc.opts.idle > 0 { + return pc.opts.idle + } + return StreamIdleTimeout() } // errStreamStop is the internal sentinel for a clean stream end. @@ -141,23 +171,42 @@ func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, strea if err := validateRequestContent(req.Messages); err != nil { return nil, "", err } + if err := validateRequestControls(req); err != nil { + return nil, "", err + } if model == "" { model = req.Model } if model == "" { return nil, "", &ConfigError{Msg: "no model set (ChatRequest.Model empty and no ChatClient model)"} } + body, url, err := pc.buildFormatBody(req, model, stream) + if err != nil { + return nil, "", err + } + body, err = mergeExtra(body, req.Extra) + return body, url, err +} + +// buildFormatBody serializes a validated request in the provider's format. +func (pc *providerClient) buildFormatBody(req *ChatRequest, model string, stream bool) ([]byte, string, error) { switch pc.cfg.Format { case FormatAnthropic: body, err := buildAnthropicRequest(req, model, stream) return body, pc.base + "/v1/messages", err case FormatGemini: body, err := buildGeminiRequest(req, model, stream) + // The model rides the URL path: escape it so a model id can never + // rewrite the path or smuggle a query. + m := neturl.PathEscape(model) if stream { - return body, fmt.Sprintf("%s/v1beta/models/%s:streamGenerateContent?alt=sse", pc.base, model), err + return body, fmt.Sprintf("%s/v1beta/models/%s:streamGenerateContent?alt=sse", pc.base, m), err } - return body, fmt.Sprintf("%s/v1beta/models/%s:generateContent", pc.base, model), err + return body, fmt.Sprintf("%s/v1beta/models/%s:generateContent", pc.base, m), err default: // FormatOpenAI + if pc.cfg.Quirks.NoJSONSchema && req.ResponseFormat != nil && req.ResponseFormat.Type == ResponseJSONSchema { + return nil, "", &ConfigError{Msg: pc.cfg.ID + " supports ResponseFormat json_object only, not json_schema"} + } if useResponsesAPI(pc.learn, pc.cfg.Format, model, req) { body, err := json.Marshal(buildResponsesRequest(req, model, stream)) return body, pc.base + "/responses", err @@ -171,8 +220,56 @@ func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, strea } } -// setAuthHeaders applies format-specific authentication headers. Keys are -// never logged. +// extraReserved are body keys Extra may not set: "stream" drives the SDK's +// buffered/SSE handling, and the model, conversation, system prompt and tool +// catalog are validated by the SDK (roles, content, controls), so replacing +// them would bypass that validation. +var extraReserved = map[string]bool{ + "stream": true, "model": true, + "messages": true, "contents": true, "input": true, + "system": true, "systemInstruction": true, "instructions": true, + "tools": true, +} + +// mergeExtra overlays ChatRequest.Extra onto a marshaled body: top-level +// keys replace the SDK's own. "stream" is reserved because the SDK's +// buffered/SSE handling depends on it. +func mergeExtra(body []byte, extra map[string]any) ([]byte, error) { + if len(extra) == 0 { + return body, nil + } + var m map[string]json.RawMessage + if err := json.Unmarshal(body, &m); err != nil { + return nil, err + } + for k, v := range extra { + if k == "" || extraReserved[k] { + return nil, &ConfigError{Msg: fmt.Sprintf("ChatRequest.Extra key %q is reserved", k)} + } + raw, err := json.Marshal(v) + if err != nil { + return nil, &ConfigError{Msg: fmt.Sprintf("ChatRequest.Extra[%q]: %v", k, err)} + } + m[k] = raw + } + return json.Marshal(m) +} + +// setHeaders applies format-specific authentication headers, then the +// provider's custom Headers (last, so they can override; an empty value +// removes the header). Keys and header values are never logged. +func (pc *providerClient) setHeaders(h http.Header) { + pc.setAuthHeaders(h) + for k, v := range pc.cfg.Headers { + if v == "" { + h.Del(k) + continue + } + h.Set(k, v) + } +} + +// setAuthHeaders applies format-specific authentication headers. func (pc *providerClient) setAuthHeaders(h http.Header) { switch pc.cfg.Format { case FormatAnthropic: @@ -268,7 +365,7 @@ func (pc *providerClient) post(ctx context.Context, client *http.Client, url str } req.Header.Set("Content-Type", "application/json") req.Header.Set("Accept", "application/json") - pc.setAuthHeaders(req.Header) + pc.setHeaders(req.Header) resp, err := client.Do(req) if err != nil { @@ -356,111 +453,73 @@ func billingExhausted(e *APIError) bool { strings.Contains(m, "exceeded your current quota") } -// retryDelay picks Retry-After when present, else exponential backoff. -func retryDelay(ra time.Duration, attempt int) time.Duration { - if ra > 0 { - return ra - } - return backoffDelay(attempt + 1) -} - // ── buffered chat ──────────────────────────────────────────────────────── -// call runs a buffered chat completion with retries. +// call runs a buffered chat completion with retries. The request timeout +// is the wall-clock budget of the whole call, retries included — the same +// contract as streaming — so a failing provider can never hold a caller for +// timeout × attempts. func (pc *providerClient) call(ctx context.Context, req *ChatRequest, model string) (*ChatResult, error) { - var ( - lastErr error - rateErr *APIError // last 429 - rateRA time.Duration - ) - for attempt := 0; attempt <= maxRetries; attempt++ { - if err := ctx.Err(); err != nil { - return nil, err - } + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) + defer cancel() + var res *ChatResult + err := pc.withRetry(ctx, func() (time.Duration, error) { body, url, err := pc.buildChatRequest(req, model, false) if err != nil { - return nil, err + return 0, terminal(err) } data, ra, err := pc.post(ctx, pc.buffered(), url, body) if err != nil { - var apiErr *APIError - if errors.As(err, &apiErr) { - switch { - case apiErr.Status == http.StatusTooManyRequests: - if billingExhausted(apiErr) { - // Permanent: only a recharge fixes this. - return nil, apiErr - } - rateErr, rateRA, lastErr = apiErr, ra, apiErr - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(ra, attempt)) { - // The sleep died with the context (deadline or - // cancel); still surface the 429 — the caller - // needs Status/RetryAfter to plan the retry. - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - continue - } - case apiErr.Retryable && attempt < maxRetries: - lastErr = apiErr - if !retrySleep(ctx, retryDelay(ra, attempt)) { - return nil, ctx.Err() - } - continue - case apiErr.Status == http.StatusBadRequest && len(req.Tools) > 0 && - !pc.learn.forceResponses.Load() && responsesRequired(apiErr): - // GPT-5.6+ (and some 5.4/5.5 payloads) reject - // effort+tools on Chat Completions; retry on /responses - // so reasoning stays on. - pc.engageLearn(&pc.learn.forceResponses, LearnResponses, apiErr) - lastErr = apiErr - if attempt < maxRetries { - continue - } - return nil, apiErr - case apiErr.Status == http.StatusBadRequest && len(req.Tools) > 0 && - !pc.learn.forceNoneEffort.Load() && reasoningEffortRejected(apiErr): - // Learn the constraint once; retry immediately with - // effort pinned to "none". - pc.engageLearn(&pc.learn.forceNoneEffort, LearnNoneEffort, apiErr) - lastErr = apiErr - if attempt < maxRetries { - continue - } - return nil, apiErr - } - if rateErr != nil && !apiErr.Retryable && apiErr.Status != http.StatusTooManyRequests { - // A definitive failure after earlier 429s. - return nil, apiErr - } - if rateErr != nil { - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - return nil, apiErr - } - // Transport error — retryable. - lastErr = err - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(0, attempt)) { - return nil, ctx.Err() - } - continue - } - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", maxRetries+1, err) + return ra, err } - return pc.parseResponse(data, url) + res, err = pc.parseResponse(data, url, pc.jsonTool(req)) + return 0, terminal(err) + }, func(apiErr *APIError) bool { + return pc.learnFromChatError(req, apiErr) + }) + if err != nil { + return nil, err } - if rateErr != nil { - return nil, &RateLimitError{APIError: *rateErr, Attempts: maxRetries + 1, RetryAfter: rateRA} + return res, nil +} + +// learnFromChatError engages the buffered path's learn-once fallbacks for +// a 400 and reports whether the request should be retried immediately. +func (pc *providerClient) learnFromChatError(req *ChatRequest, apiErr *APIError) bool { + if apiErr.Status != http.StatusBadRequest || len(req.Tools) == 0 { + return false + } + switch { + case !pc.learn.forceResponses.Load() && responsesRequired(apiErr): + // GPT-5.6+ (and some 5.4/5.5 payloads) reject effort+tools on Chat + // Completions; retry on /responses so reasoning stays on. + pc.engageLearn(&pc.learn.forceResponses, LearnResponses, apiErr) + return true + case !pc.learn.forceNoneEffort.Load() && reasoningEffortRejected(apiErr): + // Learn the constraint once; retry immediately with effort pinned + // to "none". + pc.engageLearn(&pc.learn.forceNoneEffort, LearnNoneEffort, apiErr) + return true + } + return false +} + +// jsonTool is the Anthropic JSON-mode tool for req ("" elsewhere). +func (pc *providerClient) jsonTool(req *ChatRequest) string { + if pc.cfg.Format != FormatAnthropic { + return "" } - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", maxRetries+1, lastErr) + return anthropicJSONToolName(req) } -// parseResponse dispatches format-specific buffered parsing. -func (pc *providerClient) parseResponse(data []byte, url string) (*ChatResult, error) { +// parseResponse dispatches format-specific buffered parsing. jsonTool +// names the Anthropic JSON-mode tool to fold into Content ("" = off). +func (pc *providerClient) parseResponse(data []byte, url, jsonTool string) (*ChatResult, error) { switch pc.cfg.Format { case FormatAnthropic: - return parseAnthropicResponse(data) + res, err := parseAnthropicResponse(data) + foldAnthropicJSON(res, jsonTool) + return res, err case FormatGemini: return parseGeminiResponse(data) default: @@ -499,18 +558,20 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode defer cancel() mapper := pc.mapper() + n := pc.opts.retry.attempts() var ( lastErr error rateErr *APIError rateRA time.Duration ) - for attempt := 0; attempt <= maxRetries; attempt++ { + for attempt := 0; attempt < n; attempt++ { if err := deadlineCtx.Err(); err != nil { break } if pc.learn.forceBuffered.Load() { - cancel() - return pc.call(ctx, req, model) + // Another client learned buffered mode mid-call: finish within + // this call's remaining budget, never a fresh one. + return pc.call(deadlineCtx, req, model) } body, url, err := pc.buildChatRequest(req, model, true) if err != nil { @@ -521,7 +582,7 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode attemptMapper = mapResponsesStreamEvent } - out := pc.attemptStream(deadlineCtx, url, body, attemptMapper, onDelta, len(req.Tools) > 0) + out := pc.attemptStream(deadlineCtx, url, body, attemptMapper, onDelta, len(req.Tools) > 0, pc.jsonTool(req)) switch { case out.bufferedResponse: // A successful HTTP response already contains this generation. @@ -537,12 +598,8 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode // partial result (a retry would duplicate user-visible output). return out.result, out.err case out.learnRetry: // learn-once applied; retry without backoff - if out.apiErr != nil { - lastErr = out.apiErr - } else if out.err != nil { - lastErr = out.err - } - if attempt < maxRetries { + lastErr = out.apiErr // every learn trigger is a provider 400 + if attempt < n-1 { continue } // Learn trigger fired on the final attempt: surface the cause @@ -553,7 +610,7 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode rateErr, rateRA = out.apiErr, out.retryAfter } lastErr = out.apiErr - if attempt < maxRetries && retrySleep(deadlineCtx, retryDelay(out.retryAfter, attempt)) { + if attempt < n-1 && retrySleep(deadlineCtx, pc.retryDelay(out.retryAfter, attempt)) { continue } if rateErr != nil { @@ -562,24 +619,21 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode return nil, out.apiErr case out.apiErr != nil: return nil, out.apiErr - case out.err != nil && attempt < maxRetries: + case out.err != nil && attempt < n-1: lastErr = out.err - if retrySleep(deadlineCtx, retryDelay(0, attempt)) { + if retrySleep(deadlineCtx, pc.retryDelay(0, attempt)) { continue } - if err := deadlineCtx.Err(); err != nil { - return nil, err // interrupted by deadline/cancel, not exhaustion - } - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", attempt+1, lastErr) + return nil, deadlineCtx.Err() // interrupted by deadline/cancel, not exhaustion default: return nil, out.err } } if rateErr != nil { - return nil, &RateLimitError{APIError: *rateErr, Attempts: maxRetries + 1, RetryAfter: rateRA} + return nil, &RateLimitError{APIError: *rateErr, Attempts: n, RetryAfter: rateRA} } if lastErr != nil { - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", maxRetries+1, lastErr) + return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", n, lastErr) } return nil, deadlineCtx.Err() } @@ -602,14 +656,14 @@ func (o *streamOutcome) success() bool { // attemptStream performs one streaming attempt. learnEffort enables the // reasoning_effort learn-once trigger (set when the request carries tools). -func (pc *providerClient) attemptStream(ctx context.Context, url string, body []byte, mapper streamEventMapper, onDelta func(Delta) error, learnEffort bool) streamOutcome { +func (pc *providerClient) attemptStream(ctx context.Context, url string, body []byte, mapper streamEventMapper, onDelta func(Delta) error, learnEffort bool, jsonTool string) streamOutcome { req, rerr := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) if rerr != nil { return streamOutcome{err: &ConfigError{Msg: "build request: " + rerr.Error()}} } req.Header.Set("Content-Type", "application/json") req.Header.Set("Accept", "text/event-stream") - pc.setAuthHeaders(req.Header) + pc.setHeaders(req.Header) resp, derr := pc.streamHTTP.Do(req) if derr != nil { @@ -650,12 +704,13 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] if len(data) > maxResponseSize { return streamOutcome{bufferedResponse: true, err: fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize)} } - result, err := pc.parseResponse(data, url) + result, err := pc.parseResponse(data, url, jsonTool) return streamOutcome{bufferedResponse: true, result: result, err: err} } acc := newStreamAccum() - perr := pumpSSE(ctx, resp.Body, streamIdleTimeout, func(data []byte) error { + acc.jsonTool = jsonTool + perr := pumpSSE(ctx, resp.Body, pc.idleTimeout(), func(data []byte) error { if string(bytes.TrimSpace(data)) == "[DONE]" { return errStreamStop } @@ -677,10 +732,11 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] switch { case perr == nil, errors.Is(perr, errStreamStop): - if perr == nil && pc.cfg.Format != FormatGemini && acc.finishReason == "" { - // Gemini completes at EOF; every other format has an explicit - // completion signal that never arrived — the provider dropped - // the stream. Never surface this as an empty success. + if perr == nil && !acc.sawFinish { + // Every format sends an explicit completion signal ([DONE], + // finish_reason, message_stop, finishReason, response.completed) + // before EOF. It never arrived: the provider dropped the stream. + // Never surface this as a silent (partial or empty) success. if acc.emitted { return streamOutcome{ result: acc.result(), @@ -713,39 +769,72 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] // toolCallAccum assembles one tool call from streaming fragments: the first // fragment for an index carries id/name; argument fragments concatenate. +// pos is the call's position in ChatResult.ToolCalls — the canonical +// Delta.ToolIndex on every format. type toolCallAccum struct { + pos int id string name string + sig string args strings.Builder } +// thinkingAccum assembles one signed reasoning segment (ThinkingBlock). +type thinkingAccum struct { + text strings.Builder + sig string + redacted string +} + // streamAccum assembles a ChatResult from SSE chunks across formats. type streamAccum struct { content strings.Builder reasoning strings.Builder calls []*toolCallAccum callIndex map[int]*toolCallAccum + thinking []*thinkingAccum + thinkingIndex map[int]*thinkingAccum finishReason string + sawFinish bool // provider sent a completion signal (even an unmapped one) usage Usage emitted bool // any delta delivered to the consumer - thinkingSignature string // anthropic signature_delta capture + thinkingSignature string // legacy single signature (last block's) + jsonTool string // Anthropic JSON-mode synthetic tool name ("" = off) + jsonBlock int // content-block index of the JSON-mode tool_use (-1 = none) } func newStreamAccum() *streamAccum { - return &streamAccum{callIndex: make(map[int]*toolCallAccum)} + return &streamAccum{ + callIndex: make(map[int]*toolCallAccum), + thinkingIndex: make(map[int]*thinkingAccum), + jsonBlock: -1, + } } -// call returns the accumulator for tool-call index, creating it in order. +// call returns the accumulator for provider index idx, creating it in +// first-seen order. func (a *streamAccum) call(idx int) *toolCallAccum { if c, ok := a.callIndex[idx]; ok { return c } - c := &toolCallAccum{} + c := &toolCallAccum{pos: len(a.calls)} a.callIndex[idx] = c a.calls = append(a.calls, c) return c } +// block returns the thinking accumulator for provider index idx, creating +// it in first-seen order. +func (a *streamAccum) block(idx int) *thinkingAccum { + if b, ok := a.thinkingIndex[idx]; ok { + return b + } + b := &thinkingAccum{} + a.thinkingIndex[idx] = b + a.thinking = append(a.thinking, b) + return b +} + func (a *streamAccum) result() *ChatResult { res := &ChatResult{ Content: a.content.String(), @@ -754,8 +843,11 @@ func (a *streamAccum) result() *ChatResult { FinishReason: a.finishReason, Usage: a.usage, } + for _, b := range a.thinking { + res.ThinkingBlocks = append(res.ThinkingBlocks, ThinkingBlock{Text: b.text.String(), Signature: b.sig, Redacted: b.redacted}) + } for _, c := range a.calls { - res.ToolCalls = append(res.ToolCalls, ToolCall{ID: c.id, Name: c.name, Arguments: c.args.String()}) + res.ToolCalls = append(res.ToolCalls, ToolCall{ID: c.id, Name: c.name, Arguments: c.args.String(), Signature: c.sig}) } return res } diff --git a/chat_test.go b/chat_test.go index 1ff83cd..6230837 100644 --- a/chat_test.go +++ b/chat_test.go @@ -324,9 +324,7 @@ func TestCallStream_NonSSEBodyFallsBack(t *testing.T) { } func TestCallStream_IdleRetryBeforeFirstDelta(t *testing.T) { - oldIdle := streamIdleTimeout - streamIdleTimeout = 80 * time.Millisecond - t.Cleanup(func() { streamIdleTimeout = oldIdle }) + setIdleForTest(t, 80*time.Millisecond) var attempts int var mu sync.Mutex diff --git a/content.go b/content.go index 3c38eaf..5de2173 100644 --- a/content.go +++ b/content.go @@ -28,14 +28,17 @@ func validateRequestContent(messages []Message) error { } func validateMessageContent(m Message, index int) error { + if m.IsError && m.Role != RoleTool { + return &ConfigError{Msg: fmt.Sprintf("message %d: IsError requires the tool role", index)} + } if len(m.Parts) == 0 { return nil } if m.Content != "" { return &ConfigError{Msg: fmt.Sprintf("message %d: Content and Parts cannot both be set", index)} } - if m.Role != RoleUser { - return &ConfigError{Msg: fmt.Sprintf("message %d: content parts require user role", index)} + if m.Role != RoleUser && m.Role != RoleTool { + return &ConfigError{Msg: fmt.Sprintf("message %d: content parts require the user or tool role", index)} } for j, p := range m.Parts { switch p.Type { @@ -82,3 +85,29 @@ func wireMIME(m string) string { } return m } + +// partsText returns a message's text: Content, or its text parts joined by +// newlines when it carries Parts. +func partsText(m Message) string { + if len(m.Parts) == 0 { + return m.Content + } + var texts []string + for _, p := range m.Parts { + if p.Type == ContentPartText { + texts = append(texts, p.Text) + } + } + return strings.Join(texts, "\n") +} + +// imageParts returns a message's inline-image parts. +func imageParts(m Message) []ContentPart { + var out []ContentPart + for _, p := range m.Parts { + if p.Type == ContentPartImage { + out = append(out, p) + } + } + return out +} diff --git a/controls.go b/controls.go new file mode 100644 index 0000000..e2f2d1f --- /dev/null +++ b/controls.go @@ -0,0 +1,306 @@ +package llm + +import ( + "encoding/json" + "fmt" + "regexp" + "strings" +) + +// Request controls (ToolChoice, ParallelToolCalls, ResponseFormat, Seed) +// are validated once at the SDK boundary, then mapped per format by the +// request builders. Contradictory or malformed controls fail fast with a +// ConfigError instead of a provider 400. + +// schemaNameRE is the OpenAI json_schema name contract, applied to every +// format so one request stays portable. +var schemaNameRE = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`) + +// defaultSchemaName names a json_schema response format without a Name. +const defaultSchemaName = "response" + +func validateRequestControls(req *ChatRequest) error { + if tc := req.ToolChoice; tc != nil { + if len(req.Tools) == 0 { + return &ConfigError{Msg: "ToolChoice requires Tools"} + } + switch tc.Mode { + case ToolChoiceAuto, ToolChoiceNone, ToolChoiceRequired: + if tc.Name != "" { + return &ConfigError{Msg: fmt.Sprintf("ToolChoice mode %q takes no Name", tc.Mode)} + } + case ToolChoiceTool: + if tc.Name == "" { + return &ConfigError{Msg: "ToolChoice mode \"tool\" requires a Name"} + } + if !hasTool(req.Tools, tc.Name) { + return &ConfigError{Msg: fmt.Sprintf("ToolChoice names unknown tool %q", tc.Name)} + } + default: + return &ConfigError{Msg: fmt.Sprintf("unknown ToolChoice mode %q", tc.Mode)} + } + } + if rf := req.ResponseFormat; rf != nil { + switch rf.Type { + case "", ResponseText, ResponseJSONObject: + if len(rf.Schema) > 0 { + return &ConfigError{Msg: fmt.Sprintf("ResponseFormat %q takes no Schema", rf.Type)} + } + case ResponseJSONSchema: + t := strings.TrimSpace(string(rf.Schema)) + if !strings.HasPrefix(t, "{") || !json.Valid([]byte(t)) { + return &ConfigError{Msg: "ResponseFormat json_schema requires a JSON Schema object"} + } + default: + return &ConfigError{Msg: fmt.Sprintf("unknown ResponseFormat type %q", rf.Type)} + } + if rf.Name != "" && !schemaNameRE.MatchString(rf.Name) { + return &ConfigError{Msg: fmt.Sprintf("ResponseFormat Name %q must match [A-Za-z0-9_-]{1,64}", rf.Name)} + } + } + return nil +} + +func hasTool(tools []ToolDef, name string) bool { + for _, t := range tools { + if t.Name == name { + return true + } + } + return false +} + +// jsonMode reports whether rf asks for JSON output. +func (rf *ResponseFormat) jsonMode() bool { + return rf != nil && (rf.Type == ResponseJSONObject || rf.Type == ResponseJSONSchema) +} + +// schemaName is the json_schema name sent on the wire. +func (rf *ResponseFormat) schemaName() string { + if rf.Name != "" { + return rf.Name + } + return defaultSchemaName +} + +// ── OpenAI chat completions ────────────────────────────────────────────── + +type oaJSONSchema struct { + Name string `json:"name"` + Schema json.RawMessage `json:"schema"` + Strict bool `json:"strict,omitempty"` +} + +type oaResponseFormat struct { + Type string `json:"type"` + JSONSchema *oaJSONSchema `json:"json_schema,omitempty"` +} + +// openAIToolChoice renders tool_choice ("auto" | "none" | "required" | +// {"type":"function","function":{"name":…}}). +func openAIToolChoice(tc *ToolChoice) any { + if tc == nil { + return nil + } + if tc.Mode == ToolChoiceTool { + return map[string]any{"type": "function", "function": map[string]string{"name": tc.Name}} + } + return string(tc.Mode) +} + +func openAIResponseFormat(rf *ResponseFormat) *oaResponseFormat { + switch { + case !rf.jsonMode(): + return nil + case rf.Type == ResponseJSONObject: + return &oaResponseFormat{Type: "json_object"} + default: + return &oaResponseFormat{Type: "json_schema", JSONSchema: &oaJSONSchema{Name: rf.schemaName(), Schema: rf.Schema, Strict: rf.Strict}} + } +} + +// parallelToolCalls is sent only alongside tools (OpenAI rejects it +// otherwise). +func parallelToolCalls(req *ChatRequest) *bool { + if len(req.Tools) == 0 { + return nil + } + return req.ParallelToolCalls +} + +// ── OpenAI Responses ───────────────────────────────────────────────────── + +type rsTextFormat struct { + Type string `json:"type"` + Name string `json:"name,omitempty"` + Schema json.RawMessage `json:"schema,omitempty"` + Strict bool `json:"strict,omitempty"` +} + +type rsText struct { + Format rsTextFormat `json:"format"` +} + +func responsesToolChoice(tc *ToolChoice) any { + if tc == nil { + return nil + } + if tc.Mode == ToolChoiceTool { + return map[string]string{"type": "function", "name": tc.Name} + } + return string(tc.Mode) +} + +func responsesText(rf *ResponseFormat) *rsText { + switch { + case !rf.jsonMode(): + return nil + case rf.Type == ResponseJSONObject: + return &rsText{Format: rsTextFormat{Type: "json_object"}} + default: + return &rsText{Format: rsTextFormat{Type: "json_schema", Name: rf.schemaName(), Schema: rf.Schema, Strict: rf.Strict}} + } +} + +// ── Anthropic ──────────────────────────────────────────────────────────── + +type anToolChoice struct { + Type string `json:"type"` // auto | any | tool | none + Name string `json:"name,omitempty"` + DisableParallelToolUse bool `json:"disable_parallel_tool_use,omitempty"` +} + +// anthropicJSONToolDefault names the synthetic JSON-mode tool when +// ResponseFormat.Name is empty. +const anthropicJSONToolDefault = "json_response" + +// anthropicJSONToolName returns the synthetic tool that carries a JSON +// response on Anthropic ("" when JSON mode is off). Anthropic has no +// portable response_format, so JSON mode forces a tool whose input schema +// is the requested schema and folds its input back into Content. +func anthropicJSONToolName(req *ChatRequest) string { + if req == nil || !req.ResponseFormat.jsonMode() { + return "" + } + if req.ResponseFormat.Name != "" { + return req.ResponseFormat.Name + } + return anthropicJSONToolDefault +} + +// anthropicToolControls appends the JSON-mode tool when requested and +// renders tool_choice. thinking reports extended thinking, which rejects +// forced tool use. +func anthropicToolControls(req *ChatRequest, out *anRequest, thinking bool) error { + var choice *anToolChoice + if name := anthropicJSONToolName(req); name != "" { + switch { + case req.ToolChoice != nil: + return &ConfigError{Msg: "Anthropic JSON ResponseFormat forces its own tool; it cannot be combined with ToolChoice"} + case thinking: + return &ConfigError{Msg: "Anthropic JSON ResponseFormat forces tool use, which extended thinking rejects"} + case len(req.Tools) > 0: + // The forced JSON tool would make every user tool uncallable. + return &ConfigError{Msg: "Anthropic JSON ResponseFormat forces its own tool; it cannot be combined with Tools"} + case !objectSchema(req.ResponseFormat.Schema): + return &ConfigError{Msg: "Anthropic JSON ResponseFormat needs a top-level \"type\": \"object\" schema"} + } + schema := req.ResponseFormat.Schema + if len(schema) == 0 { + schema = json.RawMessage(`{"type":"object"}`) + } + out.Tools = append(out.Tools, anTool{ + Name: name, + Description: "Return the final answer as JSON matching this schema.", + InputSchema: schema, + }) + choice = &anToolChoice{Type: "tool", Name: name} + } else if tc := req.ToolChoice; tc != nil { + switch tc.Mode { + case ToolChoiceAuto: + choice = &anToolChoice{Type: "auto"} + case ToolChoiceNone: + choice = &anToolChoice{Type: "none"} + case ToolChoiceRequired: + choice = &anToolChoice{Type: "any"} + default: // ToolChoiceTool (validated) + choice = &anToolChoice{Type: "tool", Name: tc.Name} + } + if thinking && (choice.Type == "any" || choice.Type == "tool") { + return &ConfigError{Msg: "Anthropic extended thinking rejects forced tool use (ToolChoice required/tool)"} + } + } + if p := req.ParallelToolCalls; p != nil && !*p && len(out.Tools) > 0 && (choice == nil || choice.Type != "none") { + if choice == nil { + choice = &anToolChoice{Type: "auto"} + } + choice.DisableParallelToolUse = true + } + out.ToolChoice = choice + return nil +} + +// objectSchema reports whether a schema is absent (json_object) or declares +// a top-level object type — the only shape Anthropic accepts as a tool's +// input_schema. +func objectSchema(schema json.RawMessage) bool { + if len(schema) == 0 { + return true + } + var s struct { + Type any `json:"type"` + } + return json.Unmarshal(schema, &s) == nil && s.Type == "object" +} + +// foldAnthropicJSON moves the JSON-mode tool call back into Content: the +// caller asked for JSON text, not a tool call. Other tool calls stay. +func foldAnthropicJSON(res *ChatResult, name string) { + if res == nil || name == "" { + return + } + var kept []ToolCall + folded := false + for _, tc := range res.ToolCalls { + if !folded && tc.Name == name { + res.Content += tc.Arguments + folded = true + continue + } + kept = append(kept, tc) + } + if !folded { + return + } + res.ToolCalls = kept + if res.FinishReason == FinishToolCalls && len(kept) == 0 { + res.FinishReason = FinishStop + } +} + +// ── Gemini ─────────────────────────────────────────────────────────────── + +type gmFnCallingCfg struct { + Mode string `json:"mode"` // AUTO | NONE | ANY + AllowedFunctionNames []string `json:"allowedFunctionNames,omitempty"` +} + +type gmToolConfig struct { + FunctionCallingConfig gmFnCallingCfg `json:"functionCallingConfig"` +} + +func geminiToolConfig(tc *ToolChoice) *gmToolConfig { + if tc == nil { + return nil + } + switch tc.Mode { + case ToolChoiceNone: + return &gmToolConfig{FunctionCallingConfig: gmFnCallingCfg{Mode: "NONE"}} + case ToolChoiceRequired: + return &gmToolConfig{FunctionCallingConfig: gmFnCallingCfg{Mode: "ANY"}} + case ToolChoiceTool: + return &gmToolConfig{FunctionCallingConfig: gmFnCallingCfg{Mode: "ANY", AllowedFunctionNames: []string{tc.Name}}} + default: + return &gmToolConfig{FunctionCallingConfig: gmFnCallingCfg{Mode: "AUTO"}} + } +} diff --git a/coverage_edges_test.go b/coverage_edges_test.go new file mode 100644 index 0000000..8229660 --- /dev/null +++ b/coverage_edges_test.go @@ -0,0 +1,344 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" +) + +// errBody is a response body whose read fails mid-way. +type errBody struct{} + +func (errBody) Read([]byte) (int, error) { return 0, errors.New("connection reset") } +func (errBody) Close() error { return nil } + +func bodyErrTransport() http.RoundTripper { + return rtFunc(func(r *http.Request) (*http.Response, error) { + return &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"application/json"}}, Body: errBody{}}, nil + }) +} + +// A body that dies mid-read is a transport failure on every buffered path. +func TestBufferedBodyReadErrors(t *testing.T) { + fastBackoff(t) + sdk := New(WithTransport(bodyErrTransport()), WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://x"))) + ctx := context.Background() + c, _ := sdk.Chat("openai", "m") + if _, err := c.Call(ctx, userMsg()); err == nil || !strings.Contains(err.Error(), "connection reset") { + t.Errorf("Call: %v", err) + } + p, _ := sdk.Provider("openai") + if _, err := p.ListModels(ctx); err == nil || !strings.Contains(err.Error(), "connection reset") { + t.Errorf("ListModels: %v", err) + } + if _, err := sdk.Speak(ctx, "openai", "m", SpeakRequest{Text: "x", Voice: "v"}); err == nil { + t.Error("Speak: want error") + } + if _, err := sdk.Transcribe(ctx, "openai", "m", TranscribeRequest{Audio: []byte("a")}); err == nil { + t.Error("Transcribe: want error") + } +} + +// A learn-once trigger on the final allowed attempt surfaces its cause. +func TestLearnOnFinalAttempt(t *testing.T) { + srv, n := countingServer(t, http.StatusBadRequest, nil) + srv.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n.Add(1) + w.WriteHeader(400) + _, _ = fmt.Fprint(w, `{"error":{"message":"Function tools with reasoning_effort are not supported for gpt-5.6 in /v1/chat/completions. Please use /v1/responses instead."}}`) + }) + for _, stream := range []bool{false, true} { + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "gpt-5.5") + req := toolReq(nil) + var err error + if stream { + _, err = c.CallStream(context.Background(), req, func(Delta) error { return nil }) + } else { + _, err = c.Call(context.Background(), req) + } + var ae *APIError + if !errors.As(err, &ae) || ae.Status != 400 { + t.Errorf("stream=%v: err = %v", stream, err) + } + } +} + +// withRetry: a definitive error after 429s is returned as is; a final +// retryable error after 429s keeps the rate-limit signal. +func TestWithRetry_AfterRateLimit(t *testing.T) { + fastBackoff(t) + seq := func(codes ...int) *httptest.Server { + var i atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + k := int(i.Add(1)) - 1 + if k >= len(codes) { + k = len(codes) - 1 + } + w.WriteHeader(codes[k]) + _, _ = fmt.Fprint(w, `{"error":{"message":"x"}}`) + })) + t.Cleanup(srv.Close) + return srv + } + run := func(srv *httptest.Server, attempts int) error { + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: attempts}), WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "m") + _, err := c.Call(context.Background(), userMsg()) + return err + } + var ae *APIError + var rl *RateLimitError + if err := run(seq(429, 400), 3); !errors.As(err, &ae) || ae.Status != 400 || errors.As(err, &rl) { + t.Errorf("429 then 400: %v", err) + } + if err := run(seq(429, 500), 2); !errors.As(err, &rl) || rl.Attempts != 2 { + t.Errorf("429 then final 500: %v", err) + } + if got := (&terminalError{err: errors.New("bad body")}).Error(); got != "bad body" { + t.Errorf("terminalError = %q", got) + } + if terminal(nil) != nil { + t.Error("terminal(nil) must be nil") + } +} + +// Streaming: a cancelled context ends before any attempt; transport +// failures exhaust into the wrapped error; persistent 429 is typed. +func TestCallStream_LadderEdges(t *testing.T) { + fastBackoff(t) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://127.0.0.1:1")), WithRetryPolicy(RetryPolicy{MaxAttempts: 2})) + c, _ := sdk.Chat("openai", "m") + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if _, err := c.CallStream(ctx, userMsg(), func(Delta) error { return nil }); !errors.Is(err, context.Canceled) { + t.Errorf("cancelled: %v", err) + } + if _, err := c.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }); err == nil { + t.Error("unreachable host: want error") + } + srv, n := countingServer(t, http.StatusTooManyRequests, nil) + s2 := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 2}), WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c2, _ := s2.Chat("openai", "m") + var rl *RateLimitError + if _, err := c2.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }); !errors.As(err, &rl) || n.Load() != 2 { + t.Errorf("429 ladder: %v (attempts %d)", err, n.Load()) + } +} + +func TestBackoffDelayCapped_LongLadder(t *testing.T) { + fastBackoff(t) + for _, attempt := range []int{21, 40, 63, 1000} { + if d := backoffDelayCapped(attempt, time.Hour); d <= 0 || d > time.Hour { + t.Errorf("attempt %d → %v (must stay positive and capped)", attempt, d) + } + } +} + +// Same-host redirects are still followed. +func TestSameHostRedirectFollowed(t *testing.T) { + var srv *httptest.Server + srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/old/chat/completions" { + http.Redirect(w, r, srv.URL+"/chat/completions", http.StatusTemporaryRedirect) + return + } + _, _ = fmt.Fprint(w, `{"choices":[{"message":{"content":"ok"},"finish_reason":"stop"}]}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL+"/old"))) + c, _ := sdk.Chat("openai", "m") + if res, err := c.Call(context.Background(), userMsg()); err != nil || res.Content != "ok" { + t.Fatalf("res=%+v err=%v", res, err) + } +} + +func TestEngageLearnOnlyOnce(t *testing.T) { + var n int + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI}, nil, nil) + pc.opts.observer = func(LearnEvent) { n++ } + pc.engageLearn(&pc.learn.forceBuffered, LearnBuffered, nil) + pc.engageLearn(&pc.learn.forceBuffered, LearnBuffered, nil) + if n != 1 { + t.Errorf("observer fired %d times, want 1 (monotonic flag)", n) + } +} + +// A follower whose own context ends while waiting returns promptly. +func TestListModels_FollowerContextEnds(t *testing.T) { + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + <-release + _, _ = fmt.Fprint(w, `{"data":[]}`) + })) + t.Cleanup(srv.Close) + t.Cleanup(func() { close(release) }) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + p, _ := sdk.Provider("openai") + go func() { _, _ = p.ListModels(context.Background()) }() + for { + p.mu.Lock() + inflight := p.flight != nil + p.mu.Unlock() + if inflight { + break + } + time.Sleep(time.Millisecond) + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + if _, err := p.ListModels(ctx); !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("follower err = %v", err) + } +} + +func TestSmallMappingEdges(t *testing.T) { + // json_schema without Name uses the default schema name. + m, _ := buildBody(t, FormatOpenAI, "gpt-4o", &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: testSchema}}) + if m["response_format"].(map[string]any)["json_schema"].(map[string]any)["name"] != defaultSchemaName { + t.Errorf("default schema name: %v", m["response_format"]) + } + // No JSON-tool call present: the result is untouched. + res := &ChatResult{FinishReason: FinishToolCalls, ToolCalls: []ToolCall{{Name: "g"}}} + foldAnthropicJSON(res, "answer") + if len(res.ToolCalls) != 1 || res.FinishReason != FinishToolCalls { + t.Errorf("unfolded = %+v", res) + } + // Object-shaped tool errors nest; text errors wrap. + if got := string(wrapToolError(` {"code":7} `)); got != `{"error":{"code":7}}` { + t.Errorf("object error = %s", got) + } + if u := usageFromOpenAI(nil); u != (Usage{}) { + t.Errorf("nil usage = %+v", u) + } + // Anthropic: no content and no stop reason is a protocol error. + if _, err := parseAnthropicResponse([]byte(`{"content":[]}`)); err == nil { + t.Error("empty anthropic response must error") + } + // Gemini embeddings: malformed body. + pc := newProviderClient(ProviderConfig{ID: "g", Format: FormatGemini}, nil, nil) + if _, _, err := pc.parseEmbedResponse([]byte(`{bad`), 1); err == nil { + t.Error("gemini embed parse must error") + } + // Tool images across a run of tool results land in one user message. + req := &ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "a", Name: "f"}, {ID: "b", Name: "f"}}}, + {Role: RoleTool, ToolCallID: "a", Parts: []ContentPart{TextPart("one"), ImagePart("image/png", []byte{1})}}, + {Role: RoleTool, ToolCallID: "b", Parts: []ContentPart{ImagePart("image/png", []byte{2})}}, + }} + m, _ = buildBody(t, FormatOpenAI, "gpt-4o", req) + msgs := m["messages"].([]any) + if len(msgs) != 5 || msgs[4].(map[string]any)["role"] != "user" { + t.Fatalf("messages = %v", msgs) + } + if parts := msgs[4].(map[string]any)["content"].([]any); len(parts) != 4 { + t.Errorf("follow-up parts = %v", parts) + } + b, _ := json.Marshal(oaMessage{Role: "assistant", ContentParts: []oaContentPart{{Type: "text", Text: "x"}}}) + if !strings.Contains(string(b), `"content":[`) { + t.Errorf("assistant parts = %s", b) + } +} + +func TestContentPartValidationEdges(t *testing.T) { + big := make([]byte, MaxImageBytes+1) + bad := []ContentPart{ + {Type: ContentPartText, Text: "x", MIMEType: "image/png"}, + {Type: ContentPartImage, Text: "x", MIMEType: "image/png", Image: []byte{1}}, + {Type: ContentPartImage, MIMEType: "image/png"}, + {Type: ContentPartImage, MIMEType: "image/png", Image: big}, + } + for i, p := range bad { + if err := validateMessageContent(Message{Role: RoleTool, Parts: []ContentPart{p}}, 0); err == nil { + t.Errorf("part %d must be rejected", i) + } + } +} + +func TestListModelsOpenAI_CreatedAt(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = io.WriteString(w, `{"data":[{"id":"m","created":1700000000,"max_input_tokens":42}]}`) + })) + t.Cleanup(srv.Close) + ms, err := newListModels(srv.URL, FormatOpenAI) + if err != nil || len(ms) != 1 || ms[0].CreatedAt.Unix() != 1700000000 || ms[0].ContextWindow != 42 { + t.Fatalf("models=%+v err=%v", ms, err) + } +} + +// A malformed base URL is rejected at wiring time, and a request that +// cannot be built is never retried. +func TestBadBaseURL(t *testing.T) { + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://x\x7f"))) + var ce *ConfigError + if _, err := sdk.Chat("openai", "m"); !errors.As(err, &ce) { + t.Errorf("control char in base URL: %v", err) + } + fastBackoff(t) + var n atomic.Int32 + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "http://x"}, nil, nil) + err := pc.withRetry(context.Background(), func() (time.Duration, error) { + n.Add(1) + return 0, &ConfigError{Msg: "build request: bad"} + }, nil) + if !errors.As(err, &ce) || n.Load() != 1 { + t.Errorf("ConfigError retried: attempts=%d err=%v", n.Load(), err) + } +} + +func TestSpeakAndGeminiSpeechEdges(t *testing.T) { + sdk := New(WithProvider("openai", WithAPIKey("k"))) + var ce *ConfigError + if _, err := sdk.Speak(context.Background(), "openai", " ", SpeakRequest{Text: "x", Voice: "v"}); !errors.As(err, &ce) { + t.Errorf("empty model: %v", err) + } + for _, body := range []string{`{bad`, `{"candidates":[{"content":{"parts":[{"inlineData":{"mimeType":"audio/L16","data":"!!"}}]}}]}`} { + if _, _, err := parseGeminiSpeakResponse([]byte(body)); err == nil { + t.Errorf("%s: want error", body) + } + } +} + +// Streaming: a non-retryable provider error ends the call at once; a 400 +// that names no learnable constraint is returned as is on the buffered path. +func TestNonRetryableErrors(t *testing.T) { + fastBackoff(t) + srv, n := countingServer(t, http.StatusUnauthorized, nil) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "m") + var ae *APIError + if _, err := c.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }); !errors.As(err, &ae) || ae.Status != 401 || n.Load() != 1 { + t.Errorf("stream 401: %v (attempts %d)", err, n.Load()) + } + srv2, n2 := countingServer(t, http.StatusBadRequest, nil) + s2 := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv2.URL))) + c2, _ := s2.Chat("openai", "m") + if _, err := c2.Call(context.Background(), toolReq(nil)); !errors.As(err, &ae) || ae.Status != 400 || n2.Load() != 1 { + t.Errorf("plain 400 with tools: %v (attempts %d)", err, n2.Load()) + } + if !chatCompletionsRejectsExplicitReasoningWithTools("gpt-5.6") { + t.Error("gpt-5.6 rejects explicit reasoning with tools") + } +} + +// Transport failures on the speech paths are retried, then surfaced. +func TestSpeechTransportErrors(t *testing.T) { + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://127.0.0.1:1"))) + if _, err := sdk.Speak(context.Background(), "openai", "m", SpeakRequest{Text: "x", Voice: "v"}); err == nil || !strings.Contains(err.Error(), "retry exhausted") { + t.Errorf("Speak: %v", err) + } + if _, err := sdk.Transcribe(context.Background(), "openai", "m", TranscribeRequest{Audio: []byte("a")}); err == nil || !strings.Contains(err.Error(), "retry exhausted") { + t.Errorf("Transcribe: %v", err) + } +} diff --git a/dispatch_edges_test.go b/dispatch_edges_test.go index 9e44412..6bed8a9 100644 --- a/dispatch_edges_test.go +++ b/dispatch_edges_test.go @@ -214,7 +214,15 @@ func TestBuildAnthropicRequestArms(t *testing.T) { t.Errorf("anthropic request missing %q:\n%s", want, s) } } - if !strings.Contains(s, `"temperature":0`) { + if strings.Contains(s, `"temperature"`) { + t.Error("extended thinking must omit temperature") + } + req.Thinking, req.ThinkingBudget = "", 0 + b, err = buildAnthropicRequest(req, "claude-x", false) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(b), `"temperature":0`) { t.Error("negative temperature must normalize to an explicit 0") } } @@ -749,12 +757,9 @@ func TestListModelsAnthropicPageLimitExit(t *testing.T) { srv := httptestNewServer(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, `{"data":[{"id":"x"}],"has_more":true,"last_id":"x"}`) }) - got, err := newListModels(srv.URL, FormatAnthropic) - if err != nil { - t.Fatal(err) - } - if len(got) != 10 { - t.Errorf("models = %d, want 10 (page cap)", len(got)) + // The page cap is an error, never a silently truncated listing. + if _, err := newListModels(srv.URL, FormatAnthropic); !errors.Is(err, ErrModelListTruncated) { + t.Fatalf("err = %v, want ErrModelListTruncated", err) } } @@ -976,6 +981,7 @@ func TestParseOpenAIResponseToolCalls(t *testing.T) { } func TestListModelsAnthropicMidPageError(t *testing.T) { + fastBackoff(t) var n int srv := httptestNewServer(func(w http.ResponseWriter, r *http.Request) { n++ @@ -993,6 +999,7 @@ func TestListModelsAnthropicMidPageError(t *testing.T) { } func TestListModelsGeminiMidPageError(t *testing.T) { + fastBackoff(t) var n int srv := httptestNewServer(func(w http.ResponseWriter, r *http.Request) { n++ diff --git a/embed.go b/embed.go new file mode 100644 index 0000000..a78b6d8 --- /dev/null +++ b/embed.go @@ -0,0 +1,211 @@ +package llm + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + neturl "net/url" + "sort" + "strings" + "time" +) + +// ── Embeddings ─────────────────────────────────────────────────────────── +// +// Embed turns texts into vectors. Two wire formats are supported: +// - OpenAI-compatible: POST {base}/embeddings +// - Gemini: POST {base}/v1beta/models/{model}:batchEmbedContents +// +// Anthropic has no embeddings endpoint: a ConfigError. Calls share the chat +// retry ladder, whole-call budget, custom headers and error taxonomy. + +// EmbedRequest describes one embeddings call. +type EmbedRequest struct { + Inputs []string // required: one or more non-empty texts + Dimensions int // optional output dimensionality (0 = model default) +} + +// EmbedResult carries one vector per input, in input order. +type EmbedResult struct { + Embeddings [][]float64 + Model string + Usage Usage // PromptTokens when the provider reports it +} + +// Embed computes embeddings for req.Inputs with the named provider and +// model. +func (s *SDK) Embed(ctx context.Context, providerID, model string, req EmbedRequest) (*EmbedResult, error) { + if strings.TrimSpace(model) == "" { + return nil, &ConfigError{Msg: "embed request requires a model"} + } + if len(req.Inputs) == 0 { + return nil, &ConfigError{Msg: "embed request requires at least one input"} + } + for i, in := range req.Inputs { + if strings.TrimSpace(in) == "" { + return nil, &ConfigError{Msg: fmt.Sprintf("embed input %d is empty", i)} + } + } + if req.Dimensions < 0 { + return nil, &ConfigError{Msg: "embed Dimensions must not be negative"} + } + p, err := s.usableProvider(providerID) + if err != nil { + return nil, err + } + if p.cfg.Format != FormatOpenAI && p.cfg.Format != FormatGemini { + return nil, &ConfigError{Msg: "provider format " + string(p.cfg.Format) + " does not support embeddings"} + } + return s.newClient(p, s.timeout, nil).embed(ctx, model, req) +} + +type oaEmbedRequest struct { + Model string `json:"model"` + Input []string `json:"input"` + Dimensions int `json:"dimensions,omitempty"` + EncodingFormat string `json:"encoding_format"` +} + +type oaEmbedResponse struct { + Data []struct { + Index int `json:"index"` + Embedding []float64 `json:"embedding"` + } `json:"data"` + Usage *struct { + PromptTokens int `json:"prompt_tokens"` + } `json:"usage"` +} + +type gmEmbedItem struct { + Model string `json:"model"` + Content gmContent `json:"content"` + OutputDimensionality int `json:"outputDimensionality,omitempty"` +} + +type gmEmbedRequest struct { + Requests []gmEmbedItem `json:"requests"` +} + +type gmEmbedResponse struct { + Embeddings []struct { + Values []float64 `json:"values"` + } `json:"embeddings"` +} + +// buildEmbedRequest serializes the request for the provider's format. +func (pc *providerClient) buildEmbedRequest(model string, req EmbedRequest) ([]byte, string, error) { + if pc.cfg.Format == FormatGemini { + r := gmEmbedRequest{Requests: make([]gmEmbedItem, 0, len(req.Inputs))} + for _, in := range req.Inputs { + r.Requests = append(r.Requests, gmEmbedItem{ + Model: "models/" + model, + Content: gmContent{Parts: []gmPart{{Text: in}}}, + OutputDimensionality: req.Dimensions, + }) + } + b, err := json.Marshal(r) + return b, fmt.Sprintf("%s/v1beta/models/%s:batchEmbedContents", pc.base, neturl.PathEscape(model)), err + } + b, err := json.Marshal(oaEmbedRequest{Model: model, Input: req.Inputs, Dimensions: req.Dimensions, EncodingFormat: "float"}) + return b, pc.base + "/embeddings", err +} + +// parseEmbedResponse decodes vectors in input order and checks the count. +func (pc *providerClient) parseEmbedResponse(data []byte, n int) ([][]float64, Usage, error) { + if pc.cfg.Format == FormatGemini { + var r gmEmbedResponse + if err := json.Unmarshal(data, &r); err != nil { + return nil, Usage{}, fmt.Errorf("llm: parse embeddings: %w", err) + } + if len(r.Embeddings) != n { + return nil, Usage{}, fmt.Errorf("llm: embeddings: got %d vectors for %d inputs", len(r.Embeddings), n) + } + out := make([][]float64, n) + for i, e := range r.Embeddings { + out[i] = e.Values + } + return out, Usage{}, nil + } + var r oaEmbedResponse + if err := json.Unmarshal(data, &r); err != nil { + return nil, Usage{}, fmt.Errorf("llm: parse embeddings: %w", err) + } + if len(r.Data) != n { + return nil, Usage{}, fmt.Errorf("llm: embeddings: got %d vectors for %d inputs", len(r.Data), n) + } + allZero := true + for _, d := range r.Data { + if d.Index != 0 { + allZero = false + break + } + } + if allZero { + // Gateways that omit index: trust response order. + for i := range r.Data { + r.Data[i].Index = i + } + } + sort.SliceStable(r.Data, func(i, j int) bool { return r.Data[i].Index < r.Data[j].Index }) + out := make([][]float64, n) + for i, d := range r.Data { + if d.Index != i { + return nil, Usage{}, fmt.Errorf("llm: embeddings: vector index %d out of range", d.Index) + } + out[i] = d.Embedding + } + var u Usage + if r.Usage != nil { + u.PromptTokens = r.Usage.PromptTokens + } + return out, u, nil +} + +// Per-request input caps: Gemini batchEmbedContents takes at most 100 +// requests, OpenAI /embeddings at most 2048 inputs. Larger calls are split +// into consecutive batches and concatenated in input order. +const ( + geminiEmbedBatch = 100 + openAIEmbedBatch = 2048 +) + +// embed runs the request in provider-sized batches, each with the shared +// retry ladder, all under one whole-call budget. +func (pc *providerClient) embed(ctx context.Context, model string, req EmbedRequest) (*EmbedResult, error) { + model = strings.TrimPrefix(model, "models/") // Gemini ids as ListModels may echo them + batch := openAIEmbedBatch + if pc.cfg.Format == FormatGemini { + batch = geminiEmbedBatch + } + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) + defer cancel() + res := &EmbedResult{Model: model, Embeddings: make([][]float64, 0, len(req.Inputs))} + for start := 0; start < len(req.Inputs); start += batch { + part := req + part.Inputs = req.Inputs[start:min(start+batch, len(req.Inputs))] + body, url, err := pc.buildEmbedRequest(model, part) + if err != nil { + return nil, err + } + err = pc.withRetry(ctx, func() (time.Duration, error) { + data, ra, err := pc.post(ctx, pc.buffered(), url, body) + if err != nil { + return ra, err + } + vecs, usage, perr := pc.parseEmbedResponse(data, len(part.Inputs)) + if perr != nil { + // 2xx with an unusable body is a provider protocol failure: + // typed, at the actual HTTP status, never retried. + return 0, terminal(&APIError{Provider: pc.cfg.ID, Status: http.StatusOK, Message: perr.Error()}) + } + res.Embeddings = append(res.Embeddings, vecs...) + res.Usage.PromptTokens += usage.PromptTokens + return 0, nil + }, nil) + if err != nil { + return nil, err + } + } + return res, nil +} diff --git a/followups_test.go b/followups_test.go new file mode 100644 index 0000000..41b542c --- /dev/null +++ b/followups_test.go @@ -0,0 +1,317 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// ── Responses API: every reasoning item, stream error event ────────────── + +func TestParseResponsesAPI_AllReasoningItems(t *testing.T) { + res, err := parseResponsesAPI([]byte(`{"status":"completed","output":[ + {"type":"reasoning","encrypted_content":"E1","summary":[{"type":"summary_text","text":"s1"}]}, + {"type":"function_call","call_id":"c","name":"f","arguments":"{}"}, + {"type":"reasoning","encrypted_content":"E2","summary":[{"type":"summary_text","text":"s2a"},{"type":"summary_text","text":"s2b"}]}]}`)) + if err != nil { + t.Fatal(err) + } + want := []ThinkingBlock{{Text: "s1", Signature: "E1"}, {Text: "s2a\ns2b", Signature: "E2"}} + if len(res.ThinkingBlocks) != 2 || res.ThinkingBlocks[0] != want[0] || res.ThinkingBlocks[1] != want[1] { + t.Fatalf("blocks = %+v", res.ThinkingBlocks) + } + if res.ThinkingSignature != "E2" { + t.Errorf("legacy signature = %q", res.ThinkingSignature) + } +} + +func TestMapResponsesStreamEvent_ReasoningBlocksAndError(t *testing.T) { + acc := newStreamAccum() + for _, e := range []string{ + `{"type":"response.output_item.done","output_index":0,"item":{"type":"reasoning","encrypted_content":"E1","summary":[{"type":"summary_text","text":"s1"}]}}`, + `{"type":"response.output_item.done","output_index":2,"item":{"type":"reasoning","encrypted_content":"E2","summary":[]}}`, + `{"type":"response.completed","response":{"status":"completed","output":[]}}`, + } { + if _, _, err := mapResponsesStreamEvent([]byte(e), acc); err != nil { + t.Fatal(err) + } + } + res := acc.result() + if len(res.ThinkingBlocks) != 2 || res.ThinkingBlocks[0].Signature != "E1" || res.ThinkingBlocks[0].Text != "s1" || res.ThinkingBlocks[1].Signature != "E2" { + t.Fatalf("stream blocks = %+v", res.ThinkingBlocks) + } + // Fallback: blocks only in the completed response. + acc = newStreamAccum() + _, _, _ = mapResponsesStreamEvent([]byte(`{"type":"response.completed","response":{"status":"completed","output":[{"type":"reasoning","encrypted_content":"E9","summary":[]}]}}`), acc) + if r := acc.result(); len(r.ThinkingBlocks) != 1 || r.ThinkingBlocks[0].Signature != "E9" || r.ThinkingSignature != "E9" { + t.Errorf("fallback blocks = %+v", r) + } + // A top-level error event fails the stream with the provider message. + _, done, err := mapResponsesStreamEvent([]byte(`{"type":"error","code":"server_error","message":"upstream exploded"}`), newStreamAccum()) + if err == nil || !done || !strings.Contains(err.Error(), "upstream exploded") { + t.Errorf("error event: done=%v err=%v", done, err) + } +} + +func TestBuildResponsesInput_ReplaysThinkingBlocks(t *testing.T) { + _, input := buildResponsesInput(&ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, ThinkingSignature: "legacy-ignored", ThinkingBlocks: []ThinkingBlock{ + {Text: "s1", Signature: "E1"}, {Text: "unsigned"}, {Signature: "E2"}, + }, ToolCalls: []ToolCall{{ID: "c", Name: "f", Arguments: "{}"}}}, + }}) + b, _ := json.Marshal(input) + var items []map[string]any + _ = json.Unmarshal(b, &items) + var encs []string + for _, it := range items { + if it["type"] == "reasoning" { + encs = append(encs, it["encrypted_content"].(string)) + } + } + if strings.Join(encs, ",") != "E1,E2" { + t.Errorf("replayed reasoning items = %v (unsigned skipped, legacy ignored)", encs) + } +} + +// ── tool results: IsError and content parts ────────────────────────────── + +func toolResultReq(m Message) *ChatRequest { + return &ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "c1", Name: "shot", Arguments: "{}"}}}, + m, + }} +} + +func TestToolResult_IsError(t *testing.T) { + req := toolResultReq(Message{Role: RoleTool, ToolCallID: "c1", Content: "disk full", IsError: true}) + m, _ := buildBody(t, FormatAnthropic, "claude", req) + tr := m["messages"].([]any)[2].(map[string]any)["content"].([]any)[0].(map[string]any) + if tr["is_error"] != true || tr["content"] != "disk full" { + t.Errorf("anthropic tool_result = %v", tr) + } + m, _ = buildBody(t, FormatGemini, "gemini", req) + fr := m["contents"].([]any)[2].(map[string]any)["parts"].([]any)[0].(map[string]any)["functionResponse"].(map[string]any) + jsonEq(t, "gemini error response", fr["response"], `{"error":"disk full"}`) + // IsError outside the tool role is a config error. + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "https://x"}, nil, nil) + _, _, err := pc.buildChatRequest(&ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x", IsError: true}}}, "m", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("IsError on user: err = %v", err) + } +} + +func TestToolResult_ContentParts(t *testing.T) { + png := []byte{0x89, 'P', 'N', 'G'} + req := toolResultReq(Message{Role: RoleTool, ToolCallID: "c1", Parts: []ContentPart{TextPart("see image"), ImagePart("image/png", png)}}) + + // Anthropic: tool_result content blocks. + m, _ := buildBody(t, FormatAnthropic, "claude", req) + tr := m["messages"].([]any)[2].(map[string]any)["content"].([]any)[0].(map[string]any) + blocks := tr["content"].([]any) + if len(blocks) != 2 || blocks[0].(map[string]any)["text"] != "see image" || blocks[1].(map[string]any)["type"] != "image" { + t.Errorf("anthropic tool_result content = %v", tr) + } + + // OpenAI chat: text stays on the tool message; images follow in a user + // message (chat completions tool messages cannot carry images). + m, _ = buildBody(t, FormatOpenAI, "gpt-4o", req) + msgs := m["messages"].([]any) + if len(msgs) != 4 { + t.Fatalf("openai messages = %v", msgs) + } + tool := msgs[2].(map[string]any) + if tool["role"] != "tool" || tool["content"] != "see image" { + t.Errorf("openai tool message = %v", tool) + } + user := msgs[3].(map[string]any) + parts := user["content"].([]any) + if user["role"] != "user" || parts[len(parts)-1].(map[string]any)["type"] != "image_url" { + t.Errorf("openai image follow-up = %v", user) + } + + // Responses: function_call_output carries input_text + input_image. + rs := *req + rs.Tools = []ToolDef{{Name: "shot"}} + m, _ = buildBody(t, FormatOpenAI, "gpt-5.6", &rs) + var out map[string]any + for _, it := range m["input"].([]any) { + if it.(map[string]any)["type"] == "function_call_output" { + out = it.(map[string]any) + } + } + op := out["output"].([]any) + if len(op) != 2 || op[0].(map[string]any)["type"] != "input_text" || op[1].(map[string]any)["type"] != "input_image" { + t.Errorf("responses output = %v", out) + } + + // Gemini: functionResponse text + inlineData in the same user turn. + m, _ = buildBody(t, FormatGemini, "gemini", req) + gparts := m["contents"].([]any)[2].(map[string]any)["parts"].([]any) + fr := gparts[0].(map[string]any)["functionResponse"].(map[string]any) + jsonEq(t, "gemini tool text", fr["response"], `{"result":"see image"}`) + if len(gparts) != 2 || gparts[1].(map[string]any)["inlineData"] == nil { + t.Errorf("gemini parts = %v", gparts) + } + + // Parts stay forbidden on assistant/system turns. + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "https://x"}, nil, nil) + _, _, err := pc.buildChatRequest(&ChatRequest{Messages: []Message{{Role: RoleAssistant, Parts: []ContentPart{TextPart("x")}}}}, "m", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("assistant parts: err = %v", err) + } +} + +// ── unauthenticated custom provider message ────────────────────────────── + +func TestUnauthenticatedMessageNamesRealEnvKeys(t *testing.T) { + sdk := New(WithProvider("gw", WithFormat(FormatOpenAI), WithBaseURL("https://x"))) + _, err := sdk.Chat("gw", "m") + if err == nil || strings.Contains(err.Error(), "GW_API_KEY") || !strings.Contains(err.Error(), "WithEnvKeys") { + t.Errorf("custom provider without EnvKeys: %v", err) + } + sdk = New(WithProvider("gw", WithFormat(FormatOpenAI), WithBaseURL("https://x"), WithEnvKeys("GW_TOKEN"))) + if _, err := sdk.Chat("gw", "m"); err == nil || !strings.Contains(err.Error(), "GW_TOKEN") { + t.Errorf("custom provider with EnvKeys: %v", err) + } + sdk = New() + if _, err := sdk.Chat("openai", "m"); err == nil || !strings.Contains(err.Error(), "OPENAI_API_KEY") { + t.Errorf("built-in: %v", err) + } + for _, fn := range []func() error{ + func() error { + _, err := sdk.Speak(context.Background(), "openai", "m", SpeakRequest{Text: "x", Voice: "v"}) + return err + }, + func() error { + _, err := sdk.Transcribe(context.Background(), "openai", "m", TranscribeRequest{Audio: []byte("a")}) + return err + }, + func() error { + _, err := sdk.Embed(context.Background(), "openai", "m", EmbedRequest{Inputs: []string{"a"}}) + return err + }, + } { + if err := fn(); err == nil || !strings.Contains(err.Error(), "OPENAI_API_KEY") { + t.Errorf("entry point message: %v", err) + } + } +} + +// ── embeddings ─────────────────────────────────────────────────────────── + +func TestEmbed_OpenAI(t *testing.T) { + var got map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/embeddings" || r.Header.Get("Authorization") != "Bearer k" { + t.Errorf("path=%s auth=%q", r.URL.Path, r.Header.Get("Authorization")) + } + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &got) + // Out of order on purpose: the SDK sorts by index. + _, _ = fmt.Fprint(w, `{"data":[{"index":1,"embedding":[0.3,0.4]},{"index":0,"embedding":[0.1,0.2]}],"model":"emb-x","usage":{"prompt_tokens":5}}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "openai", "emb", EmbedRequest{Inputs: []string{"a", "b"}, Dimensions: 2}) + if err != nil { + t.Fatal(err) + } + if len(res.Embeddings) != 2 || res.Embeddings[0][0] != 0.1 || res.Embeddings[1][1] != 0.4 { + t.Errorf("embeddings = %v", res.Embeddings) + } + if res.Model != "emb" || res.Usage.PromptTokens != 5 { + t.Errorf("result = %+v", res) + } + if got["model"] != "emb" || got["dimensions"] != float64(2) || got["encoding_format"] != "float" { + t.Errorf("request = %v", got) + } +} + +func TestEmbed_Gemini(t *testing.T) { + var got map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1beta/models/text-embedding-004:batchEmbedContents" || r.Header.Get("x-goog-api-key") != "k" { + t.Errorf("path=%s", r.URL.Path) + } + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &got) + _, _ = fmt.Fprint(w, `{"embeddings":[{"values":[1,2]},{"values":[3,4]}]}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("gemini", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "gemini", "text-embedding-004", EmbedRequest{Inputs: []string{"a", "b"}, Dimensions: 2}) + if err != nil { + t.Fatal(err) + } + if len(res.Embeddings) != 2 || res.Embeddings[1][0] != 3 { + t.Errorf("embeddings = %v", res.Embeddings) + } + r0 := got["requests"].([]any)[0].(map[string]any) + if r0["model"] != "models/text-embedding-004" || r0["outputDimensionality"] != float64(2) { + t.Errorf("request = %v", r0) + } +} + +func TestEmbed_ErrorsAndValidation(t *testing.T) { + ctx := context.Background() + sdk := New(WithProvider("anthropic", WithAPIKey("k")), WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://127.0.0.1:1"))) + var ce *ConfigError + for name, fn := range map[string]func() error{ + "no inputs": func() error { _, err := sdk.Embed(ctx, "openai", "m", EmbedRequest{}); return err }, + "empty input": func() error { + _, err := sdk.Embed(ctx, "openai", "m", EmbedRequest{Inputs: []string{"a", " "}}) + return err + }, + "no model": func() error { _, err := sdk.Embed(ctx, "openai", "", EmbedRequest{Inputs: []string{"a"}}); return err }, + "neg dims": func() error { + _, err := sdk.Embed(ctx, "openai", "m", EmbedRequest{Inputs: []string{"a"}, Dimensions: -1}) + return err + }, + "anthropic": func() error { + _, err := sdk.Embed(ctx, "anthropic", "m", EmbedRequest{Inputs: []string{"a"}}) + return err + }, + "unknown": func() error { _, err := sdk.Embed(ctx, "nope", "m", EmbedRequest{Inputs: []string{"a"}}); return err }, + } { + if err := fn(); !errors.As(err, &ce) { + t.Errorf("%s: err = %v, want ConfigError", name, err) + } + } + + // A vector-count mismatch or malformed body is a typed protocol error. + for _, body := range []string{`{"data":[{"index":0,"embedding":[1]}]}`, `{not json`, `{"data":[{"index":5,"embedding":[1]},{"index":0,"embedding":[1]}]}`} { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = fmt.Fprint(w, body) })) + s2 := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + _, err := s2.Embed(ctx, "openai", "m", EmbedRequest{Inputs: []string{"a", "b"}}) + srv.Close() + var ae *APIError + if !errors.As(err, &ae) || ae.Status != http.StatusOK { + t.Errorf("body %s: err = %v, want APIError at 200", body, err) + } + } + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = fmt.Fprint(w, `{"embeddings":[]}`) })) + t.Cleanup(srv.Close) + s3 := New(WithProvider("gemini", WithAPIKey("k"), WithBaseURL(srv.URL))) + if _, err := s3.Embed(ctx, "gemini", "m", EmbedRequest{Inputs: []string{"a"}}); err == nil { + t.Error("gemini count mismatch must error") + } + + // Provider errors keep the shared retry taxonomy. + fastBackoff(t) + errSrv, n := countingServer(t, http.StatusBadRequest, nil) + s4 := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(errSrv.URL))) + var ae *APIError + if _, err := s4.Embed(ctx, "openai", "m", EmbedRequest{Inputs: []string{"a"}}); !errors.As(err, &ae) || ae.Status != 400 || n.Load() != 1 { + t.Errorf("400: err = %v, attempts = %d", err, n.Load()) + } +} diff --git a/gemini.go b/gemini.go index 43ed7ff..6adebd7 100644 --- a/gemini.go +++ b/gemini.go @@ -5,6 +5,7 @@ import ( "encoding/base64" "encoding/json" "fmt" + "math" neturl "net/url" "strings" ) @@ -32,6 +33,7 @@ type gmPart struct { Thought bool `json:"thought,omitempty"` FunctionCall *gmFnCall `json:"functionCall,omitempty"` FunctionResponse *gmFnResp `json:"functionResponse,omitempty"` + ThoughtSignature string `json:"thoughtSignature,omitempty"` } type gmBlob struct { MIMEType string `json:"mimeType"` @@ -59,17 +61,21 @@ type gmThinkCfg struct { } type gmGenCfg struct { - MaxOutputTokens int `json:"maxOutputTokens,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - TopP *float64 `json:"topP,omitempty"` - StopSequences []string `json:"stopSequences,omitempty"` - ThinkingConfig *gmThinkCfg `json:"thinkingConfig,omitempty"` + MaxOutputTokens int `json:"maxOutputTokens,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopP *float64 `json:"topP,omitempty"` + StopSequences []string `json:"stopSequences,omitempty"` + ThinkingConfig *gmThinkCfg `json:"thinkingConfig,omitempty"` + Seed *int `json:"seed,omitempty"` + ResponseMimeType string `json:"responseMimeType,omitempty"` + ResponseJSONSchema json.RawMessage `json:"responseJsonSchema,omitempty"` } type gmRequest struct { SystemInstruction *gmContent `json:"systemInstruction,omitempty"` Contents []gmContent `json:"contents"` Tools []gmToolGroup `json:"tools,omitempty"` + ToolConfig *gmToolConfig `json:"toolConfig,omitempty"` GenerationConfig *gmGenCfg `json:"generationConfig,omitempty"` } @@ -107,6 +113,16 @@ func wrapToolResponse(content string) json.RawMessage { return json.RawMessage(fmt.Sprintf(`{"result":%s}`, mustJSONString(content))) } +// wrapToolError renders a failed tool result as {"error": …}, the Gemini +// functionResponse convention for errors. +func wrapToolError(content string) json.RawMessage { + t := strings.TrimSpace(content) + if strings.HasPrefix(t, "{") && json.Valid([]byte(t)) { + return json.RawMessage(`{"error":` + t + `}`) + } + return json.RawMessage(fmt.Sprintf(`{"error":%s}`, mustJSONString(content))) +} + func mustJSONString(s string) string { b, _ := json.Marshal(s) return string(b) @@ -159,25 +175,29 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er toolNameByID[tc.ID] = tc.Name } } + // Thought signatures are replayed on the part they arrived on: + // per function call (Gemini 3 rejects a function-call turn + // without them) and on the text part. parts := []gmPart{} if m.Content != "" { - parts = append(parts, gmPart{Text: m.Content}) + parts = append(parts, gmPart{Text: m.Content, ThoughtSignature: m.ThinkingSignature}) } for _, tc := range m.ToolCalls { args := json.RawMessage(tc.Arguments) if len(args) == 0 { args = json.RawMessage("{}") } - parts = append(parts, gmPart{FunctionCall: &gmFnCall{Name: tc.Name, Args: args}}) + parts = append(parts, gmPart{FunctionCall: &gmFnCall{Name: tc.Name, Args: args}, ThoughtSignature: tc.Signature}) } if len(parts) == 0 { - parts = []gmPart{{Text: " "}} // Gemini rejects empty parts + // Gemini rejects empty parts. + parts = []gmPart{{Text: " ", ThoughtSignature: m.ThinkingSignature}} } out.Contents = append(out.Contents, gmContent{Role: "model", Parts: parts}) case RoleTool: // Merge the run of consecutive tool messages into one user // content with functionResponse parts. - var parts []gmPart + var parts, images []gmPart for ; i < len(req.Messages) && req.Messages[i].Role == RoleTool; i++ { tm := req.Messages[i] name := tm.ToolName @@ -190,13 +210,19 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er if name == "" { return nil, &ConfigError{Msg: "tool result for \"" + tm.ToolCallID + "\" has no ToolName and no matching assistant tool_call"} } + resp := wrapToolResponse(partsText(tm)) + if tm.IsError { + resp = wrapToolError(partsText(tm)) + } parts = append(parts, gmPart{ - FunctionResponse: &gmFnResp{ - Name: name, - Response: wrapToolResponse(tm.Content), - }, + FunctionResponse: &gmFnResp{Name: name, Response: resp}, }) + // Images from the tool ride the same user turn as inlineData. + for _, p := range imageParts(tm) { + images = append(images, gmPart{InlineData: &gmBlob{MIMEType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) + } } + parts = append(parts, images...) i-- out.Contents = append(out.Contents, gmContent{Role: "user", Parts: parts}) } @@ -233,8 +259,20 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er if tc := geminiThinkingConfig(req.Thinking, req.ThinkingBudget); tc != nil { cfg.ThinkingConfig = tc } + if req.Seed != nil && (*req.Seed > math.MaxInt32 || *req.Seed < math.MinInt32) { + return nil, &ConfigError{Msg: "Gemini Seed must fit in 32 bits"} + } + cfg.Seed = req.Seed + if req.ResponseFormat.jsonMode() { + cfg.ResponseMimeType = "application/json" + if req.ResponseFormat.Type == ResponseJSONSchema { + cfg.ResponseJSONSchema = req.ResponseFormat.Schema + } + } + out.ToolConfig = geminiToolConfig(req.ToolChoice) if cfg.MaxOutputTokens != 0 || cfg.Temperature != nil || cfg.TopP != nil || - len(cfg.StopSequences) > 0 || cfg.ThinkingConfig != nil { + len(cfg.StopSequences) > 0 || cfg.ThinkingConfig != nil || cfg.Seed != nil || + cfg.ResponseMimeType != "" { out.GenerationConfig = &cfg } return json.Marshal(out) @@ -243,9 +281,10 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er // ── response ───────────────────────────────────────────────────────────── type gmRespPart struct { - Text string `json:"text"` - Thought bool `json:"thought"` - FunctionCall *gmFnCall `json:"functionCall"` + Text string `json:"text"` + Thought bool `json:"thought"` + FunctionCall *gmFnCall `json:"functionCall"` + ThoughtSignature string `json:"thoughtSignature"` } type gmCandidate struct { @@ -262,11 +301,21 @@ type gmUsage struct { PromptTokenCount int `json:"promptTokenCount"` CandidatesTokenCount int `json:"candidatesTokenCount"` ThoughtsTokenCount int `json:"thoughtsTokenCount"` + // CachedContentTokenCount is a subset of PromptTokenCount; nil when the + // provider did not report caching. + CachedContentTokenCount *int `json:"cachedContentTokenCount"` +} + +// gmPromptFeedback reports a prompt Gemini blocked before generating (no +// candidates follow). +type gmPromptFeedback struct { + BlockReason string `json:"blockReason"` } type gmResponse struct { - Candidates []gmCandidate `json:"candidates"` - UsageMetadata gmUsage `json:"usageMetadata"` + Candidates []gmCandidate `json:"candidates"` + UsageMetadata gmUsage `json:"usageMetadata"` + PromptFeedback *gmPromptFeedback `json:"promptFeedback"` } // mapGeminiFinishReason maps finishReason to canonical values. @@ -286,12 +335,36 @@ func mapGeminiFinishReason(s string) string { } } +// geminiFinish maps finishReason and folds in the tool-call turn: Gemini +// reports STOP for a function-call response, which every other format +// reports as tool_calls. +func geminiFinish(reason string, hasCalls bool) string { + f := mapGeminiFinishReason(reason) + if f == FinishStop && hasCalls { + return FinishToolCalls + } + return f +} + +// mapGeminiUsage normalizes usageMetadata: cached content is a subset of +// promptTokenCount (moved to CacheReadTokens) and candidatesTokenCount +// excludes thoughts (added, so CompletionTokens includes reasoning as on +// every other format). func mapGeminiUsage(u gmUsage) Usage { - return Usage{ + out := Usage{ PromptTokens: u.PromptTokenCount, - CompletionTokens: u.CandidatesTokenCount, + CompletionTokens: u.CandidatesTokenCount + u.ThoughtsTokenCount, ReasoningTokens: u.ThoughtsTokenCount, } + if c := u.CachedContentTokenCount; c != nil { + out.CacheReported = true + out.CachedTokens = *c + if *c > 0 && *c <= out.PromptTokens { + out.PromptTokens -= *c + out.CacheReadTokens = *c + } + } + return out } // foldGeminiParts folds response parts into acc, emitting deltas for new @@ -310,6 +383,7 @@ func foldGeminiParts(parts []gmRespPart, acc *streamAccum) []Delta { args = "{}" } c.args.WriteString(args) + c.sig = p.ThoughtSignature deltas = append(deltas, Delta{ Kind: DeltaToolArgs, Text: args, @@ -317,7 +391,13 @@ func foldGeminiParts(parts []gmRespPart, acc *streamAccum) []Delta { ToolID: c.id, ToolName: c.name, }) + case p.ThoughtSignature != "" && p.Text == "": + // Signature-only part (streams often close with one). + acc.thinkingSignature = p.ThoughtSignature case p.Text != "": + if p.ThoughtSignature != "" { + acc.thinkingSignature = p.ThoughtSignature + } if p.Thought { acc.reasoning.WriteString(p.Text) deltas = append(deltas, Delta{Kind: DeltaReasoning, Text: p.Text}) @@ -337,13 +417,17 @@ func parseGeminiResponse(body []byte) (*ChatResult, error) { return nil, fmt.Errorf("llm: parse response: %w", err) } if len(r.Candidates) == 0 { + if r.PromptFeedback != nil && r.PromptFeedback.BlockReason != "" { + // A blocked prompt is a completed, filtered turn. + return &ChatResult{FinishReason: FinishContentFilter, Usage: mapGeminiUsage(r.UsageMetadata)}, nil + } return nil, fmt.Errorf("llm: response has no candidates") } acc := newStreamAccum() c := r.Candidates[0] foldGeminiParts(c.Content.Parts, acc) res := acc.result() - res.FinishReason = mapGeminiFinishReason(c.FinishReason) + res.FinishReason = geminiFinish(c.FinishReason, len(res.ToolCalls) > 0) res.Usage = mapGeminiUsage(r.UsageMetadata) return res, nil } @@ -351,8 +435,9 @@ func parseGeminiResponse(body []byte) (*ChatResult, error) { // ── streaming ──────────────────────────────────────────────────────────── type gmStreamChunk struct { - Candidates []gmCandidate `json:"candidates"` - UsageMetadata *gmUsage `json:"usageMetadata"` // nil = chunk carries no usage update + Candidates []gmCandidate `json:"candidates"` + PromptFeedback *gmPromptFeedback `json:"promptFeedback"` + UsageMetadata *gmUsage `json:"usageMetadata"` // nil = chunk carries no usage update } // mapGeminiStreamEvent folds one Gemini SSE chunk into acc. Chunks carry @@ -367,12 +452,19 @@ func mapGeminiStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, error) acc.usage = mapGeminiUsage(*c.UsageMetadata) } if len(c.Candidates) == 0 { + if c.PromptFeedback != nil && c.PromptFeedback.BlockReason != "" { + // Blocked prompt: a completed content_filter turn, not a + // truncated stream to retry. + acc.finishReason = FinishContentFilter + acc.sawFinish = true + } return nil, false, nil } cand := c.Candidates[0] deltas := foldGeminiParts(cand.Content.Parts, acc) if cand.FinishReason != "" { - acc.finishReason = mapGeminiFinishReason(cand.FinishReason) + acc.finishReason = geminiFinish(cand.FinishReason, len(acc.calls) > 0) + acc.sawFinish = true } return deltas, false, nil } @@ -397,7 +489,7 @@ type gmModelsPage struct { func listModelsGemini(ctx context.Context, pc *providerClient) ([]Model, error) { var out []Model pageToken := "" - for page := 0; page < 10; page++ { + for page := 0; page < maxModelPages; page++ { url := pc.base + "/v1beta/models?pageSize=100" if pageToken != "" { url += "&pageToken=" + neturl.QueryEscape(pageToken) @@ -408,7 +500,7 @@ func listModelsGemini(ctx context.Context, pc *providerClient) ([]Model, error) } var p gmModelsPage if err := json.Unmarshal(data, &p); err != nil { - return nil, fmt.Errorf("llm: parse models response: %w", err) + return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) } for _, m := range p.Models { out = append(out, Model{ @@ -424,5 +516,5 @@ func listModelsGemini(ctx context.Context, pc *providerClient) ([]Model, error) } pageToken = p.NextPageToken } - return out, nil + return nil, fmt.Errorf("%w (%d pages)", ErrModelListTruncated, maxModelPages) } diff --git a/gemini_signature_test.go b/gemini_signature_test.go new file mode 100644 index 0000000..5da66e3 --- /dev/null +++ b/gemini_signature_test.go @@ -0,0 +1,190 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestParseGeminiResponse_CapturesThoughtSignatures(t *testing.T) { + res, err := parseGeminiResponse([]byte(`{"candidates":[{"content":{"parts":[ + {"text":"thinking","thought":true}, + {"functionCall":{"name":"f","args":{}},"thoughtSignature":"SIG1"}, + {"functionCall":{"name":"g","args":{}}}]},"finishReason":"STOP"}]}`)) + if err != nil { + t.Fatal(err) + } + if len(res.ToolCalls) != 2 || res.ToolCalls[0].Signature != "SIG1" || res.ToolCalls[1].Signature != "" { + t.Fatalf("tool calls = %+v", res.ToolCalls) + } + res, _ = parseGeminiResponse([]byte(`{"candidates":[{"content":{"parts":[{"text":"hi","thoughtSignature":"TXT"}]},"finishReason":"STOP"}]}`)) + if res.ThinkingSignature != "TXT" { + t.Errorf("text signature = %q", res.ThinkingSignature) + } +} + +func TestMapGeminiStreamEvent_SignatureOnlyPart(t *testing.T) { + acc := newStreamAccum() + for _, c := range []string{ + `{"candidates":[{"content":{"parts":[{"text":"hi"}]}}]}`, + `{"candidates":[{"content":{"parts":[{"text":"","thoughtSignature":"END"}]},"finishReason":"STOP"}]}`, + } { + if _, _, err := mapGeminiStreamEvent([]byte(c), acc); err != nil { + t.Fatal(err) + } + } + if res := acc.result(); res.ThinkingSignature != "END" || res.Content != "hi" { + t.Errorf("result = %+v", res) + } +} + +func TestBuildGeminiRequest_ReplaysThoughtSignatures(t *testing.T) { + b, err := buildGeminiRequest(&ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "q"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "call_0", Name: "f", Arguments: "{}", Signature: "SIG1"}, {ID: "call_1", Name: "g", Arguments: "{}"}}}, + {Role: RoleTool, ToolCallID: "call_0", Content: "r"}, + {Role: RoleTool, ToolCallID: "call_1", Content: "r"}, + {Role: RoleAssistant, Content: "done", ThinkingSignature: "TXT"}, + {Role: RoleAssistant, ThinkingSignature: "ONLY"}, + }}, "gemini", false) + if err != nil { + t.Fatal(err) + } + var r struct { + Contents []struct { + Parts []map[string]any `json:"parts"` + } `json:"contents"` + } + if err := json.Unmarshal(b, &r); err != nil { + t.Fatal(err) + } + calls := r.Contents[1].Parts + if calls[0]["thoughtSignature"] != "SIG1" { + t.Errorf("first call part = %v", calls[0]) + } + if _, ok := calls[1]["thoughtSignature"]; ok { + t.Errorf("second call must carry no signature: %v", calls[1]) + } + if p := r.Contents[3].Parts[0]; p["text"] != "done" || p["thoughtSignature"] != "TXT" { + t.Errorf("text part = %v", p) + } + if p := r.Contents[4].Parts[0]; p["thoughtSignature"] != "ONLY" { + t.Errorf("signature-only turn = %v", p) + } +} + +// A function-call response is a tool_calls turn on every format; Gemini +// reports STOP for it. +func TestGeminiFinishReason_ToolCalls(t *testing.T) { + res, err := parseGeminiResponse([]byte(`{"candidates":[{"content":{"parts":[{"functionCall":{"name":"f","args":{}}}]},"finishReason":"STOP"}]}`)) + if err != nil { + t.Fatal(err) + } + if res.FinishReason != FinishToolCalls { + t.Errorf("buffered finish = %q, want tool_calls", res.FinishReason) + } + acc := newStreamAccum() + _, _, _ = mapGeminiStreamEvent([]byte(`{"candidates":[{"content":{"parts":[{"functionCall":{"name":"f","args":{}}}]}}]}`), acc) + _, _, _ = mapGeminiStreamEvent([]byte(`{"candidates":[{"content":{"parts":[]},"finishReason":"STOP"}]}`), acc) + if r := acc.result(); r.FinishReason != FinishToolCalls { + t.Errorf("stream finish = %q, want tool_calls", r.FinishReason) + } + // MAX_TOKENS keeps length even with a (truncated) call. + res, _ = parseGeminiResponse([]byte(`{"candidates":[{"content":{"parts":[{"functionCall":{"name":"f","args":{}}}]},"finishReason":"MAX_TOKENS"}]}`)) + if res.FinishReason != FinishLength { + t.Errorf("max tokens finish = %q", res.FinishReason) + } +} + +func geminiStreamServer(t *testing.T, body string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = fmt.Fprint(w, body) + })) + t.Cleanup(srv.Close) + return srv +} + +// A Gemini stream that ends without any finishReason is truncated, not a +// silent partial success (invariant 3). +func TestCallStream_GeminiPrematureClose(t *testing.T) { + cfg := ProviderConfig{ID: "gemini", Format: FormatGemini} + srv := geminiStreamServer(t, "data: {\"candidates\":[{\"content\":{\"parts\":[{\"text\":\"Hel\"}]}}]}\n\n") + cfg.BaseURL = srv.URL + c := newTestClient(t, cfg, srv) + res, err := c.CallStream(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}, func(Delta) error { return nil }) + if !errors.Is(err, errPrematureClose) { + t.Fatalf("err = %v, want premature close", err) + } + if res == nil || res.Content != "Hel" { + t.Errorf("partial = %+v", res) + } +} + +// An unmapped (non-canonical) finish reason still completes the stream. +func TestCallStream_UnmappedFinishReasonCompletes(t *testing.T) { + cfg := ProviderConfig{ID: "gemini", Format: FormatGemini} + srv := geminiStreamServer(t, "data: {\"candidates\":[{\"content\":{\"parts\":[{\"text\":\"x\"}]},\"finishReason\":\"OTHER\"}]}\n\n") + cfg.BaseURL = srv.URL + c := newTestClient(t, cfg, srv) + res, err := c.CallStream(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}, func(Delta) error { return nil }) + if err != nil || res.FinishReason != "" || res.Content != "x" { + t.Fatalf("res=%+v err=%v", res, err) + } + + oa := ProviderConfig{ID: "oa", Format: FormatOpenAI} + srv2 := geminiStreamServer(t, "data: {\"choices\":[{\"delta\":{\"content\":\"y\"},\"finish_reason\":\"weird\"}]}\n\n") + oa.BaseURL = srv2.URL + c2 := newTestClient(t, oa, srv2) + res, err = c2.CallStream(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}, func(Delta) error { return nil }) + if err != nil || res.Content != "y" { + t.Fatalf("openai unmapped finish: res=%+v err=%v", res, err) + } +} + +func TestGeminiModelPathEscaped(t *testing.T) { + pc := newProviderClient(ProviderConfig{ID: "g", Format: FormatGemini, BaseURL: "https://x"}, nil, nil) + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}} + for _, stream := range []bool{false, true} { + _, url, err := pc.buildChatRequest(req, "../../v1/files?x=", stream) + if err != nil { + t.Fatal(err) + } + if strings.Contains(url, "/../") || strings.Contains(url, "?x=") { + t.Errorf("model not escaped: %s", url) + } + } + _, url, _ := pc.buildChatRequest(req, "gemini-2.5-flash", false) + if url != "https://x/v1beta/models/gemini-2.5-flash:generateContent" { + t.Errorf("plain model url = %s", url) + } +} + +// A prompt blocked by Gemini safety (promptFeedback.blockReason, no +// candidates) is a completed content_filter turn, not a truncated stream. +func TestGeminiPromptBlocked(t *testing.T) { + cfg := ProviderConfig{ID: "gemini", Format: FormatGemini} + var n int + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n++ + w.Header().Set("Content-Type", "text/event-stream") + _, _ = fmt.Fprint(w, "data: {\"promptFeedback\":{\"blockReason\":\"SAFETY\"}}\n\n") + })) + t.Cleanup(srv.Close) + cfg.BaseURL = srv.URL + c := newTestClient(t, cfg, srv) + res, err := c.CallStream(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}, func(Delta) error { return nil }) + if err != nil || res.FinishReason != FinishContentFilter || n != 1 { + t.Fatalf("res=%+v err=%v attempts=%d", res, err, n) + } + r, err := parseGeminiResponse([]byte(`{"promptFeedback":{"blockReason":"PROHIBITED_CONTENT"}}`)) + if err != nil || r.FinishReason != FinishContentFilter { + t.Fatalf("buffered: res=%+v err=%v", r, err) + } +} diff --git a/gemini_test.go b/gemini_test.go index df22b85..6f755bc 100644 --- a/gemini_test.go +++ b/gemini_test.go @@ -170,7 +170,7 @@ func TestParseGeminiResponse(t *testing.T) { if res.Content != "Answer" || res.ReasoningContent != "ponder" { t.Errorf("content: %+v", res) } - if res.FinishReason != FinishStop { + if res.FinishReason != FinishToolCalls { // STOP with a function call is a tool turn t.Errorf("finish = %q", res.FinishReason) } if len(res.ToolCalls) != 1 || res.ToolCalls[0].ID == "" || res.ToolCalls[0].Name != "f" { @@ -222,10 +222,10 @@ func TestMapGeminiStreamEvent_Chunks(t *testing.T) { if len(res.ToolCalls) != 1 || res.ToolCalls[0].Name != "f" { t.Errorf("tool calls: %+v", res.ToolCalls) } - if res.FinishReason != FinishStop { + if res.FinishReason != FinishToolCalls { // STOP with a function call is a tool turn t.Errorf("finish = %q", res.FinishReason) } - if res.Usage.CompletionTokens != 3 { + if res.Usage.CompletionTokens != 3+2 { // candidates + thoughts (last chunk wins) t.Errorf("usage = %+v (last chunk wins)", res.Usage) } } diff --git a/idle_timeout_test.go b/idle_timeout_test.go index d5416cb..dc5fa56 100644 --- a/idle_timeout_test.go +++ b/idle_timeout_test.go @@ -10,7 +10,7 @@ import ( // the watchdog by accident. func TestSetStreamIdleTimeout(t *testing.T) { orig := StreamIdleTimeout() - t.Cleanup(func() { streamIdleTimeout = orig }) + t.Cleanup(func() { SetStreamIdleTimeout(orig) }) SetStreamIdleTimeout(5 * time.Second) if got := StreamIdleTimeout(); got != 5*time.Second { diff --git a/learn.go b/learn.go index a2c6c74..dab2769 100644 --- a/learn.go +++ b/learn.go @@ -75,5 +75,9 @@ func (pc *providerClient) engageLearn(flag *atomic.Bool, kind LearnKind, apiErr ev.Status = apiErr.Status ev.Message = apiErr.Message } + if pc.opts.observer != nil { + pc.opts.observer(ev) + return + } notifyLearn(ev) } diff --git a/llm.go b/llm.go index 8a87a56..2ed1a02 100644 --- a/llm.go +++ b/llm.go @@ -2,6 +2,7 @@ package llm import ( "context" + "errors" "net/http" "strings" "sync" @@ -22,6 +23,21 @@ type SDK struct { timeout time.Duration cacheTTL time.Duration rt http.RoundTripper + retry RetryPolicy + idle time.Duration + observer func(LearnEvent) +} + +// clientOpts snapshots the SDK-level knobs every provider client honors. +func (s *SDK) clientOpts() clientOpts { + return clientOpts{retry: s.retry, idle: s.idle, observer: s.observer} +} + +// newClient builds a provider client for p with the SDK's settings. +func (s *SDK) newClient(p *Provider, timeout time.Duration, learn *learnOnce) *providerClient { + pc := newProviderClientWithLearn(p.cfg, newBufferedHTTP(s.rt, timeout), newStreamHTTP(s.rt), learn) + pc.opts = s.clientOpts() + return pc } // Option configures an SDK at construction time. @@ -130,6 +146,23 @@ func WithQuirks(q Quirks) ProviderOption { return func(c *ProviderConfig) { c.Quirks = q } } +// WithHeaders adds extra HTTP headers to every request this provider makes +// (merged over earlier WithHeaders calls). An empty value removes that +// header, e.g. {"Authorization": ""} for gateways that authenticate with +// an api-key header instead. The map is copied. +func WithHeaders(h map[string]string) ProviderOption { + return func(c *ProviderConfig) { + merged := make(map[string]string, len(c.Headers)+len(h)) + for k, v := range c.Headers { + merged[k] = v + } + for k, v := range h { + merged[k] = v + } + c.Headers = merged + } +} + // WithEnvKeys sets env var names consulted for this provider's key, // primary first. func WithEnvKeys(keys ...string) ProviderOption { @@ -188,21 +221,39 @@ func (s *SDK) Provider(id string) (*Provider, error) { return p, nil } -// Chat returns a chat client bound to a provider and model. -func (s *SDK) Chat(providerID, model string) (*ChatClient, error) { +// usableProvider resolves an authenticated, validly configured provider. +func (s *SDK) usableProvider(providerID string) (*Provider, error) { p, err := s.Provider(providerID) if err != nil { return nil, err } if !p.Authenticated() { - return nil, &ConfigError{Msg: providerID + " has no API key (set " + strings.ToUpper(providerID) + "_API_KEY or use WithAPIKey)"} + return nil, &ConfigError{Msg: providerID + " has no API key (" + keyHint(p.cfg) + ")"} } if p.invalid { return nil, &ConfigError{Msg: providerID + " has an invalid configuration"} } + return p, nil +} + +// keyHint names the env vars actually consulted for cfg's key; a custom +// provider without EnvKeys reads none, so the hint never invents one. +func keyHint(cfg ProviderConfig) string { + if len(cfg.EnvKeys) == 0 { + return "use WithAPIKey, or WithEnvKeys to read it from the environment" + } + return "set " + strings.Join(cfg.EnvKeys, " or ") + ", or use WithAPIKey" +} + +// Chat returns a chat client bound to a provider and model. +func (s *SDK) Chat(providerID, model string) (*ChatClient, error) { + p, err := s.usableProvider(providerID) + if err != nil { + return nil, err + } p.chatLearnOnce.Do(func() { p.chatLearn = &learnOnce{} }) return &ChatClient{ - pc: newProviderClientWithLearn(p.cfg, newBufferedHTTP(p.sdk.rt, p.sdk.timeout), newStreamHTTP(p.sdk.rt), p.chatLearn), + pc: p.sdk.newClient(p, p.sdk.timeout, p.chatLearn), model: model, parent: p, }, nil @@ -227,8 +278,16 @@ type Provider struct { mu sync.Mutex cached []Model cachedAt time.Time + flight *modelFlight // in-progress listing shared by concurrent callers } +// String and GoString redact the provider for fmt verbs (%v, %+v, %#v): +// the API key and header values never print. +func (p *Provider) String() string { return p.cfg.String() } + +// GoString implements fmt.GoStringer with the same redaction as String. +func (p *Provider) GoString() string { return p.cfg.String() } + // ID returns the provider's registry id. func (p *Provider) ID() string { return p.cfg.ID } @@ -273,20 +332,75 @@ func (p *Provider) ListModels(ctx context.Context, opts ...ListOption) ([]Model, return nil, &ConfigError{Msg: p.cfg.ID + " has no API key"} } p.clientOnce.Do(func() { - p.listClient = newProviderClient(p.cfg, newBufferedHTTP(p.sdk.rt, 30*time.Second), newStreamHTTP(p.sdk.rt)) + p.listClient = p.sdk.newClient(p, 30*time.Second, nil) }) - models, err := p.listClient.listModels(ctx) + models, err := p.fetchModels(ctx) if err != nil { return nil, err } - p.mu.Lock() - p.cached, p.cachedAt = models, time.Now() - p.mu.Unlock() out := make([]Model, len(models)) copy(out, models) return out, nil } +// modelFlight is one in-progress upstream model listing that concurrent +// callers share. +type modelFlight struct { + done chan struct{} + models []Model + err error + leaderCanceled bool // the leader's own context ended (not an upstream failure) +} + +// errModelFetchPanicked is what followers of a leader whose fetch panicked +// receive; the panic itself propagates to the leader's caller. +var errModelFetchPanicked = errors.New("llm: model listing fetch panicked") + +// fetchModels fetches the listing once for all concurrent callers (a cold +// cache never stampedes the provider) and refreshes the cache on success. +// A follower whose leader stopped because the leader's own context ended +// fetches again under its own context instead of inheriting that +// cancellation; an upstream failure (including an upstream timeout) is the +// shared result. The flight is always cleared, even if the fetch panics. +func (p *Provider) fetchModels(ctx context.Context) ([]Model, error) { + for { + p.mu.Lock() + if f := p.flight; f != nil { + p.mu.Unlock() + select { + case <-f.done: + case <-ctx.Done(): + return nil, ctx.Err() + } + if f.leaderCanceled && ctx.Err() == nil { + continue + } + return f.models, f.err + } + f := &modelFlight{done: make(chan struct{}), err: errModelFetchPanicked} + p.flight = f + p.mu.Unlock() + p.lead(ctx, f) + return f.models, f.err + } +} + +// lead runs the leader's fetch and publishes it; the deferred release runs +// on panic too, so a crashed fetch can never wedge later callers. +func (p *Provider) lead(ctx context.Context, f *modelFlight) { + defer func() { + p.mu.Lock() + p.flight = nil + if f.err == nil { + p.cached, p.cachedAt = f.models, time.Now() + } + p.mu.Unlock() + close(f.done) + }() + f.models, f.err = p.listClient.listModels(ctx) + f.leaderCanceled = f.err != nil && ctx.Err() != nil +} + // ── ChatClient ─────────────────────────────────────────────────────────── // ChatClient runs chat completions against one provider+model pair. @@ -331,3 +445,25 @@ func (c *ChatClient) CallStream(ctx context.Context, req *ChatRequest, onDelta f } return c.pc.callStream(ctx, req, c.model, onDelta) } + +// WithRetryPolicy replaces the default retry ladder (8 attempts, 30s +// backoff cap) for every chat, speech, transcription and embedding call +// made through this SDK. Zero fields keep their defaults. +func WithRetryPolicy(p RetryPolicy) Option { return func(s *SDK) { s.retry = p } } + +// WithStreamIdleTimeout sets the SSE idle watchdog for this SDK's streams, +// taking precedence over the process-wide SetStreamIdleTimeout default. +// Non-positive values are ignored. +func WithStreamIdleTimeout(d time.Duration) Option { + return func(s *SDK) { + if d > 0 { + s.idle = d + } + } +} + +// WithLearnObserver registers a callback fired once per learn-once fallback +// engaged by this SDK's providers. When set it replaces the process-wide +// SetLearnObserver callback for this SDK. It runs on the request goroutine: +// keep it fast and never call back into the SDK from it. +func WithLearnObserver(fn func(LearnEvent)) Option { return func(s *SDK) { s.observer = fn } } diff --git a/message.go b/message.go index 43457c7..37ad2c0 100644 --- a/message.go +++ b/message.go @@ -71,6 +71,19 @@ type ToolCall struct { ID string Name string Arguments string // JSON object as a string + // Signature is the provider's opaque per-call reasoning signature + // (Gemini thoughtSignature). Replay it unchanged; Gemini 3 rejects a + // function-call turn whose signature is missing. + Signature string +} + +// ThinkingBlock is one provider-signed reasoning segment that must be +// replayed verbatim and in order: an Anthropic thinking or +// redacted_thinking block, or an OpenAI Responses reasoning item. +type ThinkingBlock struct { + Text string // reasoning text (summary on Responses); empty when redacted + Signature string // Anthropic signature / Responses encrypted_content + Redacted string // Anthropic redacted_thinking data; Text and Signature stay empty } // Message is one canonical chat message. For RoleTool messages, ToolCallID @@ -94,9 +107,17 @@ type Message struct { // require thinking to be replayed verbatim (Anthropic signature, OpenAI // Responses encrypted_content). ThinkingSignature string - ToolCalls []ToolCall - ToolCallID string - ToolName string + // ThinkingBlocks replays every signed reasoning segment of a previous + // assistant turn, in order. When set it supersedes ReasoningContent and + // ThinkingSignature on Anthropic and Responses requests. + ThinkingBlocks []ThinkingBlock + ToolCalls []ToolCall + ToolCallID string + ToolName string + // IsError marks a RoleTool result as a failed tool execution (Anthropic + // tool_result is_error; Gemini functionResponse {"error": …}). OpenAI + // formats have no flag: describe the failure in Content. + IsError bool // Cache marks this user message for Anthropic prompt caching // (cache_control ephemeral on the text block). Ignored on other // formats and on non-user roles. @@ -124,12 +145,23 @@ type ToolDef struct { Cache bool } -// Usage reports token accounting. Fields the provider does not report stay 0. -// PromptTokens is exclusive (uncached-only) after provider-specific -// normalization: OpenAI cached_tokens and DeepSeek hit/miss are subsets of -// prompt_tokens and are subtracted; Anthropic reports cache volumes -// exclusively and is left alone. Cache volumes live in the cache fields so -// budget enforcement can sum without double-counting. +// Usage reports token accounting with one meaning on every format. Fields +// the provider does not report stay 0. +// +// - PromptTokens is uncached input only. Inclusive provider counts +// (OpenAI cached_tokens, DeepSeek prompt_cache_hit_tokens, Gemini +// cachedContentTokenCount) are subtracted and moved to CacheReadTokens; +// Anthropic reports cache volumes exclusively and is left alone. +// - CacheReadTokens / CacheCreationTokens are cache reads and cache +// writes. A DeepSeek cache miss is ordinary input (billed as such), not +// a write, so it stays in PromptTokens. +// - CompletionTokens counts every generated token, reasoning included +// (Gemini's candidatesTokenCount excludes thoughts, so they are added); +// ReasoningTokens is the reasoning subset. +// - CachedTokens is the raw provider-reported cached count, kept for +// diagnostics. It is already included in CacheReadTokens — never add it. +// +// Sum with InputTokens and TotalTokens rather than by hand. type Usage struct { PromptTokens int CompletionTokens int @@ -140,11 +172,64 @@ type Usage struct { CacheReported bool } +// InputTokens is the full input volume: uncached prompt plus cache reads +// and writes. +func (u Usage) InputTokens() int { + return u.PromptTokens + u.CacheReadTokens + u.CacheCreationTokens +} + +// TotalTokens is InputTokens plus CompletionTokens. +func (u Usage) TotalTokens() int { return u.InputTokens() + u.CompletionTokens } + +// ToolChoiceMode selects how the model may use the request's tools. +type ToolChoiceMode string + +const ( + ToolChoiceAuto ToolChoiceMode = "auto" // the model decides (provider default) + ToolChoiceNone ToolChoiceMode = "none" // never call a tool + ToolChoiceRequired ToolChoiceMode = "required" // call at least one tool + ToolChoiceTool ToolChoiceMode = "tool" // call the tool named by Name +) + +// ToolChoice constrains tool use. Name is set only with ToolChoiceTool and +// must match one of the request's Tools. +type ToolChoice struct { + Mode ToolChoiceMode + Name string +} + +// ToolChoiceNamed forces a call to the named tool. +func ToolChoiceNamed(name string) *ToolChoice { + return &ToolChoice{Mode: ToolChoiceTool, Name: name} +} + +// ResponseFormatType selects the shape of the model's text output. +type ResponseFormatType string + +const ( + ResponseText ResponseFormatType = "text" // free text (default; nothing sent) + ResponseJSONObject ResponseFormatType = "json_object" // any JSON object + ResponseJSONSchema ResponseFormatType = "json_schema" // JSON matching Schema +) + +// ResponseFormat requests structured output. Name (json_schema only; +// default "response", letters/digits/_/- up to 64) names the schema; +// Schema is a JSON Schema object; Strict asks OpenAI-format providers for +// strict schema adherence. +type ResponseFormat struct { + Type ResponseFormatType + Name string + Schema json.RawMessage + Strict bool +} + // ChatRequest is the canonical request. Model is filled from the ChatClient // when empty. Thinking accepts "", "enabled", "disabled", "low", "medium", // "high", "max" and is translated per provider format. Temperature and TopP: // 0 means use the provider default (field omitted); use a negative value to -// explicitly send 0. Stop maps to each provider's stop-sequence field. +// explicitly send 0. Stop maps to each provider's stop-sequence field (the +// OpenAI Responses API has none, so it is not sent on requests diverted +// there). type ChatRequest struct { Model string Messages []Message @@ -156,6 +241,28 @@ type ChatRequest struct { Temperature float64 TopP float64 Stop []string + // ToolChoice constrains tool use (nil = provider default). It requires + // Tools. + ToolChoice *ToolChoice + // ParallelToolCalls, when set with Tools, allows or forbids several + // tool calls in one turn (OpenAI parallel_tool_calls, Anthropic + // disable_parallel_tool_use). Gemini has no equivalent and ignores it. + ParallelToolCalls *bool + // ResponseFormat requests JSON output (nil = free text). + ResponseFormat *ResponseFormat + // Seed asks for deterministic sampling where supported (OpenAI chat + // completions, Gemini). Anthropic and the Responses API ignore it. + Seed *int + // Extra holds provider-specific top-level body fields merged into the + // request JSON after the SDK's own (so they override them): an escape + // hatch for parameters the canonical shape does not model yet. The merge + // is shallow: an object value replaces the SDK's whole object (e.g. + // Gemini generationConfig). Keys go to whichever endpoint the request + // is sent to, including /responses when an OpenAI request is diverted + // there. Reserved (ConfigError): "stream" (use Call vs CallStream) and + // the keys the SDK validates — model, messages/contents/input, + // system/systemInstruction/instructions, tools. + Extra map[string]any } // ChatResult is the canonical response for both buffered and streaming calls. @@ -166,9 +273,12 @@ type ChatResult struct { // thinking). Consumers must carry it back on the next assistant Message // for tool loops to stay valid. ThinkingSignature string - ToolCalls []ToolCall - FinishReason string - Usage Usage + // ThinkingBlocks lists every signed reasoning segment in response order + // (Anthropic thinking/redacted_thinking, Responses reasoning items). + ThinkingBlocks []ThinkingBlock + ToolCalls []ToolCall + FinishReason string + Usage Usage } // DeltaKind discriminates streamed fragments. @@ -204,3 +314,24 @@ type Model struct { MaxOutputTokens int // 0 = unknown Capabilities []string } + +// AssistantMessage returns the assistant turn to append to the conversation +// before the next request. It carries every replay field (reasoning, +// signatures, thinking blocks, tool calls) so tool loops stay valid on +// every format. Slices are copied. +func (r *ChatResult) AssistantMessage() Message { + m := Message{Role: RoleAssistant} + if r == nil { + return m + } + m.Content = r.Content + m.ReasoningContent = r.ReasoningContent + m.ThinkingSignature = r.ThinkingSignature + if len(r.ThinkingBlocks) > 0 { + m.ThinkingBlocks = append([]ThinkingBlock(nil), r.ThinkingBlocks...) + } + if len(r.ToolCalls) > 0 { + m.ToolCalls = append([]ToolCall(nil), r.ToolCalls...) + } + return m +} diff --git a/models.go b/models.go index 741d69f..170d044 100644 --- a/models.go +++ b/models.go @@ -41,12 +41,16 @@ func (pc *providerClient) listModels(ctx context.Context) ([]Model, error) { if err == nil { return models, nil } + var te *terminalError + if errors.As(err, &te) { + return nil, te.err // malformed body: retrying cannot fix it + } var apiErr *APIError - if errors.As(err, &apiErr) && !apiErr.Retryable { + if (errors.As(err, &apiErr) && !apiErr.Retryable) || errors.Is(err, ErrModelListTruncated) { return nil, err } lastErr = err - if !retrySleep(ctx, backoffDelay(attempt+1)) { + if !retrySleep(ctx, pc.retryDelay(0, attempt)) { break } } @@ -60,7 +64,7 @@ func (pc *providerClient) get(ctx context.Context, url string) ([]byte, time.Dur return nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Accept", "application/json") - pc.setAuthHeaders(req.Header) + pc.setHeaders(req.Header) resp, err := pc.buffered().Do(req) if err != nil { @@ -103,7 +107,7 @@ func listModelsOpenAI(ctx context.Context, pc *providerClient) ([]Model, error) } var r oaModelsResponse if err := json.Unmarshal(data, &r); err != nil { - return nil, fmt.Errorf("llm: parse models response: %w", err) + return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) } entries := r.Data if len(entries) == 0 { @@ -129,3 +133,10 @@ func listModelsOpenAI(ctx context.Context, pc *providerClient) ([]Model, error) } return out, nil } + +// maxModelPages bounds a paginated listing (100 models per page). +const maxModelPages = 100 + +// ErrModelListTruncated reports a model listing longer than the SDK's page +// cap; the SDK never returns a silently truncated listing. +var ErrModelListTruncated = errors.New("llm: model listing exceeds the page cap") diff --git a/models_flight_test.go b/models_flight_test.go new file mode 100644 index 0000000..73196ab --- /dev/null +++ b/models_flight_test.go @@ -0,0 +1,191 @@ +package llm + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// A malformed models body is a terminal protocol failure: retrying the +// same endpoint cannot fix it. +func TestListModels_ParseErrorNotRetried(t *testing.T) { + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + var n atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n.Add(1) + _, _ = fmt.Fprint(w, `{not json`) + })) + _, err := newListModels(srv.URL, f) + srv.Close() + if err == nil { + t.Errorf("%s: want parse error", f) + } + if got := n.Load(); got != 1 { + t.Errorf("%s: requests = %d, want 1 (no retry)", f, got) + } + } +} + +// A listing longer than the page cap is an error, never a silent +// truncation. +func TestListModels_PageCapIsAnError(t *testing.T) { + cases := map[Format]string{ + FormatAnthropic: `{"data":[{"id":"x"}],"has_more":true,"last_id":"x"}`, + FormatGemini: `{"models":[{"name":"models/x"}],"nextPageToken":"P"}`, + } + for f, body := range cases { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprint(w, body) + })) + models, err := newListModels(srv.URL, f) + srv.Close() + if err == nil || !errors.Is(err, ErrModelListTruncated) { + t.Errorf("%s: err = %v (models %d), want ErrModelListTruncated", f, err, len(models)) + } + } +} + +// Concurrent cache misses share one upstream fetch. +func TestListModels_SingleFlight(t *testing.T) { + var n atomic.Int32 + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n.Add(1) + <-release + _, _ = fmt.Fprint(w, `{"data":[{"id":"m"}]}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + p, _ := sdk.Provider("openai") + var wg sync.WaitGroup + errs := make(chan error, 8) + for i := 0; i < 8; i++ { + wg.Add(1) + go func() { + defer wg.Done() + ms, err := p.ListModels(context.Background()) + if err == nil && (len(ms) != 1 || ms[0].ID != "m") { + err = fmt.Errorf("models = %+v", ms) + } + errs <- err + }() + } + deadline := time.Now().Add(2 * time.Second) + for n.Load() == 0 && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + time.Sleep(50 * time.Millisecond) // let the other callers queue up + close(release) + wg.Wait() + close(errs) + for err := range errs { + if err != nil { + t.Error(err) + } + } + if got := n.Load(); got != 1 { + t.Errorf("upstream fetches = %d, want 1", got) + } +} + +// A follower whose leader's context died fetches for itself instead of +// inheriting the leader's cancellation. +func TestListModels_SingleFlightLeaderCancelled(t *testing.T) { + var n atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if n.Add(1) == 1 { + <-r.Context().Done() // leader hangs until cancelled + return + } + _, _ = fmt.Fprint(w, `{"data":[{"id":"m"}]}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + p, _ := sdk.Provider("openai") + leaderCtx, cancel := context.WithCancel(context.Background()) + leaderErr := make(chan error, 1) + go func() { _, err := p.ListModels(leaderCtx); leaderErr <- err }() + for n.Load() == 0 { + time.Sleep(2 * time.Millisecond) + } + followerRes := make(chan []Model, 1) + followerErr := make(chan error, 1) + go func() { + ms, err := p.ListModels(context.Background()) + followerRes <- ms + followerErr <- err + }() + time.Sleep(30 * time.Millisecond) + cancel() + if err := <-leaderErr; !errors.Is(err, context.Canceled) { + t.Errorf("leader err = %v", err) + } + if err := <-followerErr; err != nil { + t.Fatalf("follower err = %v, want its own successful fetch", err) + } + if ms := <-followerRes; len(ms) != 1 { + t.Errorf("follower models = %+v", ms) + } +} + +type rtFunc func(*http.Request) (*http.Response, error) + +func (f rtFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +// A panic inside the leader's fetch (e.g. a user RoundTripper) must never +// leave the flight in place: later calls fetch again. +func TestListModels_LeaderPanicDoesNotWedge(t *testing.T) { + var calls atomic.Int32 + rt := rtFunc(func(r *http.Request) (*http.Response, error) { + if calls.Add(1) == 1 { + panic("transport exploded") + } + return &http.Response{StatusCode: 200, Header: http.Header{}, Body: io.NopCloser(strings.NewReader(`{"data":[{"id":"m"}]}`))}, nil + }) + sdk := New(WithTransport(rt), WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://x"))) + p, _ := sdk.Provider("openai") + func() { + defer func() { _ = recover() }() + _, _ = p.ListModels(context.Background()) + }() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + ms, err := p.ListModels(ctx) + if err != nil || len(ms) != 1 { + t.Fatalf("after leader panic: models=%v err=%v", ms, err) + } +} + +// An upstream timeout inside the leader's fetch is the shared result, not a +// cancellation: followers must not re-lead one after another. +func TestListModels_UpstreamTimeoutShared(t *testing.T) { + fastBackoff(t) + var calls atomic.Int32 + gate := make(chan struct{}) + rt := rtFunc(func(r *http.Request) (*http.Response, error) { + calls.Add(1) + <-gate + return nil, fmt.Errorf("dial: %w", context.DeadlineExceeded) + }) + sdk := New(WithTransport(rt), WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://x"))) + p, _ := sdk.Provider("openai") + var wg sync.WaitGroup + for i := 0; i < 4; i++ { + wg.Add(1) + go func() { defer wg.Done(); _, _ = p.ListModels(context.Background()) }() + } + time.Sleep(50 * time.Millisecond) + close(gate) + wg.Wait() + if got := calls.Load(); got != 3 { // one flight = 3 attempts + t.Errorf("upstream attempts = %d, want 3 (one shared flight)", got) + } +} diff --git a/non_sse_test.go b/non_sse_test.go index 242c0f6..3baf714 100644 --- a/non_sse_test.go +++ b/non_sse_test.go @@ -82,7 +82,7 @@ func TestAttemptStream_NonSSEBodyBounds(t *testing.T) { t.Run(tc.name, func(t *testing.T) { client := &http.Client{Transport: nonSSETransport{body: io.NopCloser(tc.reader)}} pc := newProviderClient(ProviderConfig{Format: FormatOpenAI}, client, client) - out := pc.attemptStream(context.Background(), "http://provider.test/chat/completions", []byte(`{}`), mapOpenAIStreamEvent, func(Delta) error { t.Error("unexpected delta"); return nil }, false) + out := pc.attemptStream(context.Background(), "http://provider.test/chat/completions", []byte(`{}`), mapOpenAIStreamEvent, func(Delta) error { t.Error("unexpected delta"); return nil }, false, "") if !out.bufferedResponse || out.result != nil || out.err == nil || !strings.Contains(out.err.Error(), tc.errorText) { t.Fatalf("outcome=%+v", out) } diff --git a/openai.go b/openai.go index b9e7f13..63d08b5 100644 --- a/openai.go +++ b/openai.go @@ -62,10 +62,8 @@ type oaContentPart struct { ImageURL *oaImageURL `json:"image_url,omitempty"` } -func openAIContent(m Message) any { - if len(m.Parts) == 0 { - return m.Content - } +// openAIContent renders ordered content parts as chat-completions parts. +func openAIContent(m Message) []oaContentPart { parts := make([]oaContentPart, 0, len(m.Parts)) for _, p := range m.Parts { if p.Type == ContentPartImage { @@ -95,28 +93,36 @@ type oaStreamOptions struct { } type oaRequest struct { - Model string `json:"model"` - Messages []oaMessage `json:"messages"` - Tools []oaToolDef `json:"tools,omitempty"` - MaxTokens int `json:"max_tokens,omitempty"` - MaxCompletionTokens int `json:"max_completion_tokens,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - TopP *float64 `json:"top_p,omitempty"` - Stop []string `json:"stop,omitempty"` - Stream bool `json:"stream,omitempty"` - StreamOptions *oaStreamOptions `json:"stream_options,omitempty"` - ReasoningEffort string `json:"reasoning_effort,omitempty"` - IncludeReasoning *bool `json:"include_reasoning,omitempty"` - Thinking *oaThinking `json:"thinking,omitempty"` + Model string `json:"model"` + Messages []oaMessage `json:"messages"` + Tools []oaToolDef `json:"tools,omitempty"` + MaxTokens int `json:"max_tokens,omitempty"` + MaxCompletionTokens int `json:"max_completion_tokens,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopP *float64 `json:"top_p,omitempty"` + Stop []string `json:"stop,omitempty"` + Stream bool `json:"stream,omitempty"` + StreamOptions *oaStreamOptions `json:"stream_options,omitempty"` + ReasoningEffort string `json:"reasoning_effort,omitempty"` + IncludeReasoning *bool `json:"include_reasoning,omitempty"` + Thinking *oaThinking `json:"thinking,omitempty"` + ToolChoice any `json:"tool_choice,omitempty"` + ParallelToolCalls *bool `json:"parallel_tool_calls,omitempty"` + ResponseFormat *oaResponseFormat `json:"response_format,omitempty"` + Seed *int `json:"seed,omitempty"` } // buildOpenAIRequest renders the canonical request in OpenAI format. func buildOpenAIRequest(cfg ProviderConfig, req *ChatRequest, model string, stream, includeStreamOptions bool) oaRequest { q := cfg.Quirks out := oaRequest{ - Model: model, - Stop: req.Stop, - Stream: stream, + Model: model, + Stop: req.Stop, + Stream: stream, + ToolChoice: openAIToolChoice(req.ToolChoice), + ParallelToolCalls: parallelToolCalls(req), + ResponseFormat: openAIResponseFormat(req.ResponseFormat), + Seed: req.Seed, } // OpenAI o-series/gpt-5 models reject max_tokens in favor of // max_completion_tokens; everyone else keeps the classic parameter. @@ -144,21 +150,34 @@ func buildOpenAIRequest(cfg ProviderConfig, req *ChatRequest, model string, stre appendSystem(m.Content) } } - for _, m := range req.Messages { + var toolImages []ContentPart // pending images from the current run of tool results + for i, m := range req.Messages { switch m.Role { case RoleSystem: // already folded into the system message continue case RoleTool: - c := m.Content + // Chat-completions tool messages carry text only: the text parts + // stay on the tool message and images ride a user message right + // after the run of tool results. IsError has no wire field here. + c := partsText(m) + if c == "" && len(m.Parts) > 0 { + c = "(image result attached below)" // some providers reject empty tool content + } msgs = append(msgs, oaMessage{Role: "tool", Content: &c, ToolCallID: m.ToolCallID}) - case RoleAssistant: - om := oaMessage{Role: "assistant"} - if len(m.Parts) == 0 { - om.Content = openAIText(m) - } else { - om.ContentParts = openAIContent(m).([]oaContentPart) + if imgs := imageParts(m); len(imgs) > 0 { + toolImages = append(toolImages, TextPart("Images returned by tool call "+m.ToolCallID+":")) + toolImages = append(toolImages, imgs...) + } + if i+1 < len(req.Messages) && req.Messages[i+1].Role == RoleTool { + continue } + if len(toolImages) > 0 { + msgs = append(msgs, oaMessage{Role: "user", ContentParts: openAIContent(Message{Parts: toolImages})}) + toolImages = nil + } + case RoleAssistant: + om := oaMessage{Role: "assistant", Content: openAIText(m)} // Parts are user/tool only // DeepSeek thinking mode requires the reasoning_content key on every // replayed assistant turn once the request carries tools — including // turns where the provider returned no reasoning of its own. Omitting @@ -193,7 +212,7 @@ func buildOpenAIRequest(cfg ProviderConfig, req *ChatRequest, model string, stre if len(m.Parts) == 0 { om.Content = openAIText(m) } else { - om.ContentParts = openAIContent(m).([]oaContentPart) + om.ContentParts = openAIContent(m) } msgs = append(msgs, om) } @@ -395,6 +414,7 @@ type oaRespUsage struct { PromptTokensDetails *oaPromptDetails `json:"prompt_tokens_details"` PromptCacheHitTokens int `json:"prompt_cache_hit_tokens"` PromptCacheMissTokens int `json:"prompt_cache_miss_tokens"` + TopCachedTokens int `json:"cached_tokens"` // Moonshot/Kimi: top-level, subset of prompt_tokens } // usageFromOpenAI maps a chat-completions usage object onto canonical @@ -421,21 +441,35 @@ func usageFromOpenAI(u *oaRespUsage) Usage { if u.CacheCreationTokens > 0 || u.CacheReadTokens > 0 { out.CacheReported = true } - // DeepSeek native fields: a hit is prompt content read from cache; a - // miss is newly processed content that DeepSeek then caches for - // future requests, i.e. a cache write. - if u.PromptCacheHitTokens > 0 || u.PromptCacheMissTokens > 0 { - out.CacheReadTokens += u.PromptCacheHitTokens - out.CacheCreationTokens += u.PromptCacheMissTokens + switch { + case u.PromptCacheHitTokens > 0 || u.PromptCacheMissTokens > 0: + // DeepSeek native fields (authoritative when present; DeepSeek may + // echo the hit count as cached_tokens too). A hit is prompt content + // read from cache; a miss is ordinary uncached input and stays in + // PromptTokens. Guarded so hostile payloads never go negative. out.CacheReported = true - } - if u.PromptTokensDetails != nil && u.PromptTokensDetails.CachedTokens > 0 && - u.PromptTokensDetails.CachedTokens <= out.PromptTokens { + if hit := u.PromptCacheHitTokens; hit <= out.PromptTokens { + out.PromptTokens -= hit + out.CacheReadTokens += hit + } + case u.CacheReadTokens > 0: + // Anthropic-named field from a gateway, already in CacheReadTokens. + // When the OpenAI-shaped cached_tokens rides along, prompt_tokens + // follows OpenAI (inclusive) semantics and both fields describe the + // same read: subtract it from the prompt once, never add it twice. + if d := u.PromptTokensDetails; d != nil && d.CachedTokens > 0 && d.CachedTokens <= out.PromptTokens { + out.PromptTokens -= d.CachedTokens + } + case u.PromptTokensDetails != nil && u.PromptTokensDetails.CachedTokens > 0 && + u.PromptTokensDetails.CachedTokens <= out.PromptTokens: out.PromptTokens -= u.PromptTokensDetails.CachedTokens out.CacheReadTokens += u.PromptTokensDetails.CachedTokens - } - if total := u.PromptCacheHitTokens + u.PromptCacheMissTokens; total > 0 && total <= out.PromptTokens { - out.PromptTokens -= total + case u.TopCachedTokens > 0 && u.TopCachedTokens <= out.PromptTokens: + // Moonshot/Kimi report the cached subset top-level. + out.CacheReported = true + out.CachedTokens = u.TopCachedTokens + out.PromptTokens -= u.TopCachedTokens + out.CacheReadTokens += u.TopCachedTokens } return out } @@ -563,12 +597,13 @@ func mapOpenAIStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, done b if tc.Function.Name != "" { call.name = tc.Function.Name } - accDelta := Delta{Kind: DeltaToolArgs, Text: tc.Function.Arguments, ToolIndex: idx, ToolID: call.id, ToolName: call.name} + accDelta := Delta{Kind: DeltaToolArgs, Text: tc.Function.Arguments, ToolIndex: call.pos, ToolID: call.id, ToolName: call.name} call.args.WriteString(tc.Function.Arguments) deltas = append(deltas, accDelta) } if ch.FinishReason != "" { acc.finishReason = mapOpenAIFinishReason(ch.FinishReason) + acc.sawFinish = true } } return deltas, false, nil diff --git a/passthrough_test.go b/passthrough_test.go new file mode 100644 index 0000000..26db110 --- /dev/null +++ b/passthrough_test.go @@ -0,0 +1,158 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" +) + +// WithHeaders reaches every request the provider makes; an empty value +// removes a header the SDK would otherwise send (e.g. Authorization for +// api-key gateways). +func TestWithHeaders_AllEntryPoints(t *testing.T) { + var mu sync.Mutex + seen := map[string]http.Header{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + key := r.URL.Path + if strings.Contains(r.Header.Get("Accept"), "event-stream") { + key += "#stream" + } + seen[key] = r.Header.Clone() + mu.Unlock() + switch { + case strings.HasSuffix(r.URL.Path, "/models"): + _, _ = fmt.Fprint(w, `{"data":[{"id":"m"}]}`) + case strings.HasSuffix(r.URL.Path, "/audio/speech"): + w.Header().Set("Content-Type", "audio/mpeg") + _, _ = w.Write([]byte("ID3")) + case strings.HasSuffix(r.URL.Path, "/audio/transcriptions"): + _, _ = fmt.Fprint(w, `{"text":"hi"}`) + case strings.Contains(r.Header.Get("Accept"), "event-stream"): + w.Header().Set("Content-Type", "text/event-stream") + _, _ = fmt.Fprint(w, "data: {\"choices\":[{\"delta\":{\"content\":\"x\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n") + default: + _, _ = fmt.Fprint(w, `{"choices":[{"message":{"content":"ok"},"finish_reason":"stop"}]}`) + } + })) + t.Cleanup(srv.Close) + + sdk := New(WithProvider("gw", WithFormat(FormatOpenAI), WithBaseURL(srv.URL), WithAPIKey("secret-key"), + WithHeaders(map[string]string{"X-Title": "app", "api-key": "secret-key"}), + WithHeaders(map[string]string{"Authorization": "", "HTTP-Referer": "https://example.test"}))) + c, err := sdk.Chat("gw", "m") + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + if _, err := c.Call(ctx, userMsg()); err != nil { + t.Fatal(err) + } + if _, err := c.CallStream(ctx, userMsg(), func(Delta) error { return nil }); err != nil { + t.Fatal(err) + } + p, _ := sdk.Provider("gw") + if _, err := p.ListModels(ctx); err != nil { + t.Fatal(err) + } + if _, err := sdk.Speak(ctx, "gw", "tts", SpeakRequest{Text: "hi", Voice: "v"}); err != nil { + t.Fatal(err) + } + if _, err := sdk.Transcribe(ctx, "gw", "stt", TranscribeRequest{Audio: []byte("a")}); err != nil { + t.Fatal(err) + } + mu.Lock() + defer mu.Unlock() + for _, path := range []string{"/chat/completions", "/chat/completions#stream", "/models", "/audio/speech", "/audio/transcriptions"} { + h, ok := seen[path] + if !ok { + t.Errorf("%s: no request seen", path) + continue + } + if h.Get("X-Title") != "app" || h.Get("HTTP-Referer") != "https://example.test" || h.Get("api-key") != "secret-key" { + t.Errorf("%s: custom headers missing: %v", path, h) + } + if _, ok := h["Authorization"]; ok { + t.Errorf("%s: empty value must remove Authorization", path) + } + } + // Headers never leak through String(). + if s := p.Config().String(); strings.Contains(s, "secret-key") || strings.Contains(s, "app") { + t.Errorf("String() leaks headers: %s", s) + } +} + +// WithHeaders copies its map: later caller mutation never changes the SDK. +func TestWithHeaders_CopiesMap(t *testing.T) { + h := map[string]string{"X-A": "1"} + sdk := New(WithProvider("openai", WithHeaders(h))) + h["X-A"] = "2" + p, _ := sdk.Provider("openai") + if got := p.Config().Headers["X-A"]; got != "1" { + t.Errorf("header = %q, want 1", got) + } +} + +func TestChatRequestExtra(t *testing.T) { + req := &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "x"}}, + Extra: map[string]any{"user": "u-1", "metadata": map[string]any{"k": "v"}, "max_tokens": 7}, + } + req.MaxTokens = 100 + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + m, _ := buildBody(t, f, "m", req) + if m["user"] != "u-1" || m["metadata"].(map[string]any)["k"] != "v" { + t.Errorf("%s: extra fields missing: %v", f, m) + } + if f != FormatGemini && m["max_tokens"] != float64(7) { + t.Errorf("%s: Extra must override SDK fields, max_tokens = %v", f, m["max_tokens"]) + } + } + // Responses path too. + rs := *req + rs.Tools = []ToolDef{{Name: "f"}} + m, url := buildBody(t, FormatOpenAI, "gpt-5.6", &rs) + if !strings.HasSuffix(url, "/responses") || m["user"] != "u-1" { + t.Errorf("responses extra: %s %v", url, m) + } + + bad := []map[string]any{ + {"stream": true}, // owned by Call vs CallStream + {"x": func() {}}, // not JSON + {"": 1}, // empty key + } + for i, extra := range bad { + r := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, Extra: extra} + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "https://x"}, nil, nil) + _, _, err := pc.buildChatRequest(r, "m", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("case %d: err = %v, want ConfigError", i, err) + } + } +} + +// Extra reaches the wire on a real call. +func TestChatRequestExtra_OnTheWire(t *testing.T) { + var body map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &body) + _, _ = fmt.Fprint(w, `{"choices":[{"message":{"content":"ok"},"finish_reason":"stop"}]}`) + })) + t.Cleanup(srv.Close) + c := newTestClient(t, ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: srv.URL}, srv) + if _, err := c.Call(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, Extra: map[string]any{"service_tier": "flex"}}); err != nil { + t.Fatal(err) + } + if body["service_tier"] != "flex" { + t.Errorf("body = %v", body) + } +} diff --git a/provider.go b/provider.go index 3c96441..f568a2d 100644 --- a/provider.go +++ b/provider.go @@ -1,6 +1,7 @@ package llm import ( + neturl "net/url" "strings" ) @@ -44,6 +45,10 @@ type Quirks struct { // thinking.type=disabled outright (GLM-5.3 always reasons; the // documented migration is {type: enabled} + reasoning_effort "low"). ForceThinking []string + // NoJSONSchema: the provider accepts response_format json_object but + // rejects json_schema (DeepSeek); a json_schema ResponseFormat fails fast + // with a ConfigError instead of a provider 400. + NoJSONSchema bool // AnthropicVersion is the anthropic-version header value required by // FormatAnthropic providers ("2023-06-01"). AnthropicVersion string @@ -61,6 +66,12 @@ type ProviderConfig struct { // (aliases after). Resolution stops at the first non-empty value. EnvKeys []string Quirks Quirks + // Headers are extra HTTP headers sent on every request to this + // provider (OpenRouter attribution, OpenAI-Organization, anthropic-beta, + // api-key gateways, …). They are applied last, so they can override the + // SDK's own headers; an empty value removes that header. Like APIKey, + // header values are never logged or included in String(). + Headers map[string]string } func (c ProviderConfig) String() string { @@ -77,6 +88,9 @@ func (c ProviderConfig) String() string { return b.String() } +// GoString keeps %#v from printing the API key and header values. +func (c ProviderConfig) GoString() string { return c.String() } + func strconvBool(b bool) string { if b { return "true" @@ -152,7 +166,7 @@ func builtinProviders() []ProviderConfig { Format: FormatOpenAI, BaseURL: "https://api.deepseek.com", EnvKeys: []string{"DEEPSEEK_API_KEY"}, - Quirks: Quirks{ThinkingObject: true, EchoReasoningWithTools: true}, + Quirks: Quirks{ThinkingObject: true, EchoReasoningWithTools: true, NoJSONSchema: true}, }, { ID: "zai", @@ -200,6 +214,9 @@ func validateProviderConfig(c ProviderConfig) error { if !strings.HasPrefix(c.BaseURL, "http://") && !strings.HasPrefix(c.BaseURL, "https://") { return &ConfigError{Msg: c.ID + ": base URL must start with http:// or https://"} } + if _, err := neturl.Parse(c.BaseURL); err != nil { + return &ConfigError{Msg: c.ID + ": base URL is malformed"} + } return nil } diff --git a/request_options_test.go b/request_options_test.go new file mode 100644 index 0000000..a9b783d --- /dev/null +++ b/request_options_test.go @@ -0,0 +1,310 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" +) + +// buildBody renders req for a format (chat-completions unless model routes +// to /responses) and decodes the JSON body. +func buildBody(t *testing.T, f Format, model string, req *ChatRequest) (map[string]any, string) { + t.Helper() + pc := newProviderClient(ProviderConfig{ID: string(f), Format: f, BaseURL: "https://x", Quirks: Quirks{ReasoningEffort: f == FormatOpenAI}}, nil, nil) + b, url, err := pc.buildChatRequest(req, model, false) + if err != nil { + t.Fatalf("%s build: %v", f, err) + } + var m map[string]any + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + return m, url +} + +func toolReq(choice *ToolChoice) *ChatRequest { + return &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "x"}}, + Tools: []ToolDef{{Name: "f", Parameters: json.RawMessage(`{"type":"object"}`)}, {Name: "g"}}, + ToolChoice: choice, + } +} + +func jsonEq(t *testing.T, name string, got any, want string) { + t.Helper() + var w any + if err := json.Unmarshal([]byte(want), &w); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(got, w) { + g, _ := json.Marshal(got) + t.Errorf("%s = %s, want %s", name, g, want) + } +} + +func TestToolChoice_PerFormat(t *testing.T) { + cases := []struct { + choice *ToolChoice + openai, responses, anthropic, gemini string + }{ + {&ToolChoice{Mode: ToolChoiceAuto}, `"auto"`, `"auto"`, `{"type":"auto"}`, `{"functionCallingConfig":{"mode":"AUTO"}}`}, + {&ToolChoice{Mode: ToolChoiceNone}, `"none"`, `"none"`, `{"type":"none"}`, `{"functionCallingConfig":{"mode":"NONE"}}`}, + {&ToolChoice{Mode: ToolChoiceRequired}, `"required"`, `"required"`, `{"type":"any"}`, `{"functionCallingConfig":{"mode":"ANY"}}`}, + {ToolChoiceNamed("f"), `{"type":"function","function":{"name":"f"}}`, `{"type":"function","name":"f"}`, + `{"type":"tool","name":"f"}`, `{"functionCallingConfig":{"mode":"ANY","allowedFunctionNames":["f"]}}`}, + } + for _, c := range cases { + m, _ := buildBody(t, FormatOpenAI, "gpt-4o", toolReq(c.choice)) + jsonEq(t, "openai tool_choice", m["tool_choice"], c.openai) + m, url := buildBody(t, FormatOpenAI, "gpt-5.6", toolReq(c.choice)) + if !strings.HasSuffix(url, "/responses") { + t.Fatalf("gpt-5.6 + tools must route to /responses: %s", url) + } + jsonEq(t, "responses tool_choice", m["tool_choice"], c.responses) + m, _ = buildBody(t, FormatAnthropic, "claude", toolReq(c.choice)) + jsonEq(t, "anthropic tool_choice", m["tool_choice"], c.anthropic) + m, _ = buildBody(t, FormatGemini, "gemini", toolReq(c.choice)) + jsonEq(t, "gemini toolConfig", m["toolConfig"], c.gemini) + } + // Unset: no field anywhere. + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + m, _ := buildBody(t, f, "m", toolReq(nil)) + if _, ok := m["tool_choice"]; ok { + t.Errorf("%s: tool_choice must be absent when unset", f) + } + if _, ok := m["toolConfig"]; ok { + t.Errorf("%s: toolConfig must be absent when unset", f) + } + } +} + +func TestRequestControls_Validation(t *testing.T) { + bad := []*ChatRequest{ + {ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}}, // no tools + toolReq(&ToolChoice{Mode: "sometimes"}), + toolReq(&ToolChoice{Mode: ToolChoiceTool}), // no name + toolReq(&ToolChoice{Mode: ToolChoiceTool, Name: "missing"}), // unknown tool + toolReq(&ToolChoice{Mode: ToolChoiceAuto, Name: "f"}), // name on non-tool mode + {ResponseFormat: &ResponseFormat{Type: "yaml"}}, // unknown type + {ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema}}, // no schema + {ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`[1]`)}}, + {ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{bad`)}}, + {ResponseFormat: &ResponseFormat{Type: ResponseJSONObject, Schema: json.RawMessage(`{}`)}}, + {ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "has space", Schema: json.RawMessage(`{}`)}}, + } + for i, req := range bad { + if req.Messages == nil { + req.Messages = []Message{{Role: RoleUser, Content: "x"}} + } + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + pc := newProviderClient(ProviderConfig{ID: "x", Format: f, BaseURL: "https://x"}, nil, nil) + _, _, err := pc.buildChatRequest(req, "m", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("case %d %s: err = %v, want ConfigError", i, f, err) + } + } + } +} + +func TestParallelToolCallsAndSeed(t *testing.T) { + no := false + seed := 42 + req := toolReq(nil) + req.ParallelToolCalls = &no + req.Seed = &seed + + m, _ := buildBody(t, FormatOpenAI, "gpt-4o", req) + if m["parallel_tool_calls"] != false || m["seed"] != float64(42) { + t.Errorf("openai: parallel=%v seed=%v", m["parallel_tool_calls"], m["seed"]) + } + m, _ = buildBody(t, FormatOpenAI, "gpt-5.6", req) + if m["parallel_tool_calls"] != false { + t.Errorf("responses: parallel=%v", m["parallel_tool_calls"]) + } + m, _ = buildBody(t, FormatAnthropic, "claude", req) + jsonEq(t, "anthropic tool_choice", m["tool_choice"], `{"type":"auto","disable_parallel_tool_use":true}`) + req.ToolChoice = &ToolChoice{Mode: ToolChoiceRequired} + m, _ = buildBody(t, FormatAnthropic, "claude", req) + jsonEq(t, "anthropic tool_choice", m["tool_choice"], `{"type":"any","disable_parallel_tool_use":true}`) + m, _ = buildBody(t, FormatGemini, "gemini", req) + if m["generationConfig"].(map[string]any)["seed"] != float64(42) { + t.Errorf("gemini seed: %v", m["generationConfig"]) + } + + // Without tools, parallel_tool_calls is never sent (OpenAI rejects it). + plain := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, ParallelToolCalls: &no} + m, _ = buildBody(t, FormatOpenAI, "gpt-4o", plain) + if _, ok := m["parallel_tool_calls"]; ok { + t.Error("parallel_tool_calls without tools must be omitted") + } + m, _ = buildBody(t, FormatAnthropic, "claude", plain) + if _, ok := m["tool_choice"]; ok { + t.Error("anthropic: no tool_choice without tools") + } +} + +// Forced tool use is incompatible with Anthropic extended thinking. +func TestAnthropic_ForcedToolWithThinkingRejected(t *testing.T) { + for _, choice := range []*ToolChoice{{Mode: ToolChoiceRequired}, ToolChoiceNamed("f")} { + req := toolReq(choice) + req.Thinking = "low" + _, err := buildAnthropicRequest(req, "claude", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("%+v: err = %v, want ConfigError", choice, err) + } + } + req := toolReq(&ToolChoice{Mode: ToolChoiceAuto}) + req.Thinking = "low" + if _, err := buildAnthropicRequest(req, "claude", false); err != nil { + t.Errorf("auto + thinking must pass: %v", err) + } +} + +var testSchema = json.RawMessage(`{"type":"object","properties":{"a":{"type":"integer"}},"required":["a"]}`) + +func TestResponseFormat_PerFormat(t *testing.T) { + schemaReq := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema, Strict: true}} + objReq := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}} + + m, _ := buildBody(t, FormatOpenAI, "gpt-4o", schemaReq) + jsonEq(t, "openai response_format", m["response_format"], + `{"type":"json_schema","json_schema":{"name":"answer","schema":`+string(testSchema)+`,"strict":true}}`) + m, _ = buildBody(t, FormatOpenAI, "gpt-4o", objReq) + jsonEq(t, "openai json_object", m["response_format"], `{"type":"json_object"}`) + + // Responses API (tools + gpt-5.6 routes there). + rs := *schemaReq + rs.Tools = []ToolDef{{Name: "f"}} + m, _ = buildBody(t, FormatOpenAI, "gpt-5.6", &rs) + jsonEq(t, "responses text.format", m["text"], + `{"format":{"type":"json_schema","name":"answer","schema":`+string(testSchema)+`,"strict":true}}`) + ro := *objReq + ro.Tools = []ToolDef{{Name: "f"}} + m, _ = buildBody(t, FormatOpenAI, "gpt-5.6", &ro) + jsonEq(t, "responses json_object", m["text"], `{"format":{"type":"json_object"}}`) + + m, _ = buildBody(t, FormatGemini, "gemini", schemaReq) + gc := m["generationConfig"].(map[string]any) + if gc["responseMimeType"] != "application/json" { + t.Errorf("gemini mime = %v", gc["responseMimeType"]) + } + jsonEq(t, "gemini responseJsonSchema", gc["responseJsonSchema"], string(testSchema)) + m, _ = buildBody(t, FormatGemini, "gemini", objReq) + gc = m["generationConfig"].(map[string]any) + if gc["responseMimeType"] != "application/json" || gc["responseJsonSchema"] != nil { + t.Errorf("gemini json_object = %v", gc) + } + + // Anthropic: a forced synthetic tool carries the schema. + m, _ = buildBody(t, FormatAnthropic, "claude", schemaReq) + jsonEq(t, "anthropic tool_choice", m["tool_choice"], `{"type":"tool","name":"answer"}`) + tools := m["tools"].([]any) + last := tools[len(tools)-1].(map[string]any) + if last["name"] != "answer" { + t.Fatalf("anthropic json tool = %v", last) + } + jsonEq(t, "anthropic json tool schema", last["input_schema"], string(testSchema)) + m, _ = buildBody(t, FormatAnthropic, "claude", objReq) + tools = m["tools"].([]any) + if tools[0].(map[string]any)["name"] != anthropicJSONToolDefault { + t.Errorf("anthropic default json tool name = %v", tools[0]) + } + + // Text is the default: nothing is sent. + txt := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, ResponseFormat: &ResponseFormat{Type: ResponseText}} + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + m, _ := buildBody(t, f, "m", txt) + if m["response_format"] != nil || m["tools"] != nil || m["generationConfig"] != nil { + t.Errorf("%s: text format must send nothing: %v", f, m) + } + } +} + +func TestResponseFormat_AnthropicConflicts(t *testing.T) { + rf := &ResponseFormat{Type: ResponseJSONSchema, Name: "f", Schema: testSchema} + cases := []*ChatRequest{ + {Tools: []ToolDef{{Name: "f"}}, ResponseFormat: rf}, // name collision + {Tools: []ToolDef{{Name: "g"}}, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, ResponseFormat: rf}, // forced tool vs ToolChoice + {Thinking: "low", ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, // forced tool vs thinking + } + for i, req := range cases { + req.Messages = []Message{{Role: RoleUser, Content: "x"}} + _, err := buildAnthropicRequest(req, "claude", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("case %d: err = %v, want ConfigError", i, err) + } + } +} + +// The JSON tool call is folded back into Content on both paths. +func TestResponseFormat_AnthropicResultFolding(t *testing.T) { + srvBody := `{"content":[{"type":"tool_use","id":"t","name":"answer","input":{"a":1}}],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":2}}` + sse := strings.Join([]string{ + `event: message_start` + "\n" + `data: {"type":"message_start","message":{"usage":{"input_tokens":1}}}`, + `data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"t","name":"answer"}}`, + `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"a\":"}}`, + `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"1}"}}`, + `data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":2}}`, + `data: {"type":"message_stop"}`, + }, "\n\n") + "\n\n" + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.Contains(r.Header.Get("Accept"), "event-stream") { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = fmt.Fprint(w, sse) + return + } + _, _ = fmt.Fprint(w, srvBody) + })) + t.Cleanup(srv.Close) + c := newTestClient(t, ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL}, srv) + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema}} + + res, err := c.Call(context.Background(), req) + if err != nil { + t.Fatal(err) + } + if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { + t.Errorf("buffered = %+v", res) + } + + var deltas []Delta + res, err = c.CallStream(context.Background(), req, func(d Delta) error { deltas = append(deltas, d); return nil }) + if err != nil { + t.Fatal(err) + } + if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { + t.Errorf("stream = %+v", res) + } + for _, d := range deltas { + if d.Kind != DeltaContent { + t.Errorf("JSON mode must stream content deltas only, got %+v", d) + } + } +} + +// A real tool call alongside JSON mode is kept as a tool call. +func TestFoldAnthropicJSON_KeepsOtherCalls(t *testing.T) { + res := &ChatResult{FinishReason: FinishToolCalls, ToolCalls: []ToolCall{{Name: "answer", Arguments: `{"a":1}`}, {Name: "g", Arguments: "{}"}}} + foldAnthropicJSON(res, "answer") + if res.Content != `{"a":1}` || len(res.ToolCalls) != 1 || res.ToolCalls[0].Name != "g" || res.FinishReason != FinishToolCalls { + t.Errorf("folded = %+v", res) + } + foldAnthropicJSON(nil, "answer") // nil-safe + same := &ChatResult{Content: "x"} + foldAnthropicJSON(same, "") + if same.Content != "x" { + t.Error("inactive JSON mode must not touch the result") + } +} diff --git a/responses.go b/responses.go index d11646b..7ce81e7 100644 --- a/responses.go +++ b/responses.go @@ -74,21 +74,24 @@ type rsFunctionCallItem struct { type rsFunctionOutputItem struct { Type string `json:"type"` CallID string `json:"call_id"` - Output string `json:"output"` + Output any `json:"output"` // string, or input_text/input_image parts } type rsRequest struct { - Model string `json:"model"` - Instructions string `json:"instructions,omitempty"` - Input []any `json:"input"` - Tools []rsTool `json:"tools,omitempty"` - MaxOutputTokens int `json:"max_output_tokens,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - TopP *float64 `json:"top_p,omitempty"` - Stream bool `json:"stream,omitempty"` - Store *bool `json:"store,omitempty"` - Include []string `json:"include,omitempty"` - Reasoning *rsReasoning `json:"reasoning,omitempty"` + Model string `json:"model"` + Instructions string `json:"instructions,omitempty"` + Input []any `json:"input"` + Tools []rsTool `json:"tools,omitempty"` + MaxOutputTokens int `json:"max_output_tokens,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopP *float64 `json:"top_p,omitempty"` + Stream bool `json:"stream,omitempty"` + Store *bool `json:"store,omitempty"` + Include []string `json:"include,omitempty"` + Reasoning *rsReasoning `json:"reasoning,omitempty"` + ToolChoice any `json:"tool_choice,omitempty"` + ParallelToolCalls *bool `json:"parallel_tool_calls,omitempty"` + Text *rsText `json:"text,omitempty"` } func boolPtr(v bool) *bool { return &v } @@ -159,6 +162,10 @@ func buildResponsesRequest(req *ChatRequest, model string, stream bool) rsReques Stream: stream, Store: boolPtr(false), Include: []string{"reasoning.encrypted_content"}, + // Seed and Stop have no Responses equivalent and are not sent. + ToolChoice: responsesToolChoice(req.ToolChoice), + ParallelToolCalls: parallelToolCalls(req), + Text: responsesText(req.ResponseFormat), } if req.MaxTokens > 0 { out.MaxOutputTokens = req.MaxTokens @@ -194,6 +201,15 @@ func buildResponsesRequest(req *ChatRequest, model string, stream bool) rsReques return out } +// responsesReasoningItem renders one replayed reasoning item. +func responsesReasoningItem(summary, encrypted string) rsReasoningItem { + sum := make([]rsSummaryText, 0, 1) + if summary != "" { + sum = append(sum, rsSummaryText{Type: "summary_text", Text: summary}) + } + return rsReasoningItem{Type: "reasoning", EncryptedContent: encrypted, Summary: sum} +} + func buildResponsesInput(req *ChatRequest) (instructions string, input []any) { var sys []string appendSys := func(text string) { @@ -222,16 +238,17 @@ func buildResponsesInput(req *ChatRequest) (instructions string, input []any) { // reasoning as an encrypted reasoning item gated on // ThinkingSignature, so Quirks.EchoReasoningWithTools has nothing to // act on here and no reasoning_content key exists to echo. - if m.ThinkingSignature != "" { - sum := make([]rsSummaryText, 0) - if m.ReasoningContent != "" { - sum = append(sum, rsSummaryText{Type: "summary_text", Text: m.ReasoningContent}) + if len(m.ThinkingBlocks) > 0 { + // Every signed reasoning item, in order. Unsigned blocks + // cannot be replayed statelessly (store=false) and are + // skipped. + for _, tb := range m.ThinkingBlocks { + if tb.Signature != "" { + input = append(input, responsesReasoningItem(tb.Text, tb.Signature)) + } } - input = append(input, rsReasoningItem{ - Type: "reasoning", - EncryptedContent: m.ThinkingSignature, - Summary: sum, - }) + } else if m.ThinkingSignature != "" { + input = append(input, responsesReasoningItem(m.ReasoningContent, m.ThinkingSignature)) } for _, tc := range m.ToolCalls { input = append(input, rsFunctionCallItem{ @@ -248,7 +265,7 @@ func buildResponsesInput(req *ChatRequest) (instructions string, input []any) { input = append(input, rsFunctionOutputItem{ Type: "function_call_output", CallID: m.ToolCallID, - Output: m.Content, + Output: responsesContent(m), }) } } @@ -332,13 +349,16 @@ func chatResultFromResponses(r *rsResponse) *ChatResult { switch item.Type { case "reasoning": if item.EncryptedContent != "" { + // Legacy single-signature field: the last item's. res.ThinkingSignature = item.EncryptedContent } - for _, s := range item.Summary { - if s.Text != "" { - summaries = append(summaries, s.Text) - } + text := reasoningSummary(item.Summary) + if text != "" { + summaries = append(summaries, text) } + // Every item is kept: a tool turn can carry several reasoning + // items, and each must be replayed. + res.ThinkingBlocks = append(res.ThinkingBlocks, ThinkingBlock{Text: text, Signature: item.EncryptedContent}) case "message": for _, p := range item.Content { if p.Type == "output_text" && p.Text != "" { @@ -361,10 +381,28 @@ func chatResultFromResponses(r *rsResponse) *ChatResult { return res } +// reasoningSummary joins one reasoning item's summary texts. +func reasoningSummary(sum []rsSummaryText) string { + var parts []string + for _, s := range sum { + if s.Text != "" { + parts = append(parts, s.Text) + } + } + return strings.Join(parts, "\n") +} + func responsesFinishReason(r *rsResponse, hasTools bool) string { switch r.Status { case "incomplete": - if r.IncompleteDetails != nil && strings.Contains(strings.ToLower(r.IncompleteDetails.Reason), "max_output") { + reason := "" + if r.IncompleteDetails != nil { + reason = strings.ToLower(r.IncompleteDetails.Reason) + } + switch { + case strings.Contains(reason, "content_filter"): + return FinishContentFilter + case strings.Contains(reason, "max_output"): return FinishLength } if hasTools { @@ -390,11 +428,8 @@ func parseResponsesAPI(body []byte) (*ChatResult, error) { return nil, fmt.Errorf("llm: provider error: %s", r.Error.Message) } if r.Status == "failed" { - msg := "responses failed" - if r.Error != nil && r.Error.Message != "" { - msg = r.Error.Message - } - return nil, fmt.Errorf("llm: provider error: %s", msg) + // An error message, when present, was returned just above. + return nil, fmt.Errorf("llm: provider error: responses failed") } return chatResultFromResponses(&r), nil } @@ -403,12 +438,25 @@ func parseResponsesAPI(body []byte) (*ChatResult, error) { type rsStreamEvent struct { Type string `json:"type"` + Message string `json:"message"` // top-level "error" event Delta string `json:"delta"` OutputIndex int `json:"output_index"` Item *rsOutputItem `json:"item"` Response *rsResponse `json:"response"` } +// addResponsesReasoning records one reasoning item as a ThinkingBlock (and +// the legacy last-signature field). +func addResponsesReasoning(acc *streamAccum, idx int, item *rsOutputItem) { + b := acc.block(idx) + b.text.Reset() + b.text.WriteString(reasoningSummary(item.Summary)) + b.sig = item.EncryptedContent + if item.EncryptedContent != "" { + acc.thinkingSignature = item.EncryptedContent + } +} + func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, done bool, err error) { var ev rsStreamEvent if err := json.Unmarshal(data, &ev); err != nil { @@ -431,7 +479,7 @@ func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, don deltas = append(deltas, Delta{ Kind: DeltaToolArgs, Text: ev.Delta, - ToolIndex: ev.OutputIndex, + ToolIndex: call.pos, ToolID: call.id, ToolName: call.name, }) @@ -446,21 +494,23 @@ func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, don } } case "response.output_item.done": - if ev.Item != nil && ev.Item.Type == "reasoning" && ev.Item.EncryptedContent != "" { - acc.thinkingSignature = ev.Item.EncryptedContent + if ev.Item != nil && ev.Item.Type == "reasoning" { + addResponsesReasoning(acc, ev.OutputIndex, ev.Item) } case "response.completed", "response.incomplete": + acc.sawFinish = true if ev.Response != nil { if ev.Response.Usage != nil { acc.usage = usageFromResponses(ev.Response.Usage) } hasTools := len(acc.calls) > 0 acc.finishReason = responsesFinishReason(ev.Response, hasTools) - if acc.thinkingSignature == "" { - for _, item := range ev.Response.Output { - if item.Type == "reasoning" && item.EncryptedContent != "" { - acc.thinkingSignature = item.EncryptedContent - break + if len(acc.thinking) == 0 { + // Items not announced via output_item.done: take them from + // the final response. + for i := range ev.Response.Output { + if item := &ev.Response.Output[i]; item.Type == "reasoning" { + addResponsesReasoning(acc, i, item) } } } @@ -474,6 +524,13 @@ func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, don } } return deltas, true, nil + case "error": + // Top-level stream error event (no response object). + msg := ev.Message + if msg == "" { + msg = "responses stream error" + } + return nil, true, fmt.Errorf("llm: provider error: %s", msg) case "response.failed": msg := "responses failed" if ev.Response != nil && ev.Response.Error != nil && ev.Response.Error.Message != "" { diff --git a/responses_edges_test.go b/responses_edges_test.go new file mode 100644 index 0000000..d144b66 --- /dev/null +++ b/responses_edges_test.go @@ -0,0 +1,169 @@ +package llm + +import ( + "encoding/json" + "strings" + "testing" +) + +func TestResponsesFinishReason_Table(t *testing.T) { + inc := func(reason string) *rsResponse { + r := &rsResponse{Status: "incomplete"} + if reason != "" { + r.IncompleteDetails = &struct { + Reason string `json:"reason"` + }{Reason: reason} + } + return r + } + cases := []struct { + name string + r *rsResponse + hasTools bool + want string + }{ + {"completed", &rsResponse{Status: "completed"}, false, FinishStop}, + {"completed tools", &rsResponse{Status: "completed"}, true, FinishToolCalls}, + {"max output", inc("max_output_tokens"), true, FinishLength}, + {"content filter", inc("content_filter"), false, FinishContentFilter}, + {"content filter with tools", inc("content_filter"), true, FinishContentFilter}, + {"incomplete unknown, tools", inc("other"), true, FinishToolCalls}, + {"incomplete no details", inc(""), false, FinishLength}, + {"failed", &rsResponse{Status: "failed"}, true, ""}, + {"cancelled", &rsResponse{Status: "cancelled"}, false, ""}, + } + for _, c := range cases { + if got := responsesFinishReason(c.r, c.hasTools); got != c.want { + t.Errorf("%s: %q, want %q", c.name, got, c.want) + } + } +} + +func TestParseResponsesAPI_Errors(t *testing.T) { + for body, want := range map[string]string{ + `{bad`: "parse responses", + `{"error":{"message":"quota"}}`: "quota", + `{"status":"failed"}`: "responses failed", + `{"status":"failed","error":{"message":""}}`: "responses failed", + } { + if _, err := parseResponsesAPI([]byte(body)); err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("%s: err = %v, want %q", body, err, want) + } + } + res, err := parseResponsesAPI([]byte(`{"status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"hi"},{"type":"refusal","text":"x"}]}], + "usage":{"input_tokens":10,"output_tokens":4,"input_tokens_details":{"cached_tokens":6},"output_tokens_details":{"reasoning_tokens":2}}}`)) + if err != nil { + t.Fatal(err) + } + if res.Content != "hi" || res.FinishReason != FinishStop { + t.Errorf("result = %+v", res) + } + if u := res.Usage; u.PromptTokens != 4 || u.CacheReadTokens != 6 || u.ReasoningTokens != 2 || u.InputTokens() != 10 { + t.Errorf("usage = %+v", u) + } + if u := usageFromResponses(nil); u != (Usage{}) { + t.Errorf("nil usage = %+v", u) + } +} + +func TestMapResponsesStreamEvent_Edges(t *testing.T) { + if _, _, err := mapResponsesStreamEvent([]byte(`{bad`), newStreamAccum()); err == nil { + t.Error("malformed event must error") + } + // Terminal events without a response object fall back on accumulator state. + cases := []struct { + events []string + want string + }{ + {[]string{`{"type":"response.completed"}`}, FinishStop}, + {[]string{`{"type":"response.incomplete"}`}, FinishLength}, + {[]string{`{"type":"response.output_item.added","output_index":0,"item":{"type":"function_call","call_id":"c","name":"f"}}`, `{"type":"response.completed"}`}, FinishToolCalls}, + } + for i, c := range cases { + acc := newStreamAccum() + var done bool + for _, e := range c.events { + _, d, err := mapResponsesStreamEvent([]byte(e), acc) + if err != nil { + t.Fatal(err) + } + done = d + } + if !done || acc.finishReason != c.want || !acc.sawFinish { + t.Errorf("case %d: done=%v finish=%q", i, done, acc.finishReason) + } + } + for ev, want := range map[string]string{ + `{"type":"error"}`: "responses stream error", + `{"type":"response.failed"}`: "responses failed", + `{"type":"response.failed","response":{"error":{"message":"boom"}}}`: "boom", + } { + _, done, err := mapResponsesStreamEvent([]byte(ev), newStreamAccum()) + if !done || err == nil || !strings.Contains(err.Error(), want) { + t.Errorf("%s: done=%v err=%v", ev, done, err) + } + } + // Reasoning summary and output text deltas accumulate; empty deltas emit nothing. + acc := newStreamAccum() + for _, e := range []string{ + `{"type":"response.reasoning_summary_text.delta","delta":"r"}`, + `{"type":"response.reasoning_summary_text.delta","delta":""}`, + `{"type":"response.output_text.delta","delta":"t"}`, + `{"type":"response.output_text.delta","delta":""}`, + } { + _, _, _ = mapResponsesStreamEvent([]byte(e), acc) + } + if r := acc.result(); r.ReasoningContent != "r" || r.Content != "t" { + t.Errorf("deltas = %+v", r) + } +} + +func TestBuildResponsesRequest_Fields(t *testing.T) { + no := false + req := &ChatRequest{ + System: []SystemBlock{{Text: "base\n"}}, + Messages: []Message{{Role: RoleSystem, Content: "inband"}, {Role: RoleUser, Content: "q"}, {Role: RoleAssistant, Content: "a"}, {Role: RoleTool, ToolCallID: "c", Content: "r"}}, + Tools: []ToolDef{{Name: "f"}}, + MaxTokens: 99, + Temperature: -1, + TopP: -1, + Thinking: "disabled", + ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, + ParallelToolCalls: &no, + } + b, _ := json.Marshal(buildResponsesRequest(req, "gpt-4.1", false)) + var m map[string]any + _ = json.Unmarshal(b, &m) + if m["instructions"] != "base\n\ninband" || m["max_output_tokens"] != float64(99) || m["temperature"] != float64(0) || m["top_p"] != float64(0) { + t.Errorf("request = %v", m) + } + r := m["reasoning"].(map[string]any) + if r["effort"] != "none" || r["summary"] != nil { + t.Errorf("disabled reasoning = %v", r) + } + // Reasoning-only models never get sampling fields. + b, _ = json.Marshal(buildResponsesRequest(req, "gpt-5.6", false)) + m = nil + _ = json.Unmarshal(b, &m) + if m["temperature"] != nil || m["top_p"] != nil { + t.Errorf("gpt-5.6 sampling = %v / %v", m["temperature"], m["top_p"]) + } + for in, want := range map[string]string{"": "medium", "enabled": "medium", "low": "low", "max": "max", "disabled": "none"} { + if got := responsesEffort(in); got != want { + t.Errorf("effort(%q) = %q, want %q", in, got, want) + } + } + // Empty conversation still sends an input array. + if _, in := buildResponsesInput(&ChatRequest{}); in == nil || len(in) != 0 { + t.Errorf("empty input = %#v", in) + } + // gpt-5.4 diverts only with explicit thinking; gpt-5.6 always with tools. + tools := &ChatRequest{Tools: []ToolDef{{Name: "f"}}} + if useResponsesAPI(nil, FormatOpenAI, "gpt-5.4", tools) { + t.Error("gpt-5.4 without thinking must stay on chat completions") + } + tools.Thinking = "high" + if !useResponsesAPI(nil, FormatOpenAI, "gpt-5.4", tools) || !useResponsesAPI(nil, FormatOpenAI, "gpt-6", tools) { + t.Error("explicit thinking + tools must divert gpt-5.4 / gpt-6") + } +} diff --git a/retry.go b/retry.go index f2f8073..a90c46b 100644 --- a/retry.go +++ b/retry.go @@ -2,6 +2,8 @@ package llm import ( "context" + "errors" + "fmt" "math/rand/v2" "net/http" "strconv" @@ -70,6 +72,20 @@ var backoffUnit = time.Second // (1-based retry number) with ±20% jitter. The cap applies after jitter, // so the result never exceeds maxRetryBackoff. func backoffDelay(attempt int) time.Duration { + return backoffDelayCapped(attempt, maxRetryBackoff) +} + +// backoffDelayCapped is backoffDelay with an explicit cap. +func backoffDelayCapped(attempt int, maxBackoff time.Duration) time.Duration { + if attempt > 20 { + // 2^20 units already dwarfs any sane cap; a larger shift would + // overflow time.Duration (a long MaxAttempts ladder must never + // wrap to a zero or negative backoff). + attempt = 20 + } + if attempt < 0 { + return 0 + } d := time.Duration(1< 0 { @@ -78,8 +94,8 @@ func backoffDelay(attempt int) time.Duration { if d < 0 { d = 0 } - if d > maxRetryBackoff { - d = maxRetryBackoff + if d > maxBackoff { + d = maxBackoff } return d } @@ -99,3 +115,134 @@ func retrySleep(ctx context.Context, d time.Duration) bool { return true } } + +// RetryPolicy configures the retry ladder shared by chat (buffered and +// streaming), speech, transcription and embeddings. Zero fields keep the +// defaults. +type RetryPolicy struct { + // MaxAttempts is the total number of attempts including the first + // (default 8). 1 disables retries. + MaxAttempts int + // MaxBackoff caps one exponential backoff step (default 30s). + // Retry-After from the server is honored independently (capped at + // 120s). + MaxBackoff time.Duration +} + +// attempts resolves MaxAttempts (non-positive → default). +func (p RetryPolicy) attempts() int { + if p.MaxAttempts > 0 { + return p.MaxAttempts + } + return maxRetries + 1 +} + +// backoffCap resolves MaxBackoff (non-positive → default). +func (p RetryPolicy) backoffCap() time.Duration { + if p.MaxBackoff > 0 { + return p.MaxBackoff + } + return maxRetryBackoff +} + +// terminalError marks an attempt failure that must never be retried +// (request assembly or response parsing): withRetry unwraps and returns it. +type terminalError struct{ err error } + +func (e *terminalError) Error() string { return e.err.Error() } + +// terminal wraps err as non-retryable (nil stays nil). +func terminal(err error) error { + if err == nil { + return nil + } + return &terminalError{err: err} +} + +// withRetry runs op under the client's retry policy — the single buffered +// ladder shared by chat, speech, transcription and embeddings. op returns +// the Retry-After hint alongside its error. learn, when non-nil, is offered +// every non-retryable *APIError and reports whether a learn-once fallback +// was engaged, in which case the request is retried immediately. +// +// Classification: 429 billing exhaustion fails fast; other 429s back off +// and surface as *RateLimitError when the ladder or the context ends; +// retryable statuses and transport errors back off; anything else (and any +// terminal error) is returned as is. +func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duration, error), learn func(*APIError) bool) error { + n := pc.opts.retry.attempts() + var ( + rateErr *APIError // last 429 + rateRA time.Duration + ) + for attempt := 0; attempt < n; attempt++ { + if err := ctx.Err(); err != nil { + return err + } + ra, err := op() + if err == nil { + return nil + } + var te *terminalError + if errors.As(err, &te) { + return te.err + } + var ce *ConfigError + if errors.As(err, &ce) { + return err // the request cannot be built: retrying cannot help + } + last := attempt == n-1 + var apiErr *APIError + if !errors.As(err, &apiErr) { + // Transport error — retryable. + if cerr := ctx.Err(); cerr != nil { + // Interrupted by deadline/cancel, not exhaustion. + return cerr + } + if last { + return fmt.Errorf("llm: retry exhausted (%d attempts): %w", n, err) + } + if !retrySleep(ctx, pc.retryDelay(0, attempt)) { + return ctx.Err() + } + continue + } + switch { + case apiErr.Status == http.StatusTooManyRequests: + if billingExhausted(apiErr) { + // Permanent: only a recharge fixes this. + return apiErr + } + rateErr, rateRA = apiErr, ra + if !last { + if !retrySleep(ctx, pc.retryDelay(ra, attempt)) { + // The sleep died with the context (deadline or cancel); + // still surface the 429 — the caller needs + // Status/RetryAfter to plan the retry. + return &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} + } + continue + } + case apiErr.Retryable && !last: + if !retrySleep(ctx, pc.retryDelay(ra, attempt)) { + return ctx.Err() + } + continue + case learn != nil && learn(apiErr): + // Learn-once fallback engaged: retry immediately. + if !last { + continue + } + return apiErr + } + if rateErr != nil && !apiErr.Retryable && apiErr.Status != http.StatusTooManyRequests { + // A definitive failure after earlier 429s. + return apiErr + } + if rateErr != nil { + return &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} + } + return apiErr + } + return ctx.Err() // unreachable: attempts() >= 1 +} diff --git a/retry_policy_test.go b/retry_policy_test.go new file mode 100644 index 0000000..060a67c --- /dev/null +++ b/retry_policy_test.go @@ -0,0 +1,199 @@ +package llm + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "sync" + "sync/atomic" + "testing" + "time" +) + +// countingServer answers every request with status and counts hits. +func countingServer(t *testing.T, status int, header map[string]string) (*httptest.Server, *atomic.Int32) { + t.Helper() + var n atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n.Add(1) + for k, v := range header { + w.Header().Set(k, v) + } + w.WriteHeader(status) + _, _ = w.Write([]byte(`{"error":{"message":"boom"}}`)) + })) + t.Cleanup(srv.Close) + return srv, &n +} + +func fastBackoff(t *testing.T) { + t.Helper() + old := backoffUnit + backoffUnit = time.Millisecond + t.Cleanup(func() { backoffUnit = old }) +} + +func userMsg() *ChatRequest { + return &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}} +} + +// WithRetryPolicy bounds the attempt count on every retrying entry point. +func TestWithRetryPolicy_MaxAttempts(t *testing.T) { + fastBackoff(t) + srv, n := countingServer(t, http.StatusInternalServerError, nil) + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 2}), + WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, err := sdk.Chat("openai", "m") + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + check := func(name string, want int32) { + t.Helper() + if got := n.Swap(0); got != want { + t.Errorf("%s: attempts = %d, want %d", name, got, want) + } + } + if _, err := c.Call(ctx, userMsg()); err == nil { + t.Error("Call: want error") + } + check("Call", 2) + if _, err := c.CallStream(ctx, userMsg(), func(Delta) error { return nil }); err == nil { + t.Error("CallStream: want error") + } + check("CallStream", 2) + if _, err := sdk.Speak(ctx, "openai", "tts", SpeakRequest{Text: "hi", Voice: "v"}); err == nil { + t.Error("Speak: want error") + } + check("Speak", 2) + if _, err := sdk.Transcribe(ctx, "openai", "stt", TranscribeRequest{Audio: []byte("a")}); err == nil { + t.Error("Transcribe: want error") + } + check("Transcribe", 2) + + one := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c1, _ := one.Chat("openai", "m") + _, _ = c1.Call(ctx, userMsg()) + check("MaxAttempts=1", 1) +} + +// The default ladder is unchanged: 8 attempts. +func TestRetryPolicy_Defaults(t *testing.T) { + var p RetryPolicy + if p.attempts() != maxRetries+1 || p.backoffCap() != maxRetryBackoff { + t.Fatalf("defaults = %d / %v", p.attempts(), p.backoffCap()) + } + p = RetryPolicy{MaxAttempts: -3, MaxBackoff: -time.Second} + if p.attempts() != maxRetries+1 || p.backoffCap() != maxRetryBackoff { + t.Fatalf("negative values must fall back to defaults: %d / %v", p.attempts(), p.backoffCap()) + } +} + +func TestRetryPolicy_MaxBackoffCapsDelay(t *testing.T) { + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI}, nil, nil) + pc.opts.retry = RetryPolicy{MaxBackoff: 5 * time.Millisecond} + for attempt := 0; attempt < 10; attempt++ { + if d := pc.retryDelay(0, attempt); d > 5*time.Millisecond { + t.Fatalf("attempt %d delay %v exceeds cap", attempt, d) + } + } + // Retry-After is a server instruction and is honored over the cap. + if d := pc.retryDelay(time.Second, 0); d != time.Second { + t.Errorf("Retry-After delay = %v", d) + } +} + +// A buffered call's retries share one wall-clock budget (the request +// timeout), exactly like streaming — never timeout × attempts. +func TestCall_WholeCallBudget(t *testing.T) { + srv, _ := countingServer(t, http.StatusServiceUnavailable, map[string]string{"Retry-After": "1"}) + sdk := New(WithRequestTimeout(300*time.Millisecond), + WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "m") + start := time.Now() + _, err := c.Call(context.Background(), userMsg()) + if el := time.Since(start); el > 900*time.Millisecond { + t.Fatalf("buffered call ran %v, want ≤ the 300ms budget (+slack)", el) + } + if !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("err = %v, want deadline exceeded", err) + } +} + +// WithStreamIdleTimeout is per SDK and wins over the process default. +func TestWithStreamIdleTimeout(t *testing.T) { + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + w.(http.Flusher).Flush() + <-release + })) + t.Cleanup(srv.Close) + t.Cleanup(func() { close(release) }) + sdk := New(WithStreamIdleTimeout(50*time.Millisecond), WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "m") + start := time.Now() + _, err := c.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }) + if !errors.Is(err, ErrIdleTimeout) { + t.Fatalf("err = %v, want ErrIdleTimeout", err) + } + if el := time.Since(start); el > 2*time.Second { + t.Errorf("idle watchdog fired after %v, want ~50ms", el) + } + if New(WithStreamIdleTimeout(-1)).idle != 0 { + t.Error("non-positive idle timeout must be ignored") + } +} + +// WithLearnObserver is per SDK and supersedes the process-wide observer. +func TestWithLearnObserver(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"},"finish_reason":"stop"}]}`)) + })) + t.Cleanup(srv.Close) + var global atomic.Int32 + SetLearnObserver(func(LearnEvent) { global.Add(1) }) + t.Cleanup(func() { SetLearnObserver(nil) }) + + var mu sync.Mutex + var got []LearnEvent + sdk := New(WithLearnObserver(func(ev LearnEvent) { mu.Lock(); got = append(got, ev); mu.Unlock() }), + WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, _ := sdk.Chat("openai", "m") + if _, err := c.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }); err != nil { + t.Fatal(err) + } + mu.Lock() + defer mu.Unlock() + if len(got) != 1 || got[0].Kind != LearnBuffered || got[0].Provider != "openai" { + t.Errorf("SDK observer events = %+v", got) + } + if global.Load() != 0 { + t.Error("process-wide observer must not fire when the SDK has its own") + } +} + +// SetStreamIdleTimeout is safe to call while streams read the default. +func TestSetStreamIdleTimeout_RaceFree(t *testing.T) { + orig := StreamIdleTimeout() + t.Cleanup(func() { SetStreamIdleTimeout(orig) }) + var wg sync.WaitGroup + for i := 0; i < 4; i++ { + wg.Add(2) + go func() { defer wg.Done(); SetStreamIdleTimeout(time.Second) }() + go func() { defer wg.Done(); _ = StreamIdleTimeout() }() + } + wg.Wait() +} + +// setIdleForTest pins the process-wide idle watchdog for one test. +func setIdleForTest(t *testing.T, d time.Duration) { + t.Helper() + old := StreamIdleTimeout() + SetStreamIdleTimeout(d) + t.Cleanup(func() { SetStreamIdleTimeout(old) }) +} diff --git a/retry_test.go b/retry_test.go index da90f45..fde1a00 100644 --- a/retry_test.go +++ b/retry_test.go @@ -157,16 +157,14 @@ func TestPumpSSE_HandlerErrorStops(t *testing.T) { } func TestPumpSSE_IdleTimeout(t *testing.T) { - old := streamIdleTimeout - streamIdleTimeout = 30 * time.Millisecond - defer func() { streamIdleTimeout = old }() + setIdleForTest(t, 30*time.Millisecond) // A reader that never produces events. r, w := io.Pipe() defer r.Close() defer w.Close() start := time.Now() - err := pumpSSE(context.Background(), r, streamIdleTimeout, func([]byte) error { return nil }) + err := pumpSSE(context.Background(), r, StreamIdleTimeout(), func([]byte) error { return nil }) if err != ErrIdleTimeout { t.Fatalf("err = %v, want ErrIdleTimeout", err) } @@ -176,9 +174,7 @@ func TestPumpSSE_IdleTimeout(t *testing.T) { } func TestPumpSSE_KeepaliveResetsIdle(t *testing.T) { - old := streamIdleTimeout - streamIdleTimeout = 60 * time.Millisecond - defer func() { streamIdleTimeout = old }() + setIdleForTest(t, 60*time.Millisecond) r, w := io.Pipe() defer r.Close() @@ -192,7 +188,7 @@ func TestPumpSSE_KeepaliveResetsIdle(t *testing.T) { w.Close() }() var got []string - if err := pumpSSE(context.Background(), r, streamIdleTimeout, func(d []byte) error { + if err := pumpSSE(context.Background(), r, StreamIdleTimeout(), func(d []byte) error { got = append(got, string(d)) return nil }); err != nil { diff --git a/review_fixes_test.go b/review_fixes_test.go new file mode 100644 index 0000000..96db57d --- /dev/null +++ b/review_fixes_test.go @@ -0,0 +1,227 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" +) + +// A cancelled single-attempt call reports the context error, never +// "retry exhausted". +func TestWithRetry_LastAttemptContextError(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + rt := rtFunc(func(r *http.Request) (*http.Response, error) { + cancel() + return nil, r.Context().Err() + }) + sdk := New(WithTransport(rt), WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://x"))) + c, _ := sdk.Chat("openai", "m") + _, err := c.Call(ctx, userMsg()) + if !errors.Is(err, context.Canceled) || strings.Contains(err.Error(), "retry exhausted") { + t.Errorf("err = %v", err) + } +} + +// Anthropic-named and OpenAI-named cache fields describing the same read +// are never summed; Moonshot's top-level cached_tokens is understood. +func TestUsageFromOpenAI_GatewayShapes(t *testing.T) { + u := usageFromOpenAI(&oaRespUsage{PromptTokens: 1000, CacheReadTokens: 600, PromptTokensDetails: &oaPromptDetails{CachedTokens: 600}}) + if u.InputTokens() != 1000 || u.CacheReadTokens != 600 || u.PromptTokens != 400 { + t.Errorf("gateway both-shapes usage = %+v (input %d)", u, u.InputTokens()) + } + u = usageFromOpenAI(&oaRespUsage{PromptTokens: 1000, TopCachedTokens: 300}) + if u.PromptTokens != 700 || u.CacheReadTokens != 300 || !u.CacheReported { + t.Errorf("moonshot cached_tokens usage = %+v", u) + } +} + +// DeepSeek only accepts json_object: json_schema fails fast instead of a 400. +func TestResponseFormat_JSONSchemaQuirk(t *testing.T) { + var deepseek ProviderConfig + for _, c := range builtinProviders() { + if c.ID == "deepseek" { + deepseek = c + } + } + if !deepseek.Quirks.NoJSONSchema { + t.Fatal("deepseek must carry Quirks.NoJSONSchema") + } + deepseek.BaseURL = "https://x" + pc := newProviderClient(deepseek, nil, nil) + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: testSchema}} + var ce *ConfigError + if _, _, err := pc.buildChatRequest(req, "deepseek-chat", false); !errors.As(err, &ce) { + t.Errorf("json_schema on deepseek: err = %v", err) + } + req.ResponseFormat = &ResponseFormat{Type: ResponseJSONObject} + if _, _, err := pc.buildChatRequest(req, "deepseek-chat", false); err != nil { + t.Errorf("json_object on deepseek: %v", err) + } +} + +func TestAnthropic_ControlEdges(t *testing.T) { + no := false + // disable_parallel_tool_use never rides a "none" choice. + req := toolReq(&ToolChoice{Mode: ToolChoiceNone}) + req.ParallelToolCalls = &no + m, _ := buildBody(t, FormatAnthropic, "claude", req) + jsonEq(t, "none choice", m["tool_choice"], `{"type":"none"}`) + + var ce *ConfigError + // JSON mode forces its own tool, so user tools could never be called. + withTools := toolReq(nil) + withTools.ResponseFormat = &ResponseFormat{Type: ResponseJSONObject} + if _, err := buildAnthropicRequest(withTools, "claude", false); !errors.As(err, &ce) { + t.Errorf("JSON mode + tools: err = %v", err) + } + // Anthropic input_schema must be an object schema. + arr := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{"type":"array"}`)}} + if _, err := buildAnthropicRequest(arr, "claude", false); !errors.As(err, &ce) { + t.Errorf("non-object schema: err = %v", err) + } +} + +func TestGemini_SeedRange(t *testing.T) { + big := 1 << 40 + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, Seed: &big} + var ce *ConfigError + if _, err := buildGeminiRequest(req, "g", false); !errors.As(err, &ce) { + t.Errorf("out-of-range seed: err = %v", err) + } +} + +// An image-only tool result keeps a non-empty tool message on chat +// completions. +func TestOpenAI_ImageOnlyToolResultPlaceholder(t *testing.T) { + req := toolResultReq(Message{Role: RoleTool, ToolCallID: "c1", Parts: []ContentPart{ImagePart("image/png", []byte{1})}}) + m, _ := buildBody(t, FormatOpenAI, "gpt-4o", req) + if c := m["messages"].([]any)[2].(map[string]any)["content"]; c == "" { + t.Errorf("image-only tool content must not be empty: %v", c) + } +} + +func TestEmbed_GeminiModelPrefixAndBatching(t *testing.T) { + var calls atomic.Int32 + var paths []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + paths = append(paths, r.URL.EscapedPath()) + b, _ := io.ReadAll(r.Body) + var req gmEmbedRequest + _ = json.Unmarshal(b, &req) + var sb strings.Builder + sb.WriteString(`{"embeddings":[`) + for i, it := range req.Requests { + if i > 0 { + sb.WriteString(",") + } + if it.Model != "models/text-embedding-004" { + t.Errorf("body model = %q", it.Model) + } + fmt.Fprintf(&sb, `{"values":[%d]}`, len(it.Content.Parts[0].Text)) + } + sb.WriteString(`]}`) + _, _ = fmt.Fprint(w, sb.String()) + })) + t.Cleanup(srv.Close) + inputs := make([]string, 250) + for i := range inputs { + inputs[i] = strings.Repeat("a", i+1) + } + sdk := New(WithProvider("gemini", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "gemini", "models/text-embedding-004", EmbedRequest{Inputs: inputs}) + if err != nil { + t.Fatal(err) + } + if calls.Load() != 3 { + t.Errorf("batches = %d, want 3 (100 per call)", calls.Load()) + } + if paths[0] != "/v1beta/models/text-embedding-004:batchEmbedContents" { + t.Errorf("path = %s", paths[0]) + } + for i, v := range res.Embeddings { + if int(v[0]) != i+1 { + t.Fatalf("vector %d out of order: %v", i, v) + } + } +} + +// Gateways that omit index (all zero) are trusted in response order. +func TestEmbed_OpenAIMissingIndex(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprint(w, `{"data":[{"embedding":[1]},{"embedding":[2]}]}`) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "openai", "m", EmbedRequest{Inputs: []string{"a", "b"}}) + if err != nil || res.Embeddings[1][0] != 2 { + t.Fatalf("res=%+v err=%v", res, err) + } +} + +// Credentials never follow a redirect to another host. +func TestNoCrossHostRedirect(t *testing.T) { + var leaked atomic.Int32 + evil := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("x-api-key") != "" || r.Header.Get("api-key") != "" { + leaked.Add(1) + } + _, _ = fmt.Fprint(w, `{"content":[{"type":"text","text":"x"}],"stop_reason":"end_turn"}`) + })) + t.Cleanup(evil.Close) + origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, strings.Replace(evil.URL, "127.0.0.1", "localhost", 1)+r.URL.Path, http.StatusTemporaryRedirect) + })) + t.Cleanup(origin.Close) + fastBackoff(t) + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("anthropic", WithAPIKey("secret"), WithBaseURL(origin.URL), WithHeaders(map[string]string{"api-key": "secret"}))) + c, _ := sdk.Chat("anthropic", "claude") + _, _ = c.Call(context.Background(), userMsg()) + if leaked.Load() != 0 { + t.Error("credentials were forwarded to a redirect target on another host") + } +} + +// fmt verbs on a Provider or config never print the key or header values. +func TestProviderFormattingRedactsSecrets(t *testing.T) { + sdk := New(WithProvider("openai", WithAPIKey("SECRETKEY123"), WithHeaders(map[string]string{"X-Gw": "HDRSECRET456"}))) + p, _ := sdk.Provider("openai") + for _, s := range []string{fmt.Sprintf("%v", p), fmt.Sprintf("%+v", p), fmt.Sprintf("%#v", p), fmt.Sprintf("%#v", p.Config()), fmt.Sprint(p.Config())} { + if strings.Contains(s, "SECRETKEY123") || strings.Contains(s, "HDRSECRET456") { + t.Errorf("secret leaked: %s", s) + } + } +} + +// Extra cannot replace the conversation or tool catalog the SDK validated. +func TestChatRequestExtra_StructuralKeysReserved(t *testing.T) { + for _, k := range []string{"messages", "contents", "input", "system", "systemInstruction", "instructions", "tools", "model"} { + r := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, Extra: map[string]any{k: []any{}}} + pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "https://x"}, nil, nil) + var ce *ConfigError + if _, _, err := pc.buildChatRequest(r, "m", false); !errors.As(err, &ce) { + t.Errorf("Extra[%q]: err = %v, want ConfigError", k, err) + } + } +} + +// A control character in the upload filename can never inject a +// multipart header. +func TestTranscribe_FilenameControlChars(t *testing.T) { + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL("http://127.0.0.1:1"))) + _, err := sdk.Transcribe(context.Background(), "openai", "m", TranscribeRequest{Audio: []byte("a"), Filename: "a.wav\r\nX-Injected: 1"}) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("err = %v, want ConfigError", err) + } +} diff --git a/stt.go b/stt.go index 08b82a5..1330861 100644 --- a/stt.go +++ b/stt.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "encoding/json" - "errors" "fmt" "io" "mime/multipart" @@ -57,20 +56,18 @@ func (s *SDK) Transcribe(ctx context.Context, providerID, model string, req Tran if len(req.Audio) > maxTranscribeAudioBytes { return nil, &ConfigError{Msg: "transcribe audio exceeds 25MB limit"} } + if strings.ContainsFunc(req.Filename+req.MIMEType, func(r rune) bool { return r < 0x20 || r == 0x7f }) { + // Written into multipart part headers: a CR/LF would inject one. + return nil, &ConfigError{Msg: "transcribe Filename and MIMEType must not contain control characters"} + } if strings.TrimSpace(model) == "" { return nil, &ConfigError{Msg: "transcribe request requires a model"} } - p, err := s.Provider(providerID) + p, err := s.usableProvider(providerID) if err != nil { return nil, err } - if !p.Authenticated() { - return nil, &ConfigError{Msg: providerID + " has no API key (set " + strings.ToUpper(providerID) + "_API_KEY or use WithAPIKey)"} - } - if p.invalid { - return nil, &ConfigError{Msg: providerID + " has an invalid configuration"} - } - pc := newProviderClient(p.cfg, newBufferedHTTP(s.rt, s.timeout), nil) + pc := s.newClient(p, s.timeout, nil) return pc.transcribe(ctx, model, req) } @@ -130,8 +127,8 @@ func (pc *providerClient) buildTranscribeRequest(model string, req TranscribeReq return buf.Bytes(), pc.base + "/audio/transcriptions", nil } -// transcribe runs the STT request against one provider with retry -// semantics identical to the buffered chat path. +// transcribe runs the STT request against one provider with the shared +// retry ladder and whole-call budget of the buffered chat path. func (pc *providerClient) transcribe(ctx context.Context, model string, req TranscribeRequest) (*TranscribeResult, error) { body, url, err := pc.buildTranscribeRequest(model, req) if err != nil { @@ -139,70 +136,33 @@ func (pc *providerClient) transcribe(ctx context.Context, model string, req Tran } ctype := "multipart/form-data; boundary=" + multipartBodyBoundary(body) - var ( - lastErr error - rateErr *APIError - rateRA time.Duration - ) - for attempt := 0; attempt <= maxRetries; attempt++ { - if err := ctx.Err(); err != nil { - return nil, err - } + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) + defer cancel() + var res *TranscribeResult + err = pc.withRetry(ctx, func() (time.Duration, error) { data, ra, err := pc.postMultipart(ctx, url, body, ctype) if err != nil { - var apiErr *APIError - if errors.As(err, &apiErr) { - switch { - case apiErr.Status == http.StatusTooManyRequests && billingExhausted(apiErr): - return nil, apiErr - case apiErr.Status == http.StatusTooManyRequests: - rateErr, rateRA, lastErr = apiErr, ra, apiErr - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(ra, attempt)) { - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - continue - } - case apiErr.Retryable && attempt < maxRetries: - lastErr = apiErr - if !retrySleep(ctx, retryDelay(ra, attempt)) { - return nil, ctx.Err() - } - continue - } - if rateErr != nil && !apiErr.Retryable && apiErr.Status != http.StatusTooManyRequests { - return nil, apiErr - } - if rateErr != nil { - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - return nil, apiErr - } - // Transport error — retryable. - lastErr = err - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(0, attempt)) { - return nil, ctx.Err() - } - continue - } - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", maxRetries+1, err) + return ra, err } - res, perr := parseTranscribeResponse(data) + r, perr := parseTranscribeResponse(data) if perr != nil { // 2xx with an undecodable body is a provider protocol // failure — surface it through the typed error taxonomy // at the actual HTTP status (never a plain fmt.Errorf). - return nil, &APIError{ + return 0, terminal(&APIError{ Provider: pc.cfg.ID, Status: http.StatusOK, Message: perr.Error(), - } + }) } - res.Model = model - return res, nil + r.Model = model + res = r + return 0, nil + }, nil) + if err != nil { + return nil, err } - return nil, lastErr + return res, nil } // multipartBodyBoundary extracts the boundary from a multipart writer's @@ -248,7 +208,7 @@ func (pc *providerClient) postMultipart(ctx context.Context, url string, body [] return nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Content-Type", ctype) - pc.setAuthHeaders(req.Header) + pc.setHeaders(req.Header) resp, err := pc.buffered().Do(req) if err != nil { diff --git a/tool_index_test.go b/tool_index_test.go new file mode 100644 index 0000000..c244eb4 --- /dev/null +++ b/tool_index_test.go @@ -0,0 +1,57 @@ +package llm + +import "testing" + +// Delta.ToolIndex is the call's position in ChatResult.ToolCalls on every +// format, never a provider content-block or output-item index. +func TestDeltaToolIndex_IsToolCallPosition(t *testing.T) { + type step struct { + mapper streamEventMapper + events []string + } + cases := map[string]step{ + "anthropic": {mapAnthropicStreamEvent, []string{ + `{"type":"content_block_start","index":0,"content_block":{"type":"thinking"}}`, + `{"type":"content_block_start","index":1,"content_block":{"type":"text"}}`, + `{"type":"content_block_start","index":2,"content_block":{"type":"tool_use","id":"a","name":"f"}}`, + `{"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"{}"}}`, + `{"type":"content_block_start","index":3,"content_block":{"type":"tool_use","id":"b","name":"g"}}`, + `{"type":"content_block_delta","index":3,"delta":{"type":"input_json_delta","partial_json":"{}"}}`, + }}, + "responses": {mapResponsesStreamEvent, []string{ + `{"type":"response.output_item.added","output_index":0,"item":{"type":"reasoning"}}`, + `{"type":"response.output_item.added","output_index":1,"item":{"type":"function_call","call_id":"a","name":"f"}}`, + `{"type":"response.function_call_arguments.delta","output_index":1,"delta":"{}"}`, + `{"type":"response.output_item.added","output_index":2,"item":{"type":"function_call","call_id":"b","name":"g"}}`, + `{"type":"response.function_call_arguments.delta","output_index":2,"delta":"{}"}`, + }}, + "openai": {mapOpenAIStreamEvent, []string{ + `{"choices":[{"delta":{"tool_calls":[{"index":0,"id":"a","function":{"name":"f","arguments":"{}"}}]}}]}`, + `{"choices":[{"delta":{"tool_calls":[{"index":1,"id":"b","function":{"name":"g","arguments":"{}"}}]}}]}`, + }}, + } + for name, tc := range cases { + acc := newStreamAccum() + seen := map[string]int{} + for _, e := range tc.events { + ds, _, err := tc.mapper([]byte(e), acc) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + for _, d := range ds { + if d.Kind == DeltaToolArgs { + seen[d.ToolID] = d.ToolIndex + } + } + } + res := acc.result() + if len(res.ToolCalls) != 2 { + t.Fatalf("%s: calls = %+v", name, res.ToolCalls) + } + for pos, call := range res.ToolCalls { + if got, ok := seen[call.ID]; !ok || got != pos { + t.Errorf("%s: delta ToolIndex for %s = %d, want position %d", name, call.ID, got, pos) + } + } + } +} diff --git a/transport.go b/transport.go index 18c87b5..2e9a9e0 100644 --- a/transport.go +++ b/transport.go @@ -41,7 +41,7 @@ func newBufferedHTTP(rt http.RoundTripper, timeout time.Duration) *http.Client { if timeout <= 0 { timeout = DefaultTimeout } - return &http.Client{Timeout: timeout, Transport: rt} + return &http.Client{Timeout: timeout, Transport: rt, CheckRedirect: sameHostRedirect} } // newStreamHTTP returns a client with no whole-request timeout: a @@ -49,5 +49,16 @@ func newBufferedHTTP(rt http.RoundTripper, timeout time.Duration) *http.Client { // calls enforce a hard wall-clock deadline plus an idle watchdog via // context instead (chat.go). func newStreamHTTP(rt http.RoundTripper) *http.Client { - return &http.Client{Timeout: 0, Transport: rt} + return &http.Client{Timeout: 0, Transport: rt, CheckRedirect: sameHostRedirect} +} + +// sameHostRedirect refuses to follow a redirect to another host: Go strips +// only Authorization/Cookie across hosts, so x-api-key, x-goog-api-key and +// custom Headers would otherwise reach the redirect target. The redirect +// response itself is returned (and surfaces as an APIError). +func sameHostRedirect(req *http.Request, via []*http.Request) error { + if len(via) >= 10 || req.URL.Host != via[0].URL.Host { + return http.ErrUseLastResponse + } + return nil } diff --git a/tts.go b/tts.go index 34c201c..bfffe41 100644 --- a/tts.go +++ b/tts.go @@ -5,7 +5,6 @@ import ( "context" "encoding/base64" "encoding/json" - "errors" "fmt" "io" "net/http" @@ -51,17 +50,11 @@ func (s *SDK) Speak(ctx context.Context, providerID, model string, req SpeakRequ if strings.TrimSpace(req.Voice) == "" { return nil, &ConfigError{Msg: "speak request requires a Voice (both supported wire formats require one server-side)"} } - p, err := s.Provider(providerID) + p, err := s.usableProvider(providerID) if err != nil { return nil, err } - if !p.Authenticated() { - return nil, &ConfigError{Msg: providerID + " has no API key (set " + strings.ToUpper(providerID) + "_API_KEY or use WithAPIKey)"} - } - if p.invalid { - return nil, &ConfigError{Msg: providerID + " has an invalid configuration"} - } - pc := newProviderClient(p.cfg, newBufferedHTTP(s.rt, s.timeout), nil) + pc := s.newClient(p, s.timeout, nil) return pc.speak(ctx, model, req) } @@ -93,8 +86,9 @@ func (pc *providerClient) buildSpeakRequest(model string, req SpeakRequest) ([]b } } -// speak runs the TTS request against one provider with retry semantics -// identical to the buffered chat path (binary body, no SSE). +// speak runs the TTS request against one provider with the shared retry +// ladder and whole-call budget of the buffered chat path (binary body, no +// SSE). func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequest) (*SpeakResult, error) { body, url, mimeHint, err := pc.buildSpeakRequest(model, req) if err != nil { @@ -102,54 +96,13 @@ func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequ } gemini := pc.cfg.Format == FormatGemini - var ( - lastErr error - rateErr *APIError - rateRA time.Duration - ) - for attempt := 0; attempt <= maxRetries; attempt++ { - if err := ctx.Err(); err != nil { - return nil, err - } + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) + defer cancel() + var res *SpeakResult + err = pc.withRetry(ctx, func() (time.Duration, error) { data, ctype, ra, err := pc.postAudio(ctx, url, body) if err != nil { - var apiErr *APIError - if errors.As(err, &apiErr) { - switch { - case apiErr.Status == http.StatusTooManyRequests && billingExhausted(apiErr): - return nil, apiErr - case apiErr.Status == http.StatusTooManyRequests: - rateErr, rateRA, lastErr = apiErr, ra, apiErr - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(ra, attempt)) { - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - continue - } - case apiErr.Retryable && attempt < maxRetries: - lastErr = apiErr - if !retrySleep(ctx, retryDelay(ra, attempt)) { - return nil, ctx.Err() - } - continue - } - if rateErr != nil && !apiErr.Retryable && apiErr.Status != http.StatusTooManyRequests { - return nil, apiErr - } - if rateErr != nil { - return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} - } - return nil, apiErr - } - // Transport error — retryable. - lastErr = err - if attempt < maxRetries { - if !retrySleep(ctx, retryDelay(0, attempt)) { - return nil, ctx.Err() - } - continue - } - return nil, fmt.Errorf("llm: retry exhausted (%d attempts): %w", maxRetries+1, err) + return ra, err } if gemini { audio, mime, perr := parseGeminiSpeakResponse(data) @@ -157,11 +110,11 @@ func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequ // 2xx with no audio parts is a provider protocol // failure — surface it through the typed error // taxonomy (APIError at the actual HTTP status). - return nil, &APIError{ + return 0, terminal(&APIError{ Provider: pc.cfg.ID, Status: http.StatusOK, Message: perr.Error(), - } + }) } data, ctype = audio, mime } else if strings.HasPrefix(strings.TrimSpace(ctype), "application/json") { @@ -169,15 +122,19 @@ func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequ // JSON error envelope. Never hand JSON bytes back as // audio: parse the envelope through the shared httpError // path. - return nil, pc.httpError(http.StatusOK, data) + return 0, terminal(pc.httpError(http.StatusOK, data)) } - return &SpeakResult{ + res = &SpeakResult{ Audio: data, Model: model, MIMEType: resolveAudioMIME(ctype, mimeHint), - }, nil + } + return 0, nil + }, nil) + if err != nil { + return nil, err } - return nil, lastErr + return res, nil } // resolveAudioMIME prefers the provider's Content-Type; when absent it @@ -199,7 +156,7 @@ func (pc *providerClient) postAudio(ctx context.Context, url string, body []byte return nil, "", 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Content-Type", "application/json") - pc.setAuthHeaders(req.Header) + pc.setHeaders(req.Header) resp, err := pc.buffered().Do(req) if err != nil { diff --git a/usage_cache_test.go b/usage_cache_test.go index 428538d..8e6b7c1 100644 --- a/usage_cache_test.go +++ b/usage_cache_test.go @@ -13,7 +13,8 @@ import ( // Anthropic reports cache tokens exclusively (input_tokens excludes them). // OpenAI (prompt_tokens_details.cached_tokens) and DeepSeek // (prompt_cache_hit_tokens + prompt_cache_miss_tokens = prompt_tokens) -// report them inclusively, as subsets of prompt_tokens. +// report them inclusively, as subsets of prompt_tokens; a DeepSeek miss is +// ordinary uncached input. // // Usage.PromptTokens must be exclusive ("uncached" input) on every // provider, with cache volumes carried in CacheReadTokens / @@ -109,14 +110,15 @@ func TestUsageCache_DeepSeekHitMiss(t *testing.T) { if err != nil { t.Fatal(err) } - if res.Usage.PromptTokens != 0 { - t.Errorf("PromptTokens = %d, want 0 (prompt 1000 = hit 750 + miss 250; every token is cache-accounted)", res.Usage.PromptTokens) + // A miss is ordinary uncached input (billed as such), not a cache write. + if res.Usage.PromptTokens != 250 { + t.Errorf("PromptTokens = %d, want 250 (prompt 1000 - hit 750)", res.Usage.PromptTokens) } if res.Usage.CacheReadTokens != 750 { t.Errorf("CacheReadTokens = %d, want 750", res.Usage.CacheReadTokens) } - if res.Usage.CacheCreationTokens != 250 { - t.Errorf("CacheCreationTokens = %d, want 250", res.Usage.CacheCreationTokens) + if res.Usage.CacheCreationTokens != 0 { + t.Errorf("CacheCreationTokens = %d, want 0", res.Usage.CacheCreationTokens) } if !res.Usage.CacheReported { t.Error("CacheReported = false, want true (DeepSeek hit/miss present)") diff --git a/usage_normalize_test.go b/usage_normalize_test.go new file mode 100644 index 0000000..238b7fb --- /dev/null +++ b/usage_normalize_test.go @@ -0,0 +1,91 @@ +package llm + +import "testing" + +// One canonical meaning across formats: +// - PromptTokens: uncached input only +// - CacheReadTokens / CacheCreationTokens: cache volumes (exclusive) +// - CompletionTokens: every generated token, reasoning included +// - ReasoningTokens: subset of CompletionTokens +// - InputTokens() / TotalTokens(): the provider's full totals +func TestUsage_CanonicalAcrossFormats(t *testing.T) { + type want struct{ prompt, read, create, completion, reasoning, input, total int } + check := func(name string, u Usage, w want) { + t.Helper() + if u.PromptTokens != w.prompt || u.CacheReadTokens != w.read || u.CacheCreationTokens != w.create || + u.CompletionTokens != w.completion || u.ReasoningTokens != w.reasoning || + u.InputTokens() != w.input || u.TotalTokens() != w.total { + t.Errorf("%s: usage = %+v input=%d total=%d, want %+v", name, u, u.InputTokens(), u.TotalTokens(), w) + } + } + + // DeepSeek: a cache miss is ordinary (uncached) input, not a cache write. + check("deepseek", usageFromOpenAI(&oaRespUsage{ + PromptTokens: 1000, CompletionTokens: 40, PromptCacheHitTokens: 750, PromptCacheMissTokens: 250, + }), want{prompt: 250, read: 750, completion: 40, input: 1000, total: 1040}) + + // DeepSeek may echo the hit count as cached_tokens too: never twice. + check("deepseek+details", usageFromOpenAI(&oaRespUsage{ + PromptTokens: 1000, CompletionTokens: 40, PromptCacheHitTokens: 750, PromptCacheMissTokens: 250, + PromptTokensDetails: &oaPromptDetails{CachedTokens: 750}, + }), want{prompt: 250, read: 750, completion: 40, input: 1000, total: 1040}) + + // OpenAI: cached_tokens is a subset of prompt_tokens. + check("openai", usageFromOpenAI(&oaRespUsage{ + PromptTokens: 300, CompletionTokens: 30, PromptTokensDetails: &oaPromptDetails{CachedTokens: 200}, + CompletionTokensDetails: oaUsageDetails{ReasoningTokens: 10}, + }), want{prompt: 100, read: 200, completion: 30, reasoning: 10, input: 300, total: 330}) + + // Gemini: cachedContentTokenCount is a subset of promptTokenCount and + // candidatesTokenCount excludes thoughts. + cached := 30 + g := mapGeminiUsage(gmUsage{PromptTokenCount: 100, CandidatesTokenCount: 10, ThoughtsTokenCount: 5, CachedContentTokenCount: &cached}) + check("gemini", g, want{prompt: 70, read: 30, completion: 15, reasoning: 5, input: 100, total: 115}) + if !g.CacheReported { + t.Error("gemini: CacheReported must be true when cachedContentTokenCount is present") + } + if g := mapGeminiUsage(gmUsage{PromptTokenCount: 5}); g.CacheReported { + t.Error("gemini: no cache field → CacheReported false") + } + + // Anthropic reports cache volumes exclusively. + check("anthropic", usageFromAnthropic(anUsage{InputTokens: 10, OutputTokens: 7, CacheReadTokens: 20, CacheCreationTokens: 5}), + want{prompt: 10, read: 20, create: 5, completion: 7, input: 35, total: 42}) +} + +// Gemini usage flows through buffered parsing. +func TestParseGeminiResponse_CachedUsage(t *testing.T) { + res, err := parseGeminiResponse([]byte(`{"candidates":[{"content":{"parts":[{"text":"x"}]},"finishReason":"STOP"}], + "usageMetadata":{"promptTokenCount":100,"candidatesTokenCount":10,"thoughtsTokenCount":5,"cachedContentTokenCount":30}}`)) + if err != nil { + t.Fatal(err) + } + if res.Usage.PromptTokens != 70 || res.Usage.CacheReadTokens != 30 || res.Usage.CompletionTokens != 15 { + t.Errorf("usage = %+v", res.Usage) + } +} + +// Anthropic's message_delta carries cumulative usage (cache volumes may +// only arrive there); non-zero values update the accumulator. +func TestMapAnthropicStreamEvent_MessageDeltaUsage(t *testing.T) { + acc := newStreamAccum() + for _, e := range []string{ + `{"type":"message_start","message":{"usage":{"input_tokens":10,"output_tokens":1}}}`, + `{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":7,"input_tokens":10,"cache_read_input_tokens":20,"cache_creation_input_tokens":5}}`, + } { + if _, _, err := mapAnthropicStreamEvent([]byte(e), acc); err != nil { + t.Fatal(err) + } + } + u := acc.result().Usage + if u.PromptTokens != 10 || u.CompletionTokens != 7 || u.CacheReadTokens != 20 || u.CacheCreationTokens != 5 || !u.CacheReported { + t.Errorf("usage = %+v", u) + } + // A delta that omits input/cache fields keeps message_start's values. + acc = newStreamAccum() + _, _, _ = mapAnthropicStreamEvent([]byte(`{"type":"message_start","message":{"usage":{"input_tokens":10,"cache_read_input_tokens":3}}}`), acc) + _, _, _ = mapAnthropicStreamEvent([]byte(`{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":7}}`), acc) + if u := acc.result().Usage; u.PromptTokens != 10 || u.CacheReadTokens != 3 || u.CompletionTokens != 7 { + t.Errorf("sparse delta usage = %+v", u) + } +}