Skip to content

UsageInfo (chat completion usage) is missing prompt_tokens_details #240

Description

@baptistejamin

Hey there

Mistral is now supporting prompt caching, however, there is 0 support for this with this SDK.

Per the prompt caching docs, completion responses report cached tokens at usage.prompt_tokens_details.cached_tokens:

"usage": {
  "prompt_tokens": 1013,
  "total_tokens": 1043,
  "completion_tokens": 30,
  "prompt_tokens_details": { "cached_tokens": 1008 }
}

However, ChatCompletionResponse is typed as UsageInfo, and UsageInfo does not declare prompt_tokens_details.

Seems you rely on Speakeasy and codegen to build this SDK so it should be a quick win ;)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions