Skip to content

feat: LLM Cost Tracking and Budgeting feature in passmark#41

Open
devhimanshuu wants to merge 1 commit intobug0inc:mainfrom
devhimanshuu:features
Open

feat: LLM Cost Tracking and Budgeting feature in passmark#41
devhimanshuu wants to merge 1 commit intobug0inc:mainfrom
devhimanshuu:features

Conversation

@devhimanshuu
Copy link
Copy Markdown

🌟 Summary

Introduces an enterprise-grade LLM Cost Tracking & Budgeting system. This prevents "bill shock" by strictly monitoring API costs across all parallel test workers and halting execution if a user-defined budget is exceeded.

❓ Why is this needed?

AI-driven tests can accumulate high API costs rapidly, especially during loops or parallel test execution in CI/CD. Teams need strict, observable cost controls to confidently run Passmark in production pipelines without fear of runaway expenses.

🛠️ Key Features

  • Distributed Budget Enforcer: Added maxCostPerRun to configure(). If the budget is breached, it safely aborts the suite with a BudgetExceededError.
  • Global Worker Sync (Redis): Uses Redis (INCRBYFLOAT) with a 24-hour TTL to precisely aggregate costs across all parallel Playwright workers, ensuring true global limits.
  • Custom Pricing Overrides: Added a pricing config option to define custom token rates (useful for enterprise gateways or self-hosted models).
  • OpenTelemetry Observability: Automatically attaches token counts and cost metrics (ai.usage.cost) to active OpenTelemetry traces for rich Datadog/Axiom dashboarding.
  • CI/CD Artifacts: Generates a passmark-cost-<pid>.json summary in the uploads directory and logs the total cost upon process exit.
  • Universal Tracking: Intercepts Vercel AI SDK usage and raw OpenAI Responses API (CUA mode) calls universally.

💻 Usage Example

import { configure } from "passmark";

configure({
  maxCostPerRun: 2.00, // Halts the test suite if AI cost exceeds $2.00 globally
  pricing: {
    // Optional: override default pricing
    "my-internal-model": { input: 0.10, output: 0.20 }
  }
});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant