test: cover ai-evals API drift and CCM anomaly v2 wiring#552
Draft
cursor[bot] wants to merge 1 commit into
Draft
test: cover ai-evals API drift and CCM anomaly v2 wiring#552cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
Add regression tests for recent production fixes that lacked dispatch and schema contract coverage: - AI Evals (#529): dimension on metric create, thumbs_up on annotations, online_eval metric_set_id migration away from deprecated metric_ids - CCM anomalies (#533): anomalyListExtract, v2 list body builder (zero thresholds, time filters, ordering), drill-down path routing Co-authored-by: Rohan Gupta <thisrohangupta@users.noreply.github.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds regression tests for two recently merged production changes that shipped without dispatch/schema contract coverage.
Risky behavior now covered
AI Evals (#529) — prevents reintroduction of control-plane API drift:
eval_metriccreate requires and forwardsdimension(422 without it)eval_annotationcreate/update expose and preservethumbs_up(includingfalse)online_evalevaluate usesmetric_set_id+judge_llm_connector_refinstead of deprecatedmetric_idsCCM anomalies (#533) — guards v2 API migration:
anomalyListExtractprojection shapecost_anomalyv2 list body builder: defaults, explicit time range vstime_filter, zero thresholds (!= null), ordering/searchcost_anomaly_drilldownpath routing (details vs time-series vs sub-item list) andskipCompactround-tripTest files added/updated
tests/registry/ai-evals.test.ts— control-plane drift + online_eval dispatch teststests/registry/ccm-anomaly.test.ts— new v2 anomaly dispatch and drill-down teststests/registry/ccm-extractors.test.ts—anomalyListExtractunit testsWhy these tests materially reduce regression risk
Both fixes changed request body shapes and API paths that agents rely on at runtime. Schema-only checks would miss dispatch bugs (e.g. dropping
thumbs_up: false, reverting tometric_ids, or using truthy checks that dropmin_amount: 0). These tests assert the exact HTTP path/body the registry emits.Type of Change
Checklist
pnpm testpasses (67 tests in touched files)