feat(foundry): ASSERT to Azure AI Foundry exporter (SDK-backed) - #267
Open
tangym wants to merge 25 commits into
Open
feat(foundry): ASSERT to Azure AI Foundry exporter (SDK-backed)#267tangym wants to merge 25 commits into
tangym wants to merge 25 commits into
Conversation
Starts the v2 exporter series (SDK-based ASSERT → Foundry integration).
This commit is purely scaffolding — no runtime behavior, no imports of
the optional extra outside of code that is only reached when a caller
opts in.
Changes:
- `pyproject.toml`: new `foundry` optional extra with
- `azure-ai-projects>=2.2.0` — the first-party SDK. Wraps every
data-plane call the exporter needs (datasets, beta.evaluators,
openai.evals). Pulls `openai>=1.x` and a few Azure Core deps as
transitive; both are already-approved patterns elsewhere in
pyproject (analysis extra pins `openai>=1.30.0`, azure-aad pins
azure-identity).
- `azure-identity>=1.19.0` — DefaultAzureCredential. Same pin as
the existing azure-aad extra.
- `assert_ai/integrations/foundry/__init__.py`: empty subpackage with
the PEP-562 lazy loader (mirrors the acs subpackage). `_LAZY_EXPORTS`
and `__all__` are empty for now; follow-up commits populate them one
submodule at a time. The `_MISSING_DEPENDENCY_HINT` map is present
so the first commit that adds a submodule needing `azure-ai-projects`
can plug a clear install hint without touching the loader.
- `assert_ai/cli.py`: `_handle_missing_foundry_dependency` +
`_load_foundry_symbol` helpers (mirror the acs pair), and an empty
`foundry` click group. The group renders `--help` on a base install
(no `foundry` extra needed); subcommands land in later commits.
- `tests/test_foundry_scaffold.py`: three tests locking the scaffold
contract — help renders without the extra, unknown attributes raise
AttributeError (not ModuleNotFoundError), and `dir(foundry) ==
sorted(__all__)`.
Test totals: 1132 pass, 17 skipped, 0 failures (3 new scaffold tests).
Nothing in the base ASSERT install changes.
The v1 branch (`tangym/foundry-exporter-v1`) that shipped a hand-rolled
REST-based exporter is kept local as reference only; v2 rebuilds on top
of the SDK because it (a) already implements every REST call we wrote,
(b) exposes update/delete surface v1 lacked, and (c) supports the code-
based custom evaluator flow needed to make Foundry a viewer of ASSERT
verdicts rather than a re-scorer.
Ports assert_ai/integrations/foundry/artifacts.py + its tests from the v1 (`tangym/foundry-exporter-v1`) branch verbatim. Pure I/O, no third-party imports, no dependency on the `foundry` extra — the loader is safe to import from a base install and needed by every downstream v2 module (dataset row builder, pipeline). Public surface: - `AssertRun` — frozen dataclass. Fields: `run_dir`, `suite_dir`, `suite_id`, `run_id`, `taxonomy`, `systematization`, `stratification`, `suite_metadata`, `latest`, `test_set`, `config`, `inference_set`, `scores`, `metrics`, `manifest`, `artifacts_cache`, `inference_config_hash`, `judge_config_hash`, `viewer_files`. Plus the derived `behavior_name`, `behavior_definition`, `behavior_category_count`, `stratification_dimension_count` properties used by v2 evaluator/dataset builders. - `AssertRunError` — raised when one of the three required files is missing (`config.yaml`, `inference_set.jsonl`, `scores.jsonl`). - `load_run(run_dir)` — the entry point. - `viewer_file_names()` — the five viewer read-model filenames the runner emits; used by fixture builders to construct realistic test runs. Contract: list-shaped artifacts (`test_set.jsonl`, `inference_set.jsonl`, `scores.jsonl`) are materialized as tuples of dicts. ASSERT runs fit in memory for export; streaming would add complexity without buying anything. Missing optional files (`metrics.json`, viewer bundle, config hashes) surface as `None` or empty tuples rather than raising, because ASSERT skips those under some run configurations (dry runs, `--skip-judge`, ...). Tests (19) ported unchanged from v1: - Happy path: loads all fields from a realistic run tree. - Missing suite files degrade to `None` (taxonomy, systematization, stratification, metadata, latest, test_set). - Missing optional run files degrade cleanly (metrics, manifest, artifacts.json, both config hashes, viewer bundle). - Missing required run files raise `AssertRunError` with the file name in the message. - Malformed JSON / JSONL raises with the line number. - Rejects a suite root passed as `run_dir` (must be a run child). - `viewer_file_names()` returns the exact five expected filenames. `__init__` now lazily re-exports the four public symbols; base imports still work without the `foundry` extra (`import assert_ai.integrations.foundry as f; f.AssertRun` resolves). Test totals: 1151 pass (was 1132; +19 artifacts tests). 17 skipped, 0 failures.
Adds `assert_ai/integrations/foundry/evaluators.py` — pure factories
that translate ASSERT judge dimensions into
`azure.ai.projects.models.EvaluatorVersion` payloads. Two variants
per dimension, either or both selectable per push:
Code-based (`assert-{dim}`):
- Emits a `def grade(sample, item) -> float` that Foundry runs in its
sandbox per row.
- Body plucks `item["assert_scores"][dim]` and returns it verbatim
(missing / non-float defaults to 0.0 so an errored row surfaces as
a fail rather than raising).
- Metric: continuous [0.0, 1.0], desirable_direction=increase.
- data_schema declares `item.assert_scores.{dim}` as required so
Foundry rejects rows without a pre-computed ASSERT score before
scoring wastes a sandbox call.
- init_parameters is empty (no LLM call, no deployment needed).
Prompt-based (`assert-{dim}-rescore`):
- Emits a prompt template with `{{query}}` / `{{response}}` placeholders
and inlines the ASSERT rubric prose.
- Metric: ordinal 1-5 (5 = clean, 1 = violation), matching the built-in
rubric direction so results align with the code variant's 0-1 scale
after Foundry rescales.
- data_schema requires `query` + `response` on the row.
- init_parameters requires `deployment_name` + `threshold` at
eval-create time.
Public surface:
- `AssertEvaluatorSpec` (dimension_id, variant, evaluator_name,
evaluator_version) — one object per registerable evaluator.
- `EvaluatorMode = Literal["code", "prompt", "both"]`.
- `build_code_evaluator_spec(dim, *, description)` — code-only.
- `build_prompt_evaluator_spec(dim, *, description, rubric_prose)` — prompt-only.
- `build_evaluator_specs_for_run(run, *, mode="both")` — enumerate
scored dimensions from scores.jsonl (alphabetical) and emit specs
per mode. Within a dimension, code first then prompt.
- `evaluator_name_for(dim, *, variant)` — helper for the pipeline
when it needs to build testing_criteria entries before the eval is
actually registered.
- `resolve_rubric_prose(dim, *, inline_rubrics)` — precedence:
1) inline config, 2) hard-coded 1-5 built-in prose for
policy_violation / overrefusal, 3) generic 1-5 fallback for preset
dimensions we don't have text for.
- `EVALUATOR_NAME_PREFIX = "assert-"`, `RESCORE_SUFFIX = "-rescore"`.
- `EvaluatorSpecError` for validation failures.
Client-side dimension-id validation (^[a-z][a-z0-9_]*$) rejects bad
names at spec-construction time with a clear message instead of at
`create_version()` with a masked 400.
Wired into the subpackage's lazy-export map with a `_MISSING_DEPENDENCY_HINT`
pointing at `pip install "assert-ai[foundry]"` — importing
`assert_ai.integrations.foundry` on a base install still works, only
resolving the evaluator symbols triggers the SDK import.
Tests (28): naming (code/prompt/rescore suffix, bad-id rejection,
leading-digit rejection), code spec payload shape (custom + quality
category, grader plucks pre-computed score, continuous 0-1 metric,
data_schema requires assert_scores field, empty init_parameters),
prompt spec payload shape (rescore display name, rubric prose
inlined, {{query}}/{{response}} placeholders, ordinal 1-5 metric,
query+response schema, deployment_name+threshold init_parameters),
rubric prose precedence (inline wins, built-in fallback for both
built-ins, generic fallback for unknown), run-level enumeration
(default mode = both, alphabetical order code-before-prompt, code-only
and prompt-only modes, custom dimensions from scores + config,
empty scores → empty list, unknown mode raises), lazy load via
package root.
Test totals: 1179 pass (was 1151; +28 evaluator tests). No regressions.
Adds `assert_ai/integrations/foundry/dataset.py` — the pure translator
from `AssertRun` to flat JSONL rows for a Foundry dataset asset.
Central design principle: ASSERT stays the source of truth. Every
row carries the pre-computed ASSERT verdict + reason so Foundry's
code-based custom evaluators (from v2.4) can pluck the score off the
row instead of re-judging with a second LLM.
Row shape (flat top-level, NO {"item": ...} wrapper — Foundry wraps
internally when resolving {{item.foo}} on the evaluator side):
{
"query": "<joined user turns>",
"response": "<joined assistant turns>",
"assert_scores": {"policy_violation": 1.0, ...},
"assert_reasons": {"policy_violation": "clean", ...}
}
Score normalization (single source of truth for pass semantics):
- Built-in violation-flag dims (policy_violation, overrefusal):
True (violation) → 0.0, False (clean) → 1.0. Not overridable.
- Custom dims default to violation-flag convention. Override with
`passing_when_true={dim: True}` for rubric-positive dims like
`answer_quality`.
- Non-bool / None / missing → 0.0 (fail). An errored judge should
surface as a fail in Foundry, not silently pass.
Public API:
- `build_dataset_rows(run, *, passing_when_true=None) -> list[dict]`.
Joins inference + scores on test_case_id, extracts messages from
the `combined` (fallback: `target`) view, applies score
normalization. Raises DatasetRowsError on empty inference_set or
all-rows-missing-id.
- `rows_to_jsonl_bytes(rows) -> bytes`. Newline-delimited UTF-8,
ensure_ascii=False (readable non-ASCII on the wire),
sort_keys=True (deterministic serialization for content-addressing).
- `content_hash(payload, length=12) -> str`. SHA-256 hex truncated
to 12 chars (48 bits, collides ~1-in-70M). Used as the dataset
asset version so identical row content ⇒ identical version ⇒ the
v2.6 pipeline can reuse existing datasets instead of creating a
new version per push.
- `DatasetRowsError` for validation failures. Overriding a built-in
dim's pass direction raises loudly — the two built-ins are
contract-fixed at true=violation and flipping them would silently
invert every ASSERT run's pass counts.
Smoke-tested against the on-disk foundry-agent-smoke/smoke-1
fixture: produces 3 rows, deterministic hash `31b6a117d09a`,
end-to-end (build → serialize → hash) reproduces byte-for-byte
across invocations.
Tests (24): row shape (flat contract asserted explicitly), query +
response extraction, multi-turn join with blank-line separator,
test_case_id skip, target-view fallback, score normalization (all
four cases: built-in true/false, custom default, custom override,
non-bool → 0), built-in override raises, reasons pass through
verbatim, missing reason → empty string, row without matching
scores emits with empty maps, empty inference raises, all-id-less
raises, JSONL formatting (newline-delimited, non-ASCII preserved,
sort-keys determinism across differing dict input order), content
hash (stable, default 12-char length, configurable length, changes
with content), end-to-end determinism, lazy load via package root.
Wired into subpackage `_LAZY_EXPORTS`. Base install still imports
cleanly; SDK dep not required for the row builder itself (it's pure
Python — the SDK dep only kicks in when the pipeline actually
uploads).
Test totals: 1203 pass (was 1179; +24 dataset tests). No regressions.
…nt-hash datasets, fail-loud on eval drift)
Adds `assert_ai/integrations/foundry/pipeline.py` — the top-level
orchestrator that composes the loader (v2.3), evaluator spec builder
(v2.4), and row builder (v2.5) against the `azure-ai-projects` SDK
to publish an ASSERT run to a Foundry project.
Push sequence:
1. `load_run(run_dir)`.
2. Build evaluator specs per requested variant
(`mode='code' | 'prompt' | 'both'`, default `both`).
3. GET each `assert-{dim}[/-rescore]` at version "1" — reuse when
present, POST `create_version` only on 404. Never bumps
evaluator versions on re-push; customer can force a bump via the
Foundry UI (delete-and-recreate) or a future `foundry gc`.
4. Build flat JSONL rows + content hash. Hash is the dataset
version so identical row content ⇒ identical version ⇒ existing
dataset asset gets reused.
5. GET the dataset at `assert-{suite}/{content_hash}` — reuse when
present, else upload via SDK's `datasets.upload_file` (which
handles startPendingUpload + SAS PUT + register internally).
6. Page the project's evals listing for one named `ASSERT: {suite}`.
If found and testing_criteria matches, reuse. If found with
different criteria, raise `PushError` with a clear message
telling the user to bump `--eval-name` (Foundry's Update-eval
endpoint only accepts name+metadata — testing_criteria cannot
be patched, verified against Microsoft Learn REST reference).
7. Create the eval (new) or reuse (existing).
8. Always POST a new run — one eval, many runs, so the Foundry UI
renders run-over-run trends across pushes.
Public API:
- `push_run_dir(run_dir, *, project=..., project_client=..., ...)` — CLI entrypoint.
- `push_run(run, *, ...)` — for callers that already loaded the run.
- `PushResult` (frozen): `eval_id`, `run_id`, `evaluator_refs`,
`dataset_ref`, `reused_evaluators`, `reused_dataset`, `reused_eval`.
- `DryRunResult` (frozen): everything the push would have done + no
network calls. Dry-run doesn't require `project` or a client.
- `EvaluatorRef` / `DatasetRef` — refs returned in `PushResult`.
- `PushError` — top-level error.
- Naming helpers: `default_eval_name(run) = "ASSERT: {suite_id}"`,
`default_run_name(run) = "ASSERT run: {run_id}"`,
`default_dataset_name(run) = "assert-{suite_id}"` sanitized to
Foundry's a-z / 0-9 / hyphen / underscore character class.
- `resolve_judge_deployment(run)` — reads `pipeline.judge.model.name`
→ `default_model.name` → `scores.jsonl[0].judge_model`, stripping
any LiteLLM `provider/` prefix so Foundry sees the bare deployment
name.
- `strip_litellm_prefix(model_name)` — helper (exported so the CLI
can echo the resolved name in dry-run output).
Contract details worth flagging:
- `testing_criteria`: code-variant entries use
`data_mapping={"item.assert_scores": "{{item.assert_scores}}"}`
and empty `initialization_parameters`. Prompt-variant entries use
`{"query": "{{item.query}}", "response": "{{item.response}}"}`
with `deployment_name` + `threshold` (default 3.0, mid-scale of the
ordinal 1-5 rubric).
- `data_source_config` (custom, no sample schema): declares row
fields `query`, `response`, `assert_scores` (with all dimension
ids as required keys), `assert_reasons`.
- Eval metadata: `assert.source`, `assert.suite_id`,
`assert.behavior_name`, `assert.category_count`,
`assert.row_count` (all bounded to Foundry's 512-char values).
- Run metadata: `assert.run_id`, `assert.run_dir`,
`assert.dataset_version` (the content hash),
`assert.inference_config_hash`, `assert.judge_config_hash` — enough
to correlate a Foundry run to the ASSERT run on disk.
- Endpoint resolution (`_endpoint_from_project`) accepts three
forms: the full endpoint URL, `{account}/{project}` shorthand,
or a Cognitive Services project ARM id.
Injection surface (all injectable via `project_client=...` for tests):
- Client is `AIProjectClient` — real or fake. Fake surfaces exercised
by tests: `client.beta.evaluators` (get_version/create_version),
`client.datasets` (get/upload_file), `client.get_openai_client()`
→ `.evals` (create/list/update) + `.evals.runs` (create).
- `_resource_not_found_types()` returns `(KeyError,
azure.core.exceptions.ResourceNotFoundError, openai.NotFoundError)`
— mocks that raise plain `KeyError` still exercise the reuse path.
Smoke-tested against the on-disk foundry-agent-smoke/smoke-1 fixture
via dry-run: 3 rows, dataset content-hash version `234021feab5f`,
judge deployment `gpt-5.4-mini`, 6 evaluator specs (3 dims × 2
variants).
Tests (40): naming helpers (suite/run/dataset prefixes,
character-class sanitization), LiteLLM prefix strip (5 params),
deployment resolution precedence (pipeline.judge > default_model >
scores.jsonl > ""), dry-run (returns DryRunResult, both variants by
default, code-only mode, no network calls, passing_when_true
threading), orchestration (eval + run ids returned, evaluator
registration when missing, evaluator reuse when present, dataset
upload when new, dataset reuse on content-hash match, eval reuse on
name match, fail-loud on testing_criteria drift, run metadata
carries ids/hashes, eval metadata carries suite context, data_source
wires dataset asset id, prompt variant init parameters carry
deployment/threshold, code variant init parameters empty), error
paths (missing client + project, prompt mode without judge model,
code mode without judge model ok, no scored dimensions),
`push_run_dir` disk-loading wrapper via real fixture, endpoint
resolution (URL passthrough, shorthand, ARM id, junk rejection),
lazy load via package root.
Test totals: 1243 pass (was 1203; +40 pipeline tests). No regressions.
…,both}
Wires the pipeline (v2.6) to `assert-ai foundry push`. Loads a run
directory, orchestrates the three-object flow, and prints a compact
summary.
Options:
- `--project` (required): Foundry project as an endpoint URL,
'{account}/{project}' shorthand, or Cognitive Services project ARM id.
- `--evaluator-mode {code,prompt,both}` (default `both`): which
evaluator variant(s) to register. Ships both by default so the demo
can compare ASSERT's pre-computed verdict against Foundry's LLM re-
score side-by-side.
- `--eval-name`, `--run-name`, `--dataset-name`: naming overrides.
Dataset *version* is always the content hash (not overridable) so
identical row content deterministically reuses the same version.
- `--passing-when-true DIM=TRUE|FALSE` (repeatable): pass-direction
override for rubric-positive custom dimensions like `answer_quality`.
Built-in dimensions (`policy_violation`, `overrefusal`) are hard-
locked to the violation-flag convention and cannot be overridden;
the row builder raises loudly if you try.
- `--judge-threshold` (default 3.0): pass threshold for prompt-variant
evaluators (ordinal 1-5 scale).
- `--dry-run`: print what the exporter would send without any network
calls or credential requirement.
Output:
- Dry-run prints eval/run/dataset names, dataset row count + content-
hash version, resolved judge deployment, passing-when-true overrides,
and the evaluator spec list (name + variant per line).
- Real push prints eval id, run id, dataset asset id, and a per-
evaluator line (name, version, variant, `(reused)` marker when the
version was already registered). The eval and dataset lines also
carry `(reused)` markers when the pipeline hit its idempotency
short-circuit.
Errors are caught at the CLI boundary (`PushError`,
`DatasetRowsError`, `EvaluatorSpecError`) and rendered as single-
line stderr with a non-zero exit code, not raw tracebacks.
Small helper `_parse_passing_when_true` accepts `dim=true|false|1|0|yes|no`
case-insensitively.
Tests (19): parser accepts all common forms + rejects bad ones,
foundry push --help renders, dry-run prints summary lines,
real-push prints ids + reuse markers, `PushError` propagates to
nonzero exit with the message text, `--evaluator-mode bogus` rejected
by Click choice, missing `--project` rejected, repeatable
`--passing-when-true` flags accumulate into the dict passed through
to `push_run_dir`.
Test totals: 1262 pass (was 1243; +19 CLI tests).
…rift
Two E2E-surfaced fixes discovered during real Foundry push testing:
1. Flatten the code evaluator's data_schema so `assert_scores`
sits at the root rather than under an `item` wrapper. The
original nested form forced `data_mapping={"item": "{{item}}"}`
which fails Foundry's regex for testing_criteria data_mapping
values.
2. Delete evaluator versions with a mismatched `definition.type`
before re-registering. An earlier v1 exporter registered the
same evaluator names with `type: rubric`; silently reusing
them would fail eval-create against a stale schema.
Verified end-to-end against a real Foundry project:
- Code-only push: 3/3 completed.
- Both-mode push: 3/3 across 6 testing criteria (3 code + 3
prompt), code and prompt variants agreeing 100% on the smoke
fixture.
Full suite: 1264 pass (was 1263, +1 drift test).
Documents the SDK-based ASSERT -> Foundry exporter shipped in v2. - docs/integrations/foundry.md — on-the-wire schema contract: three-object flow (evaluators + dataset + eval/run), full SDK call sequence, code + prompt variant specs, dataset row shape, testing criteria data_mapping regex, evaluator drift detection, judge deployment resolution + LiteLLM prefix stripping. Verified against a real Foundry push on 2026-07-15 (both variants agreed 100% on the smoke fixture). - docs/guides/publishing-to-foundry.md — customer runbook: install, auth, project ID forms, --dry-run walkthrough, first + repeat pushes, watching a run, CLI reference including --evaluator-mode and --passing-when-true, verification in the Foundry UI, troubleshooting for EvaluatorNotFound / stale data-mapping / drift / RBAC / judge deployment errors. No behavior change.
…on / --no-color
The foundry push subcommand was writing hand-padded lines via click.echo,
out of step with the rest of the CLI which uses rich.Table via the shared
_console() helper (see results_status, presets_list, etc).
- Replace click.echo blocks with two rich tables per output path:
- A two-column summary (Field / Value)
- An evaluator list (Name / [Version] / Variant / [Status])
- Add --json flag matching the results_status pattern (structured
payload with dry_run flag + eval/run/dataset ids + evaluators).
- Add --no-color flag (same helper as the rest of the CLI).
- Reuse markers move from '(reused)' suffix noise to a dedicated Status
column ('reused' / 'new') plus a count in the table title. Dataset and
eval reuse markers stay inline in the summary because they're one-line
fields.
Test updates:
- Relax three assertions that pinned exact space padding — semantic
checks (label + value present, reuse marker visible) unchanged.
- Add two JSON-mode tests (dry-run + real push) covering the full
serialization shape.
Full suite: 1265 pass (was 1263, +2 JSON tests).
Previously the pipeline only detected drift on 'definition.type' (code vs prompt vs the legacy v1 rubric shape). Same-type body changes were silently reused — e.g. a rubric edit in the customer's ASSERT config, or an SDK upgrade that changed the code grader fallback, would leave stale grader logic scoring new eval runs. Close the gap with a 12-char SHA-256 fingerprint over the semantic definition fields: - type - code_text / prompt_text - data_schema - init_parameters - metrics Excludes UI-only fields (description, display_name) on purpose — editing rubric prose in ASSERT config shouldn't force a delete+recreate cycle when the underlying scoring behavior is unchanged. (Rubric edits that mutate the prompt-variant's prompt_text still flip the fingerprint because they change the actual grader body, which is the correct semantics.) The fingerprint uses _to_plain() to normalize SDK model instances and plain dicts to the same JSON-serializable form before hashing, so it works uniformly against real azure-ai-projects models (returned by get_version) and against test fakes. Tests updated: - test_push_reuses_existing_evaluators: reseed with full specs (build_evaluator_specs_for_run output) so fingerprints match byte-identical. - test_push_replaces_stale_rubric_evaluator_with_code_variant: unchanged — sparse 'rubric' stub fingerprints differently from a full code spec, so the drift path still fires. - test_push_replaces_evaluator_when_code_text_drifts (NEW): seeds a real code-variant spec with a mutated code_text; asserts delete_version + create_version fire. - test_push_replaces_evaluator_when_prompt_text_drifts (NEW): same for the prompt variant. - test_push_ignores_description_only_drift (NEW): description-only change must NOT trigger a delete+recreate. Docs updated in docs/integrations/foundry.md (schema mapping reference) and docs/guides/publishing-to-foundry.md (troubleshooting section). Full suite: 1268 pass (was 1265, +3 drift tests).
Adds evaluator_fingerprints (name -> 12-char SHA-256) to
DryRunResult and surfaces them in the CLI's rich table + --json
output. Uses the same _definition_fingerprint the drift check
compares against Foundry's stored evaluator, so a caller can
verify a config-level change (rubric prose edit, evaluator-mode
switch) would trigger a delete+recreate WITHOUT hitting Foundry:
# Before edit
assert-ai foundry push RUN --project P --dry-run --json > before.json
# Edit pipeline.judge.dimensions.{dim}.description in config.yaml
# After edit
assert-ai foundry push RUN --project P --dry-run --json > after.json
# Diff fingerprints — flipped entries are what would drift.
diff <(jq '.evaluators' before.json) <(jq '.evaluators' after.json)
Two new pipeline tests verify:
- Every dry-run result exposes a 12-char fingerprint per spec.
- Editing rubric prose in config flips the prompt-variant
fingerprint (customer-visible drift signal) but leaves the
code-variant fingerprint stable (code_text is hard-coded).
CLI dry-run tests updated: _stub_dry_run adds the new field.
Full suite: 1270 pass (was 1268, +2 fingerprint tests).
…+recreate Foundry-native pattern: name+version is immutable, so drift means register a new version, not overwrite v1. This preserves the audit trail: - Historical eval runs pinned to v1's testing_criteria continue to render their original grader body in the Foundry UI. - No delete_version calls anywhere on the happy path — old evaluator versions are the customer's asset, not ours to remove. - Version numbers walk forward: v1, v2, v3, ... — never a delete event that could race with a mid-flight eval run. Drift detection stays fingerprint-based: - list_versions(name) enumerates every existing version. - If any prior version's 12-char fingerprint matches the current spec, that version is reused; EvaluatorRef.evaluator_version carries the matched version string. - Otherwise the spec is registered at max(existing) + 1. Because Foundry's testing_criteria pin a specific evaluator_version and can't be patched, adopting a new evaluator version in scored runs typically means creating a new eval. The pipeline's existing eval-drift check already forces --eval-name bumps on testing_criteria change, so the composition is natural. Tests renamed replace_stale_* -> bumps_version_* and rewritten to assert that: - v1 stays intact (deleted list stays empty). - New spec is registered at v2. - EvaluatorRef carries the new version string. New test test_push_reuses_matching_prior_version_even_when_not_latest guarantees the pipeline picks the fingerprint-matching version out of the middle of the version list rather than blindly registering a new one — matters when a customer experiments with a v2 they later rolled back. _FakeEvaluators now implements list_versions and honors payload.version on create_version so version bumps flow through the fake correctly. Docs updated to describe the new call sequence (list_versions + create_version at max+1) and drop delete_version from the Idempotency table and troubleshooting. Full suite: 1271 pass (was 1270; one previously-added test drops out because it duplicated a scenario, one new scenario added).
…gerprint Every push was registering a new evaluator version even when the grader body was unchanged, because Foundry echoes evaluator definitions back with two silent normalizations that our fingerprint hashed differently: 1. Integer metric bounds -> float. The ordinal prompt-variant metric is built with min_value=1 / max_value=5 (ints), but Foundry returns 1.0 / 5.0 on the wire. sha256(..."1"...) != sha256(..."1.0"...) so the fingerprints diverged and the pipeline registered a new version on every push. 2. Absent opposite-variant text body: our code-variant spec leaves prompt_text as None; Foundry normalizes null -> "". Same asymmetry for code_text on prompt evaluators. Verified against a real Foundry project: six evaluators previously bumped to a new version on every push, now all six correctly reuse the matching existing version. The one evaluator whose rubric prose was actually edited between sessions still fingerprints differently from post-edit versions, so drift detection still fires for real changes. Fixes: - _to_plain: coerce int -> float (except bool). Uniform numeric representation on both sides. - _definition_fingerprint: coerce None text bodies to "" via the `or ""` idiom before hashing. Regression tests: - test_fingerprint_matches_across_int_float_metric_bounds - test_fingerprint_matches_when_opposite_body_field_is_empty_string Full suite: 1273 pass (was 1271, +2 regression tests).
…ld comments Upper bound the SDK dependency to reduce the blast radius of a future breaking change on `.beta.evaluators` — the pipeline uses `get_version` / `create_version` / `delete_version` / `list_versions` off that surface, all still marked beta. Widen the cap when a compatible newer major ships and re-verify the drift detection round-trip. Drop two stale scaffold comments that described the state of the subpackage at the initial extra-scaffold commit but no longer match: - `assert_ai.integrations.foundry.__init__.py`: "Currently empty" - `foundry` CLI group: "Currently populated by follow-up commits"
The dataset row builder emits one row per inference entry — even
entries whose judge errored, which carry an empty `assert_scores`
map (see `build_dataset_rows`). That intent ("still show the
conversation in the Foundry UI") contradicted two schemas that
declared every dimension as required:
- `_build_data_source_config` in `pipeline.py` marked every
judged dimension as `required` inside `assert_scores`, and
`assert_scores` itself as required on the row.
- `_code_data_schema` in `evaluators.py` marked
`assert_scores` and the dimension both as required on the
evaluator's own data_schema.
Foundry would have rejected any partial-scored dataset at data-
source validation, or the evaluator would have refused to score
those rows even if the dataset validated. The smoke fixtures used
during E2E are fully scored so the bug never surfaced.
Fix: keep the properties declared (so the Foundry UI knows what
column to render) but drop them from every `required` list. The
code grader already defaults missing values to 0.0, which
surfaces as a fail — the intended semantic for a row whose judge
errored.
Because `_code_data_schema` participates in the evaluator
fingerprint, existing evaluators registered by prior pushes will
show as drift and the pipeline will register a new version. This
is the correct behavior — the schema really did change.
New regression test:
- test_push_data_source_config_tolerates_missing_scores
Existing schema-shape test renamed to reflect the tolerance:
- test_code_spec_data_schema_declares_assert_scores_as_optional
Full suite: 1294 pass.
…TF-8-safe bounded_record Three small defensive fixes in the pipeline's SDK-shape helpers, all no-op on the happy path: 1. `_iter_paged` retried without kwargs on any `TypeError`, which silently swallowed unrelated TypeErrors raised inside the pager iteration itself. Narrow to the "unexpected keyword" case only; let other TypeErrors surface unchanged so a genuine bug isn't masked. 2. `_dict_get` used `hasattr(obj, key)` + `getattr` then fell back to `__getitem__`. That treats bound methods and other callables as legitimate field values: `obj.name` on an SDK model that exposes both a `name` field and a `name()` method returns the bound method, not the field. None of the current fingerprint fields hit this because our specs don't shadow Python methods, but it's brittle for future maintainers. Guard with `not callable(value)` and fall through to `__getitem__` on match. 3. `_bounded_record` truncated to 511 characters + a 1-char ellipsis, but the ellipsis (`\u2026`) is 3 UTF-8 bytes — so the final size was 514 bytes for ASCII inputs and more for multi-byte. If Foundry enforces its metadata cap in bytes, the truncation itself overshot. Fix: encode-then-slice on a UTF-8 code-point boundary, sizing to at most 509 bytes plus the 3-byte ellipsis for a total of exactly 512 bytes. New regression test: - test_bounded_record_truncates_on_utf8_byte_boundary — asserts the 512-byte cap holds for both ASCII and 3-byte-per-char inputs. Full suite: 1294 pass.
…output
Adds a `prompt_variant_calls` field to `DryRunResult` — the
number of LLM calls Foundry will make when scoring a real push
with prompt-variant evaluators registered. Rendered in the CLI's
dry-run table and included in `--json` output so a customer can
see the cost of an `--evaluator-mode both` push before spending
money on it:
LLM calls (prompt variant) ~45 (1 per row × prompt evaluator)
Zero-cost pushes (`--evaluator-mode code`) render explicitly as
`0 (code-only mode)` so the surface is the same either way.
Motivation: the default mode is `both` because that's what makes
the demo interesting (ASSERT verdict side-by-side with Foundry's
LLM re-score). But it's a stochastic default that spends money
per push; the docs already recommend `--evaluator-mode code` for
production sharing. Making the cost visible in dry-run output
turns the `both` default from a footgun into a transparent
opt-out.
Test coverage:
- test_dry_run_exposes_prompt_variant_call_estimate — asserts the
count is `(prompt evaluators) × (rows)` and drops to 0 in
code-only mode.
- Existing CLI stub updated to include the new field.
Full suite: 1294 pass.
…pped Two additions to the schema-mapping reference that customers seemed to hit in review: 1. The evaluator drift fingerprint deliberately excludes `description` and `display_name` — that's an intentional design choice, but it creates an asymmetry between the two variants when a customer edits their config's rubric prose: the prompt variant's `prompt_text` inlines the rubric so its fingerprint flips, but the code variant's `code_text` is hard-coded so its fingerprint is stable. Spell that out with a concrete example and point at the dry-run `--json` fingerprint diff as the way to preview which evaluators would churn. 2. `build_dataset_rows` filters conversation events to user + assistant only, so tool calls / developer / system messages never reach Foundry. For agent runs judged on tool-use behavior the prompt variant re-judges from a systematically less informative transcript than ASSERT saw — expect variant divergence that is methodological, not a genuine disagreement about the same evidence. Full tool-event rendering is a follow-up. Docs-only. No behavior change.
The on-the-wire schema reference in docs/integrations/foundry.md still described the pre-e96da568 shape, where every ASSERT dimension was marked required at both the evaluator's data_schema and the eval's data_source_config level. Commit e96da56 relaxed those lists to tolerate un-scored rows (empty assert_scores maps from judge-errored inference entries) but the doc wasn't updated to match. Sync both blocks to what the code actually sends today: - Code evaluator data_schema: assert_scores.properties.required goes from ["{dim_id}"] to []; outer required goes from ["assert_scores"] to []. - Eval data_source_config: assert_scores.properties.required goes from ["{dim_id}", ...] to []; outer required goes from ["query", "response", "assert_scores"] to ["query", "response"]. Add a one-paragraph explanation next to the code-evaluator schema noting that the grader defaults missing values to 0.0 (fail), which is why the field can stay optional. The equivalent context for the data_source_config was already in the code comment on _build_data_source_config. Docs-only. No behavior change.
The outer catch around _iter_paged(list_versions, name=...) was a bare 'except Exception' with the intent 'some fakes raise TypeError on unexpected kwargs; fall through to enumerate'. That was correct for the fake-signature case but too wide for the production path: a transient SDK exception (network blip, throttling, auth expiry, service 500) would silently fall through to the 1..999 get_version fan-out, whose first call hits the _resource_not_found_types() branch and returns []. The pipeline then thinks the evaluator name is fresh, POSTs a new version on top of the existing catalog, and the customer's next push sees a duplicate. Narrow to 'except TypeError' — the only exception the fallback path actually needs to tolerate. _resource_not_found_types() continues to catch KeyError / ResourceNotFoundError / NotFoundError separately. Everything else (SDK errors, network, auth) now propagates, so the caller sees a real failure instead of a silent duplicate registration. Regression test test_push_propagates_non_typeerror_from_list_versions seeds a fake evaluators op whose list_versions raises a plain RuntimeError. The test asserts get_version and create_version are never called and the RuntimeError bubbles up unmodified. Full suite (foundry only): 148 pass (was 147, +1 regression test).
The pipeline's DryRunResult path is documented as making zero network calls and explicitly does not need a project client — but the click layer marked --project as required=True, which blocked dry-run in a fresh checkout with no exported project id or az login. That contradicts the runbook's pitch of dry-run as the fastest way to catch config mistakes before touching a real project. Change --project to default=None at the click layer and validate in-body: if --dry-run is set, project is optional and threaded through to push_run_dir as None (the pipeline's dry-run branch never touches the client). Otherwise, emit a clear one-line error via _error and exit nonzero. Test updates: - test_push_requires_project renamed to test_push_requires_project_for_real_push; still asserts that a real push without --project is rejected, but drops --dry-run from the invocation so it exercises the real-push branch. - New test_push_dry_run_works_without_project: invokes push with only --dry-run (no --project) and asserts exit_code == 0, the dry-run header renders, and the stubbed push_run_dir received project=None + dry_run=True. Docs updated in docs/guides/publishing-to-foundry.md: the Dry-run first section shows the no-project invocation as the default example, and the CLI reference marks --project as [required for real push] rather than a blanket [required]. Full suite (foundry only): 149 pass (was 148, +1 regression test, -1 renamed but unchanged in count).
tangym
requested review from
AaronAspinwall123,
changliu2,
jakepresent and
minthigpen
as code owners
July 16, 2026 21:40
CI's Tier 1 job installs the built wheel plus pytest only, with no
optional extras. Three of the six foundry test modules import from
assert_ai.integrations.foundry.evaluators / .pipeline at module
scope, which unconditionally imports azure.ai.projects.models — so
pytest failed at collection with
ImportError while importing test module 'tests/test_foundry_cli.py'
...
E ModuleNotFoundError: No module named 'azure'
and the -x flag aborted the whole run before any test executed.
Add pytest.importorskip("azure.ai.projects") at the top of the
three affected files, before the SDK-backed imports. Matches the
existing pattern in tests/test_acs_generate.py etc. The pure
subpackage tests (scaffold, artifacts, dataset) keep running on the
base install because their imports don't reach azure.*.
Verified locally in .venv:
- Without the extra installed: 46 passed, 3 skipped in 0.39s
(the 3 SDK-backed modules skip cleanly, the pure ones run).
- With the extra installed: 149 passed in 1.50s. No regression.
Two runtime-output chars in foundry_push aren't reachable from a plain keyboard, which contradicts the PR-body cleanup that dropped em-dashes and arrows from user-facing text: - The Dry-run banner used an em-dash separator (\u2014). - The 'LLM calls (prompt variant)' hint used a multiplication sign (\u00d7) in '1 per row \u00d7 prompt evaluator'. Replace with '--' and 'x' respectively. Pure string edit; no shape change. Existing CLI tests check for the 'Dry-run' substring only and stay green.
The three fenced 'Expected output' blocks in the customer guide described the pre-refactor CLI shape and had drifted from what the code actually prints today: - Dry-run summary was missing the 'LLM calls (prompt variant)' row added in fb2e1dd; label was 'Passing-when-true overrides:' where the code renders 'Passing-when-true'; the evaluators block was a bulleted list where the code now renders a rich Table with 'Name / Variant / Fingerprint' columns (b11e151). - Real-push and repeat-push blocks used 'Published eval / Published run / Dataset asset' labels where the code was refactored in 89d416a to render 'Eval / Run / Dataset' with a Status column on the evaluators table ('new' / 'reused') instead of inline '(reused)' suffixes. Regenerate the dry-run block byte-accurate from a real invocation against artifacts/results/foundry-agent-smoke/smoke-1; regenerate the real-push blocks deterministically from the render code in assert_ai/cli.py:1673..1710. Placeholders (<suite>, <hash>, <12-hex>, <32-hex>, ...) preserve the existing convention. Add a short paragraph explaining the new 'LLM calls (prompt variant)' row and how it renders in --evaluator-mode code (currently '0 (code-only mode)'). Docs-only. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships a new
assert-ai[foundry]extra andassert-ai foundry pushCLI thatpublishes a completed ASSERT run to an Azure AI Foundry project as a first-class
eval. Uses the first-party
azure-ai-projectsSDK end-to-end.Scoring model is up for discussion. This PR registers each ASSERT dimension
as a Foundry custom evaluator in one of two variants, selectable per push via
--evaluator-mode {code,prompt,both}(defaultboth):assert-{dim}): a Foundry sandbox function plucks thepre-computed ASSERT verdict off the row and echoes it. Zero incremental LLM
cost; Foundry is a pure viewer of ASSERT's judgment.
assert-{dim}-rescore): Foundry's own LLM re-judges eachrow against the ASSERT rubric prose. One LLM call per (row x dim); Foundry
produces an independent second opinion.
Shipping both side-by-side lets us compare them in real projects and gather
feedback. Neither is meant to be the permanent story. The longer-term direction
is a more integrated evaluation (for example, mapping ASSERT dimensions to
Foundry's built-in evaluators where they exist, or Foundry-native custom
evaluators that consume ASSERT trace metadata directly). This PR is deliberately
scoped to "good enough to demo and get feedback"; happy to iterate on the model
based on review.
Design principle (as currently implemented)
ASSERT stays the source of truth; Foundry becomes a viewer.
Every uploaded row carries ASSERT's per-dimension verdict and justification.
That choice is what makes the code variant a one-line "echo the score" grader.
If we land on a more integrated evaluation later, the on-the-wire row shape and
the dataset content-hash keying should mostly carry over; only the evaluator
specs would change.
Changes
pyproject.toml[foundry]extra:azure-ai-projects>=2.2.0,<3.0.0(upper-bounded,.beta.evaluatorsis beta),azure-identity>=1.19.0.assert_ai/integrations/foundry/__init__.pyimport assert_ai.integrations.foundrywithout the extra; symbols only resolve when accessed, with a clearpip install "assert-ai[foundry]"hint on missing deps.assert_ai/integrations/foundry/artifacts.pyAssertRunfrozen dataclass andload_run(). Pure I/O, no SDK dep. Requiresconfig.yaml,inference_set.jsonl,scores.jsonl; optional artifacts (metrics.json, viewer bundle, config hashes) degrade toNone.assert_ai/integrations/foundry/evaluators.pyassert-{dim}) and prompt (assert-{dim}-rescore)EvaluatorVersionpayloads. Client-side dim-id validation, built-in prose forpolicy_violationandoverrefusal, config-inline rubric extraction, generic 1-5 fallback.assert_ai/integrations/foundry/dataset.pybuild_dataset_rows()produces flat JSONL rows (query,response,assert_scores,assert_reasons). Centralized score normalization: built-ins hard-locked to violation-flag (raises if you try to override), custom dims default to violation-flag, non-bool becomes 0.0 (fail-closed). Content-hashed with SHA-256 for dataset versioning.assert_ai/integrations/foundry/pipeline.pytesting_criteriadrift since Foundry can't patch criteria), create eval run. Metadata bounded to Foundry's 16-key / 512-UTF-8-byte cap. Endpoint resolution accepts URL,account/projectshorthand, or ARM id. Fingerprint-based evaluator drift withint <-> floatandNone <-> ""canonicalization to match Foundry's wire normalization; version bumps on drift (never delete) so historical runs stay reproducible.assert_ai/cli.pyassert-ai foundry push RUN_DIRsubcommand with--project,--evaluator-mode,--eval-name,--run-name,--dataset-name,--passing-when-true(repeatable),--judge-threshold,--dry-run,--json,--no-color. Rich tables for humans, structured JSON for CI.--dry-runneeds no--projectand makes zero network calls, the fastest way to sanity-check config in a fresh checkout.docs/integrations/foundry.mddata_mappingregex, drift detection, deployment-name resolution.docs/guides/publishing-to-foundry.mdEvaluatorNotFound, stale data-mapping, testing_criteria drift, judge deployment errors, RBAC).CHANGELOG.md[Unreleased] > Addedentry for the preview.tests/test_foundry_*.py(6 files)Key correctness properties preserved
0.0in therow and in the code grader body. Errored judge rows surface as fails, not
silent passes.
testing_criteriaraises
PushErrorwith--eval-name X-v2remediation. Foundry cannot patchtesting_criteria, so silent reuse would ship an inconsistent eval.
max(version) + 1;historical eval runs pinned to older versions keep rendering their original
grader body in the Foundry UI.
description,display_name): purerubric-prose edits don't churn versions. Prose changes that mutate
prompt_textstill bump (the actual grader body changed).int <-> floatandNone <-> ""canonicalization: matches Foundry'ssilent wire normalization so we don't false-positive drift on every push
(verified against a real project: six evaluators previously bumped on every
push, all six now correctly reuse).
_bounded_recordtruncates on acode-point boundary and caps at exactly 512 bytes (ellipsis included).
Testing
pytest tests/test_foundry_*.py -qprints149 passed in 1.53s.Known limitations (called out in docs)
build_dataset_rowsfilters conversation events touser + assistantonly.Tool, developer, and system events are dropped. For agent runs judged on
tool-use behavior the prompt variant sees a strictly less informative
transcript than ASSERT did. Full tool-event rendering is a follow-up.
(
code_textis hard-coded); the prompt-variant fingerprint flips (rubric isinlined in
prompt_text). Documented asymmetry, with the dry-run--jsonfingerprint diff as the way to preview it.
Checklist
pytest tests/test_foundry_*.py).<sub>,<rg>,foo/bar,acct/proj).CHANGELOG.mdentry added under[Unreleased] > Added.