Version: 1.0.0-rc.5 | License: Apache 2.0
Upstream status (each item live-checked against GitHub on 2026-09-27; evidence and full history in spec/ADOPTION.md). Every merged upstream change listed here started with the EvalPort maintainer, either as the PR author or as the person who filed the originating issue. None is an independent adoption by the upstream project. Listed in Inspect AI's community extensions (PR #4797, merged 2026-08-11): a six-line entry in docs/extensions/extensions.yml, currently filed under "Frameworks". It is a docs listing, not a code integration. Merged upstream, authored by the EvalPort maintainer: an adapter in IBM/ares under experimental-plugins/ares-openeval-adapter/ (PR #583, merged 2026-09-04), and an optional export script, scripts/export_openeval.py, in ClawBench (PR #336, merged 2026-09-08). Merged upstream, generated by the project's bot: PraisonAI's automated triage agent added to_evalport()/from_evalport()/report_to_evalport() to praisonaiagents.eval (PR #4667, merged 2026-09-02). It did this in response to issue #4275, which the EvalPort maintainer filed. That module uses its own field names (cases, case_id, evalport_version), so its output does not currently pass validate_suite()/validate_result_set(). Open, not merged: a TruLens connector, PR #2757, authored by the EvalPort maintainer. It needs a rebase, a CLA signature, and two failing checks fixed before it can merge. Until then, trulens-openeval-adapter ships the same conversion as a standalone package in this repo. This repo: 71 standalone adapter packages live in adapters/, each with its own test suite run in CI (68 written by the EvalPort maintainer and 3 by external contributors, see CONTRIBUTORS.md; 70 are Python packages and one, EvalPort.AgentEval, is .NET). These are converters maintained here, not integrations adopted by the upstream frameworks. They include Agenta (built at the explicit request of the Agenta-AI maintainer, Agenta-AI/agenta#6222), Parea AI (a first-time external contributor's PR #19, merged after review and an independent re-run of its test suite), Literal AI and Humanloop (both from external contributors, see CONTRIBUTORS.md), Hugging Face evaluate, EleutherAI lm-evaluation-harness, Hugging Face lighteval (the library behind the HF Open LLM Leaderboard), OpenCompass, FinanceBench (a real benchmark dataset, not a live SDK), Athina, DeepEval, and Galileo. Some were shaped by upstream maintainer feedback: Journeyman (codechu/journeyman's maintainer walked the mapping through two rounds of corrections before signing off, journeyman#1), TB-Science (scope agreed with a harbor-framework maintainer in discussion #1600; the upstream PR #1652 is still open with no approving review), ARES (IBM's red-teaming framework, built after a maintainer pointed to experimental-plugins/ in IBM/ares#563), agent-skills-eval (built at the maintainer's explicit direction, darkrishabh/agent-skills-eval#34, including a caution against mislabeling deterministic tool-assertion rows as LLM-judge results that shaped how the adapter classifies them), humanbound (built at the maintainer's explicit invitation, humanbound/humanbound#131), FrontierOR (a prototype the FrontierOR maintainer asked for in Minw913/FrontierOR#5 as a basis for discussion; not adopted or integrated upstream), QWED (design reviewed and approved by the QWED maintainer in QWED-AI/qwed-verification#325; the implementation itself has not yet been reviewed by QWED, and it is not adopted upstream), and FlashRAG (built after the FlashRAG maintainer declined to host it in-repo and invited an independent adapter instead, RUC-NLPIR/FlashRAG#246; not adopted or integrated upstream). See Framework Adapters below.
EvalPort is an open specification for portable LLM evaluation test cases, graders, suites, and results. It enables evaluation datasets to be shared across frameworks (DeepEval, Promptfoo, Ragas, Inspect AI, LangSmith, Braintrust, OpenAI Evals, MLflow) without loss of semantic fidelity.
examples/interop/ has three offline, CI-tested scripts that run the real DeepEval, DSPy and Haystack packages:
- Dataset round trip: benchmark suites to
LLMTestCase/dspy.Exampleand back, with a field-level diff. - Results round trip: DeepEval
evaluate()to ResultSet to Haystack and back; all per-case scores agree. - Three frameworks, one table: the same outputs scored by three frameworks and joined with
ResultSet.group.
Each script prints what survives and what doesn't; the README lists every lossy field.
# Install CLI
npm install -g evalport-cli
# Create an eval suite
evalport init my-eval-suite
# Validate
evalport validate my-eval-suite.json
# Convert from Promptfoo
evalport convert promptfoo openeval config.json output.json
# Run a suite against a real provider — always dry-run first to see estimated cost
evalport run examples/basic-suite.json --provider openai --model gpt-4o-mini --dry-run
evalport run examples/basic-suite.json --provider openai --model gpt-4o-mini --output results.json
# Python SDK
pip install evalport-sdkfrom openeval.validate import validate_suite
result = validate_suite({
"version": "1.0.0",
"id": "my_suite",
"graders": [{"id": "gr1", "type": "exact_match"}],
"test_cases": [{"id": "tc1", "input": "Hello", "expected_output": "Hi", "graders": ["gr1"]}]
})
print(result.valid) # TrueIn CI / pre-commit: add - uses: adhabnr-ux/evalport@main to a workflow to validate every **/*.evalport.json and annotate invalid files on the PR,
or add the evalport-validate hook from this repo to .pre-commit-config.yaml. Both run the evalport-validate CLI from the Python SDK.
See docs/github-action.md.
EvalPort is pre-1.0 and actively shaped by outside contributors — the fastest ways in, from least to most involved:
- Have an opinion on an open design question?
spec/SPEC.mdtracks what's genuinely unresolved. The first five questions raised on that list — suite/result signing, a formal conformance test suite, resuming interrupted runs, whetherllm_judgeinjection mitigations should be mandatory, and (most recently, Discussion #22) howResultSetshould represent repeated-attempt evals (num_repetitions, epochs) — have gone through this exact process and landed as shipped spec changes; the table shows how each one resolved. Five RFCs are open for comment right now;docs/rfcs/README.mdindexes every RFC so far, open and landed, and says what makes a useful comment. No prior EvalPort contribution is required, just a considered opinion:- Discussion #47:
Result.constraint_violations, hard constraints that fail a result without dropping it from denominators. Waiting on a real AOBench fixture before it closes. - Discussion #49: FAILED vs. UNVERIFIED (
Result.verdict). Comment period ends 2026-10-11. - Discussion #67: rate-based
GraderResult.trials/successes. Comment period ends 2026-10-11. - Discussion #108: unknown fields, strict validation with an opt-in for lenient consumption. Comment period ends 2026-10-13.
- Discussion #118: judge identity, declared vs. observed (
metadata.openeval.judge). Comment period ends 2026-10-17.
- Discussion #47:
- Want to ship a framework adapter? Issue #6 is the map.
adapters/autogen-openeval-adapteris the reference shape to copy:to_openeval(),from_openeval(), tests against the real validator, a README. - Want to propose a spec change? Open a Discussion titled
[Spec Change] <what and why>— the full process (comment period, what "consensus" means, when the spec lead's sign-off is required) is inspec/SPEC.md's Governance section. - How people actually become collaborators: ship something real and tested, engage substantively, get invited — see
CONTRIBUTORS.mdfor who's done that so far. It's not gatekept; a merged, tested PR is what clears the bar.
Full details in the Contributing Guide.
openeval/
├── spec/
│ ├── SPEC.md # Full specification
│ ├── ADOPTION.md # Adoption strategy
│ ├── CRITIQUE.md # Self-critique and hostile review
│ ├── schemas/ # JSON Schemas (4 files)
│ ├── examples/ # Example suites and conversions
├── sdk/
│ ├── typescript/ # evalport-sdk (npm)
│ └── python/ # openeval (PyPI)
├── cli/ # evalport-cli
├── api/ # Example REST API server
├── docs/
│ ├── getting-started/ # 5-minute quickstart
│ ├── grader-reference/ # All 11 grader types
│ ├── migration-guides/ # Promptfoo, DeepEval, Inspect AI
│ ├── api/ # REST API docs
│ ├── blog/ # Launch posts
│ └── landing-page.html # Landing page
├── examples/ # Example eval suites
├── adapters/ # Standalone to_openeval()/from_openeval() packages per framework
├── benchmarks/ # 14 public benchmarks converted to validated EvalPort suites
├── .github/ # CI, CONTRIBUTING, issue templates
├── LICENSE # Apache 2.0
└── README.md # This file
evalport run executes an EvalPort suite against a real model provider and produces a spec-valid, self-validated ResultSet — no separate harness, no glue code. It's the CLI's headline command:
evalport run suite.json --provider openai --model gpt-4o-mini --dry-run # estimate cost first, spend nothing
evalport run suite.json --provider anthropic --model claude-3-5-sonnet-20241022 --output results.json- Two providers out of the box — OpenAI and Anthropic — plus any OpenAI-compatible endpoint (local inference servers, proxies, other vendors) via
--api-base. - Tier 1 graders run locally, with zero external dependencies:
exact_match,contains,regex,json_schema(a hand-written draft-07-ish validator),json_path(a hand-written JSONPath subset evaluator). - Tier 2 graders call an API:
llm_judge/model graded, andsemantic_similarity(cosine similarity over embeddings — always via an OpenAI-compatible endpoint, independent of--provider, since Anthropic has no public embeddings API). - Unsupported grader types clean-skip, per the spec's "Custom grader handling" rule (
code,human,customare recorded asskippedwithmetadata.skip_reason: "unsupported_grader_type"— the run never aborts because of one grader it doesn't know how to execute). --dry-runestimates cost before spending anything — token and USD estimates per test case and in total, with warnings when a model's pricing isn't in the known table. Always run this first and get sign-off on the estimate before running for real.- Retries with backoff on retryable provider errors (HTTP 429/5xx); non-retryable errors (bad auth, malformed request) fail fast instead of repeating.
--parallel <n>for concurrent test cases,--limit <n>to run a subset,--output <path>to write results incrementally as cases complete (so a long run's progress survives an interruption).- Every
ResultSetit produces is validated against the SDK's ownvalidateResultSet()before being written —evalport runrefuses to emit output that fails its own spec.
See cli/README.md for the full flag reference.
| Type | Description |
|---|---|
exact_match |
String equality |
contains |
Substring check |
regex |
Regex match |
semantic_similarity |
Embedding cosine similarity |
llm_judge |
LLM-as-judge with prompt template |
json_schema |
JSON Schema validation |
json_path |
JSONPath extraction + comparison |
code |
Custom grading function (sandboxed) |
human |
Human review |
model graded |
Alias for llm_judge (OpenAI Evals compat) |
custom |
Framework-specific handler |
| From | Status |
|---|---|
| Promptfoo → EvalPort | ✅ CLI + SDK |
| DeepEval → EvalPort | ✅ Python SDK |
| Inspect AI → EvalPort | ✅ Python SDK |
| OpenAI Evals → EvalPort | ✅ Python SDK |
benchmarks/ converts 14 well-known public benchmarks — GSM8K, ARC, BoolQ, HellaSwag, WinoGrande, CommonsenseQA, PIQA, TruthfulQA, MMLU, HumanEval, MBPP, SQuAD 2.0, DROP, and BIG-Bench Hard — into 22 individually-valid EvalPort suites (8,012 test cases total), every one of them passing the real validate_suite() validator in CI. Every benchmark's license was independently verified before inclusion; see benchmarks/LICENSES.md for the full attribution table. Start with benchmarks/README.md for the full index and quickstart.
adapters/ has standalone to_openeval()/from_openeval() packages for converting real evaluation results to and from EvalPort, one per framework, each independently installable and tested against the real validator:
- Getting Started
- Grader Type Reference
- Migration Guides
- REST API
- Full Specification
- Adoption Strategy
- Hostile Critique
- Contributing Guide
- Governance & the RFC process
- Open Design Questions — live Discussions on unresolved spec questions, no prior contribution required
- GitHub Discussions
- Contributors
- Launch Blog Post
- Landing Page
Apache 2.0 — see LICENSE