Thanks for looking. jeval is early; small, well-tested changes are the most useful kind.
pnpm install
pnpm check # lint, typecheck, build, unit tests (offline)Node 22 and pnpm 10 (packageManager in package.json). No TypeSafe key is needed for any of the above.
- Core stays independent of React, Next.js, Supabase and the TypeSafe SDK. Provider-specific code lives in
packages/provider-jev. - Never fall back silently. A failed real provider must surface as
error; the fixture provider is only ever chosen explicitly. - Expected labels and metadata are never sent to the judge. A test asserts this; keep it green.
- Usage is per request. Do not attribute one batched request's tokens to several checks.
- No invented rationales. Reports show the criterion, the selected outcome, probabilities, the rule applied in code and the evidence ids. Nothing more.
- Escape everything that ends up in HTML reports or the website.
- No telemetry, no performance claims, no fabricated numbers. Benchmarks come from recorded runs or are labelled simulated.
- Bug: open an issue with the Bug report template. Include the failing command, the case id, rubric id and the
answersblock from the run JSON when it concerns judge behaviour. Never paste real customer data or API keys. - Feature or new check kind: open an issue with the Feature request template first, with an example case and the outcome you expect. Agreeing on the outcome before the code saves everyone time.
- Question: use Discussions, not issues.
- Security: use the private advisory form linked in
SECURITY.md. - Pull request: fork, branch from
main(fix/…,feat/…,docs/…), keep it focused, fill in the PR template, and make sure CI is green. Title in the imperative ("Add OTel span import", "Fix review count in compare"). A maintainer reviews within a few days; small PRs get merged fastest. Squash-merge is the default, so commit granularity inside the PR does not matter.
- Add or update a test in the package you touch (
vitest), or an e2e test for site journeys (apps/web/e2e). - If you change a rubric's criterion, bump its
version;comparerefuses to align across versions. - Run
pnpm check. For CLI or export changes also runpnpm verify:packed. - Keep generated runs,
.envfiles and credentials out of commits (.gitignorecovers the usual paths).
If Jev decides something surprising on the example data, open an issue with the case id, rubric id, the probabilities from the run JSON and what you expected. Do not paste real customer data.