Repository navigation
[FT] Add a format-neutral post-run export hook #1397
Description
Issue encountered
LightEval exposes structured results through EvaluationTracker and writes results_*.json plus optional detail artifacts. Downstream consumers currently appear to need either a custom wrapper/subclass or post-run discovery and parsing of those files. That couples integrations to output paths and filenames, and makes it difficult to receive the exact completed-run payload together with references to the artifacts that were actually written.
This affects provenance systems, experiment registries, audit/evidence pipelines, custom dashboards, and other consumers that need more than a second copy of aggregate metrics.
Solution/Feature
Provide a small, opt-in, format-neutral extension point after a run is finalized and its configured artifacts are saved. For example, a callback protocol or pluggable exporter could receive a structured context containing:
- the final results mapping already produced by
EvaluationTracker; - a run identifier/timestamp and completion status;
- stable references (including fsspec-compatible paths where applicable) to the results and detail artifacts actually written; and
- enough error isolation that an exporter failure is reported without corrupting LightEval's primary results.
The hook should not require LightEval to adopt a universal external schema or add format-specific logic. It would only expose a stable lifecycle boundary and structured run context so downstream packages can implement their own exporters.
Possible alternatives
- Continue scanning output directories and parsing
results_*.jsonafter each run. This works, but depends on file layout and makes artifact correlation harder. - Subclass or wrap
EvaluationTracker.save(). This is possible for programmatic callers but is less convenient for CLI runs and can track internal implementation details. - Add individual built-in integrations. A generic hook seems smaller and avoids coupling LightEval to downstream formats.
This is related to but distinct from #1318: that proposal concerns OpenEval import/export, while this request is for a format-neutral extension point usable by any downstream consumer.
One concrete consumer is the non-normative AIREP LightEval exporter, which currently parses saved LightEval results and hashes supplied detail/log files into an evidence manifest. A generic hook would remove that post-hoc coupling without adding AIREP-specific behavior to LightEval. AIREP is experimental and is not proposed as LightEval's output format.
Activity
KanishqGandharv219 commentedon Oct 11, 2026
I would be interested in helping with a small prototype for this hook. My background includes Python RAG evaluation and Hugging Face integrations. The useful boundary seems to be after configured artifacts are saved, with tests for disabled-hook behaviour and exporter failure isolation. Before implementing anything, do the maintainers favour a callback on EvaluationTracker or a separate exporter interface? I would keep the first version format-neutral and avoid introducing a new external schema. This is a proposal; I have not yet audited all save callers.