Skip to content

[FT] Add a format-neutral post-run export hook #1397

Open

Description

@halvrenofviryel

Issue encountered

LightEval exposes structured results through EvaluationTracker and writes results_*.json plus optional detail artifacts. Downstream consumers currently appear to need either a custom wrapper/subclass or post-run discovery and parsing of those files. That couples integrations to output paths and filenames, and makes it difficult to receive the exact completed-run payload together with references to the artifacts that were actually written.

This affects provenance systems, experiment registries, audit/evidence pipelines, custom dashboards, and other consumers that need more than a second copy of aggregate metrics.

Solution/Feature

Provide a small, opt-in, format-neutral extension point after a run is finalized and its configured artifacts are saved. For example, a callback protocol or pluggable exporter could receive a structured context containing:

  • the final results mapping already produced by EvaluationTracker;
  • a run identifier/timestamp and completion status;
  • stable references (including fsspec-compatible paths where applicable) to the results and detail artifacts actually written; and
  • enough error isolation that an exporter failure is reported without corrupting LightEval's primary results.

The hook should not require LightEval to adopt a universal external schema or add format-specific logic. It would only expose a stable lifecycle boundary and structured run context so downstream packages can implement their own exporters.

Possible alternatives

  • Continue scanning output directories and parsing results_*.json after each run. This works, but depends on file layout and makes artifact correlation harder.
  • Subclass or wrap EvaluationTracker.save(). This is possible for programmatic callers but is less convenient for CLI runs and can track internal implementation details.
  • Add individual built-in integrations. A generic hook seems smaller and avoids coupling LightEval to downstream formats.

This is related to but distinct from #1318: that proposal concerns OpenEval import/export, while this request is for a format-neutral extension point usable by any downstream consumer.

One concrete consumer is the non-normative AIREP LightEval exporter, which currently parses saved LightEval results and hashes supplied detail/log files into an evidence manifest. A generic hook would remove that post-hoc coupling without adding AIREP-specific behavior to LightEval. AIREP is experimental and is not proposed as LightEval's output format.

Activity

  1. KanishqGandharv219 commented on Oct 11, 2026

    @KanishqGandharv219

    I would be interested in helping with a small prototype for this hook. My background includes Python RAG evaluation and Hugging Face integrations. The useful boundary seems to be after configured artifacts are saved, with tests for disabled-hook behaviour and exporter failure isolation. Before implementing anything, do the maintainers favour a callback on EvaluationTracker or a separate exporter interface? I would keep the first version format-neutral and avoid introducing a new external schema. This is a proposal; I have not yet audited all save callers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions