Skip to content

Add optional Parquet destinations and Croissant dataset metadata #59

Description

@ericpaulsen

Add bounded Parquet output and an MLCommons Croissant sidecar after canonical training-row validation. JSONL and the Sediment manifest remain canonical.

Preserve nested messages, omitted values, source evidence, and recipe semantics. Decision: define how operators supply the dataset's actual license and rights metadata; the software license must not be assumed to license captured data.

Acceptance criteria:

  • Physical schemas derive from the canonical schema catalog and preserve supported values losslessly.
  • Unsupported unions or types refuse before publication rather than coercing values or inventing sentinels.
  • Writers consume bounded batches and have a peak-memory regression.
  • Metadata records writer/version, canonical schema identity, row counts, hashes, source bundle identity, and policy.
  • Croissant describes resources, fields, splits, declared dataset licensing, and hashes; Sediment Provenance remains in the canonical manifest.
  • Round-trip tests preserve semantic rows for supported export objectives; byte equality across writer versions is not promised.
  • Optional destination dependencies remain outside the Fact store and Derivation runtime.

Relevant files:

  • packages/export/sediment_export/schema_contracts.py
  • packages/export/sediment_export/jsonl.py
  • packages/export/sediment_export/bounded_training.py
  • docs/exports/training-exports.md

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageA maintainer evaluates the issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions