Skip to content

Curation layer over raw_records - #33

Closed
lgutschow wants to merge 1 commit into
mainfrom
feat/curation-layer
Closed

Curation layer over raw_records#33
lgutschow wants to merge 1 commit into
mainfrom
feat/curation-layer

Conversation

@lgutschow

Copy link
Copy Markdown
Contributor

What

Builds the derived curation layer fresh on the immutable raw_records
layer, alongside — never replacing — the existing consolidation path. Raw stays
read-only; the curated layer only ever inserts into curated_* and is fully
rebuildable from raw.

How

Migration 008_curation.sql promotes the scaffolded curated_* tables to
load-bearing:

  • curated_nodes gains scope + lineage columns (project_id, mode,
    importance, decay_class, event_time) and a scope/kind index.
  • New curated_embeddings (node_id, model, embedding vector(384)) mirroring
    raw_embeddings, with an IVFFlat cosine index.
  • Lineage indexes on curated_edges for reverse derived_from / supersedes
    lookups.

curation module — three passes, each scoped to a single
user/project/mode tuple and never crossing it:

  • promote_working_to_episodic — every active working raw record not already
    curated becomes an episodic node plus a derived_from edge back to the raw
    id. Idempotent: the edge is the "already curated" marker.
  • distill_semantic — clusters active episodic nodes by entity overlap
    (entities extracted on the fly from the source raw content, since raw carries
    no entities column), then asks the configured provider to distill each
    cluster into semantic facts, each linked by derived_from edges to every
    source raw id. Degrades to a logged no-op when the provider can't distill.
  • rebuild — wipes a user's curated set and re-derives from raw across every
    project/mode bucket, proving the rebuildable-from-raw contract.

Every new node is embedded into curated_embeddings (best-effort, never fatal).

Endpoints (glass-box, behind the existing bearer auth):

  • GET /records/:id/derivations — curated nodes derived from a raw id.
  • POST /records/rebuild and the admin /api/curate trigger — scoped rebuild.
  • A separate background scheduler task runs the rebuild on an interval, listing
    users straight from raw_records so it never touches the legacy path.

Tests

cargo test green. New coverage: promote (+ idempotency), entity-overlap
distill with all sources linked, graceful skip without distill capability,
deterministic rebuild, raw immutability under curation, per-user scope
isolation, and the derivations read. cargo fmt --all --check clean; clippy
clean on the new code; release build passes.

Build the derived curation layer fresh on the immutable raw_records
layer, alongside (never replacing) the existing consolidation path.
Raw stays read-only; curation only ever inserts into curated_* and is
fully rebuildable from raw.

Migration 008 promotes the scaffolded curated_* tables to load-bearing:
scope + lineage columns on curated_nodes (project_id, mode, importance,
decay_class, event_time), a curated_embeddings table mirroring
raw_embeddings (one row per node+model, IVFFlat index), and lineage
indexes on curated_edges for reverse (derived_from) lookups.

The curation module implements three passes, each scoped to a single
user/project/mode tuple and never crossing it:

- promote_working_to_episodic: every active working raw record that
  isn't already curated becomes an episodic node plus a derived_from
  edge back to the raw id. Idempotent — the edge is the "already
  curated" marker.
- distill_semantic: clusters active episodic nodes by entity overlap
  (entities extracted on the fly from the source raw content), then
  asks the configured provider to distill each cluster into semantic
  facts, each with derived_from edges to every source raw id. Degrades
  gracefully to a logged no-op when the provider can't distill.
- rebuild: wipes a user's curated_* and re-derives from raw across
  every project/mode bucket, proving the rebuildable-from-raw contract.

Every new node is embedded into curated_embeddings (best-effort, never
fatal). Glass-box endpoints: GET /records/:id/derivations lists the
curated nodes derived from a raw id; POST /records/rebuild and the
admin /api/curate trigger drive a scoped rebuild behind the existing
bearer auth. A separate background scheduler task runs the rebuild on
an interval, listing users straight from raw_records so it never
touches the legacy path.

Tests cover promote (+ idempotency), entity-overlap distill (with all
sources linked), graceful skip without distill capability,
deterministic rebuild, raw immutability under curation, per-user scope
isolation, and the derivations read.
@lgutschow lgutschow closed this Jul 24, 2026
@lgutschow
lgutschow deleted the feat/curation-layer branch July 24, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant