Curation layer over raw_records - #33
Closed
lgutschow wants to merge 1 commit into
Closed
Conversation
Build the derived curation layer fresh on the immutable raw_records layer, alongside (never replacing) the existing consolidation path. Raw stays read-only; curation only ever inserts into curated_* and is fully rebuildable from raw. Migration 008 promotes the scaffolded curated_* tables to load-bearing: scope + lineage columns on curated_nodes (project_id, mode, importance, decay_class, event_time), a curated_embeddings table mirroring raw_embeddings (one row per node+model, IVFFlat index), and lineage indexes on curated_edges for reverse (derived_from) lookups. The curation module implements three passes, each scoped to a single user/project/mode tuple and never crossing it: - promote_working_to_episodic: every active working raw record that isn't already curated becomes an episodic node plus a derived_from edge back to the raw id. Idempotent — the edge is the "already curated" marker. - distill_semantic: clusters active episodic nodes by entity overlap (entities extracted on the fly from the source raw content), then asks the configured provider to distill each cluster into semantic facts, each with derived_from edges to every source raw id. Degrades gracefully to a logged no-op when the provider can't distill. - rebuild: wipes a user's curated_* and re-derives from raw across every project/mode bucket, proving the rebuildable-from-raw contract. Every new node is embedded into curated_embeddings (best-effort, never fatal). Glass-box endpoints: GET /records/:id/derivations lists the curated nodes derived from a raw id; POST /records/rebuild and the admin /api/curate trigger drive a scoped rebuild behind the existing bearer auth. A separate background scheduler task runs the rebuild on an interval, listing users straight from raw_records so it never touches the legacy path. Tests cover promote (+ idempotency), entity-overlap distill (with all sources linked), graceful skip without distill capability, deterministic rebuild, raw immutability under curation, per-user scope isolation, and the derivations read.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Builds the derived curation layer fresh on the immutable
raw_recordslayer, alongside — never replacing — the existing consolidation path. Raw stays
read-only; the curated layer only ever inserts into
curated_*and is fullyrebuildable from raw.
How
Migration
008_curation.sqlpromotes the scaffoldedcurated_*tables toload-bearing:
curated_nodesgains scope + lineage columns (project_id,mode,importance,decay_class,event_time) and a scope/kind index.curated_embeddings (node_id, model, embedding vector(384))mirroringraw_embeddings, with an IVFFlat cosine index.curated_edgesfor reversederived_from/supersedeslookups.
curationmodule — three passes, each scoped to a singleuser/project/modetuple and never crossing it:promote_working_to_episodic— every activeworkingraw record not alreadycurated becomes an
episodicnode plus aderived_fromedge back to the rawid. Idempotent: the edge is the "already curated" marker.
distill_semantic— clusters active episodic nodes by entity overlap(entities extracted on the fly from the source raw content, since raw carries
no entities column), then asks the configured provider to distill each
cluster into
semanticfacts, each linked byderived_fromedges to everysource raw id. Degrades to a logged no-op when the provider can't distill.
rebuild— wipes a user's curated set and re-derives from raw across everyproject/mode bucket, proving the rebuildable-from-raw contract.
Every new node is embedded into
curated_embeddings(best-effort, never fatal).Endpoints (glass-box, behind the existing bearer auth):
GET /records/:id/derivations— curated nodes derived from a raw id.POST /records/rebuildand the admin/api/curatetrigger — scoped rebuild.users straight from
raw_recordsso it never touches the legacy path.Tests
cargo testgreen. New coverage: promote (+ idempotency), entity-overlapdistill with all sources linked, graceful skip without distill capability,
deterministic rebuild, raw immutability under curation, per-user scope
isolation, and the derivations read.
cargo fmt --all --checkclean;clippyclean on the new code; release build passes.