Data-lake curation and operation layers over raw records - #34
Merged
Conversation
Build the derived curation layer fresh on the immutable raw_records layer, alongside (never replacing) the existing consolidation path. Raw stays read-only; curation only ever inserts into curated_* and is fully rebuildable from raw. Migration 008 promotes the scaffolded curated_* tables to load-bearing: scope + lineage columns on curated_nodes (project_id, mode, importance, decay_class, event_time), a curated_embeddings table mirroring raw_embeddings (one row per node+model, IVFFlat index), and lineage indexes on curated_edges for reverse (derived_from) lookups. The curation module implements three passes, each scoped to a single user/project/mode tuple and never crossing it: - promote_working_to_episodic: every active working raw record that isn't already curated becomes an episodic node plus a derived_from edge back to the raw id. Idempotent — the edge is the "already curated" marker. - distill_semantic: clusters active episodic nodes by entity overlap (entities extracted on the fly from the source raw content), then asks the configured provider to distill each cluster into semantic facts, each with derived_from edges to every source raw id. Degrades gracefully to a logged no-op when the provider can't distill. - rebuild: wipes a user's curated_* and re-derives from raw across every project/mode bucket, proving the rebuildable-from-raw contract. Every new node is embedded into curated_embeddings (best-effort, never fatal). Glass-box endpoints: GET /records/:id/derivations lists the curated nodes derived from a raw id; POST /records/rebuild and the admin /api/curate trigger drive a scoped rebuild behind the existing bearer auth. A separate background scheduler task runs the rebuild on an interval, listing users straight from raw_records so it never touches the legacy path. Tests cover promote (+ idempotency), entity-overlap distill (with all sources linked), graceful skip without distill capability, deterministic rebuild, raw immutability under curation, per-user scope isolation, and the derivations read.
Build a RAPTOR-style summary tree over the curated layer and rank the
active feed by an Ebbinghaus decay/importance signal, all derived and
rebuildable on top of the immutable raw layer.
- Migration: summary levels (level > 0, kind 'summary', 'summarizes'
edges) and a level-scope index; extend the reference-weight table with
a polymorphic ref_kind ('raw' | 'curated'), a named decay_class, and a
per-ref half-life override. Nothing is ever deleted; decay only demotes.
- Summaries: cluster level-0 nodes by entity overlap, summarize each
cluster (provider distillation, with an extractive highest-importance
fallback when no model is configured), write a parent node at the next
level with edges to every child, embed it, and recurse until a level is
small enough or a depth cap is hit. Scope-bounded and rebuilt from
scratch each pass. rebuild() now regenerates the tree too.
- Decay: R = exp(-t / S) from hours-since-access and a per-class
half-life. On retrieval, returned nodes have their access clock reset
and weight recomputed.
- Retrieval: /records/context now feeds curated summaries first and
backstops with raw, blending semantic + keyword + recency + importance
+ decay with named default weights, behind the same request/response
shape. Adds GET /records/summaries for the summary-tree view.
- Tests: multi-level tree with summarizes edges + rebuild reproduction,
extractive fallback, summary-first and decay-ordered retrieval, access
resets the decay clock, tombstone-not-delete, scope isolation, raw
immutability, and the decay formula against known values.
Promote reference identity onto the immutable raw layer and add a HippoRAG-style entity pointer table, both derived and rebuildable on top of raw. A reference is a named mutable cell (identity = user_id/state_kind/state_key); its current value is the terminal state_object row not superseded by any newer row, and that row's payload always holds the complete value so a current-value read needs no chain walk. - Migration: add nullable, non-mutating state_kind/state_key columns to raw_records, filled from the payload by a BEFORE INSERT trigger (the append-only UPDATE/DELETE guard is untouched) with a one-time backfill, plus a partial identity index. The immutability contract is preserved: the columns are written once at insert, never updated. - Migration: an entity_index (user_id, entity, record_id) pointer table with a forward-hop index, and an optional label on curated_edges so an 'entity' edge can carry the entity string without smearing the node table with synthetic entity nodes. - Curation: promotion now populates the entity index and emits labelled 'entity' edges; the distill pass clusters from the index instead of re-extracting every time, falling back to on-the-fly extraction when the index has no row for a record yet. rebuild() wipes and deterministically repopulates the index. - References: a new raw-native surface under /records/state — read the terminal current value, list every key of a kind, walk the supersede chain oldest to newest, and write a new complete value (an append-only row superseding the prior terminal via the normal ingest path). The legacy state routes are untouched. - Retrieval: /records/context gives a matching reference a small additive bias on present-tense/current queries so it surfaces above episodic noise, with the response shape unchanged. - Tests: append-only create/patch with terminal read and full-chain history, raw immutability intact, scope isolation by user, the entity index populated by curation and used for clustering, deterministic rebuild repopulation, and the present-tense reference bias.
The catalog is a store registry over the lake. The raw and curated layers auto-register as stores with a live schema and record count; a user can register operational/external stores that publish slices into the lake. GET /catalog groups stores by kind with counts and a lineage note; POST/GET/PUT/DELETE /catalog/stores manage registered stores; and POST /catalog/stores/:id/sync pulls a store's published facts into the raw layer through the idempotent import path (dedup on source + published-fact id). Proposals let the system surface an action or insight for a human to decide on — it never acts on its own. POST /proposals records a proposed row citing raw-record evidence (validated as existing and owned); approve/deny are the operator's decision and executed is the host reporting it carried the action out. There is no endpoint or code path by which the system executes an action itself; the lifecycle guards keep the transitions terminal-correct. Adds MCP tools flashback_catalog and flashback_propose (no execute tool), and admin pages for the store map and the review queue.
Modes (cognitive registers) become a real axis of the memory model. Each mode pins a fastembed embedder + vector dimension, so a record is embedded in its register's geometry and only ever compared against records in the same geometry. - migration: a `modes` table (per-user, PK (user_id, name)) seeded with the built-in general/code/journal/research registers under a template user; nullable per-dimension embedding columns (embedding_768, embedding_1024) plus partial IVFFlat cosine indexes on raw_embeddings and curated_embeddings, with the existing 384 column made nullable/partial. A record writes exactly one column. - multi-embedder infra: the NLP service holds extra embedders keyed by model, lazily loaded; embed_for_mode(embedder_key, text) returns (dim, vector). The extraction schema gains a `mode` field so an LLM provider can auto-classify a record's register; the heuristic returns none and the default register wins. - ingest resolves a record's mode by precedence (caller override -> LLM auto-classify -> user default -> general), embeds with that register's embedder, and writes the matching embedding column; import does the same batched per embedder. - retrieval embeds the query in the requested register and reads its column, scoping exactly to that mode. A cross-register request (all, or several named modes) degrades to keyword + entity + recency and returns a visible degraded flag and warning. - curation and hierarchical summaries embed curated nodes in their scope's register; clustering and distillation stay within (user, mode) and never merge across registers. - modes API: GET/POST /modes and GET/PUT/DELETE /modes/:name (built-ins protected from deletion), behind bearer auth. The MCP record/recall tools accept an optional mode. The admin memories list gains a register filter.
… proposals, modes
Remove the pre-raw memory store and everything that only served it. The
raw_records -> curated_* -> catalog / proposals / references / modes lineage is
now the single data world.
- Drop the memories/core_memory/consolidation_runs tables from the migration set
so a fresh database builds only the canonical schema; renumber migrations to a
clean contiguous sequence.
- Delete the modules, routes, and background scheduler that read the retired
store; keep the raw-derived curation pipeline and its scheduler.
- Trim the MCP tools to the raw-native surface: record, recall, catalog, propose,
lineage (re-pointed to /records/{id}/lineage), and reference get/set/list/
history over /records/state/*.
- Re-home the admin UI onto raw + curated: dashboard counts, a raw record list
with a native mode filter, a curated view, references, the embedding map over
raw_embeddings, and a curation trigger.
- Delete the tests for the removed paths; keep the rest green.
Security + robustness hardening across the new-world code: - Admin record detail: scope the supersede-chain recursive CTE (and the single-record fetch) to the caller so the chain can never surface another user's rows even if a supersede pointer crosses a user boundary. - Cap the previously unbounded terminal state-object admin list. - Bound the curation pipeline: promotion, distillation, the entity-index read, and per-level summarization now pull a bounded batch (configurable), so one rebuild can't load a whole corpus into memory or feed an unbounded quadratic clusterer. A large backlog drains over successive passes. - Guard an unchecked first-element index in distillation. - Percent-encode reference kind/state_key path segments in the MCP proxy so a crafted key can't reshape the request path. - De-duplicate the graph projection work on the polled map endpoint. Tests: cross-user chain access is excluded; state list is capped; oversized query/context limits clamp; batch cap parses/defaults correctly; segment encoding escapes path-significant and non-ASCII bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the curation and operation layers on top of the immutable raw-records store, and collapses the service onto a single canonical data model.
Curation (derived, rebuildable from raw)
raw_records: working records promote to episodic curated nodes; entity-overlap clusters distill into semantic facts; every curated node carriesderived_fromlineage edges back to the raw records it came from. Wiping curation and rebuilding from raw reproduces the same derived set.summarizesedges.R = e^(−t/S)); access resets the clock; nothing is deleted — decay only lowers ranking./records/contextblends semantic similarity, keyword overlap, recency, importance, and decay, and feeds curated summaries first with raw records as the backstop.References as first-class
(user, kind, key); the current value is the terminal node of a forward-only supersede chain, read in O(1) with no chain walk. Writes are append-only./records/state/*endpoints: current value, list, history, append-value write.Operation — catalog + propose-don't-act
Modes (cognitive registers)
One canonical world
Hardening
Tests
~175 tests covering curation promote/distill/rebuild + lineage, summary trees, the decay curve, reference terminal/history + immutability, the entity index, catalog auto-registration + sync idempotency, proposal transitions + propose-only enforcement, mode-scoped embedding/retrieval + cross-mode degradation, and cross-user access rejection.
Not covered here (needs real-world integration/tuning)
pgvector index sizing + probe tuning under load; the multi-embedder memory footprint on target hardware; the curation batch-cap default; provider (remote model) configuration; and end-to-end load/soak testing. These are deployment/ops decisions, not correctness gaps. A fresh database is required (the migration set was rebuilt).