Skip to content

Add derived fields (FieldSpec.derivedFn) for Stores and Cube Views - #4701

Draft
lbwexler wants to merge 8 commits into
developfrom
cube-derived-fields
Draft

lbwexler wants to merge 8 commits into
developfrom
cube-derived-fields

Conversation

@lbwexler

@lbwexler lbwexler commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Adds derived fields: a Field computed from the record's other values, read through a getter on record data so it is never loaded, parsed or written, and always current with its inputs. One declaration works on a plain Store, the Cube's own store, and a connected store, so apps can share a field dictionary across all three.

{name: 'notional', aggregator: 'SUM', dependsOn: ['qty', 'price'], derivedFn: d => d.qty * d.price}
{name: 'pnlBps', dependsOn: ['pnl', 'notional'], derivedFn: d => (d.pnl / d.notional) * 10000}
  • FieldSpec.derivedFn + required dependsOn (may be []). A dependsOn naming a field the Store or Cube does not declare throws at construction. Store installs getters on its record-data prototype; raw parsing and modifyRecords ignore derived fields, getModifiedValues excludes them.
  • On a CubeField the function runs wherever the field is not aggregated. With an aggregator it derives each leaf and rolls up (market value, notional). Without one it derives every level from that row's aggregates (PnL in bps, margin %, VWAP) - the ratio-of-sums case that previously required a custom aggregator and a full rebuild per tick. Parent rows get a getter rather than a slot. Derived fields may read other derived fields.
  • Query pulls dependsOn fields in, as it does dimensions. On a data-only tick the View closes the producer's changedFields over dependsOn before diffing leaves (so a tick naming only price still re-sums a derived notional), and reports level-derived fields as changed to connected stores whenever an input moved.
  • Use-case: incremental weighted averages and other ratios of sums (Support incremental aggregators that read other fields on the row (e.g. weighted average) #4747). {name: 'priceQty', aggregator: 'SUM', dependsOn: ['price', 'qty'], derivedFn: d => d.price * d.qty} plus {name: 'vwap', dependsOn: ['priceQty', 'qty'], derivedFn: d => d.qty ? d.priceQty / d.qty : null} keeps the View on the incremental dataOnly path with built-in SUMs - no custom aggregator, no dependsOnChildrenOnly: false, and no changes to the Aggregator / RowUpdate protocol. Leaf edits via Cube.modifyRecordsAsync() recompute the product automatically, since it is a getter rather than a stored value.
  • 💥 Connected stores are now always projectionOnly - the View sets the flag, and projectionOnly: false or processRawData on a connected store throws. This is what lets a shared field definition carry a derivedFn harmlessly into a connected store: projections adopt values and never derive. Pulled forward from Data: calculated fields for Store and Cube View, connected stores via StoreConfig.view (#4620, #4637) #4638.

Perf: stores and Views without derived fields are unchanged from develop (benched, dense-record path). Those with them pay ~0.35µs per dense record/parent row created, for a __proto__-bearing clone that stays in V8 fast-properties mode. Verified with %HasFastProperties on real rows - the generated-class approach from #4638 produces dictionary-mode parent rows and was not used.

v88.0.0 shipped while this PR was open - its CHANGELOG entries now target 89.0.0-SNAPSHOT, with a new docs/upgrade-notes/v89-upgrade-notes.md for the connected-store change.

Not in scope: percent-of-total and other result-set-dependent values, which need lazy evaluation (see #4638). Follow-up noted: 'NULL'-aggregated fields still take a parent-row slot where a constant getter would do.

Toolbox PR: xh/toolbox#909

Hoist P/R Checklist

  • Caught up with develop branch as of last change.
  • Added CHANGELOG entry, or determined not required.
  • Reviewed for breaking changes, added breaking-change label + CHANGELOG if so.
  • Updated doc comments / prop-types, or determined not required.
  • Reviewed and tested on Mobile, or determined not required. (Shared data layer - no platform code.)
  • Created Toolbox branch / PR, or determined not required.

* Field.derivedFn computes a value from the record's other values, named in the required dependsOn, via a getter on record data - never loaded, parsed or written.
* On a CubeField the function also runs on every View row where the field is not aggregated: with an aggregator it derives leaves and rolls up, without one it derives each level from that row's aggregates. A Query including a derived field includes its inputs.
* Stores connected to a Cube View are now always projectionOnly - the View sets the flag, and conflicting config throws.
- `Store` and `RowDataGenerator` each hold one `{data, proto}` template, built in a single pass over fields with a per-field `addDerivedGetter`, replacing the shared defaults object that doubled as the dense-record prototype
- Comment the fixpoint loop over level-derived fields in `View.applyDataUpdate`
- Changelog wording
…setup

- `Store.parseFields` throws on a `dependsOn` naming an unknown field, covering plain Stores and Cubes alike; `Query.withDependencies` now skips quietly
- Inline the single-use derived-getter helper in `RowDataGenerator.buildParentTemplate`
- Comment the level-derived `changedFields` pass in `View.applyDataUpdate`
@lbwexler
lbwexler requested a review from haynesjm42 September 19, 2026 04:03
…ates

develop's #4723 narrows the leaf diff on a data-only update to the fields named in the producer's changedFields. Derived fields are never stored, so a producer cannot name them - a tick naming only `price` would skip a derived `notional` (qty * price) and leave its SUM stale up the tree.

View.withDerivedDependents() closes a set of field names over the derived query fields that read them, transitively. dataOnlyUpdate applies it to the incoming changedFields before diffing leaves, and to the outgoing changed.fields so consumers hear about level-derived values (getters, never diffed) whenever an input moved. Replaces the post-hoc level-derived widening, which after the merge was mutating the producer's own set.
@lbwexler

Copy link
Copy Markdown
Member Author

Follow-ups lifted from the (now closed) #4638 are tracked in #4748: read-only enforcement for derived fields, closing Store changedFields over dependsOn as the View now does, a canAggregateFn guard on level-derived fields, and doc carry-overs. None block this PR.

…ers (#4748 items 1, 3, 4)

- modifyRecords() throws on a write to a derived field; columns bound to one are never editable.
- CubeField throws when a level-derived field sets canAggregateFn.
- READMEs: read-by-name, pure/stable-return guidance; replace WeightedAverageAggregator example with the SUM/SUM/ratio decomposition.
- Store.updateData() adds every derived field reading a named input to the changedFields hint, so grids sorted on a derived column re-sort.
- Shared withDerivedDependents() util in data/impl; View uses it instead of its own copy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support incremental aggregators that read other fields on the row (e.g. weighted average)

1 participant