Data: calculated fields for Store and Cube View, connected stores via StoreConfig.view (#4620, #4637)

CHANGELOG.md

@@ -12,6 +12,38 @@
   3. Plain ASCII punctuation only. Use " - " for in-sentence breaks, never an em dash.
 -->
 
+## 89.0.0-SNAPSHOT - unreleased
+
+### 💥 Breaking Changes (upgrade difficulty: 🟢 LOW - connected stores)
+
+See [`docs/upgrade-notes/v89-upgrade-notes.md`](docs/upgrade-notes/v89-upgrade-notes.md) for
+detailed, step-by-step upgrade instructions with before/after code examples.
+
+* Stores now connect to a Cube `View` at construction via `StoreConfig.view`, replacing
+  `ViewConfig.stores` and `View.setStores()` (see `View.addStore()`/`removeStore()` for transient
+  detach). Connected stores are built as `projectionOnly` projections adopting view rows by
+  reference - conflicting config (`projectionOnly: false`, `processRawData`, `digestSpec`,
+  `idEncodesTreePath`) throws. Route edits through the Cube (e.g. `Cube.modifyRecordsAsync()`).
+* A connected store's `fields` are now reconciled to its View's query fields at connection and on
+  query changes - view-published data is described by the query's own `CubeField`s, superseding
+  any same-named app or grid-inferred field, with app-declared extras preserved. Field metadata
+  read off the store (types, `displayName`s, calculated status) now flows from the Cube. A
+  store-layer `calculatedFn` field sharing a view field's name throws at connection.
+
+### 🎁 New Features
+
+* Added `FieldSpec.calculatedFn` - declare Store fields computed on the client from each record's
+  other values and the Store, with no source data or server round-trip required. Values are
+  computed lazily on read - always current, with minimal memory and load-time overhead - and
+  support sorting, filtering, and export like any other field. Grids repaint calculated columns
+  automatically as data changes, and calculated fields are read-only for editing.
+* Added `CubeFieldSpec.calculatedFn` - the Cube-layer form of the same concept, computed on View
+  rows with the View's `AggregationContext`. Recommended for globally-dependent values such as
+  percent-of-total or ratios of sums such as a weighted average, where a custom aggregator would
+  slow updates to the entire View - calculated fields keep Views on their fastest incremental
+  update path. `AggregationContext.filteredRecords`
+  is readable from these functions and always current.
+
 ## 88.0.0 - 2026-09-28
 
 ### 💥 Breaking Changes (upgrade difficulty: 🔴 HIGH - TC39 decorators + Rsbuild, AG Grid 36, MobX 7, removals)

cmp/grid/Grid.ts

@@ -715,8 +715,9 @@ export class GridLocalModel extends HoistModel {
             transaction = newRs.diffFrom(prevRs);
         model.diagnostics.noteGenTransaction(transaction, newRs, prevRs, start);
 
-        const applyStart = performance.now();
-        if (!this.transactionIsEmpty(transaction)) {
+        const applyStart = performance.now(),
+            isEmptyTxn = this.transactionIsEmpty(transaction);
+        if (!isEmptyTxn) {
             this.transactionMgr.apply(transaction, prevRs, newRs);
         } else if (!prevRs) {
             // AG Grid needs rowData (even if empty) to exit its initial loading state.
@@ -743,6 +744,26 @@ export class GridLocalModel extends HoistModel {
             }
         }
 
+        const calcNames = store.calculatedFieldNames;
+        if (calcNames.size) {
+            // Calculated values can move via inputs outside transacted rows - repaint their columns.
+            const columns = model
+                .getVisibleLeafColumns()
+                .filter(c => calcNames.has(c.field))
+                .map(c => c.colId);
+            if (!isEmpty(columns)) agApi.refreshCells({columns});
+
+            // An empty sync (e.g. summary-only) could still require calc cols resort.
+            if (
+                isEmptyTxn &&
+                prevRs &&
+                !model.externalSort &&
+                model.sortBy.some(s => calcNames.has(model.getColumn(s.colId)?.field))
+            ) {
+                this.transactionMgr.noteSortStale();
+            }
+        }
+
         if (!isEmpty(transaction.add) || !isEmpty(transaction.remove)) {
             wait().then(() => this.syncSelection());
         }

cmp/grid/columns/Column.ts

@@ -383,7 +383,9 @@ export interface ColumnSpec {
 
     /**
      * True to make cells in this column editable, or a function to determine on a
-     * record-by-record basis.
+     * record-by-record basis. Ignored for columns bound to a calculated field
+     * ({@link FieldSpec.calculatedFn}) - such values are computed at read time and never
+     * editable.
      */
     editable?: boolean | ColumnEditableFn;
 
@@ -773,6 +775,7 @@ export class Column {
     isEditableForRecord(record: StoreRecord): boolean {
         const {editable, gridModel} = this;
         if (!record) return false;
+        if (record.store.calculatedFieldNames.has(this.field)) return false;
         return isFunction(editable)
             ? editable({record, store: record.store, gridModel, column: this})
             : editable;

cmp/grid/impl/GridTransactionManager.ts

@@ -47,8 +47,9 @@ export class GridTransactionManager extends HoistBase {
     @managed
     private sortScheduler: DeferredWorkScheduler;
 
-    // Rows updated by suppressed transactions since the last sort - null when order is current.
-    private pendingSortIds: Set<StoreRecordId> = null;
+    // Rows updated by suppressed transactions since the last sort, or 'full' when order may be
+    // stale beyond any tracked rows - null when order is current.
+    private pendingSort: Set<StoreRecordId> | 'full' = null;
 
     // Cached provable sort paths - undefined = stale, null = sort not provably value-based.
     private _sortPaths: Array<Some<string>> | null | undefined;
@@ -79,7 +80,7 @@ export class GridTransactionManager extends HoistBase {
         });
         try {
             agApi.applyTransaction(transaction);
-            if (!suppress) this.pendingSortIds = null;
+            if (!suppress) this.pendingSort = null;
         } finally {
             agApi.updateGridOptions({
                 suppressModelUpdateAfterUpdateTransaction: false,
@@ -88,6 +89,15 @@ export class GridTransactionManager extends HoistBase {
         }
     }
 
+    /**
+     * Note current row order may be stale with no transaction to prove otherwise - e.g. a
+     * calculated sort value moved by a summary-only update. Schedules a paced full re-sort.
+     */
+    noteSortStale() {
+        this.pendingSort = 'full';
+        this.sortScheduler.scheduleAsync();
+    }
+
     //------------------------
     // Implementation
     //------------------------
@@ -119,7 +129,7 @@ export class GridTransactionManager extends HoistBase {
 
         // With a flush pending, current order is stale - a delta merge would preserve the
         // staleness, so any refresh must be full (which resolves the pending flush, per apply).
-        if (this.pendingSortIds) return 'full';
+        if (this.pendingSort) return 'full';
 
         const changedCount = update.length + add.length + remove.length;
         return newRs.count > 0 && changedCount / newRs.count < this.deltaSortRatio
@@ -136,6 +146,12 @@ export class GridTransactionManager extends HoistBase {
         const sortPaths = this.getSortPaths();
         if (!sortPaths) return false;
 
+        // Calculated sort values can move via inputs outside any updated row - nothing provable.
+        const calcNames = this.model.store.calculatedFieldNames;
+        if (calcNames.size && sortPaths.some(p => calcNames.has(isArray(p) ? p[0] : p))) {
+            return false;
+        }
+
         if (changedFields) {
             return sortPaths.every(p => !changedFields.has(isArray(p) ? p[0] : p));
         }
@@ -191,25 +207,31 @@ export class GridTransactionManager extends HoistBase {
     }
 
     private notePendingSort(updates: StoreRecord[]) {
-        const ids = (this.pendingSortIds ??= new Set());
-        updates.forEach(rec => ids.add(rec.id));
+        const {pendingSort} = this;
+        // A pending full sort already covers these rows - no need to track them.
+        if (pendingSort !== 'full') {
+            const ids = pendingSort ?? (this.pendingSort = new Set());
+            updates.forEach(rec => ids.add(rec.id));
+        }
         this.sortScheduler.scheduleAsync();
     }
 
     private flushPendingSort() {
-        const {model, pendingSortIds} = this,
+        const {model, pendingSort} = this,
             latestRs = model._syncedRs;
-        // Not ready - leave ids pending; the next transaction will run 'full' and resolve them.
-        if (!pendingSortIds || !latestRs || !model.isReady) return;
-        this.pendingSortIds = null;
+        // Not ready - leave the sort pending; the next transaction will run 'full' and resolve it.
+        if (!pendingSort || !latestRs || !model.isReady) return;
+        this.pendingSort = null;
 
         const start = performance.now(),
             {agApi} = model,
             update = [];
-        pendingSortIds.forEach(id => {
-            const rec = latestRs.getById(id);
-            if (rec) update.push(rec);
-        });
+        if (pendingSort !== 'full') {
+            pendingSort.forEach(id => {
+                const rec = latestRs.getById(id);
+                if (rec) update.push(rec);
+            });
+        }
 
         if (update.length && update.length / latestRs.count < this.deltaSortRatio) {
             agApi.updateGridOptions({deltaSort: true});

data/Field.ts

@@ -5,14 +5,16 @@
  * Copyright © 2026 Extremely Heavy Industries Inc.
  */
 
-import {XH} from '@xh/hoist/core';
+import {PlainObject, XH} from '@xh/hoist/core';
 import {RuleLike} from '@xh/hoist/data/validation/Types';
 import {isLocalDate, LocalDate} from '@xh/hoist/utils/datetime';
 import {withDefault} from '@xh/hoist/utils/js';
 import {Rule} from './validation/Rule';
 import equal from 'fast-deep-equal';
 import {isDate, isString, toNumber, isFinite, startCase, isFunction, castArray} from 'lodash';
 import DOMPurify from 'dompurify';
+import type {Store} from './Store';
+import type {CubeCalculatedFn} from './cube/CubeField';
 
 /**
  * Constructor arguments for a Hoist data package Field.
@@ -64,8 +66,51 @@ export interface FieldSpec {
      * building secured internal apps with large datasets and tight performance tolerances.
      */
     enableXssProtection?: boolean;
+
+    /**
+     * Function computing this field's value at read time from the record's other values and the
+     * Store, making this a *calculated* field - derived on the client rather than loaded:
+     *
+     * ```ts
+     * {
+     *     name: 'pctCommission',
+     *     calculatedFn: (data, store) =>
+     *         (data.commission / store.summaryRecords[0]?.data.commission) * 100
+     * }
+     * ```
+     *
+     * Values are read via lazy prototype getters on record `data` - never stored, parsed, or
+     * compared for record reuse, and always current, even when inputs live outside the record
+     * (e.g. a summary denominator). Grids repaint calculated columns after each transaction,
+     * and a `FieldFilter` on one triggers a full re-filter. (`FunctionFilter`s are opaque and
+     * may need a manual {@link Store.refreshFilter}.)
+     *
+     * Calculated fields are read-only ({@link Store.modifyRecords} throws, columns are never
+     * editable, `type` is display-only) and invisible to own-property enumeration - read values
+     * by name, or via {@link StoreRecord.getValues}. Keep the fn pure and fast (it runs per
+     * cell paint and per sort comparison), return primitives or stable references, and avoid
+     * cycles when reading other calculated fields.
+     *
+     * See {@link CubeFieldSpec.calculatedFn} for the Cube View form - the union type keeps
+     * `CubeFieldSpec` assignable wherever `FieldSpec` is accepted; on a plain Store, always
+     * supply the {@link StoreCalculatedFn} form.
+     */
+    calculatedFn?: StoreCalculatedFn | CubeCalculatedFn;
 }
 
+/**
+ * Function computing a Store-level calculated field value at read time.
+ * See {@link FieldSpec.calculatedFn}.
+ */
+export type StoreCalculatedFn = (data: PlainObject, store: Store) => any;
+
+/**
+ * Function computing a calculated field value at read time - the union of the layer-specific
+ * signatures declared by {@link FieldSpec.calculatedFn} (Store) and `CubeFieldSpec.calculatedFn`
+ * (Cube View).
+ */
+export type CalculatedFn = (data: any, context: any) => any;
+
 /**
  * Metadata for an individual data field within a {@link StoreRecord}.
  *
@@ -76,6 +121,11 @@ export class Field {
         return true;
     }
 
+    /** True for {@link CubeField} instances - see that subclass. */
+    get isCubeField() {
+        return false;
+    }
+
     readonly name: string;
     readonly type: FieldType;
     readonly displayName: string;
@@ -85,6 +135,19 @@ export class Field {
     readonly rules: Rule[];
     readonly enableXssProtection: boolean;
 
+    /**
+     * Function computing this field's value at read time, marking it as a calculated field.
+     * Layer-specific signatures - see {@link FieldSpec.calculatedFn} (Store) and
+     * `CubeFieldSpec.calculatedFn` (Cube View). Not readonly to support subclass assignment
+     * and anticipated runtime updates to calculated field specs.
+     */
+    calculatedFn: CalculatedFn;
+
+    /** True if this field's value is computed at read time - see {@link FieldSpec.calculatedFn}. */
+    get isCalculated(): boolean {
+        return !!this.calculatedFn;
+    }
+
     constructor({
         name,
         type = 'auto',
@@ -93,7 +156,8 @@ export class Field {
         defaultValue = null,
         isDimension = false,
         rules = [],
-        enableXssProtection = XH.appSpec.enableXssProtection
+        enableXssProtection = XH.appSpec.enableXssProtection,
+        calculatedFn = null
     }: FieldSpec) {
         this.name = name;
         this.type = type;
@@ -103,6 +167,7 @@ export class Field {
         this.rules = this.processRuleSpecs(rules);
         this.enableXssProtection = enableXssProtection;
         this.defaultValue = this.parseValueInternal(defaultValue);
+        this.calculatedFn = calculatedFn;
     }
 
     parseVal(val: any): any {

data/README.md

@@ -433,6 +433,51 @@ apps with large datasets. Set `enableXssProtection` per field, or app-wide via
 | `'tags'` | String array | Splits comma-separated |
 | `'pwd'` | Password | Marks as sensitive |
 
+### Calculated Fields
+
+Declare a field with a `calculatedFn` to compute its value on the client at read time, from the
+record's other field values and the Store - no source data or server round-trip required:
+
+```typescript
+const store = new Store({
+    fields: [
+        'commission',
+        {
+            name: 'pctCommission',
+            calculatedFn: (data, store) =>
+                (data.commission / store.summaryRecords[0]?.data.commission) * 100
+        }
+    ]
+});
+```
+
+Calculated values are read through lazy prototype getters on record `data` objects - never
+stored, parsed, or included in the equality/digest comparisons used to detect unchanged records,
+and always current when read. Key characteristics:
+
+- **Read-only** - grid columns bound to calculated fields are never editable, and
+  `modifyRecords()` throws on any attempt to write one. `type` is display/metadata only, as
+  parsing never applies.
+- **Works with `projectionOnly`** - record `data` becomes a generated wrapper over the adopted
+  raw object, adding the calculated getters with no per-record copy of source values.
+- **Read by name, never enumerated** - calculated values live behind prototype getters, invisible
+  to own-property enumeration: `Object.keys()`, spread and `JSON.stringify()` omit them (record
+  `data` should never be enumerated in any case). `StoreRecord.getValues()` returns a plain-object
+  copy of all field values, calculated included.
+- **Automatic grid repaint** - grids bound to the Store refresh columns displaying calculated
+  fields after each transaction, repainting visible cells whose value moved via an input outside
+  their own row (e.g. a summary denominator).
+- **Automatic filter refresh** - a `FieldFilter` testing a calculated field triggers a full
+  re-filter on each transaction, keeping membership current. `FunctionFilter`s are opaque to this
+  detection - one reading calculated values may require a manual `refreshFilter()`.
+- Sorting and exporting read through the getters and work naturally - keep `calculatedFn` a fast,
+  pure function, as it can run once per visible cell per paint and once per comparison when
+  sorting. Prefer returning primitives or stable references - a fresh object or array per read
+  defeats the value-equality check grids use to skip repainting unchanged cells.
+
+See `CubeFieldSpec.calculatedFn` (`data/cube/README.md`) for the Cube-layer form of the same
+concept, computed on View rows with an `AggregationContext`.
+
 ## Filter System
 
 **Files**: `filter/Filter.ts`, `filter/FieldFilter.ts`, `filter/CompoundFilter.ts`, `filter/FunctionFilter.ts`
@@ -961,21 +1006,17 @@ parses and owns. Store then uses each incoming raw object *as* its record's `dat
 This collapses the usual two objects per row to one, and skips the per-row parse on every load and
 update.
 
-Use this config for stores connected to a Cube `View`, or fed by an endpoint that returns data in
-its final client-side form. A View logs a warning when a connected store leaves the config unset.
-Set it explicitly to `false` to opt out and silence that warning.
+Use this config for stores fed by an endpoint that returns data in its final client-side form.
+Stores connected to a Cube `View` (constructed with `StoreConfig.view`) are always projections -
+the flag is set automatically, and conflicting config throws.
 
 ```typescript
-const store = new Store({
-    fields: [...],
-    projectionOnly: true
-});
-
 const view = cube.createView({
     query: {dimensions: ['region', 'product']},
-    stores: store,
     connect: true
 });
+
+const store = new Store({view, fields: [...]});
 ```
 
 This mode carries real constraints: