Enable gradient explanations in client

CHANGELOG.md

@@ -11,12 +11,15 @@ these packages. All five distributions share one version and one entry.
 
 - `SDMClient.in_process()` runs `kumo-tabular` and `kumo-relational` in this
   Python process with structured-data-models, through the same calls as a NIM:
-  predictions, fitted sessions, rankings and forecasts. It loads each model
+  predictions, fitted sessions, rankings, forecasts and explanations. It loads each model
   from its published weights on first use, or uses one passed in. The request
   is prepared as it is for a NIM and the model run on it directly, the context
   fitted as the NIM fits it. A request the model cannot answer is refused with
   a `StructuredDataError` whose code says why. Install it with the
-  `in-process` extra.
+  `in-process` extra. In-process explanations are written against the released
+  sdm 0.2.1 API: the converter reads a table's numerical columns from
+  `TableTensor.columns` by stype, not the newer `numerical_columns` property.
+  Revisit when the pinned sdm moves past 0.2.1.
 - A Snowflake session the client opens names itself to Snowflake as
   `nvidia/kumo-sdm`, so Snowflake can attribute its queries. An application
   named any other way is kept: an `application` argument, one in the
@@ -29,6 +32,17 @@ these packages. All five distributions share one version and one entry.
   installs on a laptop rather than on Linux only. The engine ships no source
   distribution, so those wheels remain the full set of platforms it supports:
   `structured-data-models-client` itself is pure Python and stays broader.
+- `kumo-tabular` accepts `explanation` in `outputs`.
+- `result.explanations` maps each explained row's key to its explanation: the
+  row's `id`, `(id, forecast_step)` on a forecast, or `row_index` when the
+  request has no entity id. Each entry is a plain dict of `outputs` and, on
+  `kumo-tabular`, the row's own columns under `instance`, or, on
+  `kumo-relational`, every table, the entity's own included, under `related`,
+  whose rows always carry their values as sent; `keys` is present when the
+  table's key column is one the model receives, i.e. a column a relationship
+  joins on; otherwise rows are identified by their values. It is kept out of
+  `result.to_pandas()`; `result.explanations.to_pandas()` flattens it into one
+  line per row, output and column, with each row's `hop` from the entity.
 
 ### Changed
 
@@ -42,6 +56,25 @@ these packages. All five distributions share one version and one entry.
   value that cannot be read as its type, such as a `run_mode` that is not
   text, is refused with `INVALID_REQUEST` before sending, where a NIM
   answered `422 VALIDATION_FAILED`.
+- A `kumo-relational` explanation is no longer refused above the 1,000-row
+  context budget. That limit belonged to an earlier deployment; the Kumo
+  Relational SDM NIM explains under either budget.
+- An explanation covers every scored row, on both models. A `kumo-tabular`
+  explanation over more than one row and a `kumo-relational` one over more
+  than one entity are no longer refused before sending. This needs a NIM newer
+  than the shipping releases (Relational 1.1.0, Tabular 1.0.0), which refuse
+  `explanation`. The first release that serves it will be named here when it
+  is cut.
+
+### Deprecated
+
+- `result.explanation` and `PredictionResult(explanation=...)` are deprecated
+  in favour of `result.explanations` and `explanations=`, warn with
+  `DeprecationWarning`. To upgrade, read a row's
+  explanation as `result.explanations[key]`, under the key naming that row in
+  `result.to_pandas()`. `result.explanation` returns the first record's raw
+  explanation only; the `{'by_forecast_step': ...}` it held for a forecast is
+  gone.
 
 ### Fixed
 

docs/about/architecture.md

@@ -500,7 +500,7 @@ result = client.predict(
 )
 
 result.to_pandas()  # always a DataFrame
-result.explanation  # None unless you asked for it
+result.explanations  # empty unless you asked for them
 ```
 
 Explanations are an **output field**, not a method. Ask for one with
@@ -537,7 +537,7 @@ It is not what any one deployment answers; see below.
 | `scores` | ✅ | ❌ |
 | `rankings` | ✅ | ❌ |
 | `embeddings` | ✅ | ❌ |
-| `explanation` | ✅ | ❌ |
+| `explanation` | ✅ | ✅ |
 
 Asking a model for a field it does not declare fails client-side, naming what
 it does declare.
@@ -564,7 +564,7 @@ So `scores`, `rankings`, `embeddings` and `explanation` are declared by the
 client and refused by that deployment on every task. That is a fact about this
 build, not about the fields: `explanation` has been measured working against
 another relational deployment, which returns an explanation the client parses
-and hands back on `PredictionResult.explanation`. A deployment that refuses it is
+and hands back on `PredictionResult.explanations`. A deployment that refuses it is
 narrower than the model family, not the client declaring something nothing
 serves.
 
@@ -581,6 +581,49 @@ served either — a forecast is sent as a regression per step, and a bare
 so a deployment that gains or loses an output fails the suite rather than
 leaving the document wrong.
 
+### Explanations are keyed by row, off the frame
+
+Every scored row is explained, and `result.explanations` maps each row's key
+to its explanation: the `id` it carries in `to_pandas()`, `(id,
+forecast_step)` for a forecast, and `row_index` when the request has no entity
+key. The same rule holds on both models. A client opened with
+`SDMClient.in_process()` computes the same record itself. The shape of an entry is in
+[Prediction output](../reference/prediction-output.md#explanations).
+
+**Keys, not positions.** The endpoint locates every explained row by its
+position in the tables as submitted. The caller never sees those tables: the
+related ones are gathered by this client, and a keyless one is identified on
+the wire only by the engine's own synthetic node id. So the client turns each
+position back into what the caller can recognise. The predicted row's position
+is dropped, since the mapping's key already names it. A related row always
+carries its values as sent; `keys` is present when the table's key column is
+one the model receives, i.e. a column a relationship joins on; otherwise rows
+are identified by their values.
+
+**Translated per call, before merging.** A request split across several calls
+numbers its positions from zero in each call. Read after merging, the second
+call's rows would point into the first call's tables. Each call's explanations
+are translated while that call's own tables are still at hand, and only then
+merged.
+
+**Kept off the frame.** An entry is nested: outputs by columns for the row
+itself, and outputs by rows by columns for each related table. `to_pandas()`
+stays one row per answer, as the task decides, and the explanations sit beside
+it on the result. `result.explanations.to_pandas()` flattens them into a table
+of their own when one is wanted, with each related row's `hop` from the entity
+worked out from the relationships and foreign-key values the client sent.
+
+**The engine's question row is dropped.** The engine sends a question row
+of its own, the entity key and anchor time with no caller columns, beside the
+entity's own table, and the endpoint explains both. The client drops that
+row's block when it reads the reply, because nothing in it is the caller's, so
+a `kumo-relational` entry is only `related`, with the entity's own table at
+hop 0.
+
+A forecast is explained a step at a time, under `(id, forecast_step)`, rather
+than aggregated across steps: each step is its own row in `to_pandas()`, and
+its explanation is keyed the same way.
+
 ---
 
 ## 6. Adding a model

docs/reference/limits.md

@@ -27,8 +27,6 @@ Both endpoints enforce sizes, and most of these are the relational one's.
 | calls a question asked in parts makes | relational | 200, one per candidate or step | at `materialize()`, as `sdm_client.decompose.MOST_PARTS` |
 | request body | relational | 30 MiB per call | at `materialize()` |
 | request body | relational, through Databricks model serving | 16 MB per call, the gateway's own decimal limit | before sending, on the wrapped envelope |
-| an explanation | relational | exactly one entity | before sending |
-| an explanation | relational | only within the 1,000-row context budget | before sending |
 | hops | relational | 6 | at `materialize()` |
 | tables | relational | 15 | against the graph the request carries, not the schema it was built from |
 
@@ -38,6 +36,9 @@ deployment sets, such as the longest string a cell may hold or the most
 distinct values a column may have. Those protect a shared server, and are
 not applied when the model runs in your own process.
 
+**An explanation has no row limit of its own.** Every scored row is
+explained, on either model.
+
 A relational request for more entities than one call scores is not refused.
 `materialize()` gathers it in batches of at most that many, each with a
 context of its own, sampled around that batch's rows rather than copied from

docs/reference/prediction-output.md

@@ -6,7 +6,7 @@ explanation.
 ```python
 result = client.predict(request, model=KumoRelational())
 result.to_pandas()  # always a frame
-result.explanation  # None unless you asked for it
+result.explanations  # empty unless you asked for them
 result.metadata  # whatever the endpoint reported
 ```
 
@@ -117,3 +117,158 @@ joined = to_score.reset_index(names='row_index').merge(answers, on='row_index')
 | `quantiles` | `quantile_<level>`, one per level |
 | `scores`, `embeddings` | one column holding a list |
 | `rankings` | one column holding a list of dicts |
+
+## Explanations
+
+Ask for one by adding `'explanation'` to `outputs`. There is no row limit:
+every scored row is explained. A NIM and a client opened with
+`SDMClient.in_process()` both serve it, with the same request and the same
+`result.explanations`.
+
+```python
+result = client.predict(request, model=KumoRelational(),
+                        outputs=['prediction', 'probabilities', 'explanation'])
+```
+
+### Keys
+
+`result.explanations` maps a row's key to that row's explanation. It is kept
+out of `result.to_pandas()`. `result.explanation` is deprecated, see
+`result.explanations`; there is no `{'by_forecast_step': ...}`.
+
+The key is whatever identifies the row in `to_pandas()`, by the same rule on
+both models:
+
+| Request | Key | Example |
+|---|---|---|
+| has an entity id, none repeated | the row's `id`, the same value as in `to_pandas()['id']` | `result.explanations['user_5']` |
+| forecast | `(id, forecast_step)`, or `(row_index, forecast_step)` when ids repeat | `result.explanations[('user_5', 3)]` |
+| no id column, or its ids repeat | `row_index` | `result.explanations[0]` |
+
+### One entry
+
+Each entry is a plain nested dict:
+
+```python
+{
+  'outputs':  [...],                        # the explained scores, in order
+  'instance': {                             # kumo-tabular only
+      'columns':   [...],                   # model's preprocessed numerical columns
+      'gradients': [[...], ...],            # [output][column]
+  },
+  'related': {                              # kumo-relational only: every table,
+                                            # the entity's own included
+      '<table>': {
+          'keys':      [...],               # each row's key; only when a relationship joins on the key column
+          'rows':      [{...}, ...],        # each row's values as sent to the model
+          'columns':   [...],               # model's preprocessed numerical columns
+          'gradients': [[[...], ...], ...], # [output][row][column], rows aligned with keys and rows
+      },
+  },
+}
+```
+
+`outputs` is copied from the endpoint:
+
+- classification gives every class probability in declared order, such as
+  `{'kind': 'probability', 'class': 'yes'}`;
+- regression gives the returned prediction,
+  `{'kind': 'prediction', 'statistic': 'mean'}` (or `'median'`), plus each
+  returned quantile when `quantiles` is requested.
+
+A gradient is the change in that output score per unit change in that
+preprocessed column. It is a diagnostic, not a causal effect or a
+feature-importance ranking.
+
+`columns` are the model's preprocessed numerical columns, not the columns you
+sent. Categoricals and datetimes may appear as derived numeric columns, and a
+table without numerical features has an empty `columns`.
+
+### Which tables get gradients
+
+| Model | Tables with gradients |
+|---|---|
+| kumo-tabular | the one table: each predicted row's own columns |
+| kumo-relational | the predicted entity's own table, plus every related table in the subgraph the client gathered for that entity (all tables reached within the hop limit, up to 6) |
+
+A `kumo-tabular` entry has no `related` key: its one table is the `instance`.
+A `kumo-relational` entry has no `instance` key: the entity's own table is one
+of the tables under `related`, at hop 0.
+
+Only the rows being predicted and their related rows get gradients. The
+context rows, the labelled examples the model compares against, get none.
+
+Gradients are per row, not per path. A row reached through several paths, for
+example a product bought in two of the user's orders, appears once. Its
+gradient is its total effect through all of them, and the model gives no split
+by path.
+
+### Related rows
+
+A related table's rows always carry `rows`, each row's values as sent to the
+model. `keys` is present when the table's key column is one the model receives,
+i.e. a column a relationship joins on; otherwise rows are identified by their
+values.
+Keyed or keyless changes only how a row is labelled; both get gradients in the
+same way.
+
+### Examples
+
+Values are illustrative. `kumo-tabular`, request with an id column:
+
+```python
+result.explanations['cust_17']
+# {'outputs': [{'kind': 'probability', 'class': 'no'},
+#              {'kind': 'probability', 'class': 'yes'}],
+#  'instance': {'columns': ['age', 'balance'],
+#               'gradients': [[0.01, -0.03], [-0.01, 0.03]]}}
+```
+
+`kumo-relational`, user 5 with one order and two login events. `users` carries
+`keys` because `user_id` is the join column; `orders` and `events` are
+identified by their values. All are part of the same entry:
+
+```python
+result.explanations['user_5']
+# {'outputs': [{'kind': 'probability', 'class': 'no'},
+#              {'kind': 'probability', 'class': 'yes'}],
+#  'related': {
+#      'users':  {'keys': ['user_5'],
+#                 'rows': [{'user_id': 'user_5', 'tenure_days': 410}],
+#                 'columns': ['tenure_days'],
+#                 'gradients': [[[-0.05]], [[0.05]]]},
+#      'orders': {'rows': [{'user_id': 'user_5', 'order_id': 'order_9003', 'amount': 12.0}],
+#                 'columns': ['amount'],
+#                 'gradients': [[[-0.02]], [[0.02]]]},
+#      'events': {'rows': [{'user_id': 'user_5', 'kind': 'login', 'at': '2026-10-01T09:00:00'},
+#                          {'user_id': 'user_5', 'kind': 'login', 'at': '2026-10-03T18:20:00'}],
+#                 'columns': ['kind_login', 'at_dayofweek'],    # model-derived names, illustrative
+#                 'gradients': [[[0.01, 0.00], [0.02, -0.01]],
+#                               [[-0.01, 0.00], [-0.02, 0.01]]]}}}
+```
+
+### As one table
+
+`result.explanations.to_pandas()` gives one line per (row, output, column),
+across all entities:
+
+```
+id      table     hop  key         row  output           column       gradient
+user_5  users     0    user_5      0    probability_yes  tenure_days   0.05
+user_5  orders    1    None        0    probability_yes  amount        0.02
+user_5  products  2    prod_77     0    probability_yes  price         0.004
+user_5  events    1    None        1    probability_yes  kind_login   -0.02
+```
+
+- `table` is the table's name. On `kumo-relational` the entity's own table
+  (`users` here) is hop 0. On `kumo-tabular` the one table is shown as
+  `instance`, at hop 0.
+- `hop` is how many links the row is from the entity, 0 for the entity's own
+  table. The client works it out from the relationships and foreign-key
+  values it sent. It is `None` for a row no relationship reaches.
+- `key` is the row's entry in `keys`, or `None` when the table has no `keys`.
+  On a `kumo-tabular` `instance` line it is the row's `id`, or `None` when the
+  request has no entity key. `row` is the row's position in that entity's
+  `related[table]['rows']`, and 0 on a `kumo-tabular` `instance` line.
+- `output` names the score by the column of `result.to_pandas()` that holds
+  it: `probability_yes`, `prediction`, `quantile_0.1`.

packages/structured-data-models-client/src/sdm_client/__init__.py

@@ -30,6 +30,7 @@
     UnknownModelError,
     UnsupportedRequestError,
 )
+from sdm_client.explanations import GradientExplanations
 from sdm_client.models import KumoRelational, KumoTabular, Model
 from sdm_client.query import PredictiveQuery
 from sdm_client.request import Request, Split, Target, Task
@@ -46,6 +47,7 @@
     'SDMWarning',
     'Column',
     'FittedModel',
+    'GradientExplanations',
     'KumoRelational',
     'KumoTabular',
     'MissingExtraError',