Document user_metadata filters on artifact list requests - #7814
Open
kumare3 wants to merge 1 commit into
Open
Conversation
The comments on ListArtifactsRequest and ListArtifactNamesRequest enumerate the filters the service supports, so leaving the metadata filter out makes them wrong rather than merely incomplete. Comment-only: the filter needs no new field. common.ListRequest.filters already carries it, keyed by a "user_metadata.<key>" field name, the same way the runs service keys label filters off "labels.<key>". Implemented in unionai/cloud. Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com>
3 tasks
kumare3
added a commit
to flyteorg/flyte-sdk
that referenced
this pull request
Aug 10, 2026
> [!NOTE] > **Stacked on #1402**, which adds the `flyte.io/kind` key this filters on. Review that first. ## Why are the changes needed? #1402 made the kind discriminator writable but not *askable*. `listall` could filter on name, time and source, but not on attrs — so "find all models" still meant listing everything and filtering client-side, which is exactly what that PR declined to build a helper for. unionai/cloud#17546 adds server-side `user_metadata.<key>` filtering. This is the SDK half. ## What changes were proposed in this pull request? **API:** `Artifact.listall(kind=..., attrs=...)`. `attrs` values may be a single string or a sequence. ```python Artifact.listall(kind="model") Artifact.listall(kind="model", attrs={"framework": ["torch", "sklearn"]}) ``` **CLI:** `--kind` and a repeatable `--attr key=value` on `flyte get artifact`. ``` flyte get artifact --kind model flyte get artifact --kind model --attr framework=torch ``` Both compile to `user_metadata.<key>` filters on the existing `ListRequest.filters`, so the work happens **server-side** and paging returns matches rather than scanning. ### Decisions worth reviewing **`kind=` is folded into the attr namespace**, not implemented separately — it's shorthand for the reserved key, and the two combine into one predicate set. One mechanism, not two. **Value grouping follows the server's semantics:** values for one key ride in a *single* filter because the server ORs them; separate keys become *separate* filters because the server ANDs those. Getting this backwards would silently return the wrong set. **An empty sequence sends no filter.** `attrs={"framework": []}` means "no constraint", not "match nothing" — the alternative silently returns zero rows for what reads like an unset filter. **Forward dependency, deliberately loud.** This needs a control plane carrying unionai/cloud#17546. An older one **rejects** the request rather than returning unfiltered results — the right failure mode, since a filter that silently doesn't filter is worse than one that errors. ## How was this patch tested? ``` $ pytest tests/flyte/remote/test_artifact_attr_filter.py 7 passed $ pytest tests/flyte/test_artifact_kind.py tests/flyte/remote/test_artifact.py \ tests/flyte/cli/test_create_artifact.py tests/flyte/test_produces_artifacts.py 82 passed ``` `make fmt` clean, `make mypy` clean (831 source files). The new tests assert **the request that goes on the wire**, not the results: filtering is server-side, so a wrong field name or function returns the wrong set silently rather than failing. They cover `kind=` mapping to the reserved key, single and sequence values, separate keys becoming separate filters, `kind` + `attrs` combining, the empty-sequence case, and no filters by default. **Verified end-to-end** against a local devbox running unionai/cloud#17546, with seeded artifacts: | filter | result | |---|---| | none | 13 artifacts | | `kind=model` | 2 | | `kind=data` | 1 | | `kind in (model,data)` | 3 | | `kind!=model` | 11 | | `kind EXISTS` | 3 | | `kind NOT_EXISTS` | 10 | | `kind=model AND framework=torch` | 1 | That run also caught a **NULL-metadata bug in the backend** — negated filters were dropping every artifact with no metadata at all (`kind!=model` returned 4 instead of 11), since `NULL @> x` is NULL and `WHERE NULL` excludes. Fixed in unionai/cloud#17546. ### Labels - **added** — attr and kind filtering on artifact listing. ## Check all the applicable boxes - [x] I updated the documentation accordingly. *(docstrings on `listall`, including the control-plane requirement; CLI help text)* - [x] All new and existing tests passed. - [x] All commits are signed-off. ## Related PRs - #1402 — the `flyte.io/kind` key (base) - unionai/cloud#17546 — the server-side filter this depends on - flyteorg/flyte#7814 — IDL doc comment describing these filters Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
The comments on
ListArtifactsRequestandListArtifactNamesRequestenumerate which filters the service supports. unionai/cloud is addinguser_metadatafiltering, so leaving it out makes those comments wrong rather than merely incomplete — a caller reading them would conclude the filter doesn't exist.What changes were proposed in this pull request?
Comment-only. No new field is needed:
common.ListRequest.filtersalready carries this, keyed by auser_metadata.<key>field name — the same mechanism the runs service uses forlabels.<key>label filtering, which likewise has no label-specific IDL.Documents the supported functions (
EQUAL,NOT_EQUAL,VALUE_IN,VALUE_NOT_IN,EXISTS,NOT_EXISTS), and that values for one key are ORed while separate keys are ANDed.For
ListArtifactNamesRequestit also states the grouping semantic, which is the non-obvious part: a name is listed when any of its versions matches, represented by the newest matching version.How was this patch tested?
Comment-only — no generated code changes, no behavior change. The filtering itself is implemented and tested in unionai/cloud.
Labels
Check all the applicable boxes