Repository navigation
feat(dataproxy): mark trigger inputs on UploadInputsRequest - #8397
Merged
Merged
Conversation
Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The trigger-input marker change has no active review issues.
No open issues 🎉
3 files not reviewed
gen/go/flyteidl2/dataproxy/dataproxy_service.pb.go: Generated filegen/go/flyteidl2/dataproxy/dataproxy_service.pb.validate.go: Generated filegen/python/flyteidl2/dataproxy/dataproxy_service_pb2.py: Generated file
What changed in this PR
Adds a trigger-input marker to Flyte’s data proxy API so supporting backends can distinguish long-lived trigger inputs from per-run data. The bundled data proxy’s storage behavior remains unchanged.
Changes:
- Adds
bool for_trigger = 9outside the existing required oneofs. - Documents the marker’s intended retention semantics.
- Updates generated Go, Python, and TypeScript bindings.
| File | Description |
|---|---|
gen/ts/flyteidl2/dataproxy/dataproxy_service_pb.ts |
Adds forTrigger and updates the descriptor. |
gen/python/flyteidl2/dataproxy/dataproxy_service_pb2.pyi |
Adds field and constructor type declarations. |
gen/python/flyteidl2/dataproxy/dataproxy_service_pb2.py |
Updates the Python descriptor for the new field. |
gen/go/flyteidl2/dataproxy/dataproxy_service.pb.validate.go |
Adds a comment noting no validation rules for the marker. |
gen/go/flyteidl2/dataproxy/dataproxy_service.pb.go |
Adds ForTrigger, its getter, and descriptor metadata. |
flyteidl2/dataproxy/dataproxy_service.proto |
Defines and documents the trigger-input marker. |
Review effort: Balanced
pingsutw
approved these changes
Nov 12, 2026
AdilFayyaz
added a commit
to flyteorg/flyte-sdk
that referenced
this pull request
Nov 13, 2026
## Motivation A trigger's fixed inputs are uploaded once when the trigger is deployed, and every run the trigger creates reads that same upload. The request gave the backend no way to tell these inputs from the inputs of a single run, so it stored them with per-run data, where they can be removed while the trigger is still active. flyteorg/flyte#8397 added `UploadInputsRequest.for_trigger` so the backend can keep them for as long as the trigger exists. ## Summary - `offload_trigger_inputs` sets `for_trigger=True` on `UploadInputsRequest`. This covers both `flyte deploy` and `remote.Trigger.create`. - Bump `flyteidl2` from 2.0.54 to 2.0.55 (the first release with the field) in `pyproject.toml`, `rs_controller/pyproject.toml`, `rs_controller/Cargo.toml`, and the lockfiles. - `rs_controller/src/core.rs`: set the new `Action.override_applied` field to `false`. 2.0.55 adds the field and the struct literal no longer compiled without it; `false` keeps the current behaviour. Backends that do not know the field ignore it, so the change is safe against older servers. Existing triggers pick up the new behaviour the next time they are deployed. ## Test Plan - `tests/flyte/remote/test_trigger_create_offload.py` asserts the request carries `for_trigger`. - Full unit suite against the published `flyteidl2==2.0.55`: `make unit_test`. - `cargo check` and `cargo fmt --check` in `rs_controller`. - End to end against a backend that implements the field: deployed a cron trigger with fixed inputs, confirmed the inputs were stored apart from per-run data and that scheduled runs read them. Deploying the same trigger with the previous SDK release kept the old behaviour. Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Inputs registered with a trigger are uploaded once and then read by every run the trigger creates, for as long as the trigger exists. A run's own inputs are only needed for that run. Today both are uploaded with the same
UploadInputsRequest, so a server cannot tell them apart and has to store trigger inputs with per-run data, where any retention applied to that data eventually removes them and the trigger's runs start failing to load their inputs.What changes were proposed in this pull request?
bool for_trigger = 9toUploadInputsRequest(flyteidl2/dataproxy/dataproxy_service.proto). A client sets it when the inputs are registered with a trigger, so a server may store them separately from per-run data and keep them for the trigger's lifetime.idandtaskoneofs: servers that do not know it ignore it, and clients that do not set it behave as before.How was this patch tested?
buf lintpasses andgo build ./gen/go/flyteidl2/dataproxy/...succeeds.for_triggerset is stored apart from run data and the trigger's scheduled runs read it; an upload without it is stored as before.Labels