Skip to content

feat(dataproxy): mark trigger inputs on UploadInputsRequest - #8397

Merged
AdilFayyaz merged 1 commit into
mainfrom
adil/trigger-inputs-durable
Nov 12, 2026
Merged

AdilFayyaz merged 1 commit into
mainfrom
adil/trigger-inputs-durable

Conversation

@AdilFayyaz

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Inputs registered with a trigger are uploaded once and then read by every run the trigger creates, for as long as the trigger exists. A run's own inputs are only needed for that run. Today both are uploaded with the same UploadInputsRequest, so a server cannot tell them apart and has to store trigger inputs with per-run data, where any retention applied to that data eventually removes them and the trigger's runs start failing to load their inputs.

What changes were proposed in this pull request?

  • Add bool for_trigger = 9 to UploadInputsRequest (flyteidl2/dataproxy/dataproxy_service.proto). A client sets it when the inputs are registered with a trigger, so a server may store them separately from per-run data and keep them for the trigger's lifetime.
  • It is a plain field outside the required id and task oneofs: servers that do not know it ignore it, and clients that do not set it behave as before.
  • Regenerate the Go, Python, TypeScript and Rust code for the dataproxy package.
  • No behavior change in this repo: the bundled data proxy has a single store and does not read the field.

How was this patch tested?

  • buf lint passes and go build ./gen/go/flyteidl2/dataproxy/... succeeds.
  • End to end against a backend that honors the field: an upload with for_trigger set is stored apart from run data and the trigger's scheduled runs read it; an upload without it is stored as before.
  • Against a backend that does not know the field: the request is accepted and handled as before.

Labels

  • added

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 21:04
@AdilFayyaz AdilFayyaz added the added Merged changes that add new functionality label Nov 12, 2026
@AdilFayyaz AdilFayyaz self-assigned this Nov 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The trigger-input marker change has no active review issues.

No open issues 🎉

3 files not reviewed
  • gen/go/flyteidl2/dataproxy/dataproxy_service.pb.go: Generated file
  • gen/go/flyteidl2/dataproxy/dataproxy_service.pb.validate.go: Generated file
  • gen/python/flyteidl2/dataproxy/dataproxy_service_pb2.py: Generated file
What changed in this PR

Adds a trigger-input marker to Flyte’s data proxy API so supporting backends can distinguish long-lived trigger inputs from per-run data. The bundled data proxy’s storage behavior remains unchanged.

Changes:

  • Adds bool for_trigger = 9 outside the existing required oneofs.
  • Documents the marker’s intended retention semantics.
  • Updates generated Go, Python, and TypeScript bindings.
File Description
gen/​ts/​flyteidl2/​dataproxy/​dataproxy_service_pb.ts Adds forTrigger and updates the descriptor.
gen/​python/​flyteidl2/​dataproxy/​dataproxy_service_pb2.pyi Adds field and constructor type declarations.
gen/​python/​flyteidl2/​dataproxy/​dataproxy_service_pb2.py Updates the Python descriptor for the new field.
gen/​go/​flyteidl2/​dataproxy/​dataproxy_service.pb.validate.go Adds a comment noting no validation rules for the marker.
gen/​go/​flyteidl2/​dataproxy/​dataproxy_service.pb.go Adds ForTrigger, its getter, and descriptor metadata.
flyteidl2/​dataproxy/​dataproxy_service.proto Defines and documents the trigger-input marker.

Review effort: Balanced

@AdilFayyaz
AdilFayyaz merged commit 3bc0f4f into main Nov 12, 2026
27 of 30 checks passed
@AdilFayyaz
AdilFayyaz deleted the adil/trigger-inputs-durable branch November 12, 2026 21:57
AdilFayyaz added a commit to flyteorg/flyte-sdk that referenced this pull request Nov 13, 2026
## Motivation
A trigger's fixed inputs are uploaded once when the trigger is deployed,
and every run the trigger creates reads that same upload. The request
gave the backend no way to tell these inputs from the inputs of a single
run, so it stored them with per-run data, where they can be removed
while the trigger is still active. flyteorg/flyte#8397 added
`UploadInputsRequest.for_trigger` so the backend can keep them for as
long as the trigger exists.

## Summary
- `offload_trigger_inputs` sets `for_trigger=True` on
`UploadInputsRequest`. This covers both `flyte deploy` and
`remote.Trigger.create`.
- Bump `flyteidl2` from 2.0.54 to 2.0.55 (the first release with the
field) in `pyproject.toml`, `rs_controller/pyproject.toml`,
`rs_controller/Cargo.toml`, and the lockfiles.
- `rs_controller/src/core.rs`: set the new `Action.override_applied`
field to `false`. 2.0.55 adds the field and the struct literal no longer
compiled without it; `false` keeps the current behaviour.

Backends that do not know the field ignore it, so the change is safe
against older servers. Existing triggers pick up the new behaviour the
next time they are deployed.

## Test Plan
- `tests/flyte/remote/test_trigger_create_offload.py` asserts the
request carries `for_trigger`.
- Full unit suite against the published `flyteidl2==2.0.55`: `make
unit_test`.
- `cargo check` and `cargo fmt --check` in `rs_controller`.
- End to end against a backend that implements the field: deployed a
cron trigger with fixed inputs, confirmed the inputs were stored apart
from per-run data and that scheduled runs read them. Deploying the same
trigger with the previous SDK release kept the old behaviour.

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

added Merged changes that add new functionality flyte2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants