Repository navigation
Show org limit and entitlement refusals as readable, typed errors - #1820
Merged
katrogan merged 4 commits intoNov 18, 2026
Merged
Conversation
The server attaches a cloudidl.common.OrgLimitExceeded detail when a request would pass an org limit and an OrgEntitlementRequired detail when the org isn't entitled to a capability. Decode both (with descriptors of our own, in a private pool, since the SDK has no cloudidl protos) and: - OrgLimitExceededError and ResourceExhaustedError carry org_limit (limit, ceiling, current, requested, tier_default, upgrade_url); a new OrgEntitlementRequiredError carries entitlement, values and upgrade_url. - Their message names the limit, the numbers and where to raise it, e.g. "Org limit max_concurrent_runs (1000) reached: 1000 in use, 1 more requested. Wait for running work to finish, or contact Union to raise it: https://www.union.ai/contact". - The CLI prints that message for any command, plugin commands included, instead of the raw status or "Permission denied". A refusal without the detail (an older server) keeps today's message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
pingsutw
reviewed
Nov 17, 2026
| msg = f"Org limit {limit.limit} ({limit.ceiling}) reached: {limit.current} in use" | ||
| msg += f", {limit.requested} more requested." if limit.requested else "." | ||
| if limit.tier_default is not None and limit.tier_default != limit.ceiling: | ||
| msg += f" Its tier default is {limit.tier_default}." |
Member
There was a problem hiding this comment.
similar to https://github.com/unionai/cloud/pull/20254#discussion_r4604542133. If the value is just an integer, users might not understand what it represents.
pingsutw
reviewed
Nov 17, 2026
| raise flyte.errors.ResourceExhaustedError(e.message) from e | ||
| if (exhausted := resource_exhausted(e)) is not None: | ||
| raise exhausted from e | ||
| if e.code == Code.ABORTED: |
Member
There was a problem hiding this comment.
Should we raise new entitlement error here as well?
if (entitlement := org_entitlement_required(e)) is not None:
raise entitlement from e…ages The refusal details are now public flyteidl2.org.OrgLimitExceeded and OrgEntitlementRequired, so the SDK decodes them with the generated classes instead of descriptors of its own. A detail that only names the older cloudidl.common.OrgLimitExceeded type is still a refusal, reported with the server's message, and the Rust controller recognises both names. - OrgLimit.tier names the tier tier_default belongs to: "The Teams tier default is 2." - An enqueue refused for a missing entitlement fails the child once as a user error, like an org limit, instead of retrying and failing as a system error. - Messages say "Contact sales" to raise a limit or enable an entitlement. - The CLI error test strips color, which CI forces on. Needs a flyteidl2 release that includes flyteidl2/org/limits.proto (flyteorg/flyte#8425). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
pingsutw
previously approved these changes
Nov 18, 2026
2.0.56 adds flyteidl2/org/limits.proto (flyteorg/flyte#8425), which the org limit and entitlement decoding imports. The plugin lock files are re-locked with uv 0.13.4, as CI checks them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
pingsutw
approved these changes
Nov 18, 2026
katrogan
deleted the
katrina/eng26-1545-show-structured-limit-errors-in-the-sdk
branch
November 18, 2026 09:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When Union refuses a request because the org is at one of its limits, or isn't entitled to a capability, the SDK and CLI show the raw status ("RPC error invoking command…", "Permission denied…"). The server attaches a structured detail naming the limit; this decodes it.
What
flyteidl2.org.OrgLimitExceededandOrgEntitlementRequired(feat(idl): add org limit and entitlement error details flyte#8425) with the generatedflyteidl2classes.cloudidl.common.OrgLimitExceeded(servers from before the public detail) is still recognised by type name and reported with the server's message, as today. The Rust controller recognises both names.OrgLimitExceededErrorandResourceExhaustedErrorcarryorg_limit(flyte.errors.OrgLimit:limit,ceiling,current,requested,tier_default,upgrade_url,tier). NewOrgEntitlementRequiredError(entitlement,values,upgrade_url).max_concurrent_runsstays aResourceExhaustedError, so the controller still retries it.Org limit max_clusters (4) reached: 4 in use, 1 more requested. The Teams tier default is 2. Contact sales to raise it: https://www.union.ai/get-a-quoteOrg limit max_concurrent_runs (1000) reached: 1000 in use, 1 more requested. Wait for running work to finish, or contact sales to raise it: …This org isn't entitled to allowed_clouds (coreweave). Contact sales to enable it: …A refusal without a detail keeps today's message. Servers that don't send
tier_default,tierorupgrade_urlget the same message without those parts.Bumps
flyteidl2to 2.0.56, which addsflyteidl2/org/limits.proto(flyteorg/flyte#8425), inpyproject.toml,rs_controller/pyproject.tomlandrs_controller/Cargo.toml, with the root, plugin andCargo.locklock files.Testing
tests/internal/controllers/test_org_limits.py: decoding every field, a tier default without a tier name, unset vs 0 tier default, the older detail alone and alongside the public one, unreadable detail falls back, pickling, type names match the generated messages.tests/internal/controllers/test_org_limit_exceeded.py: an org limit or entitlement refusal on enqueue fails the child once, as a user error.tests/flyte/test_run_create_errors.py:create_runatmax_concurrent_runsand on an entitlement refusal;PERMISSION_DENIEDwithout the detail unchanged.tests/flyte/cli/test_common_errors.py: CLI output for each, with color forced on as in CI.rs_controller:cargo testfor both detail names.max_secretslowered to 1:flyte create secretprinted the readablemax_secretsmessage with the upgrade link. Not exercised against a live server:max_concurrent_runs(the server usually queues a run past the cap rather than refusing it) and entitlement refusals (no server check sends one yet); unit tests cover both.🤖 Generated with Claude Code