Skip to content

Show org limit and entitlement refusals as readable, typed errors - #1820

Merged
katrogan merged 4 commits into
mainfrom
katrina/eng26-1545-show-structured-limit-errors-in-the-sdk
Nov 18, 2026
Merged

katrogan merged 4 commits into
mainfrom
katrina/eng26-1545-show-structured-limit-errors-in-the-sdk

Conversation

@katrogan

@katrogan katrogan commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

When Union refuses a request because the org is at one of its limits, or isn't entitled to a capability, the SDK and CLI show the raw status ("RPC error invoking command…", "Permission denied…"). The server attaches a structured detail naming the limit; this decodes it.

What

  • Decodes flyteidl2.org.OrgLimitExceeded and OrgEntitlementRequired (feat(idl): add org limit and entitlement error details flyte#8425) with the generated flyteidl2 classes.
  • A refusal that only carries the older cloudidl.common.OrgLimitExceeded (servers from before the public detail) is still recognised by type name and reported with the server's message, as today. The Rust controller recognises both names.
  • OrgLimitExceededError and ResourceExhaustedError carry org_limit (flyte.errors.OrgLimit: limit, ceiling, current, requested, tier_default, upgrade_url, tier). New OrgEntitlementRequiredError (entitlement, values, upgrade_url). max_concurrent_runs stays a ResourceExhaustedError, so the controller still retries it.
  • An enqueue refused for a missing entitlement fails the child once as a user error the parent can catch, like an org limit, instead of retrying and failing as a system error.
  • Readable messages, e.g.:
    • Org limit max_clusters (4) reached: 4 in use, 1 more requested. The Teams tier default is 2. Contact sales to raise it: https://www.union.ai/get-a-quote
    • Org limit max_concurrent_runs (1000) reached: 1000 in use, 1 more requested. Wait for running work to finish, or contact sales to raise it: …
    • This org isn't entitled to allowed_clouds (coreweave). Contact sales to enable it: …
  • The CLI's shared error handler prints them for every command, plugin commands included.

A refusal without a detail keeps today's message. Servers that don't send tier_default, tier or upgrade_url get the same message without those parts.

Bumps flyteidl2 to 2.0.56, which adds flyteidl2/org/limits.proto (flyteorg/flyte#8425), in pyproject.toml, rs_controller/pyproject.toml and rs_controller/Cargo.toml, with the root, plugin and Cargo.lock lock files.

Testing

  • tests/internal/controllers/test_org_limits.py: decoding every field, a tier default without a tier name, unset vs 0 tier default, the older detail alone and alongside the public one, unreadable detail falls back, pickling, type names match the generated messages.
  • tests/internal/controllers/test_org_limit_exceeded.py: an org limit or entitlement refusal on enqueue fails the child once, as a user error.
  • tests/flyte/test_run_create_errors.py: create_run at max_concurrent_runs and on an entitlement refusal; PERMISSION_DENIED without the detail unchanged.
  • tests/flyte/cli/test_common_errors.py: CLI output for each, with color forced on as in CI.
  • rs_controller: cargo test for both detail names.
  • Full unit suite on flyteidl2 2.0.56, and the affected tests on connectrpc 0.11. ruff, mypy, ty clean.
  • Earlier, against a Union staging deployment sending the detail fields, with an org's max_secrets lowered to 1: flyte create secret printed the readable max_secrets message with the upgrade link. Not exercised against a live server: max_concurrent_runs (the server usually queues a run past the cap rather than refusing it) and entitlement refusals (no server check sends one yet); unit tests cover both.

🤖 Generated with Claude Code

The server attaches a cloudidl.common.OrgLimitExceeded detail when a request
would pass an org limit and an OrgEntitlementRequired detail when the org isn't
entitled to a capability. Decode both (with descriptors of our own, in a private
pool, since the SDK has no cloudidl protos) and:

- OrgLimitExceededError and ResourceExhaustedError carry org_limit (limit,
  ceiling, current, requested, tier_default, upgrade_url); a new
  OrgEntitlementRequiredError carries entitlement, values and upgrade_url.
- Their message names the limit, the numbers and where to raise it, e.g.
  "Org limit max_concurrent_runs (1000) reached: 1000 in use, 1 more
  requested. Wait for running work to finish, or contact Union to raise it:
  https://www.union.ai/contact".
- The CLI prints that message for any command, plugin commands included,
  instead of the raw status or "Permission denied".

A refusal without the detail (an older server) keeps today's message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
msg = f"Org limit {limit.limit} ({limit.ceiling}) reached: {limit.current} in use"
msg += f", {limit.requested} more requested." if limit.requested else "."
if limit.tier_default is not None and limit.tier_default != limit.ceiling:
msg += f" Its tier default is {limit.tier_default}."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to https://github.com/unionai/cloud/pull/20254#discussion_r4604542133. If the value is just an integer, users might not understand what it represents.

raise flyte.errors.ResourceExhaustedError(e.message) from e
if (exhausted := resource_exhausted(e)) is not None:
raise exhausted from e
if e.code == Code.ABORTED:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise new entitlement error here as well?

if (entitlement := org_entitlement_required(e)) is not None:
    raise entitlement from e

…ages

The refusal details are now public flyteidl2.org.OrgLimitExceeded and
OrgEntitlementRequired, so the SDK decodes them with the generated classes
instead of descriptors of its own. A detail that only names the older
cloudidl.common.OrgLimitExceeded type is still a refusal, reported with the
server's message, and the Rust controller recognises both names.

- OrgLimit.tier names the tier tier_default belongs to: "The Teams tier
  default is 2."
- An enqueue refused for a missing entitlement fails the child once as a user
  error, like an org limit, instead of retrying and failing as a system error.
- Messages say "Contact sales" to raise a limit or enable an entitlement.
- The CLI error test strips color, which CI forces on.

Needs a flyteidl2 release that includes flyteidl2/org/limits.proto
(flyteorg/flyte#8425).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
pingsutw
pingsutw previously approved these changes Nov 18, 2026
2.0.56 adds flyteidl2/org/limits.proto (flyteorg/flyte#8425), which the org
limit and entitlement decoding imports. The plugin lock files are re-locked
with uv 0.13.4, as CI checks them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
@katrogan
katrogan merged commit 2e821ef into main Nov 18, 2026
73 checks passed
@katrogan
katrogan deleted the katrina/eng26-1545-show-structured-limit-errors-in-the-sdk branch November 18, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants