Skip to content

fix(acp): honour the relay's retry hint on a rate-limited REST call - #5724

Open
Chessing234 wants to merge 2 commits into
block:mainfrom
Chessing234:fix/acp-honour-relay-retry-hint-5557
Open

fix(acp): honour the relay's retry hint on a rate-limited REST call#5724
Chessing234 wants to merge 2 commits into
block:mainfrom
Chessing234:fix/acp-honour-relay-retry-hint-5557

Conversation

@Chessing234

Copy link
Copy Markdown
Contributor

Partial fix for #5557 — one of the two mechanisms the report named, and I want to be clear up front about which.

What I found

The report offered two candidate mechanisms and said it could not tell them apart from outside. From inside the code, "no backoff on retry" is not it: request_with_retry (crates/buzz-acp/src/relay.rs) already sleeps 500ms / 1s / 2s with ±20% jitter between attempts. Sub-millisecond gaps at 20–42/sec cannot come from that loop, so the observed storm is fan-out — many concurrent requests each getting their own 429 — which this PR does not fix.

What it does fix is a real gap on the same path. The relay computes exactly how long a rate-limited principal must wait (enforce_http_admission, crates/buzz-relay/src/api/bridge.rs) and spends it in the 429 body:

{"error":"rate-limited: quota exceeded; retry in 4s"}

The subscription path already reads that hint — parse_rate_limit_retry_secs arms a gate and parks until it clears. The REST path parsed nothing and backed off on its own schedule, which tops out at 2s. Against a 4s hint every retry is a guaranteed second rejection, and a client that re-arrives inside its own window keeps that window alive. That is a self-sustaining component of the loop even if it is not the whole amplifier.

The change

Read the hint off the retriable response with the parser the WebSocket path already uses, and wait at least that long before the next attempt:

  • capped at 30s, so a pathological value cannot park the harness;
  • a shorter hint never shortens the existing backoff — this only ever waits longer;
  • no hint leaves today's schedule byte-for-byte unchanged, so nothing regresses on 502/503/504;
  • the warn line now names the hint when there is one, which also makes the storm legible in a log.

Not in this PR

  • The concurrency cap on in-flight publishes — the mechanism I believe actually produces 20–42 rejections/sec. That is a bigger change to the publish path and deserves its own review; happy to take it next if you agree with the read above.
  • Log rate-limiting for this path (the report's third bullet). One line per rejection is still one line per rejection.

Also worth noting: buzz-cli carries its own private copy of this parser (parse_retry_hint_text in crates/buzz-cli/src/client.rs). I left it alone rather than widen this diff into another crate, but the two could collapse onto a shared helper if you want that.

Verification

cargo fmt --all --check, cargo clippy -p buzz-acp --all-targets (clean), cargo test -p buzz-acp --lib — 778 pass, including 5 new tests. The last one parses the relay's real 429 body end to end into a delay, pinning the wire shape. I did not reproduce the live rate-limit storm, so the claim I can stand behind is the delay choice, not a measured reduction in 429 volume.

Refs #5557

The HTTP bridge path backed off on its own schedule (500ms / 1s / 2s)
and threw away what the relay said. The relay's admission limiter
computes exactly how long the principal must wait and spends it in the
429 body - "rate-limited: quota exceeded; retry in 4s" - so a harness
that retries at 2s is guaranteed another rejection, and re-arriving
early keeps its own limit window alive.

The subscription path already reads that hint via
parse_rate_limit_retry_secs and parks until it clears; the REST path
did not. Read it off the retriable response and wait at least that
long, capped at 30s so a pathological value cannot park the harness.
A shorter hint never shortens the existing backoff, and a response with
no hint keeps today's schedule exactly.

Refs block#5557

Signed-off-by: Taksh <takshkothari09@gmail.com>
Five cases: no hint leaves the schedule alone, a longer hint wins, a
shorter one does not shorten the backoff, a pathological one is capped,
and the relay's actual 429 body parses end to end into a delay.

The last one pins the wire shape against
crates/buzz-relay/src/api/bridge.rs, since the hint arrives as prose
inside a JSON error field.

Refs block#5557

Signed-off-by: Taksh <takshkothari09@gmail.com>
@Chessing234
Chessing234 requested a review from a team as a code owner August 13, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant