Skip to content

Don't fail in leader mode if job is stopping#1080

Merged
mwylde merged 2 commits into
masterfrom
worker_id_verification
Jun 6, 2026
Merged

Don't fail in leader mode if job is stopping#1080
mwylde merged 2 commits into
masterfrom
worker_id_verification

Conversation

@mwylde

@mwylde mwylde commented Jun 5, 2026

Copy link
Copy Markdown
Member

We've seen several cases now where jobs in leader mode unexpectedly get a stop command, without a log from the controller (or any controller) indicating that it sent such a command. It's not clear how this is happening—the worker id verification should prevent other controllers from talking to the wrong worker.

Currently, this causes the pipeline to fail as we were being intentionally conservative about unexpected cases.

This PR makes two changes:

  • This case is now a retryable error, rather than a fatal error
  • We've added additional logging in the worker id verification interceptor to try to shed a better light on what's happening

Open in Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@mwylde mwylde merged commit 450b20a into master Jun 6, 2026
8 checks passed
@mwylde mwylde deleted the worker_id_verification branch June 6, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants