Skip to content

fix(worker): fence async operation writes by claim generation#2908

Open
sireika wants to merge 1 commit into
vectorize-io:mainfrom
sireika:fix/fence-async-operation-claims
Open

fix(worker): fence async operation writes by claim generation#2908
sireika wants to merge 1 commit into
vectorize-io:mainfrom
sireika:fix/fence-async-operation-claims

Conversation

@sireika

@sireika sireika commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Closes #2816.

Summary

  • Carry each execution attempt's exact database claim identity, (worker_id, claimed_at), from claim readback through the worker task context.
  • Fence completion, failure, retry, defer, and unknown-task deletion writes against that exact claim.
  • Make dependent side effects conditional on winning the fence: parent aggregation, consolidation webhook enqueue, and the file-conversion-to-retain handoff.
  • Normalize Oracle timestamp strings before binding claimed_at back into compare-and-swap queries.

Why claimed_at

worker_id alone is not a claim generation: a stable worker ID can reclaim an operation while an older coroutine from the same worker is still running. The claim path now reads back the exact database-stored claimed_at value after the claim update and uses it together with worker_id for subsequent compare-and-swap predicates.

This reuses existing columns, so there is no schema migration. It also does not change persisted task payloads, executor signatures, or public APIs. Direct/non-worker call paths retain their existing idempotency guards.

Safety properties

  • A stale attempt cannot complete, fail, retry, defer, or delete a newer claim.
  • Parent aggregation and consolidation webhook delivery are skipped after claim loss.
  • Claimed-owner webhook enqueue is checked and inserted in one transaction while the operation row is locked.
  • File conversion completion and downstream retain creation remain atomic; a stale conversion creates no child task and emits no completion hook.
  • PostgreSQL and Oracle both return the exact stored claim timestamp; Oracle string timestamps are normalized before reuse.

Verification

  • ./scripts/hooks/lint.sh
  • 116 operation-completion and worker tests passed.
  • 115 file-handoff, cancellation-compatibility, and database-adapter tests passed.
  • Post-rebase critical regression run: 12 passed, including a real PostgreSQL same-worker reclaim case.
  • Independent review: no remaining Critical or Important findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fence async_operations completion writes on ownership (compare-and-swap on worker_id + status)

1 participant