Repository navigation
[Nightshift] Onboard a new source once and give back the extraction unit of a cancelled run - #306588
Conversation
…nit of a cancelled run Continuous onboarding starts a run without a reservation in the source knowledge state, so source reconciliation retried with a 409 until the run ended and then scheduled a second run for the same query. Reconciliation now treats a running or completed run that started at or after the current query revision as the onboarding of that revision. A scheduled onboarding consumes one knowledge indicators extraction unit in its first step and nothing gave it back. Cancelling a source's runs now releases the unit of each run whose budget step was allowed, on the day it was consumed, once per run.
…or runs that did not finish Reconciliation records a running or completed run of the current query as the source's reservation, so a later denied run cannot make it look unscheduled, and it waits for a run it just cancelled. The extraction unit is released only for runs that are going or cancelled, marked released once a unit is given back, and the remembered run ids are capped.
|
Pinging @elastic/nightshift (Team:nightshift) |
This comment has been minimized.
This comment has been minimized.
…atus A run dropped by the concurrency limit ends skipped and never ran, but as the newest execution of a source it made the status API report cancelled for a source whose real run completed. The onboarding status reads and the eligibility check now leave skipped executions out.
…k a release only where it was marked A run that is going may still fail or be cancelled, and reserving it kept the query from being onboarded again, so reconciliation now records a run as the reservation only once it completed. A failed read on a repeat cancel no longer drops the mark of a run an earlier cancel released.
… the cancel after marking the run The workflow engine can throw from a cancel after it already marked the run, for example when the run's wake task is busy. The release sat after the cancels, so the throw skipped it, and the retry no longer listed the run once it had ended cancelled. cancelActiveExecutions now lets every cancel settle and returns the failures, cancelBySource releases first and then rethrows.
This comment has been minimized.
This comment has been minimized.
🧪 E2E: ✅ pass · 5 checks · 4 findings ·
|
💚 Build Succeeded
Metrics [docs]Page load bundle
Unknown metric groupstotal optimizer output size
warm start memory
|
achyutjhunjhunwala
left a comment
There was a problem hiding this comment.
LGTM. I left one comment, but I'm approving to unblock.
A bulk action on the Sources page can start a run that only identifies features or only generates queries, on the same workflow and concurrency key. Reconciliation took any going or completed run of the revision for the onboarding, so a one-step run could be recorded as the reservation and the full onboarding never scheduled. It now reads the run inputs and adopts only a run with both skipFeatures and skipQueries false.
📓 Summary
When the first source is created in a space, onboarding ran twice for it. Continuous onboarding started a run right away, while the source reconciliation workflow for the same source kept asking to schedule one too. It failed with a "waiting for the previous onboarding" conflict and retried for as long as the first run lasted (94 failed steps over 8 minutes in one run). Once the first run finished, reconciliation did not know it had happened, so it started a second onboarding for the unchanged source. With the daily limit already spent, that second run was refused and ended cancelled, and as the newest run it made the Sources page show
-for a source that had been onboarded fine.Separately, a scheduled onboarding takes one knowledge indicators extraction unit from the daily limit when it starts, and nothing gave it back when the run was cancelled. Deleting a source mid-onboarding therefore burned one of the three daily units.
Reconciliation now checks the source's latest onboarding run first. If a run that is going or completed started after the source's current query, that run is the onboarding for it: reconciliation records it and stops, with no conflict and no second run. A cancelled or failed run does not count, and neither does a run it just cancelled itself for an older query. When a source's runs are cancelled, the unit of each run whose budget check was allowed is now given back on the day it was taken, once per run, unless that run finished on its own before the cancel landed.
The same race can also end with the second run dropped by the concurrency limit instead of refused by the budget. A dropped run ends
skippedand never ran, but it was still the newest run of the source, so the status API reportedcanceledfor a source whose real run had completed. The status reads and the continuous onboarding eligibility check now leave skipped runs out, so a dropped duplicate neither hides the run that executed nor counts as an onboarding attempt.🧪 Testing
FROM logs-synth-default.reconcile_sourcesteps. Before, it failed with a 409 until the onboarding finished, then started a second onboarding.GET /internal/significant_events/run_quotas. Theki_extractioncount is one lower than before the delete.