Skip to content

[Nightshift] Onboard a new source once and give back the extraction unit of a cancelled run - #306588

Merged
tonyghiani merged 7 commits into
elastic:mainfrom
tonyghiani:tonyghiani/nightshift-onboarding-dedupe-run-counts
Nov 20, 2026
Merged

tonyghiani merged 7 commits into
elastic:mainfrom
tonyghiani:tonyghiani/nightshift-onboarding-dedupe-run-counts

Conversation

@tonyghiani

@tonyghiani tonyghiani commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

📓 Summary

When the first source is created in a space, onboarding ran twice for it. Continuous onboarding started a run right away, while the source reconciliation workflow for the same source kept asking to schedule one too. It failed with a "waiting for the previous onboarding" conflict and retried for as long as the first run lasted (94 failed steps over 8 minutes in one run). Once the first run finished, reconciliation did not know it had happened, so it started a second onboarding for the unchanged source. With the daily limit already spent, that second run was refused and ended cancelled, and as the newest run it made the Sources page show - for a source that had been onboarded fine.

Separately, a scheduled onboarding takes one knowledge indicators extraction unit from the daily limit when it starts, and nothing gave it back when the run was cancelled. Deleting a source mid-onboarding therefore burned one of the three daily units.

Reconciliation now checks the source's latest onboarding run first. If a run that is going or completed started after the source's current query, that run is the onboarding for it: reconciliation records it and stops, with no conflict and no second run. A cancelled or failed run does not count, and neither does a run it just cancelled itself for an older query. When a source's runs are cancelled, the unit of each run whose budget check was allowed is now given back on the day it was taken, once per run, unless that run finished on its own before the cancel landed.

The same race can also end with the second run dropped by the concurrency limit instead of refused by the budget. A dropped run ends skipped and never ran, but it was still the newest run of the source, so the status API reported canceled for a source whose real run had completed. The status reads and the continuous onboarding eligibility check now leave skipped runs out, so a dropped duplicate neither hides the run that executed nor counts as an onboarding attempt.

🧪 Testing

  1. In a space with no sources and continuous onboarding on, create a source such as FROM logs-synth-default.
  2. Open the workflow executions for that source. There is one onboarding run, and the source reconciliation run ends without failed reconcile_source steps. Before, it failed with a 409 until the onboarding finished, then started a second onboarding.
  3. Create three sources, delete one while its onboarding runs, then read GET /internal/significant_events/run_quotas. The ki_extraction count is one lower than before the delete.

…nit of a cancelled run

Continuous onboarding starts a run without a reservation in the source knowledge state, so source reconciliation retried with a 409 until the run ended and then scheduled a second run for the same query. Reconciliation now treats a running or completed run that started at or after the current query revision as the onboarding of that revision.

A scheduled onboarding consumes one knowledge indicators extraction unit in its first step and nothing gave it back. Cancelling a source's runs now releases the unit of each run whose budget step was allowed, on the day it was consumed, once per run.
…or runs that did not finish

Reconciliation records a running or completed run of the current query as the source's reservation, so a later denied run cannot make it look unscheduled, and it waits for a run it just cancelled. The extraction unit is released only for runs that are going or cancelled, marked released once a unit is given back, and the remembered run ids are capped.
@tonyghiani tonyghiani added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Feature: Nightshift Detection Engine Raw telemetry → Knowledge Indicators → Significant Events → alert Team:nightshift Nightshift team. When scope is known, use sub-teams: sre-agent, context-and-research, investigations Feature: Nightshift labels Nov 19, 2026
@tonyghiani
tonyghiani marked this pull request as ready for review November 20, 2026 06:34
@tonyghiani
tonyghiani requested a review from a team as a code owner November 20, 2026 06:34
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

Pinging @elastic/nightshift (Team:nightshift)

@flash1293

This comment has been minimized.

…atus

A run dropped by the concurrency limit ends skipped and never ran, but as the newest execution of a source it made the status API report cancelled for a source whose real run completed. The onboarding status reads and the eligibility check now leave skipped executions out.
…k a release only where it was marked

A run that is going may still fail or be cancelled, and reserving it kept the query from being onboarded again, so reconciliation now records a run as the reservation only once it completed. A failed read on a repeat cancel no longer drops the mark of a run an earlier cancel released.
… the cancel after marking the run

The workflow engine can throw from a cancel after it already marked the run, for example when the run's wake task is busy. The release sat after the cancels, so the throw skipped it, and the retry no longer listed the run once it had ended cancelled. cancelActiveExecutions now lets every cancel settle and returns the failures, cancelBySource releases first and then rethrows.
@flash1293

This comment has been minimized.

@flash1293

Copy link
Copy Markdown
Contributor
🧪 E2E: ✅ pass · 5 checks · 4 findings · 1d39fb0

The PR works in the running serverless stack. A source created in an empty space got exactly one onboarding run and its reconciliation run finished with no failed steps (0 failed steps across the whole session). Reconciliation recognised an already-running onboarding run for the current query and stopped (debug-log proof), and cancelling a source's run (delete, disable) gave the ki_extraction unit back exactly once while completed or already-cancelled runs gave nothing back. The skipped-run filter and the engine-failure cancel path were not exercised.

Approach: API + ES + Kibana log only (server-side change, no UI). Reset the stack for clean state, ran nightshift-setup, seeded four logs-synth*-default data streams, enabled continuous onboarding, then drove real source create/update/disable/delet … (truncated) · tested 1d39fb0b59 · 2026-11-20 10:39 UTC · openrouter/anthropic/claude-sonnet-5.5 · run locally by nightshift-pr-helper

What the PR changes (tester's view)

Fix two Nightshift onboarding problems: (1) a new source was onboarded twice, because continuous onboarding and the source-reconciliation workflow both started runs (409 loop, then a second run that ended cancelled and hid the real status); reconciliation now checks the latest run first and treats a going/completed run started after the source query's revision as the onboarding. (2) A cancelled scheduled onboarding run never returned its ki_extraction daily unit; cancel now releases it once per run. Status reads and eligibility also ignore skipped runs.

Setup

  • reset-stack (clean ES), then nightshift-setup.mjs
  • PUT /internal/streams/_knowledge_indicators/continuous_ki_extraction/settings {continuousKiExtraction:{enabled:true}}
  • seeded 200 docs each into logs-synth-default, logs-synthb-default, logs-synthc-default, logs-synthd-default (scripts/seed.mjs)
  • temporary [e2e-debug] console.log lines in reconcile_source_catalog.ts (reconcile schedule closure: latest run, revision, match) and onboarding_workflow_client.ts (releaseConsumedExtractions: run id, status, days, alreadyReleased); reverted before the report
  • no mocks needed; real LLM (Elastic Inference) onboarding runs

Checks

✅ First source in an empty space gets exactly one onboarding run; reconciliation has no failed steps

Created synth (FROM logs-synth-default) with continuous onboarding on. Workflow executions for the source: one system-streams-ki-onboarding run (10:21:44 → completed 10:25:56) plus its features/queries child runs, and one system-nightshift-source-reconciliation run (10:21:35 → completed 10:21:45) whose two reconcile_source steps both completed. Failed step executions in the whole index: 0 (the PR describes 94 failed steps over 8 min before). After the onboarding completed, no second onboarding appeared, _status = completed, and a manual POST /_reconcile_source afterwards did not start a run. ki_extraction count = 1.

Evidence

f77ee1d7 system-streams-ki-onboarding completed 10:21:44 10:25:56 nightshift-source-onboarding-synth:25fb86e0...
8c262d16 system-nightshift-source-reconciliation completed 10:21:35 10:21:45
reconcile_source completed 10:21:37 (x2)
failed steps: {}
counts.ki_extraction: 1
✅ Reconcile sees an already-running onboarding run of the current revision and stops (the race from the PR)

Created synthd and immediately (within ~2 s) started an onboarding run with POST /internal/streams/{id}/onboarding/_execute (action schedule), mimicking continuous onboarding starting a run before reconciliation schedules one. The debug line in the reconcile schedule closure shows it found the running run started after the revision and matched. The reconciliation run completed in 7 s with no failed steps (no 409 loop), only ONE onboarding run exists for the source, it completed (10:30:20 → 10:35:10), _status = completed, and no second run followed. Note: the manual _execute run did not consume a ki_extraction unit (count stayed 2), which is existing behaviour of non-scheduled runs, not a PR matter. Without the debug line the early-return branch could not be told apart from the normal path, so this check rests on the temporary [e2e-debug] output.

Evidence

[e2e-debug] reconcile schedule source=synthd latestRun=1d911e84-.../running/2026-11-20T10:30:22.178Z revision=2026-11-20T10:30:18.137Z match=true
b736a1e8 system-nightshift-source-reconciliation completed 10:30:19 10:30:26
1d911e84 system-streams-ki-onboarding completed 10:30:20 10:35:10
✅ Deleting a source mid-onboarding gives its extraction unit back (once)

Created synthb and synthc after synth (ki_extraction = 3 = the daily limit). Deleted synthc while its onboarding was waiting_for_child. GET /run_quotas went 3 → 2 within 8 s and stayed 2 for the next 48 s; the onboarding and features runs ended cancelled. The debug line shows exactly one release for the run (status waiting_for_child, day 2026-11-20, alreadyReleased=false) even though the cancel can list it repeatedly while it winds down.

Evidence

before delete: ki_extraction:3
after: 2,2,2,2,2,2
[e2e-debug] release run=1d391875-... status=waiting_for_child days=["2026-11-20"] alreadyReleased=false
1d391875 system-streams-ki-onboarding cancelled 10:28:17 10:29:11
✅ Query change starts a fresh run; disabling the source mid-run releases that run's unit

After synthb had completed (count 2), updated its ES|QL: a new onboarding run started (b6722fb2) and the count went 2 → 3 (reconcile took the 'previous revision was reset' branch, latestRun undefined, as designed). Then POST /sources/{id}/_disable while it was running: the run ended cancelled, one release logged (day 2026-11-20), count 3 → 2.

Evidence

after PUT: ki_extraction 2,2,3,3,3,3
after _disable: 2,2,2,2,2
[e2e-debug] release run=b6722fb2-... status=waiting_for_child days=["2026-11-20"] alreadyReleased=false
✅ Deleting a source whose runs already completed/cancelled gives nothing back

Deleted synthb (last run already cancelled and released) and synthd (run completed). Count stayed at 2 and no further release debug line / 'Could not release' warning appeared, so a finished unit is not refunded and an already-released run is not released twice.

Evidence

ki_extraction:2 after both deletes; no new [e2e-debug] release lines

Findings

  • 👍 positive: No 409 retry storm and no duplicate onboarding for a new source
    Across four sources and the whole session there were 0 failed workflow steps and never more than one onboarding run per source revision. Linked issue: elastic/nightshift-program#2448 (open, updated 2026-11-19), points 1 and 2 are what this PR fixes; point 3 (pause reports 0 cancelled) was deliberately left out by the author.
  • 📝 note: In the normal (non-racing) create flow reconciliation itself schedules the run
    For synthb/synthc the reconcile closure found no previous run (latestRun undefined, match=false) and went on to schedule the onboarding, which is the unchanged path. The early-return branch only fired when I started a run manually right after creation; I did not manage to make the continuous-onboarding workflow itself win the race (it first ran at 10:21:26, before the first source existed).
  • 📝 note: Not exercised: skipped-run filtering and engine-failure cancel
    A run dropped by the concurrency limit (skipped) could not be produced naturally here, so the statuses filter on status/eligibility reads was only exercised implicitly (status reads worked for completed runs). The 'cancel fails after the engine marked the run' path (fault injection) was not tested.
  • 📝 note: Environment
    The stack had leftovers from other runs (sources, quota counts, no onboarding workflow) so I reset it first. Expected ERROR lines Inferred feature identification failed ... not found / Converse request was aborted appear when a source is deleted/disabled mid-run (cancelled LLM call).

Left in the environment

  • Stack was reset (clean ES) and nightshift-setup re-run
  • continuous onboarding enabled in the default space (ui setting observability:nightshiftContinuousOnboardingEnabled)
  • seeded logs-synth{,b,c,d}-default data streams; source synth still exists (synthb, synthc, synthd were deleted); ki_extraction count is 2 today
  • temporary [e2e-debug] edits in reconcile_source_catalog.ts and onboarding_workflow_client.ts reverted with git checkout and synced to the VM (git status clean); Kibana was not restarted after the revert (the dev watcher reloads on sync)

Not covered

  • skipped run (concurrency-limit drop) being ignored by status/eligibility reads
  • Engine failing the cancel after marking the run (release must still happen) via fault injection
  • Multi-node double release (per-process Set)
  • Continuous-onboarding workflow itself winning the race against reconciliation on a brand-new source
  • Runs that consumed on a previous UTC day (release on the day it was taken)

🔒 Agent traces: 14 conversations, 14 Phoenix traces (zip) · 9.9 MB · internal, Elastic employees only

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
kbnUiSharedDeps-srcJs 5.5MB 5.5MB +1.0B
vendors-heavy 2.0MB 2.0MB +2.0B
total +3.0B
Unknown metric groups

total optimizer output size

id before after diff
all 69.1MB 69.1MB +2.0B

warm start memory

id before after diff
post forced gc heap baseline - 629415290 +629415290
post forced gc heap delta - -2686944 -2686944
post forced gc heap delta standard deviation - 2783046 +2783046
post forced gc heap target - 626728346 +626728346
tail heap delta - -22004501 -22004501
total +1234235237

@achyutjhunjhunwala achyutjhunjhunwala left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I left one comment, but I'm approving to unblock.

A bulk action on the Sources page can start a run that only identifies features or only generates queries, on the same workflow and concurrency key. Reconciliation took any going or completed run of the revision for the onboarding, so a one-step run could be recorded as the reservation and the full onboarding never scheduled. It now reads the run inputs and adopts only a run with both skipFeatures and skipQueries false.
@tonyghiani
tonyghiani enabled auto-merge November 20, 2026 13:28
@tonyghiani
tonyghiani added this pull request to the merge queue Nov 20, 2026
Merged via the queue into elastic:main with commit 19c505f Nov 20, 2026
29 checks passed
@tonyghiani
tonyghiani deleted the tonyghiani/nightshift-onboarding-dedupe-run-counts branch November 20, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting Feature: Nightshift Detection Engine Raw telemetry → Knowledge Indicators → Significant Events → alert Feature: Nightshift release_note:skip Skip the PR/issue when compiling release notes Team:nightshift Nightshift team. When scope is known, use sub-teams: sre-agent, context-and-research, investigations v9.7.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants