Skip to content

[Nightshift][Detection] Stamp discovery detections from written events - #306694

Draft
crespocarlos wants to merge 9 commits into
elastic:mainfrom
crespocarlos:fix/discovery-stamp-from-written-events
Draft

crespocarlos wants to merge 9 commits into
elastic:mainfrom
crespocarlos:fix/discovery-stamp-from-written-events

Conversation

@crespocarlos

Copy link
Copy Markdown
Contributor

closes https://github.com/elastic/nightshift-program/issues/2439

Summary

Discovery stamped detections as processed from the rule uuids the agent copied into written_rule_uuids. About 1.4% of those matched no rule, so those detections went back into the queue each cycle with no error or log line.

A new step, significantEvents.resolveWrittenDetections, now works out the rules to stamp. It takes the batch detections whose events this run wrote, found through data.workflow_execution_id, and adds the agent-reported rules that exist in the batch. Reported rules that are not in the batch are dropped, warned about and sent in the _completed telemetry as unknown_reported_rule_count.

The stamping query and the _mark_processed route are unchanged.

How to test

  • Run discovery on a batch where the agent returns a rule uuid that is in no batch detection. The rule is not stamped and unknown_reported_rule_count is above 0.
  • Run it where the agent omits a rule whose event it wrote. That rule is stamped.
  • Automated: unit tests cover the resolver, the lookup, the route warning and the workflow wiring.

Checklist

Identify risks

  • If the event lookup fails or is truncated, the step falls back to the agent-reported rules, as before this change. Only a log warning records it, because I kept the lookup outcome out of telemetry.
  • resolve_written_detections has on-failure: continue, unlike classify_detections. The agent has already written its events when it runs, so a failure should not skip stamping or the report.
  • written_rule_count now counts rules stamped as processed, not rules the agent reported. The field name is unchanged, so check any dashboards that read it.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor
🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!
  • Click to trigger kibana-static-checks for this PR!

@crespocarlos crespocarlos added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Feature: Nightshift Detection Engine Raw telemetry → Knowledge Indicators → Significant Events → alert Team:nightshift-sre-agent labels Nov 20, 2026
@crespocarlos crespocarlos changed the title fix(significant_events): stamp discovery detections from written events [Nightshift][Detection] Stamp discovery detections from written events Nov 20, 2026
@crespocarlos
crespocarlos marked this pull request as ready for review November 20, 2026 13:09
@crespocarlos
crespocarlos requested a review from a team as a code owner November 20, 2026 13:09
crespocarlos and others added 3 commits November 20, 2026 14:10
…ot YAML literals

Co-authored-by: Cursor <cursoragent@cursor.com>
@crespocarlos
crespocarlos force-pushed the fix/discovery-stamp-from-written-events branch from b6f93cc to 7e06f8f Compare November 20, 2026 13:10
@crespocarlos
crespocarlos requested a review from a team as a code owner November 20, 2026 13:10
@botelastic botelastic Bot added the Team:One Workflow Team label for One Workflow (Workflow automation) label Nov 20, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@flash1293

This comment has been minimized.

@crespocarlos

crespocarlos commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

🧪 E2E: ❌ fail · 6 checks (2 failed) · 4 findings, 1 bug 🐞, 1 concern ⚠️ · 7e06f8f

Tests ran before b4a70c5 landed

… stamp rules

Co-authored-by: Cursor <cursoragent@cursor.com>
… guards

Co-authored-by: Cursor <cursoragent@cursor.com>
@kibanamachine

kibanamachine commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

🌯 Save a burrito

@crespocarlos, if you're still iterating on this PR, consider converting it to draft: 4 CI builds were cancelled in the last few hours before they finished. Drafts don't run CI automatically, and you can comment /ci whenever you want a build.

@crespocarlos
crespocarlos marked this pull request as draft November 20, 2026 14:18
@crespocarlos
crespocarlos marked this pull request as ready for review November 20, 2026 14:18
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

⏳ Build in-progress, with failures

Failed CI Steps

History

… definition

Co-authored-by: Cursor <cursoragent@cursor.com>
@crespocarlos
crespocarlos marked this pull request as draft November 20, 2026 14:38
@crespocarlos

Copy link
Copy Markdown
Contributor Author

/ci

flash1293 pushed a commit to flash1293/pr-helper-screenshots that referenced this pull request Nov 20, 2026
@flash1293

Copy link
Copy Markdown
Contributor
🧪 E2E: ✅ pass · 4 checks · 4 findings · 721adfc

The new significantEvents.resolveWrittenDetections step works in real Discovery runs. In a clean run the detection was stamped (processed_by set) and unknown_reported_rule_count was 0. With an injected agent answer that omitted the real rule and reported a bogus one, the real rule was still stamped, the bogus one was dropped, unknown_reported_rule_count=1 reached _completed, and the route logged the warning. The event-lookup failure fallback was not exercised.

Approach: Two real Discovery runs through the managed detection and discovery workflows with the real LLM (nightshift-discover.mjs --seed). Run 1 is unmodified. Run 2 has a temporary [e2e-fault] that replaces the agent-reported list so the 'agent … (truncated) · tested 721adfc5c3 · 2026-11-20 15:35 UTC · openrouter/anthropic/claude-sonnet-5.5 · run locally by nightshift-pr-helper

What the PR changes (tester's view)

Discovery used to stamp detections as processed from the rule uuids the agent copied into written_rule_uuids. About 1.4% of those matched no rule, so those detections went back into the queue silently. A new workflow step, significantEvents.resolveWrittenDetections, now derives the rules to stamp from events written by this execution (data.workflow_execution_id) plus the agent-reported rules that are in the batch. Unknown reported rules are dropped, warned about, and sent as unknown_reported_rule_count in the _completed report.

Setup

  • Seeded the synthetic claims app with node scripts/seed_sigevents_env.js --clean via nightshift-discover.mjs --seed (twice).
  • [e2e-fault] in discovery.yaml: step compute_written_rule_uuids was forced to written_rule_uuids: ['e2e-fault-bogus-rule-uuid'], then Kibana was restarted. This simulates an agent that omits the real rule and reports an unknown one. Used for run 2 only.
  • No debug lines or mock seams. The LLM, ES and workflow engine were real.

Checks

✅ Run 1 (unmodified): real Discovery run stamps the detection via the new step

Detection and discovery workflows ran with the real agent (.openai-gpt-6-luna-chat_completion). The agent answered written_rule_uuids=[6fdee8d6…]. resolve_written_detections returned stamp_rule_uuids=[6fdee8d6…], derived_count=1, unknown_reported_count=0. _mark_processed stamped the detection, and ES shows processed_by = the discovery execution id. The _completed body had written_rule_count=1 and unknown_reported_rule_count=0.

Evidence

resolve_written_detections output: {"stamp_rule_uuids":["6fdee8d6-…"],"derived_count":1,"unknown_reported_count":0}
_completed body: …"written_rule_count":1,"unknown_reported_rule_count":0…
ES detection: processed_by=15f966fc-50f5-41ab-bf5f-e4777c7cc533
✅ Agent omits a real rule and reports an unknown one (fault-injected agent answer): real rule stamped, unknown dropped and counted

Run 2 used the [e2e-fault] above, so the reported list was only e2e-fault-bogus-rule-uuid. The step still derived the real rule from the event written by this execution (derived_count=1) and counted the bogus rule (unknown_reported_count=1). write_processed_marker stamped the real detection, and ES shows processed_by = the run 2 execution id. _completed carried unknown_reported_rule_count=1. The real agent also wrote an event this run, so the event lookup via data.workflow_execution_id works.

Evidence

step input reported_rule_uuids: ["e2e-fault-bogus-rule-uuid"]
step output: {"stamp_rule_uuids":["4d094cfe-…"],"derived_count":1,"unknown_reported_count":1}
_completed body: …"written_rule_count":1,"unknown_reported_rule_count":1…

Workflows execution view of run 2: the `resolve_written_detections` step input (bogus reported uuid) and output (`derived_count` 1, `unknown_reported_count` 1).
Workflows execution view of run 2: the resolve_written_detections step input (bogus reported uuid) and output (derived_count 1, unknown_reported_count 1).

✅ `_completed` route warns when unknown reported rules arrive

Kibana logged the new warning once, for the run 2 execution id. It fired on the real _completed call made by the workflow.

Evidence

[WARN ][plugins.significantEvents] discovery-completed: 1 reported rule uuids were not in the discovery batch of d8739432-980d-4d4e-9d25-baa49d19c46c
✅ New workflow wiring executes (agent schema step, resolve, stamp-list join)

Both runs completed with status completed. In the execution tree, build_agent_answer_schema, resolve_written_detections, compute_resolved_stamp_rule_uuids and compute_stamp_rule_uuids ran. The fallback branch step (compute_fallback_stamp_rule_uuids) was skipped because the resolve step succeeded. The registered step type was accepted by the managed workflow, shown by 'No validation errors' in the editor.

Execution tree shows the new steps in order after `compute_written_rule_uuids`. Editor reports 'No validation errors'.
Execution tree shows the new steps in order after compute_written_rule_uuids. Editor reports 'No validation errors'.

Findings

  • 👍 positive: Rule stamped from the written event even when the agent did not report it
    With the agent's reported list replaced by a bogus uuid, the real detection was still marked processed. This is the PR's main fix.
  • 👍 positive: Unknown reported rules are counted and warned about
    unknown_reported_rule_count=1 was in the _completed body and the route logged the warning.
  • 📝 note: The real agent did not hallucinate a rule in either run
    The unknown-rule path was reached only through an injected fault ([e2e-fault] in discovery.yaml). The PR also adds an enum of batch rule uuids to the agent's answer schema. I did not check that the model or provider enforces it.
  • 📝 note: Event-lookup failure and truncation fallback not exercised
    I did not fault the ES lookup. The step's on-failure: continue and the fallback to reported rules therefore rest on the PR's unit tests only.

The UI diff only registers the step's public definition. The visible result is the step in the Workflows execution view, shown in the screenshots.

Left in the environment

  • discovery.yaml fault reverted with git checkout -- <file>; Kibana restarted clean afterwards.
  • Seeded sigevents data (source, KIs, rules, alerts, logs, one event, detections) remains in the stack; the latest --seed --clean replaced the first.
  • Screenshots 01 and 04 in shots/ are incidental (workflows list, Nightshift overview).

Not covered

  • Event lookup failing or truncated (MAX_RUN_VERSIONS) and the fallback to reported rules.
  • Batches with several rules where only some are written.
  • resolve_written_detections failing and the compute_fallback_stamp_rule_uuids branch.
  • Dashboards reading written_rule_count, whose meaning changed.

🔒 Agent traces: 11 conversations, 15 Phoenix traces (zip) · 16.1 MB · internal, Elastic employees only

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting Feature: Nightshift Detection Engine Raw telemetry → Knowledge Indicators → Significant Events → alert release_note:skip Skip the PR/issue when compiling release notes Team:nightshift-sre-agent Team:One Workflow Team label for One Workflow (Workflow automation)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants