Skip to content

[Fleet] Clear stale inputs_for_versions when a package drops its agent version condition - #306187

Merged
juliaElastic merged 13 commits into
elastic:mainfrom
juliaElastic:fix-stale-inputs-for-versions-305367
Nov 20, 2026
Merged

juliaElastic merged 13 commits into
elastic:mainfrom
juliaElastic:fix-stale-inputs-for-versions-305367

Conversation

@juliaElastic

@juliaElastic juliaElastic commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #305367

Summary

When a package upgrade drops conditions.agent.version, the package policy kept the inputs_for_versions cached from the old version. Agents kept receiving the stale per-version inputs, the agent policy stayed flagged has_agent_version_conditions: true, and later edits never reached them.

  • compilePackagePolicyForVersions now clears a non-empty inputs_for_versions when the package has no agent version condition (feature flag still gates it). It returns { cleared }.
  • update() and bulkUpdate() (used by package upgrades) force a full agent version conditions recompute when the cache was cleared. This also self-heals package policies already affected, on their next edit.
  • Stale #9.x variants are removed by the existing reassign and sweep logic once has_agent_version_conditions goes false.

Test plan

  • Unit tests for the clear / no-op / flag-off cases and the bulkUpdate recompute flag
  • FTR test for upgrade to a package without the condition (not run locally, needs the package registry)

🤖 Generated with Claude Code

…t version condition

Fixes elastic#305367

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@juliaElastic juliaElastic added bug Fixes for quality problems that affect the customer experience release_note:fix Team:streams-ui Streams UI team labels Nov 18, 2026
… condition

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@juliaElastic juliaElastic added libra:enable-draft Enable Libra reviews for a draft PR backport:version Backport to applied version labels v9.6.1 v9.5.8 labels Nov 18, 2026
…, strengthen FTR

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…complete FTR vars

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on for test fixture

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rowing

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@juliaElastic
juliaElastic marked this pull request as ready for review November 18, 2026 14:07
@juliaElastic
juliaElastic requested review from a team as code owners November 18, 2026 14:07
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

Pinging @elastic/streams-ui (Team:streams-ui)

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

⏳ Build in-progress, with failures

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #78 / Agent policies fleet_version_specific_policies package upgrade that drops the agent version condition should clear inputs_for_versions and has_agent_version_conditions after upgrading to a package without the condition
  • [job] [logs] FTR Configs #78 / Agent policies fleet_version_specific_policies package upgrade that drops the agent version condition should clear inputs_for_versions and has_agent_version_conditions after upgrading to a package without the condition
  • [job] [logs] FTR Configs #78 / Agent policies fleet_version_specific_policies package upgrade that drops the agent version condition should serve later edits to agents on the base policy
  • [job] [logs] FTR Configs #78 / Agent policies fleet_version_specific_policies package upgrade that drops the agent version condition should serve later edits to agents on the base policy
  • [job] [logs] FTR Configs #91 / Alerting builtin alertTypes long_running_rule long running rule writes event log document for timeout and no alerts for each rule execution that ends in timeout - some executions times out

History

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@@ -2168,6 +2252,9 @@ class PackagePolicyClientImpl implements PackagePolicyClient {
// the secret can be deleted while the old compiled doc is still the latest revision
// fleet-server reads.
await Promise.all([bumpPromise, assetRemovePromise]);
if (clearFailure) {
throw clearFailure;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagged by Claude:

bulkUpdate() throws on a partial clear failure, so the whole bulk upgrade aborts

bulkUpdate() normally reports per-item errors through failedPolicies. This throw skips secret deletion, telemetry and the failedPolicies assembly for the policies that did save. The caller then has no per-item result for any policy in the chunk, including the ones already updated.

Low risk: it needs a saved-object version conflict on one policy between the bulkGet and the bulkUpdate in the helper, so the window is narrow.

Suggestion: Push the failed-clear ids into failedPolicies (and skip the throw) so the route reports them per item. In update(), run the secret cleanup before throwing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a25c9ca. clearStaleInputsForVersions now returns per-policy failures. bulkUpdate() no longer throws: a policy whose clear failed is pushed into failedPolicies (and left out of the success list and post-update callbacks), so the route reports it per item. The agent policy recompute still runs for the caches that did clear, and secret deletion and telemetry run as before.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in 839ced0. Rather than reporting failed clears through failedPolicies, the clear is now best effort: a conflict is logged and bulkUpdate() behaves as before, secrets, telemetry and per-item results included. The next update of the package policy clears the cache again. Surfacing per-policy upgrade failures on the auto-upgrade path is tracked in #306706.

@@ -2691,6 +2798,9 @@ class PackagePolicyClientImpl implements PackagePolicyClient {
// The bump writes new .fleet-policies revisions that no longer reference the old secrets;
// without sequencing, secrets can be deleted while old compiled docs still reference them.
await Promise.all([bumpPromise, removeAssetPromise, installAssetsPromise]);
if (clearFailure) {
throw clearFailure;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagged the same as above

Suggestion: run the secret cleanup before throwing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a25c9ca. In update() the failed-clear error is now thrown after the secret cleanup, right before telemetry and the post-update callbacks, so the old secret is no longer left behind.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in 839ced0. update() no longer throws on a failed clear, so the secret cleanup runs as before.

… secrets before throwing in update

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es, strengthen tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…artial results and read failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A clear that loses a version conflict is logged instead of retried or surfaced. The next update of the package policy finds the cache again and clears it, the same self-healing that fixes already affected policies. Drops the retry, the per-policy failure reporting and the auto-upgrade logging, which move to a follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tabilize upgrade FTR

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@criamico criamico left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! LGTM

…ade FTR test

The one-shot reassignment task scheduled on package policy create could run before the agent existed, and the periodic task is disabled in this config. Wait for the agent to reach its version-specific policy before upgrading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kibanamachine

Copy link
Copy Markdown
Contributor

🌯 Save a burrito

@juliaElastic, if you're still iterating on this PR, consider converting it to draft: 3 CI builds were cancelled in the last few hours before they finished. Drafts don't run CI automatically, and you can comment /ci whenever you want a build.

@juliaElastic
juliaElastic added this pull request to the merge queue Nov 20, 2026
Merged via the queue into elastic:main with commit 7cc2e49 Nov 20, 2026
29 checks passed
@juliaElastic
juliaElastic deleted the fix-stale-inputs-for-versions-305367 branch November 20, 2026 15:01
@kibanamachine

Copy link
Copy Markdown
Contributor

Starting backport for target branches: 9.5, 9.6

https://github.com/elastic/kibana/actions/runs/44759027676

@kibanamachine

Copy link
Copy Markdown
Contributor

💚 All backports created successfully

Status Branch Result
✅ 9.5
✅ 9.6

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels bug Fixes for quality problems that affect the customer experience libra:enable-draft Enable Libra reviews for a draft PR release_note:fix Team:streams-ui Streams UI team v9.5.8 v9.6.1 v9.7.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet] Package upgrade that drops the agent version condition leaves stale inputs_for_versions

4 participants