Skip to content

[9.4](backport #8276) Fix doubled route metrics and missing request logs when OpAMP is enabled - #8294

Merged
samuelvl merged 3 commits into
9.4from
mergify/bp/9.4/pr-8276
Nov 19, 2026
Merged

samuelvl merged 3 commits into
9.4from
mergify/bp/9.4/pr-8276

Conversation

@mergify

@mergify mergify Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

What is the problem this PR solves?

When the OpAMP endpoint is enabled, Fleet Server applies the route limits two times to every request. The active and total route metrics, such as http_server_routes_checkin_active, count each request two times.

How does this PR solve the problem?

Fleet Server now serves the /v1/opamp endpoint from the same API router as the other endpoints, behind the same middleware. Each request goes through the route limits one time so the route metrics count each request one time. Requests that a route limit rejects now show in the request log and in APM with their status code.

OpAMP requests now also go through the request log, APM and the server.limits.max_connections limit like the other endpoints.

How to test this PR locally

Run the following tests:

TEST_RUN=Test_server_RouteMonitoring mage test:unit
TEST_RUN=Test_RouteMonitoring mage test:integration

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

Related issues


This is an automatic backport of pull request #8276 done by [Mergify](https://mergify.com).

…led (#8276)

(cherry picked from commit af0dc1e)

# Conflicts:
#	internal/pkg/api/router.go
#	internal/pkg/api/router_test.go
#	internal/pkg/api/server_test.go
#	internal/pkg/server/fleet_integration_test.go
@mergify
mergify Bot requested a review from a team as a code owner November 19, 2026 10:47
@mergify mergify Bot added backport conflicts There is a conflict in the backported pull request labels Nov 19, 2026
@mergify
mergify Bot requested review from swiatekm and removed request for a team November 19, 2026 10:47
@mergify mergify Bot added the backport label Nov 19, 2026
Copilot AI balanced review requested due to automatic review settings November 19, 2026 10:47
@mergify mergify Bot added the conflicts There is a conflict in the backported pull request label Nov 19, 2026
@mergify

mergify Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Cherry-pick of af0dc1e has failed:

On branch mergify/bp/9.4/pr-8276
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit af0dc1e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1795003967-fix-doubled-route-metrics-and-missing-request-logs-when-opamp-is-enabled.yaml
	modified:   internal/pkg/api/server.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/api/router.go
	both modified:   internal/pkg/api/router_test.go
	both modified:   internal/pkg/api/server_test.go
	both modified:   internal/pkg/server/fleet_integration_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify

mergify Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

This pull request is merged.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved merge conflicts, missing imports, and undefined metric and agent-tag symbols prevent the API and affected test packages from compiling.

5 open issues
What changed in this PR

Backports #8276 to 9.4 to address #8167: duplicate route metrics and missing rejection logs when OpAMP is enabled. The change places OpAMP behind Fleet Server’s shared API middleware.

Changes:

  • Registers /v1/opamp in the main router and removes the outer router’s duplicate limiter.
  • Adds unit and integration coverage for request counts, logging, and rate-limit rejections.
  • Adds a changelog entry describing the fix.
File Description
internal/​pkg/​server/​fleet_integration_test.go Adds route-monitoring integration coverage; also includes incoming agent-tag tests.
internal/​pkg/​api/​server.go Passes the OpAMP handler to the main router instead of wrapping it.
internal/​pkg/​api/​server_test.go Adds route-monitoring unit coverage and an incoming TLS reload test.
internal/​pkg/​api/​router.go Registers OpAMP behind shared middleware; includes a throttle-counting helper.
internal/​pkg/​api/​router_test.go Adds throttle-counter tests and lint-suppression comments.
changelog/​fragments/​1795003967-fix-doubled-route-metrics-and-missing-request-logs-when-opamp-is-enabled.yaml Documents corrected route metrics and rejection logging.

Review effort: Balanced

Comment thread internal/pkg/api/router.go Resolved
Comment thread internal/pkg/api/router.go Resolved
Comment thread internal/pkg/api/router_test.go Resolved
Comment thread internal/pkg/api/server_test.go Resolved
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Dependency review finished without posting an analysis comment.

If you expected a review, push a new commit to this pull request (or close and reopen it) to try again.

Workflow run: https://github.com/elastic/fleet-server/actions/runs/44521343527

@github-actions

Copy link
Copy Markdown
Contributor

TL;DR

Buildkite failed due to unresolved cherry-pick conflict markers committed in backport SHA 7fee4817, causing Go parse errors (not a flaky infra/test issue). Remove conflict markers and keep a single merged version of each affected block.

Remediation

  • Resolve cherry-pick conflicts in:
    • internal/pkg/api/router.go (around @@ -74,6 +79,41 @@)
    • internal/pkg/api/router_test.go (around @@ -104,3 +104,62 @@)
    • internal/pkg/api/server_test.go (imports and around @@ -406,3 +419,271 @@)
    • internal/pkg/server/fleet_integration_test.go (around @@ -2014,3 +2015,419 @@)
  • Remove all <<<<<<<, =======, >>>>>>> ... markers and keep valid Go syntax/imports.
  • Re-run: mage check:all and release packaging (or re-run Buildkite pipeline).
Investigation details

Root Cause

The backport commit message itself reports cherry-pick conflicts, and the patch for SHA 7fee48171463bb7ac9964049e7ff1904d81eec54 includes literal conflict markers added to source files. These markers break parsing, which cascades into all compile/package/FIPS steps.

Evidence

  • Build: https://buildkite.com/elastic/fleet-server/builds/18169
  • Failed jobs: check-ci, detect-fips-crypto-imports, package x86_64/aarch64 (+ FIPS)
  • Key log excerpts:
    • internal/pkg/api/server_test.go:659:4: expected declaration, found ctx
    • internal/pkg/api/server_test.go:689:92: illegal character U+0023 '#'
    • internal/pkg/server/fleet_integration_test.go:2018:1: expected declaration, found '<<'
    • internal/pkg/server/fleet_integration_test.go:2433:1: expected declaration, found '>>'
    • internal/pkg/api/server_test.go:16:1: missing import path
  • Commit metadata for 7fee4817 contains:
    • # Conflicts:
      • internal/pkg/api/router.go
      • internal/pkg/api/router_test.go
      • internal/pkg/api/server_test.go
      • internal/pkg/server/fleet_integration_test.go
  • Commit patch hunks show inserted markers in those files (+<<<<<<< HEAD, +=======, +>>>>>>> ...).

Verification

  • Local rerun was not executed because this workspace is at af0dc1e (clean upstream commit), while failures are on backport SHA 7fee4817.
  • Analysis is based on Buildkite failure logs plus the failing commit patch content.

Follow-up

  • After resolving markers, if any job still fails, share the new failing log section and I can do a second-pass root-cause check.

What is this? | From workflow: Observability Agentic Workflow — Status

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

9.4 serves OpAMP through a second router and does not have throttleWithCount. Keep the single-router OpAMP handler and drop the connection-rejection counter and the unrelated TLS reload test that the cherry-pick pulled in.

Drop the unused staticcheck nolint so the touched integration test file passes lint.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the OpAMP routing and shared-middleware changes.

No open issues 🎉

5 resolved since last review

Review effort: Balanced

…egration test

Lint checks the whole file because this backport changes it. NewClient and Config are deprecated in the v9 client.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the OpAMP routing and shared-middleware changes.

No open issues 🎉

Review effort: Balanced

@samuelvl
samuelvl merged commit 5aaa5fb into 9.4 Nov 19, 2026
31 checks passed
@samuelvl
samuelvl deleted the mergify/bp/9.4/pr-8276 branch November 19, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport bug Something isn't working conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants