Skip to content

fix: stop the status e2e tests demanding their old node ports back - #460

Merged
nic-6443 merged 1 commit into
masterfrom
fix/e2e-nodeport-realloc-race-20260812
Aug 12, 2026
Merged

fix: stop the status e2e tests demanding their old node ports back#460
nic-6443 merged 1 commit into
masterfrom
fix/e2e-nodeport-realloc-race-20260812

Conversation

@jarvis9443

@jarvis9443 jarvis9443 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Type of change:

  • CI/CD or Tests

What this PR does / why we need it:

The dataplane unavailable spec is flaky, and it fails like this:

[FAIL] Test Gateway API Status Test HTTPRoute Sync Status [It] dataplane unavailable
test/e2e/gatewayapi/status.go:161

[FAILED] creating service
Unexpected error:
  The Service "apisix-standalone" is invalid: spec.ports[6].nodePort:
  Invalid value: 32746: provided port is already allocated

The spec captures the dataplane Service's spec, flips the Service to ExternalName so the route sync fails, checks the SyncFailed status, then restores the captured spec. Switching to ExternalName releases the node ports the Service held, but the captured spec still names them — so the restore demands those exact port numbers back. The e2e suite runs many namespaces in parallel, so by then another Service may have been allocated one of them, and the restore is rejected.

That race is why it passes most of the time rather than never. The fix is to clear NodePort before restoring and let the API server allocate whatever is free; these tests reach the dataplane through a port-forward tunnel, so the actual port numbers are never depended on.

test/e2e/crds/v2/status.go has the same capture-and-restore, so it gets the same fix.

No new test cases: this is a fix to existing tests, and the failure is a scheduling race that isn't worth reproducing deterministically.

Pre-submission checklist:

  • Did you explain what problem does this PR solve? Or what new features have been added?
  • Have you added corresponding test cases?
  • Have you modified the corresponding document?
  • Is this PR backward compatible?

Ports apache/apisix-ingress-controller#2845, which carries the same change upstream.

@jarvis9443
fix: stop the status e2e tests demanding their old node ports back
011428f 
"dataplane unavailable" in both test/e2e/gatewayapi/status.go and
test/e2e/crds/v2/status.go flips the dataplane Service to ExternalName to
make the route sync fail, then restores the spec it captured beforehand.

Switching to ExternalName releases the node ports the Service held. The
captured spec still names them, so the restore asks for those exact
numbers back — and the e2e suite runs many namespaces in parallel, so
another Service may have been allocated one of them in the meantime:

  The Service "apisix-standalone" is invalid: spec.ports[6].nodePort:
  Invalid value: 32746: provided port is already allocated

Whether it fails depends on that race, which is why the specs pass most
of the time. Clear NodePort on restore and let the API server allocate;
these tests reach the dataplane through a port-forward tunnel, so the
port numbers are never depended on.

Ports apache/apisix-ingress-controller#2845.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b335c064-c618-4da8-8376-2122bb11286b

📥 Commits

Reviewing files that changed from the base of the PR and between aca9c49 and 011428f.

📒 Files selected for processing (2)
  • test/e2e/crds/v2/status.go
  • test/e2e/gatewayapi/status.go

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

conformance test report - apisix-standalone mode

apiVersion: gateway.networking.k8s.io/v1
date: "2026-08-12T07:33:02Z"
gatewayAPIChannel: experimental
gatewayAPIVersion: v1.6.0
implementation:
  contact:
  - https://github.com/apache/apisix-ingress-controller/issues
  organization: APISIX
  project: apisix-ingress-controller
  url: https://github.com/apache/apisix-ingress-controller.git
  version: v2.0.0
kind: ConformanceReport
mode: default
profiles:
- core:
    result: partial
    skippedTests:
    - TLSRouteHostnameIntersection
    - TLSRouteInvalidBackendRefNonexistent
    - TLSRouteInvalidBackendRefUnknownKind
    - TLSRouteSimpleSameNamespace
    statistics:
      Failed: 0
      Passed: 16
      Skipped: 4
  extended:
    result: partial
    skippedTests:
    - TLSRouteTerminateSimpleSameNamespace
    statistics:
      Failed: 0
      Passed: 3
      Skipped: 1
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - TLSRouteModeTerminate
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
    - TLSRouteModeMixed
  name: GATEWAY-TLS
  summary: Core tests partially succeeded with 4 test skips. Extended tests partially
    succeeded with 1 test skips.
- core:
    result: partial
    skippedTests:
    - HTTPRouteHTTPSListener
    - HTTPRouteInvalidBackendRefUnknownKind
    - HTTPRouteInvalidCrossNamespaceBackendRef
    - HTTPRouteInvalidNonExistentBackendRef
    - HTTPRouteListenerHostnameMatching
    - HTTPRouteMultipleGateways
    - HTTPRouteNoBackendRefs
    statistics:
      Failed: 0
      Passed: 30
      Skipped: 7
  extended:
    result: partial
    skippedTests:
    - HTTPRouteRedirectPortAndScheme
    statistics:
      Failed: 0
      Passed: 12
      Skipped: 1
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - HTTPRouteBackendProtocolWebSocket
    - HTTPRouteDestinationPortMatching
    - HTTPRouteHostRewrite
    - HTTPRouteMethodMatching
    - HTTPRoutePathRewrite
    - HTTPRoutePortRedirect
    - HTTPRouteQueryParamMatching
    - HTTPRouteRequestMirror
    - HTTPRouteResponseHeaderModification
    - HTTPRouteSchemeRedirect
    unsupportedFeatures:
    - BackendTLSPolicy
    - BackendTLSPolicySANValidation
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - HTTPRoute303RedirectStatusCode
    - HTTPRoute307RedirectStatusCode
    - HTTPRoute308RedirectStatusCode
    - HTTPRouteBackendProtocolH2C
    - HTTPRouteBackendRequestHeaderModification
    - HTTPRouteBackendTimeout
    - HTTPRouteCORS
    - HTTPRouteNamedRouteRule
    - HTTPRouteParentRefPort
    - HTTPRoutePathRedirect
    - HTTPRouteRequestMultipleMirrors
    - HTTPRouteRequestPercentageMirror
    - HTTPRouteRequestTimeout
    - HTTPRouteRetry
    - HTTPRouteRetryBackendTimeout
    - HTTPRouteRetryConnectionError
    - ListenerSet
  name: GATEWAY-HTTP
  summary: Core tests partially succeeded with 7 test skips. Extended tests partially
    succeeded with 1 test skips.
- core:
    result: partial
    skippedTests:
    - GRPCRouteListenerHostnameMatching
    statistics:
      Failed: 0
      Passed: 14
      Skipped: 1
  extended:
    result: success
    statistics:
      Failed: 0
      Passed: 1
      Skipped: 0
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
  name: GATEWAY-GRPC
  summary: Core tests partially succeeded with 1 test skips. Extended tests succeeded.
succeededProvisionalTests:
- GatewayOptionalAddressValue

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

conformance test report - apisix mode

apiVersion: gateway.networking.k8s.io/v1
date: "2026-08-12T07:33:27Z"
gatewayAPIChannel: experimental
gatewayAPIVersion: v1.6.0
implementation:
  contact:
  - https://github.com/apache/apisix-ingress-controller/issues
  organization: APISIX
  project: apisix-ingress-controller
  url: https://github.com/apache/apisix-ingress-controller.git
  version: v2.0.0
kind: ConformanceReport
mode: default
profiles:
- core:
    result: partial
    skippedTests:
    - TLSRouteHostnameIntersection
    - TLSRouteInvalidBackendRefNonexistent
    - TLSRouteInvalidBackendRefUnknownKind
    - TLSRouteSimpleSameNamespace
    statistics:
      Failed: 0
      Passed: 16
      Skipped: 4
  extended:
    result: partial
    skippedTests:
    - TLSRouteTerminateSimpleSameNamespace
    statistics:
      Failed: 0
      Passed: 3
      Skipped: 1
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - TLSRouteModeTerminate
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
    - TLSRouteModeMixed
  name: GATEWAY-TLS
  summary: Core tests partially succeeded with 4 test skips. Extended tests partially
    succeeded with 1 test skips.
- core:
    result: partial
    skippedTests:
    - HTTPRouteHTTPSListener
    - HTTPRouteInvalidBackendRefUnknownKind
    - HTTPRouteInvalidCrossNamespaceBackendRef
    - HTTPRouteInvalidNonExistentBackendRef
    - HTTPRouteListenerHostnameMatching
    - HTTPRouteMultipleGateways
    - HTTPRouteNoBackendRefs
    statistics:
      Failed: 0
      Passed: 30
      Skipped: 7
  extended:
    result: partial
    skippedTests:
    - HTTPRouteRedirectPortAndScheme
    statistics:
      Failed: 0
      Passed: 12
      Skipped: 1
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - HTTPRouteBackendProtocolWebSocket
    - HTTPRouteDestinationPortMatching
    - HTTPRouteHostRewrite
    - HTTPRouteMethodMatching
    - HTTPRoutePathRewrite
    - HTTPRoutePortRedirect
    - HTTPRouteQueryParamMatching
    - HTTPRouteRequestMirror
    - HTTPRouteResponseHeaderModification
    - HTTPRouteSchemeRedirect
    unsupportedFeatures:
    - BackendTLSPolicy
    - BackendTLSPolicySANValidation
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - HTTPRoute303RedirectStatusCode
    - HTTPRoute307RedirectStatusCode
    - HTTPRoute308RedirectStatusCode
    - HTTPRouteBackendProtocolH2C
    - HTTPRouteBackendRequestHeaderModification
    - HTTPRouteBackendTimeout
    - HTTPRouteCORS
    - HTTPRouteNamedRouteRule
    - HTTPRouteParentRefPort
    - HTTPRoutePathRedirect
    - HTTPRouteRequestMultipleMirrors
    - HTTPRouteRequestPercentageMirror
    - HTTPRouteRequestTimeout
    - HTTPRouteRetry
    - HTTPRouteRetryBackendTimeout
    - HTTPRouteRetryConnectionError
    - ListenerSet
  name: GATEWAY-HTTP
  summary: Core tests partially succeeded with 7 test skips. Extended tests partially
    succeeded with 1 test skips.
- core:
    result: partial
    skippedTests:
    - GRPCRouteListenerHostnameMatching
    statistics:
      Failed: 0
      Passed: 14
      Skipped: 1
  extended:
    result: success
    statistics:
      Failed: 0
      Passed: 1
      Skipped: 0
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
  name: GATEWAY-GRPC
  summary: Core tests partially succeeded with 1 test skips. Extended tests succeeded.
succeededProvisionalTests:
- GatewayOptionalAddressValue

@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

conformance test report

apiVersion: gateway.networking.k8s.io/v1
date: "2026-08-12T07:52:32Z"
gatewayAPIChannel: experimental
gatewayAPIVersion: v1.6.0
implementation:
  contact:
  - https://github.com/apache/apisix-ingress-controller/issues
  organization: APISIX
  project: apisix-ingress-controller
  url: https://github.com/apache/apisix-ingress-controller.git
  version: v2.0.0
kind: ConformanceReport
mode: default
profiles:
- core:
    failedTests:
    - GatewayModifyListeners
    result: failure
    statistics:
      Failed: 1
      Passed: 14
      Skipped: 0
  extended:
    result: success
    statistics:
      Failed: 0
      Passed: 1
      Skipped: 0
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
  name: GATEWAY-GRPC
  summary: Core tests failed with 1 test failures. Extended tests succeeded.
- core:
    failedTests:
    - GatewayModifyListeners
    - TLSRouteHostnameIntersection
    - TLSRouteInvalidBackendRefNonexistent
    - TLSRouteInvalidBackendRefUnknownKind
    - TLSRouteSimpleSameNamespace
    result: failure
    statistics:
      Failed: 5
      Passed: 15
      Skipped: 0
  extended:
    failedTests:
    - TLSRouteTerminateSimpleSameNamespace
    result: failure
    statistics:
      Failed: 1
      Passed: 3
      Skipped: 0
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - TLSRouteModeTerminate
    unsupportedFeatures:
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - ListenerSet
    - TLSRouteModeMixed
  name: GATEWAY-TLS
  summary: Core tests failed with 5 test failures. Extended tests failed with 1 test
    failures.
- core:
    failedTests:
    - GatewayModifyListeners
    - HTTPRouteMultipleGateways
    - HTTPRouteNoBackendRefs
    result: failure
    skippedTests:
    - HTTPRouteHTTPSListener
    statistics:
      Failed: 3
      Passed: 33
      Skipped: 1
  extended:
    result: partial
    skippedTests:
    - HTTPRouteRedirectPortAndScheme
    statistics:
      Failed: 0
      Passed: 12
      Skipped: 1
    supportedFeatures:
    - GatewayAddressEmpty
    - GatewayPort8080
    - HTTPRouteBackendProtocolWebSocket
    - HTTPRouteDestinationPortMatching
    - HTTPRouteHostRewrite
    - HTTPRouteMethodMatching
    - HTTPRoutePathRewrite
    - HTTPRoutePortRedirect
    - HTTPRouteQueryParamMatching
    - HTTPRouteRequestMirror
    - HTTPRouteResponseHeaderModification
    - HTTPRouteSchemeRedirect
    unsupportedFeatures:
    - BackendTLSPolicy
    - BackendTLSPolicySANValidation
    - GatewayBackendClientCertificate
    - GatewayFrontendClientCertificateValidation
    - GatewayFrontendClientCertificateValidationInsecureFallback
    - GatewayHTTPListenerIsolation
    - GatewayHTTPSListenerDetectMisdirectedRequests
    - GatewayInfrastructurePropagation
    - GatewayStaticAddresses
    - HTTPRoute303RedirectStatusCode
    - HTTPRoute307RedirectStatusCode
    - HTTPRoute308RedirectStatusCode
    - HTTPRouteBackendProtocolH2C
    - HTTPRouteBackendRequestHeaderModification
    - HTTPRouteBackendTimeout
    - HTTPRouteCORS
    - HTTPRouteNamedRouteRule
    - HTTPRouteParentRefPort
    - HTTPRoutePathRedirect
    - HTTPRouteRequestMultipleMirrors
    - HTTPRouteRequestPercentageMirror
    - HTTPRouteRequestTimeout
    - HTTPRouteRetry
    - HTTPRouteRetryBackendTimeout
    - HTTPRouteRetryConnectionError
    - ListenerSet
  name: GATEWAY-HTTP
  summary: Core tests failed with 3 test failures. Extended tests partially succeeded
    with 1 test skips.
succeededProvisionalTests:
- GatewayOptionalAddressValue

@jarvis9443

jarvis9443 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

More evidence for this one: #459 (an unrelated dependency bump) just hit the same flake in CI, this time at the crds/v2 site rather than the Gateway API one:

[FAIL] Test apisix.apache.org/v2 Status Test ApisixRoute Sync Status [It] dataplane unavailable
[FAILED] creating service
  The Service "apisix-standalone" is invalid: spec.ports[5].nodePort:
  Invalid value: 32293: provided port is already allocated

Same root cause, different port and different port index — which is what you'd expect from a scheduling race rather than a fixed conflict. So both call sites do get hit in practice.

@jarvis9443 jarvis9443 mentioned this pull request Aug 12, 2026
Merged
5 tasks done
@nic-6443
nic-6443 merged commit 4bf9729 into master Aug 12, 2026
22 of 23 checks passed
@nic-6443
nic-6443 deleted the fix/e2e-nodeport-realloc-race-20260812 branch August 12, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shreemaan-abhishek shreemaan-abhishek shreemaan-abhishek approved these changes
@AlinsRan AlinsRan AlinsRan approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jarvis9443 @shreemaan-abhishek @AlinsRan @nic-6443