Skip to content

fix: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment - #9713

Open
yuehaii wants to merge 11 commits into
envoyproxy:mainfrom
yuehaii:eds-initial_fetch_timeout
Open

fix: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment#9713
yuehaii wants to merge 11 commits into
envoyproxy:mainfrom
yuehaii:eds-initial_fetch_timeout

Conversation

@yuehaii

@yuehaii yuehaii commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
the initial_fetch_timeout: 0s will disable timeout in envoy. however, that setting only applies to LDS and CDS, not to EDS. It caused initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment.

the sds_config can't be added in to dynamic_resources. the config.bootstrap.v3.Bootstrap.DynamicResources only has three fields listed below. the EDS go through xDS discovery.
{
"lds_config": {...},
"cds_config": {...},
"ads_config": {...}
}
This PR set EDS default InitialFetchTimeout as 0 to mitigate the issue.

Which issue(s) this PR fixes:
Fixes #9712


PR Checklist

  • Authorship & ownership: Coding agents / AI assistants are welcome, but I have reviewed every change, understand how and why it works, can explain and maintain it, and take full responsibility for this PR. I have not submitted generated output I do not understand.
  • DCO: All commits are signed off (git commit -s). See DCO: Sign your work.
  • API agreed first: N/A. this PR does not contain API changes.
  • Required checks pass: make generate gen-check, make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, but gen-check, lint, and coverage MUST pass.)
  • Tests added/updated: New/changed code is covered by appropriate tests. N/A if this PR does not contain code changes.
  • Docs: N/A. this PR does not contain user-facing changes.
  • Release notes: For any non-trivial change, added a release-note fragment under release-notes/current/<section>/<pr-number>-<slug>.md (see release-notes/current/README.md for sections and naming). N/A if this PR does not contain non-trivial changes.
  • Generated files committed: Ran make gen-check and committed the result if API/helm charts/modules changed.
  • Scope & compatibility: The PR is reasonably scoped (no unrelated changes) and preserves backward compatibility, or any breaking change is called out above and documented in release-notes/current/breaking_changes/.
  • Codex review: Requested a Codex review and addressed all of its comments.
  • Copilot review: Requested a Copilot review and addressed all of its comments.

@yuehaii
yuehaii requested a review from a team as a code owner August 9, 2026 00:04
@netlify

netlify Bot commented Aug 9, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit 68ef96e
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a7b2ee6247d7e0008a2e04b
😎 Deploy Preview https://deploy-preview-9713--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@yuehaii

yuehaii commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

this is a follow up pr of #9532 (comment), cc @shahar-h

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 224e48b659

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/xds/translator/cluster.go
@arkodg arkodg added this to the v1.9.0 Release milestone Aug 9, 2026
@arkodg

arkodg commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

thanks, can you fix the test yaml as well

Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
@yuehaii
yuehaii force-pushed the eds-initial_fetch_timeout branch from 9dca44b to 281ab1c Compare August 10, 2026 02:08
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.27%. Comparing base (f1d26db) to head (68ef96e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9713      +/-   ##
==========================================
- Coverage   76.32%   76.27%   -0.05%     
==========================================
  Files         261      261              
  Lines       44115    44116       +1     
==========================================
- Hits        33671    33650      -21     
- Misses       8223     8244      +21     
- Partials     2221     2222       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…veHTTP/active_http/health_check_fail … expected status code to be one of [503], got 200"

the initialFetchTimeout config should not has any relation with health check.
adjust BackendHealthCheckActiveHTTP test case to make sure all failCluster endpoints go into unhealthy state before launching "health check fail" test case.

Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
@yuehaii

yuehaii commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

/retest

Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
… with cluster warming up the endpoints. need to verify in envoy proxy.

Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
@yuehaii

yuehaii commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

I am investigating why all conformance and e2e test cases always failed.

those conformance and e2e test cases failure have the same pattern. they wait for health check activity to appear (in prometheus or loki) but the data never shows up within the timeout.

with initialFetchTimeout: 0s, the cluster stays in warming state until EDS delivers endpoints.
when initialFetchTimeout is 0s, no timer fires, so onConfigUpdateFailed is never called. the cluster never exits warming via the timeout path, it must wait for EDS to actually deliver a response.

and there is a feature "envoy.reloadable_features.health_check_after_cluster_warming" declared with RUNTIME_GUARD which means it is enabled by default.

the feature "envoy.reloadable_features.health_check_after_cluster_warming" will be used as gate in setHealthChecker function which is called during cluster setup. If this feature off, the health checker will be start immediately.
this feature gate also used in onInitDone which is called after warming completes. if this feature on, the deferred health checker will be start.

so in those e2e test cases, the health check probes didn't start until onInitDone(), which was blocked by 'initialFetchTimeout: 0s'. since no timer fires onConfigUpdateFailed to exits init warming.

@yuehaii

yuehaii commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

this is an envoy proxy issue. I’ve created a pr and pending approval.

@arkodg arkodg modified the milestones: v1.9.0 Release, Backlog Aug 14, 2026
@kkk777-7 kkk777-7 mentioned this pull request Aug 14, 2026
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment

2 participants