fix: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment - #9713
fix: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment#9713yuehaii wants to merge 11 commits into
Conversation
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
this is a follow up pr of #9532 (comment), cc @shahar-h |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 224e48b659
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
thanks, can you fix the test yaml as well |
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
9dca44b to
281ab1c
Compare
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9713 +/- ##
==========================================
- Coverage 76.32% 76.27% -0.05%
==========================================
Files 261 261
Lines 44115 44116 +1
==========================================
- Hits 33671 33650 -21
- Misses 8223 8244 +21
- Partials 2221 2222 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…veHTTP/active_http/health_check_fail … expected status code to be one of [503], got 200" the initialFetchTimeout config should not has any relation with health check. adjust BackendHealthCheckActiveHTTP test case to make sure all failCluster endpoints go into unhealthy state before launching "health check fail" test case. Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
|
/retest |
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
… with cluster warming up the endpoints. need to verify in envoy proxy. Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
|
I am investigating why all conformance and e2e test cases always failed. those conformance and e2e test cases failure have the same pattern. they wait for health check activity to appear (in prometheus or loki) but the data never shows up within the timeout. with initialFetchTimeout: 0s, the cluster stays in warming state until EDS delivers endpoints. and there is a feature "envoy.reloadable_features.health_check_after_cluster_warming" declared with RUNTIME_GUARD which means it is enabled by default. the feature "envoy.reloadable_features.health_check_after_cluster_warming" will be used as gate in setHealthChecker function which is called during cluster setup. If this feature off, the health checker will be start immediately. so in those e2e test cases, the health check probes didn't start until onInitDone(), which was blocked by 'initialFetchTimeout: 0s'. since no timer fires onConfigUpdateFailed to exits init warming. |
|
this is an envoy proxy issue. I’ve created a pr and pending approval. |
What this PR does / why we need it:
the initial_fetch_timeout: 0s will disable timeout in envoy. however, that setting only applies to LDS and CDS, not to EDS. It caused initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment.
the sds_config can't be added in to dynamic_resources. the config.bootstrap.v3.Bootstrap.DynamicResources only has three fields listed below. the EDS go through xDS discovery.
{
"lds_config": {...},
"cds_config": {...},
"ads_config": {...}
}
This PR set EDS default InitialFetchTimeout as 0 to mitigate the issue.
Which issue(s) this PR fixes:
Fixes #9712
PR Checklist
git commit -s). See DCO: Sign your work.make generate gen-check,make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, butgen-check,lint, and coverage MUST pass.)release-notes/current/<section>/<pr-number>-<slug>.md(seerelease-notes/current/README.mdfor sections and naming). N/A if this PR does not contain non-trivial changes.make gen-checkand committed the result if API/helm charts/modules changed.release-notes/current/breaking_changes/.