Skip to content

fix: health check probes blocked indefinitely when EDS initialFetchTimeout is 0s - #46667

Open
yuehaii wants to merge 2 commits into
envoyproxy:mainfrom
yuehaii:eds-initialFetchTimeout
Open

fix: health check probes blocked indefinitely when EDS initialFetchTimeout is 0s#46667
yuehaii wants to merge 2 commits into
envoyproxy:mainfrom
yuehaii:eds-initialFetchTimeout

Conversation

@yuehaii

@yuehaii yuehaii commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Commit Message:
when a cluster is configured with EDS and initialFetchTimeout: 0s, the cluster stays in warming state indefinitely until EDS delivers its first response, no timeout fires to unblock it.

Additional Description:
added a bypass in ClusterImplBase::setHealthChecker(). when health_check_after_cluster_warming is enabled, health checks are still started immediately if both conditions hold:

  1. the EDS config source has initialFetchTimeout: 0s (hasZeroInitialFetchTimeout() == true)
  2. there are no pending SDS init targets (init_manager_.uninitializedCount() == 0)

Risk Level:
Low

Testing:
Will be added soon.

Release Notes:
fix health check probes being blocked indefinitely on EDS clusters configured with initialFetchTimeout: 0s. With health_check_after_cluster_warming enabled(default), health checks are no longer deferred when EDS warming has no timeout deadline.

Fixes #46666

…eck_after_cluster_warming is enabled

Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
@yuehaii

yuehaii commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@adisuissa adisuissa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Can you clarify what is the expected behavior in the case where initial-fetch-timeout is 0, no EDS response was received for that cluster (the cluster is still warming), and there is a health-check request?
My understanding is that the health-check should just fail, as with any cluster that is in a warming state, correct?
Please add regression tests to make sure the correct behavior is maintained
/wait

@yuehaii

yuehaii commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Can you clarify what is the expected behavior in the case where initial-fetch-timeout is 0, no EDS response was received for that cluster (the cluster is still warming), and there is a health-check request? My understanding is that the health-check should just fail, as with any cluster that is in a warming state, correct? Please add regression tests to make sure the correct behavior is maintained /wait

with initialFetchTimeout: 0s, the cluster stays in warming state until EDS delivers endpoints.
when initialFetchTimeout is 0s, no timer fires, so onConfigUpdateFailed is never called. the cluster never exits warming via the timeout path, it must wait for EDS to actually deliver a response.

and there is a feature "envoy.reloadable_features.health_check_after_cluster_warming" declared with RUNTIME_GUARD which means it is enabled by default.

the feature "envoy.reloadable_features.health_check_after_cluster_warming" will be used as gate in setHealthChecker function which is called during cluster setup. If this feature off, the health checker will be start immediately.
this feature gate also used in onInitDone which is called after warming completes. if this feature on, the deferred health checker will be start.

so the health check probes didn't start until onInitDone(), which was blocked by 'initialFetchTimeout: 0s'. since no timer fires onConfigUpdateFailed to exits init warming.

I will add a regression test case soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

health check probes blocked indefinitely when EDS initialFetchTimeout is 0s

2 participants