Title: [kafka_consumer] Bound in-flight admin requests (chunking and/or per-request timeout) to avoid mass simultaneous expiry
Type: Feature request
Summary:
kafka_consumer issues one AdminClient.list_consumer_group_offsets per consumer group over a single shared AdminClient, collecting all futures before awaiting any of them. The only deadline is socket.timeout.ms (from kafka_timeout); no per-request request_timeout is passed. On a large cluster, N requests are enqueued within ~1 second and all expire on the same clock, and that mass simultaneous timeout expiry triggers the librdkafka use-after-free on the shared admin coordinator path (librdkafka #4605 / fix PR #5397), aborting the whole Agent process.
Request:
Add a way to bound the fan-out, either by chunking the in-flight admin requests (a batch or concurrency limit) or by passing a staggered per-request timeout so they cannot all expire in the same tick, or both. Today there is no batch-size, concurrency, or throttle knob exposed, so the only workaround is an explicit consumer_groups list with monitor_unlisted_consumer_groups: false.
Impact:
This removes the crash trigger on large clusters without waiting for the upstream librdkafka fix (which is unmerged, and no released librdkafka / confluent-kafka-python carries it as of 2.15.0). Combined with the regex-before-fetch bug fix, it would let customers monitor large consumer-group counts safely.
Environment:
kafka_consumer 6.13.0, Agent 7.76.3, confluent-kafka-python / librdkafka 2.11.1, Kafka 3.9.0 (Strimzi), ~9,100 consumer groups.
References:
ZD 2980971, AGENT-16802. Upstream: librdkafka #4605, PR #5397. Adjacent fan-out issues: integrations-core #18983, #19564, #858.
Title: [kafka_consumer] Bound in-flight admin requests (chunking and/or per-request timeout) to avoid mass simultaneous expiry
Type: Feature request
Summary:
kafka_consumer issues one AdminClient.list_consumer_group_offsets per consumer group over a single shared AdminClient, collecting all futures before awaiting any of them. The only deadline is socket.timeout.ms (from kafka_timeout); no per-request request_timeout is passed. On a large cluster, N requests are enqueued within ~1 second and all expire on the same clock, and that mass simultaneous timeout expiry triggers the librdkafka use-after-free on the shared admin coordinator path (librdkafka #4605 / fix PR #5397), aborting the whole Agent process.
Request:
Add a way to bound the fan-out, either by chunking the in-flight admin requests (a batch or concurrency limit) or by passing a staggered per-request timeout so they cannot all expire in the same tick, or both. Today there is no batch-size, concurrency, or throttle knob exposed, so the only workaround is an explicit consumer_groups list with monitor_unlisted_consumer_groups: false.
Impact:
This removes the crash trigger on large clusters without waiting for the upstream librdkafka fix (which is unmerged, and no released librdkafka / confluent-kafka-python carries it as of 2.15.0). Combined with the regex-before-fetch bug fix, it would let customers monitor large consumer-group counts safely.
Environment:
kafka_consumer 6.13.0, Agent 7.76.3, confluent-kafka-python / librdkafka 2.11.1, Kafka 3.9.0 (Strimzi), ~9,100 consumer groups.
References:
ZD 2980971, AGENT-16802. Upstream: librdkafka #4605, PR #5397. Adjacent fan-out issues: integrations-core #18983, #19564, #858.