Repository navigation
feat(gateways): delete expired Kafka group offsets in Iggy - #4790
Open
krishvishal wants to merge 3 commits into
Open
krishvishal wants to merge 3 commits into
krishvishal wants to merge 3 commits into
Conversation
Add IGGY_KAFKA_OFFSETS_RETENTION_MINUTES and a read check on new groups.
Iggy creates a consumer group for a user who may read the topic's stream or the topic, not only for a global grant.
A gateway with only the Iggy address and password now keeps a default Java producer in order and expires old data.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4790 +/- ##
============================================
- Coverage 90.89% 90.89% -0.01%
Complexity 3878 3878
============================================
Files 1564 1564
Lines 407450 407447 -3
Branches 341308 341305 -3
============================================
- Hits 370368 370342 -26
- Misses 29706 29728 +22
- Partials 7376 7377 +1
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR address?
Closes #4513.
Rationale
A gateway on its defaults hurt stock Kafka clients. It ran in stub mode, where Produce answers a
retriable error and clients retry without end. With the bridge on, idempotent producers were off.
So a default Java producer wrote a retried batch twice, or after later batches. Topics
never expired. A single gateway never expired committed offsets, so a partition filled its 4,096
group offsets over time.
What changed?
A gateway with only the Iggy address and password ran in stub mode and stored nothing. Now the
bridge, idempotent producers and the coordinator log are on by default, also for a single gateway.
A new topic keeps records for 7 days, as in Kafka:
IGGY_KAFKA_BRIDGE_ENABLEDistrue. Stub mode needsfalse, and logs a warning.IGGY_KAFKA_IDEMPOTENT_PRODUCERSistrue.IGGY_KAFKA_TOPIC_RETENTION_MSis604800000.-1keeps records forever.IGGY_KAFKA_COORDINATOR_LOGistruewith the bridge, also withoutIGGY_KAFKA_BROKERS.