Repository navigation
Feat/dedicated broker metadata storage refresh - #349
Draft
eduardagarici wants to merge 2 commits into
Draft
eduardagarici wants to merge 2 commits into
eduardagarici wants to merge 2 commits into
Conversation
Add opt-in dedicated metadata PVCs for broker-only KRaft nodes, with fail-closed offline migration from existing data disks, retained backups, and source-disk protection until recovery. Keep controller storage unchanged. Include generated schemas, migration guidance, unit coverage, and a Kafka 3.9.2 smoke test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make enabling brokerConfig.metadataStorage the only required step: the operator now migrates broker-only nodes one at a time and guards the migration lifecycle end to end. Migration orchestration: - Add a one-per-cluster migration gate to handleRollingUpgrade. A metadata migration restart waits until no other opted-in broker is mid-migration and no other broker has offline/out-of-sync replicas, independent of concurrentBrokerRestartCountPerRack and also on the crashed-container path. - Project per-broker metadataStorageState (Ready) into status from a durable PVC annotation. - Defer metadata.log.dir in the live-mounted broker ConfigMap until no broker pod without the metadata volume remains, and refuse to create a metadata-storage pod whose ConfigMap lacks it. Migration script: - Accept version 0 meta.properties (broker.id) and a missing bootstrap.checkpoint from finalized ZooKeeper-to-KRaft brokers, while writing version 1 at the destination. - Log each migration phase. Lifecycle and admission: - Delete the metadata PVC together with data PVCs on broker removal, tolerating NotFound. - Reject data-disk removal until the broker's metadata storage is Ready. - Reject metadataStorage when any data storage is emptyDir/non-PVC. - Share kraft-metadata, metadata.log.dir and migration.broker.kRaftMode constants instead of string literals. Docs: replace the manual one-broker-at-a-time procedure with the automatic rollout and monitoring guidance; require Kafka 3.9+. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds opt-in dedicated KRaft metadata storage for broker-only nodes (
brokerConfig.metadataStorage). The operator also live-migrates existing brokers to it automatically, one broker at a time. Enabling the field in the broker-onlybrokerConfigGroupis the only required step.Today a broker's
__cluster_metadatalog is stored on a data disk (log.dirs). That ties metadata durability and I/O to topic-data disks, and makes data-disk changes risky. With this change, each opted-in broker gets a separate<cluster>-<id>-metadataPVC andmetadata.log.dir=<mountPath>/kafka. The data disks, Cruise Control capacity and topic assignments are not changed.What's included
API and admission
metadataStorage, which reuses theStorageConfigshape. It needs a PVC (pvcSpec) and rejectsemptyDirand block-mode volumes.processRoles: [broker].emptyDirdata storage on brokers that use it, because that data would be lost when the pod is replaced and could not be migrated;Ready.status.brokersState[id].metadataStorageState.Migration (
migrate-broker-metadatainit container)meta.properties. Also accepts version 0 (broker.id) with nobootstrap.checkpoint, as found on brokers already migrated from ZooKeeper to KRaft. The new copy is always written as version 1.Automatic, serialized rollout
handleRollingUpgradeallows at most one migration per cluster at a time, whateverconcurrentBrokerRestartCountPerRackis set to. The next broker waits until the previous one isReadyand no other broker has offline or out-of-sync replicas.metadata.log.diris not added to the live-mounted broker ConfigMap until no broker pod without the metadata volume remains. The operator also refuses to create a metadata-storage pod whose ConfigMap lacks the setting.Lifecycle
Docs
docs/broker-metadata-storage.md: preconditions, the automatic rollout, monitoring commands, failure handling, the fencing limitation and broker removal.config/samples/kraft/simplekafkacluster_kraft.yaml.Requirements and limitations
metadataStorage, a leftover PVC can block reconciliation.Type of Change
Checklist
Checklist notes:
• I left the contributing-guidelines box for you to tick, since only you can confirm you've read them.
• make test passed.
• Lint ( golangci-lint v2.14.0) reported 0 issues in the root module after the final change; the api/ module last passed before the final change, which left api/ untouched.
• make license-header-check last passed before the final round of changes and wasn't re-run.