Keycloakx: Automate minor/major upgrade process in cluster mode

[maarous]

Keycloak does not support minor or major version upgrades while running in cluster mode. Only patch updates are supported.
(Refer to the official Keycloak documentation for more details.)

If you attempt such an upgrade in cluster mode, you may encounter errors due to JGroups version mismatches, for example:

WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,keycloakx-1-2180(v=16.0.8)) JGRP000006: 10.151.254.47:7800: failed accepting connection from peer SSLSocket[hostname=127.0.0.6, port=51749, Session(1776862509121|TLS_AES_256_GCM_SHA384)]: java.io.IOException: 10.151.254.47:7800: readPeerAddress(): packet from /127.0.0.6:51749 has different version (5.3.16) from ours (5.5.1); discarding it

Because of this limitation, downtime is required to perform minor or major version upgrades.

Recommended upgrade procedure:

1. Scale down the StatefulSet to a single replica to disable cluster mode.
2. Perform the version upgrade.
3. Once the upgrade is complete, scale the StatefulSet back to its original number of replicas.

To simplify this process, this pull request introduces a new parameter updateHook that automates these steps.

You can see in this GIF a test of upgrading a three-instance cluster from version 26.5.6 to version 26.6.1.

---

I took this opportunity to upgrade Keycloak to the latest version, 26.6.1.

[dominiquemetz]

Hi! When trying out your feature, my local cluster of 3 Keycloaks scales down, but not up anymore.

helm upgrade --install keycloak helm-charts/charts/keycloakx --values values.yaml --namespace pr898 --set image.tag=26.6.1
level=WARN msg="upgrade failed" name=keycloak error="conflict occurred while applying object pr898/keycloak-keycloakx apps/v1, Kind=StatefulSet: Apply failed with 1 conflict: conflict with \"curl\" with subresource \"scale\" using apps/v1: .spec.replicas"
Error: UPGRADE FAILED: conflict occurred while applying object pr898/keycloak-keycloakx apps/v1, Kind=StatefulSet: Apply failed with 1 conflict: conflict with "curl" with subresource "scale" using apps/v1: .spec.replicas

The curl command modifies and takes ownership of the scale resource:

...
apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:replicas: {}
manager: curl
operation: Update
subresource: scale
...

which lets the helm upgrade command fail. Did you experience the same issues?

[maarous]

Hi! When trying out your feature, my local cluster of 3 Keycloaks scales down, but not up anymore.

helm upgrade --install keycloak helm-charts/charts/keycloakx --values values.yaml --namespace pr898 --set image.tag=26.6.1
level=WARN msg="upgrade failed" name=keycloak error="conflict occurred while applying object pr898/keycloak-keycloakx apps/v1, Kind=StatefulSet: Apply failed with 1 conflict: conflict with \"curl\" with subresource \"scale\" using apps/v1: .spec.replicas"
Error: UPGRADE FAILED: conflict occurred while applying object pr898/keycloak-keycloakx apps/v1, Kind=StatefulSet: Apply failed with 1 conflict: conflict with "curl" with subresource "scale" using apps/v1: .spec.replicas

The curl command modifies and takes ownership of the scale resource:

...
apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:replicas: {}
manager: curl
operation: Update
subresource: scale
...

which lets the helm upgrade command fail. Did you experience the same issues?

Hello @dominiquemetz , thank you for your quick reply.

Unfortunately, I haven’t been able to reproduce the issue on my side.

I tested this setup locally using Minikube and also deployed it across our vanilla Kubernetes clusters in all environments without encountering any problems.

For reference, I retested using:

• Minikube v1.35
• Helm v3.19.4

The issue still did not occur.

Here is the values file I used:

replicas: 3

command:
  - "/opt/keycloak/bin/kc.sh"
  - "start"
  - "--http-port=8080"
  - "--hostname-strict=false"
extraEnv: |
  - name: KEYCLOAK_ADMIN
    value: admin
  - name: KEYCLOAK_ADMIN_PASSWORD
    value: admin
  - name: JAVA_OPTS_APPEND
    value: >-
      -Djgroups.dns.query={{ include "keycloak.fullname" . }}-headless

affinity: ""

updateHook:
  enabled: true
  kubernetesApi:
    url: https://192.168.49.2
    port: 8443
    cidr: 192.168.49.2/32

I’ll attach the results here for reference.
Screencast from 2026-05-01 13-54-04.webm

Out of curiosity, are you using Helm v4?

[maarous]

I think it's better to disable this feature by default (for new cluster set up and for patch update that are supported natively in Keycloak HA).

I will add also some improvements.