v36.0.0 regression: exec WebSocket returns 401 Unauthorized on AKS (works in v35.0.0)

[GK-07]

What happened?

After upgrading the kubernetes Python client from 35.0.0 to 36.0.0, pod exec via kubernetes.stream.stream() consistently fails against an Azure AKS cluster with 401 Unauthorized during the WebSocket handshake. Regular REST API calls (v1.list_namespaced_pod, v1.read_namespaced_pod, etc.) continue to work with the same kubeconfig and same identity. Downgrading the library to 35.0.0 immediately restores exec functionality with no other changes.

What did you expect to happen?

stream(v1.connect_get_namespaced_pod_exec, ...) should succeed in v36.0.0 the same way it does in v35.0.0, against the same AKS cluster, with the same kubeconfig.

How can we reproduce it (as minimally and precisely as possible)?

1. AKS cluster (Private Link), kubectl exec works fine from the same host.
2. kubeconfig uses an exec: auth plugin (kubelogin with Azure AD).
3. Run the following Python script with kubernetes==36.0.0:

from kubernetes import client, config
from kubernetes.stream import stream

cfg = client.Configuration()
cfg.verify_ssl = False
config.load_kube_config(config_file="/path/to/aks/kubeconfig", client_configuration=cfg)
cfg.proxy = None
v1 = client.CoreV1Api(api_client=client.ApiClient(cfg))

# REST: works in both v35 and v36
print([p.metadata.name for p in v1.list_namespaced_pod("vault").items])

# Exec WebSocket: works in v35, fails in v36 with 401
resp = stream(
    v1.connect_get_namespaced_pod_exec,
    "hashicorp-vault-0", "vault",
    command=["/bin/sh", "-c", "vault status"],
    stderr=True, stdin=False, stdout=True, tty=False,
)
print(resp)

Anything else we need to know?

The traceback from v36.0.0:

File ".../kubernetes/stream/ws_client.py", line 552, in create_websocket
    websocket.connect(url, **connect_opt)
File ".../websocket/_handshake.py", line 158, in _get_resp_headers
    raise WebSocketBadStatusException(
websocket._exceptions.WebSocketBadStatusException: Handshake status 401 Unauthorized -+-+-
{'audit-id': '...', 'cache-control': 'no-cache, private', 'content-type': 'application/json',
 'date': '...', 'content-length': '129', 'connection': 'close'}
-+-+- b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure",
        "message":"Unauthorized","reason":"Unauthorized","code":401}\n'

The 401 response carries an audit-id, confirming the request reaches the AKS API server and is rejected at the auth layer (not a network/proxy issue).

Root cause (suspected)

PR #2486 (implement pod exec websockets v5) changed the advertised WebSocket subprotocols from a single value to a comma-separated list:

kubernetes/base/stream/ws_client.py — v35.0.0:

header.append("sec-websocket-protocol: v4.channel.k8s.io")

kubernetes/base/stream/ws_client.py — v36.0.0:

header.append("sec-websocket-protocol: %s,%s" % (V5_CHANNEL_PROTOCOL, V4_CHANNEL_PROTOCOL))
# → "sec-websocket-protocol: v5.channel.k8s.io,v4.channel.k8s.io"

The Authorization header handling is identical between the two versions, and the Authorization header is sent. However, when the AKS API server receives multiple subprotocols in Sec-WebSocket-Protocol, it appears to take a different auth code path (presumably the browser-style base64url.bearer.authorization.k8s.io.<token> lookup inside the subprotocol header) and fails to fall back to honoring the standard Authorization header, returning 401 instead of negotiating down to v4.channel.k8s.io.

This makes the regression appear specifically against cluster/control-plane versions that don't natively support the v5 subprotocol.

Workaround

Pinning kubernetes==35.0.0 resolves the issue.

Kubernetes client version

$ pip show kubernetes | grep Version
Version: 36.0.0

Server version (AKS)

*.privatelink.<region>.azmk8s.io (Azure AKS, control plane managed by Azure; happy to share exact Kubernetes minor if needed)

OS

Red Hat Enterprise Linux 9.x (containerized Python 3.12 environment)

Python version

Python 3.12.11

Python client version

kubernetes==36.0.0
websocket-client==1.9.0

Installation method

pip3 install kubernetes

Related

• PR implement pod exec websockets v5 #2486 — implement pod exec websockets v5
• KEP-4006 — Transition SPDY to WebSockets

[aojea]

So the comma separated list from the headers meets the RFC https://datatracker.ietf.org/doc/html/rfc6455#section-11.3.4 , specifically https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Sec-WebSocket-Protocol

Directives

A comma-separated list of sub-protocol names, in the order of preference. The sub-protocols may be selected from the IANA WebSocket Subprotocol Name Registry, or may be a custom name jointly understood by the client and the server.

As a response header, this is a single sub-protocol that the server selected.

What I do not know if the comma separated list requires a space between subprotocols and sec-websocket-protocol: v5.channel.k8s.io,v4.channel.k8s.io" should be sec-websocket-protocol: v5.channel.k8s.io, v4.channel.k8s.io" , does it make a difference?

; happy to share exact Kubernetes minor if needed)

yes, we need the exact version

[GK-07]

Thanks for taking a look.

Re: comma vs comma-space formatting

Monkey-patched the installed ws_client.py to use "sec-websocket-protocol: %s, %s" (with a space) and re-ran the reproducer:

• Still 401 Unauthorized.

Same traceback / response headers as the original report. So the comma formatting isn't what AKS is reacting to.

Re: AKS Kubernetes version

Client Version: v1.33.7
Kustomize Version: v5.6.0
Server Version: v1.33.1

So this AKS control plane is on v1.33.1, which is well past 1.30 where the v5 channel protocol was introduced server-side. The server should be fully capable of negotiating v5.

So the situation is

• kubectl --kubeconfig=… exec … works against this same cluster, same kubeconfig.
• Python client v35.0.0 (advertises only v4.channel.k8s.io) → exec works.
• Python client v36.0.0 (advertises v5.channel.k8s.io,v4.channel.k8s.io) → 401 Unauthorized at the WebSocket handshake.
• REST calls (list_namespaced_pod, etc.) on v36 work fine with the same Configuration / token.

So the bearer token is valid and is being honored by the server for REST. Something specifically about the v5 advertisement (rather than the auth header propagation) is causing the API server to reject the WebSocket handshake on AKS 1.33.1.

A few possibilities that might be worth investigating:

1. v5 auth path expects the token in the subprotocol header (browser-style: Sec-WebSocket-Protocol: base64url.bearer.authorization.k8s.io.<base64-token>, v5.channel.k8s.io, v4.channel.k8s.io) rather than (or in addition to) the Authorization header. kubectl may be doing this; the Python client currently only sets Authorization.
2. AKS-specific webhook/RBAC interception on the exec subresource that triggers only when v5 is selected.
3. Some interaction with the kubelogin-issued AAD token that the v5 negotiation path doesn't handle the same way as v4.

Happy to test any specific patch you'd like me to try against this cluster (e.g., dropping the Authorization header and embedding the bearer token in Sec-WebSocket-Protocol, or making v5 advertisement opt-in via Configuration). Just tell me what to monkey-patch and I'll come back with the result.

[faust64]

could it be related to #2592 & #2591 & #2584

[dakaribhhlankenship]

Have you checked if the WebSocket handshake headers are altered in v36.0.0, specifically in kubernetes.stream.stream? It might be worth logging the headers being sent during the handshake to see if anything is missing or formatted differently compared to v35.0.0.

[GK-07]

Suggested fix

I won't be able to send a PR myself (corporate process is non-trivial), but the change should be very small. Maintainers — or anyone in the community — feel free to pick this up.

Option A (preferred): update kube_config.py to match v36's generated key name

In kubernetes/base/config/kube_config.py, change every line that writes to api_key['authorization'] to write to api_key['BearerToken'] instead.

- configuration.api_key['authorization'] = "Bearer " + self.token
+ configuration.api_key['BearerToken'] = "Bearer " + self.token

Quick way to find every occurrence:

rg "api_key\['authorization'\]" kubernetes/base/config/

I verified locally with a one-line sed against the installed v36 copy and REST + WebSocket exec both work end-to-end after the change.

Option B: make auth_settings() accept the legacy key too (backward-compat)

If there's concern about breaking users who set Configuration.api_key['authorization'] directly in their own code, the alternative is to extend the generated auth_settings() to fall back to 'authorization':

def auth_settings(self):
    auth = {}
    bearer_key = (
        'BearerToken' if 'BearerToken' in self.api_key
        else 'authorization' if 'authorization' in self.api_key
        else None
    )
    if bearer_key:
        auth['BearerToken'] = {
            'type': 'api_key',
            'in': 'header',
            'key': 'authorization',
            'value': self.get_api_key_with_prefix(bearer_key),
        }
    return auth

This is the safer option for downstream consumers since it doesn't change any externally observable contract. The downside is that configuration.py is auto-generated, so the patch would need to live in the templates or post-generation step.

Recommended test

A regression test in kube_config_test.py after load_kube_config():

self.assertIn('BearerToken', cfg.auth_settings())
self.assertTrue(cfg.auth_settings()['BearerToken']['value'])

If this had existed in the v36 cut, the bug would have been caught immediately.

Workaround for users on v36 in the meantime

Pin kubernetes==35.0.0, or apply this two-line shim after load_kube_config():

if 'authorization' in cfg.api_key and 'BearerToken' not in cfg.api_key:
    cfg.api_key['BearerToken'] = cfg.api_key['authorization']

Happy to test any fix branch against the same AKS cluster.

cc: @aojea @yliaog @seans3

[jicuss]

confirming this is affected EKS as well

[bahag-klickst]

Same with GKE, tested with 36.0.1 today and had to revert to 35.0.0

[aojea]

@bahag-klickst is 36.0.1 showing the same failure?

Based on comment #2595 (comment) it seems the problem is because of the change on the BearerToken header that was fixed in that version ... can you provide a repro and the logged errors?

[bahag-klickst]

Hej @aojea

have tested it today with 36.0.1 and received the following error which does not happen with 35.0.0

Exception when calling BatchV1Api->create_namespaced_job: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '8c72d720-b643-4b85-adee-db0a067a2c7e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 27 May 2026 06:08:53 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

The code itself is quite simple

credentials, project_id = google.auth.default()
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
cluster_manager_client = ClusterManagerClient(credentials=credentials)
cluster=f"projects/{project_id}/locations/us-central1-a/clusters/clus-1337"
cluster = cluster_manager_client.get_cluster(name=cluster)
# set kubernetes config
config = client.Configuration()
config.host = f"https://{cluster.endpoint}"
config.debug = False
with NamedTemporaryFile(delete=False) as ca_cert:
    ca_cert.write(base64.b64decode(cluster.master_auth.cluster_ca_certificate))
    config.ssl_ca_cert = ca_cert.name
config.verify_ssl = True
config.api_key = {"authorization": "Bearer " + credentials.token}
client.Configuration.set_default(config)
k8s_batch_api = client.BatchV1Api()
runner_dict = {
    'apiVersion': 'batch/v1',
    'kind': 'Job',
    'metadata': {
        'name': 'test-1337',
        'namespace': 'nsp-1337',
        'annotations': {
            'cluster-autoscaler.kubernetes.io/safe-to-evict': 'false'
        },
        'labels': {
            'name': 'test-1337'
        },
    },
    'spec': {
        'template': {
            'metadata': {
                'labels': {
                    'is-busy': 'false',
                }
            },
            'spec': {
                'serviceAccountName': 'github-runner',
                'containers': [{
                    'name': 'test-1337',
                    'image': 'europe-west1-docker.pkg.dev/proj-1337/image-repo/image:stable',
                    'imagePullPolicy': 'Always',
                }],
                'restartPolicy': 'Never',
            },
        },
        'backoffLimit': 3,
        'ttlSecondsAfterFinished': 5,
    }
}
k8s_batch_api.create_namespaced_job(namespace=k8s_namespace, body=runner_dict)

Hope this helps.

Cheers,
Tim

[GK-07]

Glad to see Option A landed in 36.0.1 — load_kube_config() users (the AKS case in this issue) are now unblocked, thanks.

The recent reports from @jicuss and @bahag-klickst are exactly the case Option B was meant to cover:

config.api_key = {"authorization": "Bearer " + credentials.token}

This is a perfectly reasonable pattern used by GCP/GKE and EKS examples in the wild, and was working in v35.0.0 and earlier. Since auth_settings() only checks 'BearerToken', these users still get the silent token-drop on 36.0.1.

Would you (@aojea / @yliaog) be open to extending Configuration.auth_settings() to fall back to 'authorization' for backward compatibility? It's a 3-line generated-code change and would close the regression for all users, not just the load_kube_config() path. Suggested patch:

def auth_settings(self):
    auth = {}
    bearer_key = (
        'BearerToken' if 'BearerToken' in self.api_key
        else 'authorization' if 'authorization' in self.api_key
        else None
    )
    if bearer_key:
        auth['BearerToken'] = {
            'type': 'api_key',
            'in': 'header',
            'key': 'authorization',
            'value': self.get_api_key_with_prefix(bearer_key),
        }
    return auth

Alternatively, a release note flagging the change in api_key key name from 'authorization' → 'BearerToken' as a breaking change for direct consumers would also help — users could then update their own code consciously rather than silently lose auth.

cc: @aojea @yliaog

[yliaog]

We have the following release note to document the breaking change. Given users may take time to adapt their code, I think it is good to provide backward compatibility for certain period of time. Could someone help to create a PR?

https://github.com/kubernetes-client/python/blob/release-36.0/CHANGELOG.md#breaking-change-from-upgrading-openapi-generator-to-v660

Configuration auth uses 'BearerToken' instead of 'authorization' in api_key.

[yliaog]

https://github.com/kubernetes-client/python/pull/2585/commits initially had written both authorization and BearerToken, I think if we were to provide the backward compatibility for a certain period of time, it is better to implement that in the handwritten code like what the PR did, than in the auto generated code with a patch.

any thoughts?

[GK-07]

Thanks @yliaog that makes sense, the handwritten layer is a much cleaner place for transition logic than patching generated code.

One thing I want to make sure we're on the same page about: the dual-key write in kube_config.py (per PR #2585) only catches users who go through load_kube_config(). The GKE/EKS reports above (e.g. @bahag-klickst's example) construct Configuration directly and set api_key = {"authorization": "Bearer ..."} themselves ,that path never touches kube_config.py, so a fix there won't unblock them.

Is the intended scope of the transition shim only load_kube_config() users, with the direct-api_key users being expected to migrate per the release note? Or do we want to cover both? If the latter, I don't see a handwritten chokepoint in the request path, the migration would need to happen on read inside Configuration.auth_settings() (auto-generated) or via a Configuration subclass somewhere.

Whichever scope you settle on, happy to test against the AKS reproducer here, and would assume the GKE/EKS folks could test their setups too.

cc: @bahag-klickst @jicuss