Skip to content

fix(helm): override HINDSIGHT_API_PORT in worker StatefulSet to survive K8s service discovery#2904

Open
Fyko wants to merge 2 commits into
vectorize-io:mainfrom
Fyko:fix/worker-statefulset-port-override
Open

fix(helm): override HINDSIGHT_API_PORT in worker StatefulSet to survive K8s service discovery#2904
Fyko wants to merge 2 commits into
vectorize-io:mainfrom
Fyko:fix/worker-statefulset-port-override

Conversation

@Fyko

@Fyko Fyko commented Jul 22, 2026

Copy link
Copy Markdown

Summary

The dedicated worker StatefulSet crashes on boot inside Kubernetes because it parses HINDSIGHT_API_PORT as an int, but the hindsight-api Service makes Kubernetes inject the legacy service-discovery env var HINDSIGHT_API_PORT=tcp://<clusterIP>:8888 into every pod in the namespace.

File ".../hindsight_api/config.py", line 1912, in from_env
    port=int(os.getenv(ENV_PORT, DEFAULT_PORT)),
ValueError: invalid literal for int() with base 10: 'tcp://172.20.174.228:8888'

The api Deployment template already guards against this — it explicitly sets HINDSIGHT_API_PORT to a plain int before ranging .Values.api.env:

{{- /* Explicitly set port to override K8s service discovery env var (HINDSIGHT_API_PORT) */}}
- name: HINDSIGHT_API_PORT
  value: {{ .Values.api.service.targetPort | quote }}

The worker StatefulSet template had no equivalent, so it inherited the injected tcp://… string and crashed. This PR adds the same guard to the worker.

Root cause

get_config() runs Config.from_env() at worker startup (hindsight_api/worker/main.py). from_env() unconditionally does port=int(os.getenv("HINDSIGHT_API_PORT", DEFAULT_PORT)) (config.py:1912). When the hindsight-api Service exists, Kubernetes injects HINDSIGHT_API_PORT=tcp://<clusterIP>:8888 into sibling pods, and the int() call raises ValueError.

Value choice: worker.service.targetPort (8889)

The worker does not bind config.port (HINDSIGHT_API_PORT). Its uvicorn health/metrics server binds args.http_port, which defaults to config.worker_http_port = HINDSIGHT_API_WORKER_HTTP_PORT (default 8889, = worker.service.targetPort); the probes hit /health on 8889. config.port is parsed by from_env() but never listened on by the worker.

So the override only needs to be a parseable int. Mirroring the api template with {{ .Values.worker.service.targetPort | quote }} keeps the pattern consistent and self-documenting. There is no double-bind risk against HINDSIGHT_API_WORKER_HTTP_PORT even though both resolve to 8889, because the worker never opens a socket on config.port.

Validation

helm template ... --set worker.enabled=true now renders exactly one HINDSIGHT_API_PORT with a plain int:

- name: HINDSIGHT_API_PORT
  value: "8889"

No tcp://, no duplicate. helm lint --set worker.enabled=true passes (only the pre-existing icon is recommended info).

Secondary fix (second commit): de-duplicate worker env keys

The worker template ranged .Values.api.env then .Values.worker.env with no de-dupe, so a key present in both (e.g. HINDSIGHT_API_WORKER_POLL_INTERVAL_MS) emitted a duplicate env entry, which Kubernetes server-side apply rejects. Reproduced before the fix:

$ helm template ... --set worker.enabled=true \
    --set api.env.HINDSIGHT_API_WORKER_POLL_INTERVAL_MS=999
- name: HINDSIGHT_API_WORKER_POLL_INTERVAL_MS
  value: "999"
...
- name: HINDSIGHT_API_WORKER_POLL_INTERVAL_MS
  value: "500"

The two ranges are now merged with merge (deepCopy worker.env) api.env, so worker-specific values still win on conflict (preserving the prior intended precedence) but only one entry is emitted. Default renders are unchanged. This is a separate, droppable commit if you'd prefer to keep the PR to just the crash fix.

Note: no Chart.yaml version bump — this repo bumps chart versions via scripts/release.sh at release time, not per PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant