Skip to content

chore: bump release to v1.50.12 - #3968

Merged
chasers merged 1 commit into
mainfrom
chore/bump-1.50.12
Sep 12, 2026
Merged

chasers merged 1 commit into
mainfrom
chore/bump-1.50.12

Conversation

@chasers

@chasers chasers commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Bumps VERSION to 1.50.12. Prod runs v1.50.11 on prod-c/e/f/a since 2026-09-07, and prod-b/d/g run v1.50.7.

Why now

This release carries the fixes for the process and memory growth on prod-c (213K processes, 46 GB RSS, ~11 GB of binaries pinned under a stale S3Adaptor tree for one source). Local reproduction against the v1.50.11 tag confirmed two of the mechanisms; see .plans/2026-09-12_01_process-leak-investigation.md (not in this PR).

  • fix: stop the requested backend child #3865 — stop the requested backend child. On v1.50.11, SourceSup.stop_backend_child/2 passes its arguments to Enum.find_value/3 in the wrong order and always stops the newest backend child. Supervisor.which_children/1 lists newest first. With a detached backend that is not the newest child, every SourceSupWorker tick re-adds the attached backend and then kills it again. Locally: the detached tree survives, and the attached backend is present in 12 of 300 samples. On main both orders pass.
  • fix: bound S3 upload stalls #3944 — bound S3 upload stalls. HTTP timeouts and 3 retries instead of ExAws' default 10. Locally, one batch against a permanent 503 makes 10 attempts on v1.50.11 and 3 on main.
  • fix: retry failed S3 upload batches #3945 — retry failed S3 upload batches. Batcher demand capped at 100 and failed batches acked. Locally, a stalled bucket pulls all 5,000 queued events into GenStage buffers on v1.50.11 (40 MB pinned) and 2,000 on main (18 MB pinned).

Also in v1.50.11..main: cache warmer fixes (#3913, #3915, #3916), ClickHouse telemetry (#3950, #3952, #3962), read replica read-only connections (#3902), mapper follow-ups, and the anti-slop tooling (#3899).

Rollout

Canary first. Watch :erlang.system_info(:process_count), length(:ets.all()) and :erlang.memory(:binary) for 24 h against a non-canary node.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YQ8Y8zCvSKUFNPvkwH2WS4

Prod runs v1.50.11. This release carries the fixes for the process and memory
growth seen on prod-c since 2026-09-07:

- #3865 stops the requested backend child. On v1.50.11
  `SourceSup.stop_backend_child/2` stops the newest child instead, so a
  detached backend tree never stops and the attached backend gets killed on
  every `SourceSupWorker` tick.
- #3944 bounds S3 upload stalls with HTTP timeouts and 3 retries instead of
  ExAws' default 10.
- #3945 caps the S3 batcher demand at 100 and acks failed batches, so a
  stalled bucket no longer pulls the whole queue into GenStage buffers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQ8Y8zCvSKUFNPvkwH2WS4
@chasers
chasers merged commit 3c66a65 into main Sep 12, 2026
4 checks passed
@chasers
chasers deleted the chore/bump-1.50.12 branch September 12, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants