Skip to content

Release subscription WAL buffers between prefetch rounds - #18956

Merged
jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:codex/fix-subscription-wal-read-buffers
Oct 21, 2026
Merged

jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:codex/fix-subscription-wal-read-buffers

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

The DataNode at the supplied build exhausted its 4 GiB direct-memory limit while serving /metrics. The node had 240 subscription queues, whose paused WAL readers could each retain a direct segment buffer. This change releases those buffers between prefetch rounds while preserving reader positions and pending fragments, then reloads the current segment when a queue resumes. It also streams Prometheus snapshots as bounded 8 KiB UTF-8 chunks and reports EOF if a segment was shortened while its buffer was released.

Verification

  • Targeted DataNode/grammar tests: 62 passed; one performance test was disabled by default.
  • PrometheusReporterTest: 3 passed, including a response larger than 24 MiB under a 32 MiB direct-memory limit.
  • Full reactor test-compile -P with-zh-locale: passed.
  • The final EOF regression test could not be rerun locally: DataNode compilation fails on the existing generated relational parser/API mismatch (AstBuilder calls missing copyToStatement() methods). The full English compile encounters the same unrelated mismatch.

@jt2594838
jt2594838 merged commit 10749a2 into apache:master Oct 21, 2026
38 of 39 checks passed
@Caideyipi
Caideyipi deleted the codex/fix-subscription-wal-read-buffers branch October 21, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants