Skip to content

fix(journal): repair a torn WAL tail instead of panicking the shard - #3976

Open
numinnex wants to merge 1 commit into
masterfrom
torn_wal_on_kernel_version
Open

fix(journal): repair a torn WAL tail instead of panicking the shard#3976
numinnex wants to merge 1 commit into
masterfrom
torn_wal_on_kernel_version

Conversation

@numinnex

@numinnex numinnex commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

FileStorage::truncate is the boot-time repair for a torn metadata WAL tail.
It used compio's set_len, which submits IORING_OP_FTRUNCATE. That opcode
landed in kernel 6.9. Below it the driver probes the opcode as unsupported and
falls back to push_blocking, but shard proactors are built with
thread_pool_limit(0), so the fallback panics with the thread pool is needed but no worker thread is running. The panic fires inside dispatch, outside
catch_unwind_io.

Net effect: a crash that tore the last WAL append made the next boot kill the
shard instead of repairing it, and the repair is idempotent, so the node stayed
down across restarts.

Two conditions have to coincide, so this is not a routine path:

kernel < 6.9 IORING_OP_FTRUNCATE absent, compio falls back
AND
torn WAL tail crash mid-append, so boot calls truncate_or_fail

The affected range is 5.19 through 6.8, not everything below 6.9. Ring setup
already requires IORING_SETUP_COOP_TASKRUN and IORING_SETUP_TASKRUN_FLAG,
which need 5.19, so RHEL 9 (5.14) and stock Ubuntu 22.04 (5.15) never start the
server at all and were never exposed. What this actually broke is Debian 12 and
AL2023 (6.1), and Ubuntu 24.04 and 22.04-HWE (6.8). macOS aarch64 is exempt
because create_shard_executor keeps a blocking pool there by design.

The fix

Truncate synchronously through std::fs on the stored path, which needs
neither the opcode nor the blocking pool. This mirrors what segment recovery
already does in truncate_to. The sync_all moves inside truncate, so the
repair is durable on its own and the caller no longer pairs it with a separate
fsync. sync_all rather than sync_data because the file length is metadata,
and without it a power cut right after the repair re-presents the torn tail.

truncate is no longer async. An async fn that never awaits trips
clippy::unused_async, and the journal crate denies clippy::pedantic.
Blocking the shard thread costs nothing here: the sole caller is boot-time
repair, before the shard serves traffic.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 27, 2026
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.09677% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.68%. Comparing base (6a1c84c) to head (c06dccf).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
core/journal/src/file_storage.rs 89.65% 0 Missing and 3 partials ⚠️
core/journal/src/prepare_journal.rs 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3976       +/-   ##
=============================================
- Coverage     84.20%   67.68%   -16.53%     
- Complexity     1398     1399        +1     
=============================================
  Files          1219     1218        -1     
  Lines        173564   151840    -21724     
  Branches     140663   118939    -21724     
=============================================
- Hits         146150   102768    -43382     
- Misses        23453    45177    +21724     
+ Partials       3961     3895       -66     
Components Coverage Δ
Rust Core 63.70% <87.09%> (-21.32%) ⬇️
Java SDK 67.30% <ø> (+0.07%) ⬆️
C# SDK 75.32% <ø> (-0.04%) ⬇️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 95.97% <ø> (-0.10%) ⬇️
Go SDK 69.09% <ø> (ø)
Files with missing lines Coverage Δ
core/journal/src/prepare_journal.rs 90.60% <50.00%> (+0.15%) ⬆️
core/journal/src/file_storage.rs 74.00% <89.65%> (+6.43%) ⬆️

... and 309 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant