Skip to content

fix(cluster): make the advertised leader accept partition writes - #3985

Merged
numinnex merged 6 commits into
masterfrom
fix_view_divergence_during_bootstrap
Aug 28, 2026
Merged

fix(cluster): make the advertised leader accept partition writes#3985
numinnex merged 6 commits into
masterfrom
fix_view_divergence_during_bootstrap

Conversation

@numinnex

@numinnex numinnex commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

A client is told which node leads the cluster, sends a partition
write there, and is refused for its whole budget. The node it was
told about is genuinely the leader; the write is genuinely unroutable.

Each node runs a metadata consensus group plus one per partition,
each with its own view counter, and both pick a primary as
view % replica_count. get_cluster_metadata marks a node Leader
from the metadata plane alone and ClusterNode carries one role,
so a partition group's primary cannot be expressed on the wire.
Partition ops route within a node, never to a peer, so the
advertised leader cannot forward one either.

A group materialised with no durable state started at view 0,
naming replica 0 wherever the metadata plane had got to, so any
metadata election opened the gap for every topic created after.

Seed a fresh group from the metadata view shard 0 already publishes
for the roster's leader marking. Both planes then name the same
node by construction, with no wire or client change. The seed sets
log_view too: a primary whose log_view lags counts as
mid-transition and answers no RequestStartView. A replica that
has not seen a view defers materialising rather than reading the
unknown sentinel as 0, and that deferral counts toward the pass so
it cannot arm the reconciler's fast-skip.

This aligns the planes once, at creation. A later election on
either side parts them again and nothing forwards a write across
the gap, so the symptom returns for that group.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 28, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few things that aren't on changed lines.

nothing tells you from outside whether the seed took. the branch that applies it logs nothing, the one right above it logs, and no other surface shows a partition group's view - so the test can only check that sends work. a log line there would let it check the real thing. the view_fallback doc is stale too, it still talks about the last journaled prepare.

current_primary_index gives back a replica id, not a roster index - the name says otherwise.

the simulator covers none of this: init_partition is simulator-only and never goes through build_partition_fresh.

this only lines the two planes up once, at create time. the next election on either side splits them again, and nothing forwards writes over TCP, so the same problem comes back. worth spelling out in the description.

one question: the primary stays at log_view 0 while backups move to V. can that wedge the next view change?

Comment thread core/server/src/partition_helpers.rs Outdated
Comment thread core/server/src/partition_helpers.rs
Comment thread core/server/src/partition_helpers.rs
Comment thread core/server/src/partition_helpers.rs Outdated
Comment thread core/integration/tests/cluster/partition_primary_routing.rs
Comment thread core/integration/tests/cluster/partition_primary_routing.rs
Comment thread core/integration/tests/cluster/partition_primary_routing.rs Outdated
Comment thread core/integration/tests/cluster/partition_primary_routing.rs Outdated
Comment thread core/integration/tests/cluster/partition_primary_routing.rs Outdated
Comment thread core/server/src/partition_reconciler.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 28, 2026
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.78899% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.86%. Comparing base (0fdf9d6) to head (f484392).

Files with missing lines Patch % Lines
core/server/src/bootstrap.rs 42.85% 4 Missing ⚠️
core/consensus/src/impls.rs 97.36% 0 Missing and 1 partial ⚠️
core/server/src/partition_helpers.rs 95.65% 0 Missing and 1 partial ⚠️
core/simulator/src/lib.rs 98.41% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3985       +/-   ##
=============================================
- Coverage     84.26%   68.86%   -15.40%     
  Complexity     1398     1398               
=============================================
  Files          1219     1220        +1     
  Lines        173644   156522    -17122     
  Branches     140742   123623    -17119     
=============================================
- Hits         146317   107787    -38530     
- Misses        23369    44797    +21428     
+ Partials       3958     3938       -20     
Components Coverage Δ
Rust Core 65.37% <96.78%> (-19.72%) ⬇️
Java SDK 67.23% <ø> (ø)
C# SDK 75.32% <ø> (-0.08%) ⬇️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.07% <ø> (ø)
Go SDK 69.09% <ø> (ø)
Files with missing lines Coverage Δ
core/server/src/cluster_meta.rs 98.60% <100.00%> (ø)
core/server/src/partition_reconciler.rs 97.96% <100.00%> (+0.05%) ⬆️
core/server/src/responses.rs 91.30% <100.00%> (+0.15%) ⬆️
core/shard/src/lib.rs 81.54% <100.00%> (+0.37%) ⬆️
core/consensus/src/impls.rs 91.42% <97.36%> (+0.25%) ⬆️
core/server/src/partition_helpers.rs 75.99% <95.65%> (+0.88%) ⬆️
core/simulator/src/lib.rs 96.94% <98.41%> (-0.15%) ⬇️
core/server/src/bootstrap.rs 79.47% <42.85%> (-0.10%) ⬇️

... and 349 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@numinnex

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 28, 2026
@numinnex numinnex changed the title fix(cluster): start a fresh partition group in the metadata view fix(cluster): make the advertised leader accept partition writes Aug 28, 2026
@numinnex
numinnex merged commit db14a4b into master Aug 28, 2026
100 checks passed
@numinnex
numinnex deleted the fix_view_divergence_during_bootstrap branch August 28, 2026 14:53
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants