Skip to content

fix(cluster): recover writes after a fast partition primary rejoin - #3987

Merged
spetz merged 5 commits into
masterfrom
recover_writes
Aug 28, 2026
Merged

fix(cluster): recover writes after a fast partition primary rejoin#3987
spetz merged 5 commits into
masterfrom
recover_writes

Conversation

@spetz

@spetz spetz commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

A fast partition primary rejoin could leave the group unable to
commit new writes. A backup could adopt suffix headers without
obtaining their bodies because repair stopped at commit_max. Clients
then handled the typed non-admission response by checking metadata
leadership, even though metadata and partition groups can elect
different primaries.

Extend suffix repair to the available frontier and clear completed
sessions after the later commit catches up. Reject writes on stale
backups before local offset validation, so callers receive the typed
response required for safe rerouting.

Add one-pass roster routing for HTTP and the Rust, Node, Go, Java,
and C# clients. Coordinate concurrent Rust reconnects and prevent
ambiguous replicated writes from being replayed under a new session.

@spetz spetz added the sdk Change related to sdk (client) API label Aug 28, 2026
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 28, 2026
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.68421% with 172 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.69%. Comparing base (90e38b1) to head (377247c).

Files with missing lines Patch % Lines
core/sdk/src/tcp/tcp_client.rs 82.06% 26 Missing and 7 partials ⚠️
core/sdk/src/websocket/websocket_client.rs 76.08% 27 Missing and 6 partials ⚠️
core/sdk/src/quic/quic_client.rs 75.26% 19 Missing and 4 partials ⚠️
core/sdk/src/leader_aware.rs 94.72% 13 Missing and 7 partials ⚠️
core/server/src/http/forward.rs 86.57% 16 Missing and 4 partials ⚠️
...IggyClient/Implementations/TcpMessageStream.Vsr.cs 71.18% 10 Missing and 7 partials ⚠️
.../apache/iggy/client/async/tcp/LeaderAwareness.java 64.70% 2 Missing and 4 partials ⚠️
core/shard/src/lib.rs 94.68% 5 Missing ⚠️
...ache/iggy/client/async/tcp/AsyncIggyTcpClient.java 76.19% 3 Missing and 2 partials ⚠️
foreign/go/client/tcp/tcp_core.go 80.95% 2 Missing and 2 partials ⚠️
... and 2 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3987      +/-   ##
============================================
- Coverage     84.63%   83.69%   -0.94%     
- Complexity     1398     1405       +7     
============================================
  Files          1224     1224              
  Lines        176637   177971    +1334     
  Branches     143161   144285    +1124     
============================================
- Hits         149491   148956     -535     
- Misses        23173    25076    +1903     
+ Partials       3973     3939      -34     
Components Coverage Δ
Rust Core 85.64% <89.09%> (+0.01%) ⬆️
Java SDK 67.35% <75.55%> (+0.11%) ⬆️
C# SDK 52.48% <73.84%> (-22.91%) ⬇️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.24% <98.30%> (+1.85%) ⬆️
Go SDK 69.29% <87.30%> (+0.19%) ⬆️
Files with missing lines Coverage Δ
core/partitions/src/iggy_partition.rs 88.58% <100.00%> (+0.51%) ⬆️
core/partitions/src/journal.rs 89.25% <100.00%> (+0.72%) ⬆️
core/partitions/src/types.rs 48.95% <ø> (ø)
core/sdk/src/vsr.rs 94.01% <100.00%> (+0.10%) ⬆️
core/server/src/http.rs 93.31% <100.00%> (+0.19%) ⬆️
core/server/src/partition_reconciler.rs 97.96% <100.00%> (-0.31%) ⬇️
...SDK/IggyClient/Implementations/TcpMessageStream.cs 44.52% <100.00%> (-36.48%) ⬇️
...ache/iggy/client/async/tcp/AsyncTcpConnection.java 74.69% <100.00%> (+0.24%) ⬆️
foreign/node/src/client/client.socket.ts 97.50% <100.00%> (+0.19%) ⬆️
foreign/node/src/client/client.connection.ts 97.31% <93.75%> (-0.18%) ⬇️
... and 11 more

... and 92 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread core/sdk/src/tcp/tcp_client.rs
Comment thread core/sdk/src/tcp/tcp_client.rs Outdated
Comment thread core/shard/src/lib.rs Outdated
Comment thread core/shard/src/lib.rs Outdated
Comment thread core/sdk/src/quic/quic_client.rs Outdated
Comment thread core/sdk/src/websocket/websocket_client.rs Outdated
@spetz

spetz commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ready

Comment thread core/sdk/src/quic/quic_client.rs Outdated
Comment thread core/sdk/src/websocket/websocket_client.rs Outdated
Comment thread core/partitions/src/iggy_partition.rs
@spetz

spetz commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@spetz
spetz merged commit 996ac04 into master Aug 28, 2026
98 checks passed
@spetz
spetz deleted the recover_writes branch August 28, 2026 17:44
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 28, 2026
hubcio added a commit to chengxilo/iggy that referenced this pull request Aug 31, 2026
Merging master pulled in the partition forward path (apache#3987),
written against the pre-branch roster API: replica_ip() as an
Option, ClusterRoster.self_ip, and an infallible From for
ResolvedClusterNode. This branch validates roster IPs at config
time, so replica_ip() is plain and the conversion is a TryFrom.
Git merged the hunks without a textual conflict and left the
server crate uncompilable.

Adapt the forward path and its test helper to the resolved
roster. The unparsable-IP case in the roster walk test is gone:
such a node can no longer exist in a resolved roster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sdk Change related to sdk (client) API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants