Skip to content

RATIS-2821. LogAppenderBase#newAppendEntriesRequest should get follower nextIndex once - #1700

Open
ivandika3 wants to merge 2 commits into
apache:masterfrom
ivandika3:RATIS-2821
Open

ivandika3 wants to merge 2 commits into
apache:masterfrom
ivandika3:RATIS-2821

Conversation

@ivandika3

Copy link
Copy Markdown
Contributor

Generated-by: GPT-6.1 Sol

Please refer to https://issues.apache.org/jira/browse/RATIS-2821.

This main fix is in LogAppenderBase#newAppendEntriesRequest. However, there are also some fixes to handle cases where the nextIndex / matchIndex / snapshotIndex update from follower RPC reply will not cause another issue. For example, increaseNextIndex is replaced with updateNextIndexForAppendEntries since it's possible where nextIndex is updated and causes increaseNextIndex to faill.

Appender prepares entries 100–109
Reply handler advances nextIndex to 150
Appender tries to advance nextIndex to 110
Strict increasing-index assertion fails

The concurrency model for the indices needs to be discussed further and addressed in the future tasks since it might affect performance and might risk issues like deadlock. Possible ways

  • Use the GrpcLogAppender lock when updating any indices to ensure atomicity
  • FollowerInfoImpl can be synchronized or have a lock methods to protect the indices updates. Also support updating multiple indices atomically (e.g. nextIndex and matchIndex).

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/RATIS-2821

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2821

How was this patch tested?

UT.

@ivandika3 ivandika3 self-assigned this Nov 18, 2026
@ivandika3
ivandika3 marked this pull request as ready for review November 18, 2026 10:44
@szetszwo

Copy link
Copy Markdown
Contributor

@ivandika3 , the change looks good.

Just some minor comments from Claude:

... might risk issues like deadlock. ...

  • (Claude) I checked the new write locks in the snapshot paths for deadlock: none of them run inside another lock, and the read lock in hasAllResponse is released before installSnapshot takes the write lock.

Suggestions

  1. computeNextIndex(ignored -> getNextIndexForInconsistency(...)) (GrpcLogAppender.java:642) behaves like setNextIndex, but the lambda ignores its input, which hides that. There is also a small side effect:
    • RaftLogIndex.updateUnconditionally runs the function twice: once inside getAndUpdate (which can retry) and once more to build the log message.
    • The function reads matchIndex, which the SUCCESS path can change at any time, so the logged new value can differ from what was stored.
    • setNextIndex(getNextIndexForInconsistency(requestFirstIndex, replyNextIndex)) inside the lock would be clearer and does the same thing.
  2. updateNextIndexForSnapshot right after setSnapshotIndex is redundant. setSnapshotIndex already sets nextIndex = idx+1, so the extra call usually does nothing except add an INFO log line. It was redundant before this change too, so it's fine to drop or keep.
  3. Older issue, not introduced here: FollowerInfoImpl.setSnapshotIndex overwrites matchIndex and nextIndex unconditionally. The SUCCESS path doesn't take the lock, so wrapping the call in the lock doesn't stop a concurrent reply's progress from being overwritten with a lower value. The effect is conservative (it only re-sends entries), and your new comment ("Successful append replies may advance the indexes independently") already acknowledges it. It could be a follow-up.
  4. increaseNextIndex now has one caller: LogAppenderDefault.java:210. That's fine because that appender is synchronous.

@ivandika3

Copy link
Copy Markdown
Contributor Author

Thanks @szetszwo for the review.

(Claude) I checked the new write locks in the snapshot paths for deadlock: none of them run inside another lock, and the read lock in hasAllResponse is released before installSnapshot takes the write lock.

Yes, currently it has no observed possible deadlock. I was trying a different ways like introducing lock to FollowerInfoImpl, but this means that lock ordering need to be preserved and things like computeNextindex that can take any arbitrary function can introduce deadlock.

computeNextIndex(ignored -> getNextIndexForInconsistency(...)) (GrpcLogAppender.java:642) behaves like setNextIndex, but the lambda ignores its input, which hides that. There is also a small side effect: ...

The proposed setNextIndex replacement is not equivalent. We use the CAS retry as a way to get the latest matchIndex to recompute the getNextIndexForInconsistency. If we simply use setNextIndex, we only calculate the matchIndex once and it can be updated by a newer value instead.

updateNextIndexForSnapshot right after setSnapshotIndex is redundant. setSnapshotIndex already sets nextIndex = idx+1, so the extra call usually does nothing except add an INFO log line. It was redundant before this change too, so it's fine to drop or keep.

Good catch, removed the subsequent setSnapshotIndex.

Older issue, not introduced here: FollowerInfoImpl.setSnapshotIndex overwrites matchIndex and nextIndex unconditionally. The SUCCESS path doesn't take the lock, so wrapping the call in the lock doesn't stop a concurrent reply's progress from being overwritten with a lower value. The effect is conservative (it only re-sends entries), and your new comment ("Successful append replies may advance the indexes independently") already acknowledges it. It could be a follow-up.

Let's address it in the new follow-up task, raised RATIS-2824.

increaseNextIndex now has one caller: LogAppenderDefault.java:210. That's fine because that appender is synchronous.

Yes, correct.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants