You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
RATIS-2821. LogAppenderBase#newAppendEntriesRequest should get follower nextIndex once - #1700
This main fix is in LogAppenderBase#newAppendEntriesRequest. However, there are also some fixes to handle cases where the nextIndex / matchIndex / snapshotIndex update from follower RPC reply will not cause another issue. For example, increaseNextIndex is replaced with updateNextIndexForAppendEntries since it's possible where nextIndex is updated and causes increaseNextIndex to faill.
The concurrency model for the indices needs to be discussed further and addressed in the future tasks since it might affect performance and might risk issues like deadlock. Possible ways
Use the GrpcLogAppender lock when updating any indices to ensure atomicity
FollowerInfoImpl can be synchronized or have a lock methods to protect the indices updates. Also support updating multiple indices atomically (e.g. nextIndex and matchIndex).
(Claude) I checked the new write locks in the snapshot paths for deadlock: none of them run inside another lock, and the read lock in hasAllResponse is released before installSnapshot takes the write lock.
Suggestions
computeNextIndex(ignored -> getNextIndexForInconsistency(...)) (GrpcLogAppender.java:642) behaves like setNextIndex, but the lambda ignores its input, which hides that. There is also a small side effect:
RaftLogIndex.updateUnconditionally runs the function twice: once inside getAndUpdate (which can retry) and once more to build the log message.
The function reads matchIndex, which the SUCCESS path can change at any time, so the logged new value can differ from what was stored.
setNextIndex(getNextIndexForInconsistency(requestFirstIndex, replyNextIndex)) inside the lock would be clearer and does the same thing.
updateNextIndexForSnapshot right after setSnapshotIndex is redundant. setSnapshotIndex already sets nextIndex = idx+1, so the extra call usually does nothing except add an INFO log line. It was redundant before this change too, so it's fine to drop or keep.
Older issue, not introduced here: FollowerInfoImpl.setSnapshotIndex overwrites matchIndex and nextIndex unconditionally. The SUCCESS path doesn't take the lock, so wrapping the call in the lock doesn't stop a concurrent reply's progress from being overwritten with a lower value. The effect is conservative (it only re-sends entries), and your new comment ("Successful append replies may advance the indexes independently") already acknowledges it. It could be a follow-up.
increaseNextIndex now has one caller: LogAppenderDefault.java:210. That's fine because that appender is synchronous.
(Claude) I checked the new write locks in the snapshot paths for deadlock: none of them run inside another lock, and the read lock in hasAllResponse is released before installSnapshot takes the write lock.
Yes, currently it has no observed possible deadlock. I was trying a different ways like introducing lock to FollowerInfoImpl, but this means that lock ordering need to be preserved and things like computeNextindex that can take any arbitrary function can introduce deadlock.
computeNextIndex(ignored -> getNextIndexForInconsistency(...)) (GrpcLogAppender.java:642) behaves like setNextIndex, but the lambda ignores its input, which hides that. There is also a small side effect: ...
The proposed setNextIndex replacement is not equivalent. We use the CAS retry as a way to get the latest matchIndex to recompute the getNextIndexForInconsistency. If we simply use setNextIndex, we only calculate the matchIndex once and it can be updated by a newer value instead.
updateNextIndexForSnapshot right after setSnapshotIndex is redundant. setSnapshotIndex already sets nextIndex = idx+1, so the extra call usually does nothing except add an INFO log line. It was redundant before this change too, so it's fine to drop or keep.
Good catch, removed the subsequent setSnapshotIndex.
Older issue, not introduced here: FollowerInfoImpl.setSnapshotIndex overwrites matchIndex and nextIndex unconditionally. The SUCCESS path doesn't take the lock, so wrapping the call in the lock doesn't stop a concurrent reply's progress from being overwritten with a lower value. The effect is conservative (it only re-sends entries), and your new comment ("Successful append replies may advance the indexes independently") already acknowledges it. It could be a follow-up.
Let's address it in the new follow-up task, raised RATIS-2824.
increaseNextIndex now has one caller: LogAppenderDefault.java:210. That's fine because that appender is synchronous.
Yes, correct.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated-by: GPT-6.1 Sol
Please refer to https://issues.apache.org/jira/browse/RATIS-2821.
This main fix is in
LogAppenderBase#newAppendEntriesRequest. However, there are also some fixes to handle cases where the nextIndex / matchIndex / snapshotIndex update from follower RPC reply will not cause another issue. For example,increaseNextIndexis replaced withupdateNextIndexForAppendEntriessince it's possible wherenextIndexis updated and causesincreaseNextIndexto faill.The concurrency model for the indices needs to be discussed further and addressed in the future tasks since it might affect performance and might risk issues like deadlock. Possible ways
GrpcLogAppenderlock when updating any indices to ensure atomicityWhat changes were proposed in this pull request?
https://issues.apache.org/jira/browse/RATIS-2821
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-2821
How was this patch tested?
UT.