Repository navigation
Conversation
| private static boolean hasGroup(XceiverServerRatis s, RaftGroupId groupId) throws Exception { | ||
| for (RaftGroupId id : s.getServer().getGroupIds()) { | ||
| if (id.equals(groupId)) { | ||
| return true; | ||
| } | ||
| } | ||
| return false; | ||
| } |
There was a problem hiding this comment.
I may be missing something, but this seems to be the same as s.getServer().getGroupIds().contains(groupId). If so, we can improve the assertions like:
-assertTrue(hasGroup(peerServer, groupId));
+assertThat(peerServer.getServer().getGroupIds()).contains(groupId);to get better failure messages.
There was a problem hiding this comment.
As shown below, hasGroup is called to check for both true and false. So, assertThat(..). contains(..) does not work.
//testRemoveGroup
assertTrue(hasGroup(peerServer, groupId));
server.removeGroup(groupId, null);
assertFalse(hasGroup(server, groupId));
// For RATIS, the group is also removed from the peer; for CHAINED, only locally.
assertEquals(type == ReplicationType.CHAINED, hasGroup(peerServer, groupId));There was a problem hiding this comment.
assertThat(...).doesNotContain(...) can be used to replace assertFalse.
| // For CHAINED, each datanode adds the group from its own command; see XceiverServerRatis.addGroup. | ||
| peers.stream().filter(d -> type != ReplicationType.CHAINED && !d.getID().equals(dn.getID())) |
There was a problem hiding this comment.
type is constant, so we can skip the stream() for CHAINED
|
@adoroszlai , thanks for reviewing this! Just have pushed a commit to address your comments and some refactoring. |
|
|
||
| @SuppressWarnings("parameternumber") | ||
| public ContainerStateMachine(HddsDatanodeService hddsDatanodeService, RaftGroupId gid, | ||
| ReplicationType replicationType, |
There was a problem hiding this comment.
I believe when I read section 4.2 from the design doc: https://github.com/apache/ozone/pull/11982/changes#diff-9f932235b1dd6d8e60276cef9b9b28dffeb74ad942931a34d414e301a41c1f12R220, it pretty much says we keep the container state machine unaware of the type. The type is stored in XceiverServerRatis and looked up on demand.
However the implementation here turns into the idea of having a replication type within the SM. Does then we turn the design to a chained pipeline binds with a dedicated Ratis group and ContainerStateMachine?
There was a problem hiding this comment.
@amaliujia , adding ReplicationType to ContainerStateMachine is to avoid non-zero bcsid. If a ContainerStateMachine support both V1 and V3 containers, V1 containers will change the bcsid and cause confusion. In HDDS-17543, we will disable Raft in V3 ContainerStateMachine and disable StreamCommand in V1 ContainerStateMachine, in order to avoid mixing them.
Will update the design doc.
There was a problem hiding this comment.
... section 4.2 ....
@amaliujia , I forgot to mention that I compared the existing design in 4.2 (SCM reply) with the new Per-pipeline RocksDB marker in HDDS-17845. Per-pipeline RocksDB marker is cleaner.
Also, revised the design doc, please take a look.
Thanks, the updated patch LGTM. Let's wait for @amaliujia's review. |
There was a problem hiding this comment.
Overall LGTM. Left two comments.
Meanwhile I tried to reason the problem of the per replication type SM design. It seems to me that the problem will be double overhead from Raft side because we double the RaftGroup/SM if a DN supports both RATIS and CHAINED replication type by different ContainerSM (we may be able to avoid a mixed case in a single SM but probably has no reason to have a DN only support one kind of SM),
It is definitely a trade-off though. IIUC the benefit of this design is to reduce our implementation complexity.
|
|
||
| /** @return the {@link ReplicationType} of the given group. */ | ||
| public ReplicationType getReplicationType(RaftGroupId groupId) throws IOException { | ||
| return ((ContainerStateMachine) server.getDivision(groupId).getStateMachine()).getReplicationType(); |
There was a problem hiding this comment.
We have this addingGroups.put(groupId, type);, so do we need to go to the ContainerStateMachine to fetch the type? It looks like we can get the type directly by addingGroups.get(groupId).
| if (type == null) { | ||
| // The type is unknown when the group is reloaded on restart or added by a peer. | ||
| // TODO HDDS-17845: support restart for CHAINED. | ||
| LOG.info("{}: ReplicationType is unknown, use {}", gid, ReplicationType.RATIS); |
There was a problem hiding this comment.
I know we defer handing of the DN restart, but is this LOG.info becomes too noisy now after DN restarts?Per my understanding, the DN restart will lose the replication type from the ContainerSM as that was not persisted. After restart, every of the RaftGroup will create an unknown type log.
Then
- Maybe lower the level of this log?
- Probably we add a persisted metadata for RaftGroup then the replication type is managed by Ratis and can survive from node failures. This can be deferred (and likely we need Ratis side changes).
What changes were proposed in this pull request?
Assisted by Claude Opus 5.5.
Out of scope:
What is the link to the Apache JIRA
HDDS-17809
How was this patch tested?
By adding new tests using AI.
CI passed (before rebased with master): https://github.com/szetszwo/ozone/actions/runs/43930861415