Skip to content

HDDS-17809. Add ReplicationType to ContainerStateMachine. - #12075

Open
szetszwo wants to merge 2 commits into
apache:masterfrom
szetszwo:HDDS-17809
Open

szetszwo wants to merge 2 commits into
apache:masterfrom
szetszwo:HDDS-17809

Conversation

@szetszwo

@szetszwo szetszwo commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Assisted by Claude Opus 5.5.

  • Add ReplicationType to ContainerStateMachine to tell if it is for a RATIS or a CHAINED pipeline.
  • Change CreatePipelineCommand to pass ReplicationType to ContainerStateMachine.
  • Refactor removeGroup from ClosePipelineCommandHandler to XceiverServerRatis (an existing TODO)

Out of scope:

What is the link to the Apache JIRA

HDDS-17809

How was this patch tested?

By adding new tests using AI.

CI passed (before rebased with master): https://github.com/szetszwo/ozone/actions/runs/43930861415

@adoroszlai adoroszlai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @szetszwo for the patch.

(The fix for unrelated failure in acceptance (compat-old) is in progress: #12076).

Comment on lines +119 to +126
private static boolean hasGroup(XceiverServerRatis s, RaftGroupId groupId) throws Exception {
for (RaftGroupId id : s.getServer().getGroupIds()) {
if (id.equals(groupId)) {
return true;
}
}
return false;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be missing something, but this seems to be the same as s.getServer().getGroupIds().contains(groupId). If so, we can improve the assertions like:

-assertTrue(hasGroup(peerServer, groupId));
+assertThat(peerServer.getServer().getGroupIds()).contains(groupId);

to get better failure messages.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As shown below, hasGroup is called to check for both true and false. So, assertThat(..). contains(..) does not work.

//testRemoveGroup
    assertTrue(hasGroup(peerServer, groupId));

    server.removeGroup(groupId, null);

    assertFalse(hasGroup(server, groupId));
    // For RATIS, the group is also removed from the peer; for CHAINED, only locally.
    assertEquals(type == ReplicationType.CHAINED, hasGroup(peerServer, groupId));

@adoroszlai adoroszlai Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assertThat(...).doesNotContain(...) can be used to replace assertFalse.

Comment on lines +126 to +127
// For CHAINED, each datanode adds the group from its own command; see XceiverServerRatis.addGroup.
peers.stream().filter(d -> type != ReplicationType.CHAINED && !d.getID().equals(dn.getID()))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type is constant, so we can skip the stream() for CHAINED

@szetszwo

Copy link
Copy Markdown
Contributor Author

@adoroszlai , thanks for reviewing this! Just have pushed a commit to address your comments and some refactoring.


@SuppressWarnings("parameternumber")
public ContainerStateMachine(HddsDatanodeService hddsDatanodeService, RaftGroupId gid,
ReplicationType replicationType,

@amaliujia amaliujia Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe when I read section 4.2 from the design doc: https://github.com/apache/ozone/pull/11982/changes#diff-9f932235b1dd6d8e60276cef9b9b28dffeb74ad942931a34d414e301a41c1f12R220, it pretty much says we keep the container state machine unaware of the type. The type is stored in XceiverServerRatis and looked up on demand.

However the implementation here turns into the idea of having a replication type within the SM. Does then we turn the design to a chained pipeline binds with a dedicated Ratis group and ContainerStateMachine?

@szetszwo szetszwo Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amaliujia , adding ReplicationType to ContainerStateMachine is to avoid non-zero bcsid. If a ContainerStateMachine support both V1 and V3 containers, V1 containers will change the bcsid and cause confusion. In HDDS-17543, we will disable Raft in V3 ContainerStateMachine and disable StreamCommand in V1 ContainerStateMachine, in order to avoid mixing them.

Will update the design doc.

@szetszwo szetszwo Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... section 4.2 ....

@amaliujia , I forgot to mention that I compared the existing design in 4.2 (SCM reply) with the new Per-pipeline RocksDB marker in HDDS-17845. Per-pipeline RocksDB marker is cleaner.

Also, revised the design doc, please take a look.

@adoroszlai

Copy link
Copy Markdown
Contributor

thanks for reviewing this! Just have pushed a commit to address your comments and some refactoring.

Thanks, the updated patch LGTM. Let's wait for @amaliujia's review.

@szetszwo
szetszwo requested a review from amaliujia November 18, 2026 17:39

@amaliujia amaliujia left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Left two comments.

Meanwhile I tried to reason the problem of the per replication type SM design. It seems to me that the problem will be double overhead from Raft side because we double the RaftGroup/SM if a DN supports both RATIS and CHAINED replication type by different ContainerSM (we may be able to avoid a mixed case in a single SM but probably has no reason to have a DN only support one kind of SM),

It is definitely a trade-off though. IIUC the benefit of this design is to reduce our implementation complexity.


/** @return the {@link ReplicationType} of the given group. */
public ReplicationType getReplicationType(RaftGroupId groupId) throws IOException {
return ((ContainerStateMachine) server.getDivision(groupId).getStateMachine()).getReplicationType();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this addingGroups.put(groupId, type);, so do we need to go to the ContainerStateMachine to fetch the type? It looks like we can get the type directly by addingGroups.get(groupId).

if (type == null) {
// The type is unknown when the group is reloaded on restart or added by a peer.
// TODO HDDS-17845: support restart for CHAINED.
LOG.info("{}: ReplicationType is unknown, use {}", gid, ReplicationType.RATIS);

@amaliujia amaliujia Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we defer handing of the DN restart, but is this LOG.info becomes too noisy now after DN restarts?Per my understanding, the DN restart will lose the replication type from the ContainerSM as that was not persisted. After restart, every of the RaftGroup will create an unknown type log.

Then

  1. Maybe lower the level of this log?
  2. Probably we add a persisted metadata for RaftGroup then the replication type is managed by Ratis and can survive from node failures. This can be deferred (and likely we need Ratis side changes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants