Skip to content

HDDS-17724. Limit command-status report size in datanode heartbeats. - #11962

Draft
sadanand48 wants to merge 23 commits into
apache:masterfrom
sadanand48:HDDS-17724
Draft

sadanand48 wants to merge 23 commits into
apache:masterfrom
sadanand48:HDDS-17724

Conversation

@sadanand48

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

HDDS-16723 added a count-based limit for reports in datanode heartbeats, but it did not limit their serialized size. A small number of command-status reports containing accumulated delete-block ACKs could still exceed ipc.maximum.data.length.

This PR:

  • Adds hdds.heartbeat.max.command.status.report.size, defaulting to 64MB.
  • Limits command-status data to the smaller of: the configured limit; or the remaining space within 90% of ipc.maximum.data.length.
  • Adds command-status reports after other heartbeat fields so their available budget reflects existing payload size.
  • Splits aggregate reports only between complete CommandStatus entries.
  • Never splits an individual delete ACK or its result list.
  • Leaves deferred reports queued for subsequent scheduled heartbeats.
  • Removes immediate follow-up heartbeats to avoid burst RPCs during backlog recovery.
  • Preserves selected reports and ordering when an RPC fails.
  • Adds per-endpoint metrics for queued, sent, deferred, requeued, and oversized command-status bytes.
  • Adds tests covering byte budgeting, splitting, ordering, retry, and heartbeat scheduling.
  • The existing report-count limit remains in place for full reports and ICR backlogs. This change adds a complementary byte-based safeguard for command-status reports.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17724

How was this patch tested?

Unit tests

Developed with help of Cursor (GPT-5.6 Sol)

…/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sadanand48 for the patch

@sumitagrawl sumitagrawl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sadanand48 given few comment

@sadanand48
sadanand48 requested a review from smengcl November 9, 2026 16:10
@sadanand48
sadanand48 requested a review from smengcl November 12, 2026 11:25

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@sumitagrawl Do you want to take another look?

@adoroszlai
adoroszlai marked this pull request as ready for review November 13, 2026 07:41
@adoroszlai

Copy link
Copy Markdown
Contributor

Thanks @sadanand48 for the patch.

Workflows are not currently being run on this fork due to the scale of GitHub Actions usage. A repository maintainer can choose to re-enable these

Can you please re-enable workflows in your fork? Afterwards you may want to manually disable some:

@sadanand48
sadanand48 marked this pull request as draft November 13, 2026 09:04
Comment on lines +79 to +80
public static final String HDDS_IPC_MAXIMUM_DATA_LENGTH =
"ipc.maximum.data.length";

@adoroszlai adoroszlai Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sadanand48 TestOzoneConfigurationFields failed. ipc.maximum.data.length should not be declared in HddsConfigKeys (since ozone-default.xml does not have it, nor should it), but in a non-config class (which is not checked by TestOzoneConfigurationFields). For now it can be moved to e.g. HeartbeatEndpointTask.

@adoroszlai

Copy link
Copy Markdown
Contributor

@sadanand48 Please don't re-run failed workflow of the outdated commit (853c5f8), it cancels the workflow for the newer commit (d35de55).

@sadanand48

Copy link
Copy Markdown
Contributor Author

I had a discussion with @sumitagrawl where he proposed approach 2 - the publisher-side alternative and attached the patch so we can compare both approaches before deciding.

Approach 1: split while constructing the heartbeat (the current PR)

CommandStatusReportPublisher remains unchanged and publishes one aggregate CommandStatusReportsProto.
During heartbeat construction, the aggregate is split only between complete CommandStatus entries.
The split uses the actual remaining heartbeat byte budget for that SCM endpoint.
The unsent remainder replaces the original queued report and is sent in later heartbeats.
Advantages:

  • Smaller, more localized change.
  • No change to the generic ReportPublisher API.
  • Uses the available heartbeat space efficiently.
  • Naturally handles different remaining budgets for different SCM endpoints.

Trade-off:
-StateContext needs splitting/remainder handling.
-The aggregate may be split independently for each endpoint.

Approach 2: split in CommandStatusReportPublisher

  • Change ReportPublisher#getReport() to getReports().
  • CommandStatusReportPublisher creates multiple reports bounded by actual serialized size.
  • All chunks are queued during the same publisher execution.
  • Heartbeat construction only admits complete queued reports that fit its remaining count and byte budgets.

Advantages:

  • Clear separation: the publisher creates bounded reports, while heartbeat construction only performs admission.
  • Simpler StateContext selection logic with no protobuf remainder replacement.
  • Chunking is performed once before reports are added to endpoint queues.

Trade-offs:

  • Broader API change affecting every report publisher.
  • It still requires heartbeat-side size accounting because FCRs, ICRs and actions consume the same RPC payload.
  • Less efficient packing: for example, a complete 30 MB report is deferred when only 20 MB remains, even if some statuses inside it could have fitted.
    It does not solve an individual CommandStatus exceeding the safe RPC size; both approaches have that limitation.
    Therefore, publisher-side splitting is mainly an ownership and simplification choice, not an additional wire-size guarantee

@sumitagrawl @smengcl @aryangupta1998 could you please share your thoughts to conclude on this one. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants