Skip to content

HDDS-17463. Publish GitHub Actions summary for coverage check - #12054

Merged
adoroszlai merged 7 commits into
apache:masterfrom
kjs1095:HDDS-17463
Nov 17, 2026
Merged

adoroszlai merged 7 commits into
apache:masterfrom
kjs1095:HDDS-17463

Conversation

@kjs1095

@kjs1095 kjs1095 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

From JIRA description

Coverage check (runs for push builds) publishes report and posts to Sonar (only in Apache). It could also post a GitHub Actions summary to make it easier to quickly understand coverage of a commit without downloading the bundle.

Scope

  • New script parses the JaCoCo XML report produced by coverage.sh and prints a Markdown summary
  • The coverage job in ci.yml gets a new step after the archive step that appends the output to $GITHUB_STEP_SUMMARY
  • Unit tests in test_coverage_summary.py, run through a bats wrapper like junit_summary.bats.

Markdown coverage summary content

(Refer screenshots in section How was this patch tested?)

  • Overall coverage table
    • two-row table for lines and branches (coverage, missed, total)
    • one line count of untested methods and classes
  • Download link to artifact
  • Collapsed <details> block with per-package table
    • sorted by missed lines + missed branches, so the biggest gap come first

Design decisions

  • Combined coverage using SonarCloud's formula : (covered lines + covered branches) / (lines + branches) [1]
    • This is a very common standard, and the numbers are easy to understand.
  • Ignore the INSTRUCTION and COMPLEXITY JaCoCo counters
    • These two numbers are similar to LINE and BRANCH. [2] These two metrics will not give any more helpful insights.
  • Output the package level coverage table
    • Master has 316 packages (2026/11) and the size of the resulting package-level table is ~21KB, well inside the 1MB summary limit. [3]
    • Why just package-level?
      • Class-level metrics provide too much information. I believe package-level already give us big hint about which module worth to enhance coverage.
    • Why sorted by missed lines + missed branches?
      • Sorting by percentage would put tiny packages (a few constants) at the top

Reference

[1] https://docs.sonarsource.com/sonarqube-cloud/managing-your-projects/metric-definitions
[2] https://www.jacoco.org/jacoco/trunk/doc/counters.html
[3] https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands#step-isolation-and-limits

Co-authored-by: Claude Code (Claude Fable 5.1) noreply@anthropic.com

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17463

How was this patch tested?

Local ran:

[ozone] $ cd dev-support/ci && python3 -m unittest test_coverage_summary -v

test_invalid_report (test_coverage_summary.TestMain) ... ok
test_missing_report (test_coverage_summary.TestMain) ... ok
test_prints_summary (test_coverage_summary.TestMain) ... ok
test_combined_uses_sonar_formula (test_coverage_summary.TestParseReport) ... ok
test_overall_counters (test_coverage_summary.TestParseReport) ... ok
test_packages_counters (test_coverage_summary.TestParseReport) ... ok
test_percent (test_coverage_summary.TestPercent) ... ok
test_no_packages (test_coverage_summary.TestRenderSummary) ... ok
test_overall_section (test_coverage_summary.TestRenderSummary) ... ok
test_package_table_sorted_by_missed (test_coverage_summary.TestRenderSummary) ... ok
test_report_url (test_coverage_summary.TestRenderSummary) ... ok

----------------------------------------------------------------------
Ran 11 tests in 0.009s

OK

Fork CI ran CI Run Link

  • all checks passed
  • rat.sh check passed

Screenshot

Item Screenshot
Coverage Summary Screenshot 2026-11-15 at 16 42 48
Expand Package-level Screenshot 2026-11-15 at 16 43 09

kjs1095 and others added 5 commits November 15, 2026 15:11
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… in CI

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…en report

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot added the CI label Nov 15, 2026

@adoroszlai adoroszlai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kjs1095 for the patch, LGTM.

Comment on lines +18 to +21
@test "coverage_summary.py unit tests" {
cd "${BATS_TEST_DIRNAME}"
run python3 -m unittest test_coverage_summary
[ "$status" -eq 0 ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For follow-up: I think we should add a separate check for Python tests to avoid the need to hook these into the bats check.

@kjs1095 kjs1095 Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

While checking dev-support/ci/selective_ci_checks.sh, contributors need to understand that any new Python utility requires its own .bats file. (Or the test code will not be executed). It's unnecessary cognitive load.

Happy to take this as a follow-up!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File HDDS-17840.

@adoroszlai adoroszlai changed the title HDDS-17463. Publish coverage summary to GitHub Actions job summary HDDS-17463. Publish GitHub Actions summary for coverage check Nov 15, 2026

@chihsuan chihsuan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! @kjs1095 This looks good overall. I left two small inline comments.

same formula as SonarCloud

FYI, on master 8220773 this PR's summary showed 81.8%, while Sonar showed 79.9%. The gap mostly comes from coverage.sh class selection, so I filed HDDS-17836. No change needed here.

with mock.patch.dict(os.environ, {}, clear=True):
self.assertNotIn("📦 Full HTML report: [download artifact]", self.render())

def test_package_table_sorted_by_missed(self):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this doesn't really test sorting by missed, since the samples are already in name order. It still passes with key=lambda p: p.name. Could we add a case where the two orders disagree?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

Will add cases for sorting by missed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised by 2240eb2

Comment thread dev-support/ci/coverage_summary.py Resolved
@adoroszlai
adoroszlai requested a review from chihsuan November 17, 2026 10:58
@adoroszlai

adoroszlai commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Thanks @kjs1095 for updating the patch.

The failed kubernetes (ozone-ha) check can be safely ignored for this PR.

@kjs1095

kjs1095 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Updated

I updated the follow-up Jira link (I copied the wrong ticket link).
Thanks @adoroszlai for pointing out this.

====
Hi @adoroszlai ,
Thanks for the review and feedback!

I filed HDDS-17840 as a follow-up.
We can continue the problem description and proposal discussion in that ticket.

@kjs1095
kjs1095 requested a review from adoroszlai November 17, 2026 11:04
@kjs1095

kjs1095 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Hi @chihsuan ,
Thanks for the review and feedback.

I responded to each feedback with commits.
PTAL

@adoroszlai

Copy link
Copy Markdown
Contributor

I filed HDDS-17836 as a follow-up.

I guess it's HDDS-17840.

@chihsuan chihsuan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! @kjs1095 LGTM

@adoroszlai
adoroszlai merged commit d265f9b into apache:master Nov 17, 2026
5 checks passed
@adoroszlai

Copy link
Copy Markdown
Contributor

Thanks @kjs1095 for the patch, @chihsuan for the review.

@kjs1095
kjs1095 deleted the HDDS-17463 branch November 17, 2026 20:22
@adoroszlai

Copy link
Copy Markdown
Contributor

🎉 https://github.com/apache/ozone/actions/runs/44127889626/attempts/1#summary-134325487829

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants