Skip to content

Clarify Production List and Submission rules - #361

Merged
seattleplus merged 3 commits into
mainfrom
clarify-production-and-submission-rules
Oct 2, 2026
Merged

seattleplus merged 3 commits into
mainfrom
clarify-production-and-submission-rules

Conversation

@seattleplus

Copy link
Copy Markdown
Contributor

Summary

Clarifies and improves the readability of the Production List and Submission rules in the_lists.php and rules_submission.php:

  • Production List Overview & Eligibility (the_lists.php):
    • Moves the core definition of a "Production System" to the top of the Production List Overview section and adds a Production List Terms subheading.
    • Recommends engaging with the IO500 Steering Committee at least 3 months prior to list release for new Production List storage systems, award contenders, or eligibility questions.
    • Clarifies that benchmarks must run on the exact file system namespace and configuration used by Production Applications, and that eligibility requirements must be maintained to remain on the Production List.
    • Clarifies that Production Data must represent the majority of data stored in the storage system, and that all Single Point of Failure mechanisms must be enabled during the benchmark run.
  • Reproducibility Overview (the_lists.php):
    • Breaks out the Fully Reproducible criteria into an explicit bulleted list covering exact version availability, public architectural design details, multi-region availability, and the 3-month production deployment window.
    • Fixes wording, grammar, and copy-paste issues across the list and reproducibility descriptions.
  • Submission Rules (rules_submission.php):
    • Clarifies the definition of persistent storage (surviving a permanent data center power outage).
    • Fixes HTML list nesting so the source-code modification policy and 300-second stonewall rule are top-level items rather than nested under read-after-write semantics.

@seattleplus
seattleplus requested review from a team as code owners September 30, 2026 18:59
@jeanbez jeanbez added this to the SC26 milestone Sep 30, 2026
jeanbez
jeanbez previously approved these changes Sep 30, 2026
@gflofst
gflofst self-requested a review October 1, 2026 11:38
gflofst
gflofst previously approved these changes Oct 1, 2026
Comment thread templates/Pages/the_lists.php Outdated
introducing a storage system not currently on the Production List,
targeting an award, or having any questions regarding eligibility
should review their submission details with the IO500 Steering
Committee at least 3 months prior to the release of a new list. Early

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a high bar - that would make any new contributor ineligible for SC this year, so it seems late to make this kind of rule change. We don't even get the CFS out 3 months in advance.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the same thing, but didn't have a better answer. How do we get them to engage early so we don't have to delist on problems? The other thing the 3 month standard would impose on us is code changes would be frozen then too. I'm not thrilled with that. We could make it work, but it is a change for us.

Maybe drop this item for now, but the onus is on us to process new submiters and high placement first giving the benefit of the doubt to the lower places. We still reserve the right to change a ranking after publishing the list.

We might want to encourage submissions at the start of the submission period to ensure we have adequate time to work through any new submission problems. Lack of responsiveness on their part during validation makes the entry potentially ineligible as it couldn't be validated in time. We need to give ourselves space to respond as well.

Any other ideas?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about "It is advisable...."?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is any expectation that code is frozen 3 months in advance of submission. It just means "your product has to exist for some time before it can be considered production". 3 months isn't going to make the difference between research and development. I do think that industry presence, customers, etc. are important, and TBH Sugon has a lot of that already.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely no intention of freezing code. "advisable to review their submission details" is all I'm saying here. The goal of this text is to just have them reach out in advance so we can give them early advice so we avoid any last minute disappointments. I added "It is advisable", instead of 'should', any other update to the text you can think of to indicate it is simply request for early engagement?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to be clear we are talking about the metadata rather than the score. Maybe, "it is advisable to engage with the committee three months in advance with descriptive information other than the final submission score if the score is likely to place highly or win as well as all new entrants. This will give time for the committee to work with the submitters to potentially achieve the desired list categorization (i.e., entry on the production list)".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the input, I updated with some new text based on your suggestion,
" The Production List sets a very high bar for inclusion.
It is advisable to engage with the committee at least three months
in advance with descriptive information (no score required) of your submission to work
through any initial eligibility issues for the Production List.
Early engagement ensures the steering committee has sufficient time
to engage with submitters on their desired list categorization
(i.e., entry on the production list). This is especially
encouraged for submitters introducing a storage system not currently
on the Production List or believe their their submission score has the
potential to win an IO500 award."

@seattleplus
seattleplus dismissed stale reviews from gflofst and jeanbez via f66c996 October 1, 2026 16:49
Revised guidelines for engaging with the IO500 Steering Committee regarding submissions to the Production List.
@seattleplus
seattleplus requested a review from adilger October 1, 2026 22:47
@seattleplus
seattleplus merged commit d8f6730 into main Oct 2, 2026
2 checks passed
@seattleplus
seattleplus deleted the clarify-production-and-submission-rules branch October 2, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants