[HeCBench] Dataset Provisioning [1/7]: Prepare run script - #2484
Conversation
The script had grown by accretion: one flat body of loops and inline
conditionals, mixed quoting styles, and lower-case locals that read the same as
the environment variables the run is configured with.
Adopt the shape run_composable-kernels.sh already uses in this directory, so the
two suite runners read alike:
- named functions in "function name {" form for the steps that stand on their
own -- the compiler cross-check, and the build-and-run of one benchmark,
- UpperCamelCase locals against ALLCAPS for the variables a caller may
override, which now tells the two apart at a glance,
- braced expansions and quoted paths throughout,
- a block comment per function saying why it exists, replacing the inline
remarks that had drifted from the code they described.
The benchmark loop keeps only what it is about: pick the model, take the
directories, run each, count the verdicts. The work that follows adds to this
shape rather than to the old one.
No behaviour change. Every message, log line, results.txt entry and exit status
is byte-identical; verified by running the old and the new script over nineteen
scenarios -- both models and each alone, an unknown and an empty model list, six
shapes of HECBENCH_LIST including a path, a trailing slash and a wildcard, a
LAUNCHER, a timeout, a failing clean, a missing checkout and a missing src --
and diffing their stdout, stderr and results.txt.
AI-assisted.
Two ways the file could mislead a reader, both of them older than the work that follows. A refusal exits before the results file is touched, so a missing checkout left the previous run's PASS lines and TOTAL block in place, with nothing to distinguish them from this run's. Invalidate the file at the start of the run, ahead of every refusal, rather than after the checks that precede it. tee reports a failed write on its own stderr and nowhere else, so a results file that could not be written - a read-only checkout, a full disk - produced a complete PASS report on the terminal, exit 0, and no file to read it back from. Truncate it once, before the first line is teed to it, and refuse the run when that does not work. AI-assisted.
The runner needed a checkout someone else had made, so it could not be the first thing run on a fresh machine: clone_test.sh had to go first, and forgetting it showed up as "HeCBench not found" rather than as a missing step. Clone it when it is not there. The clone lands in a staging directory and is moved into place only once git reports success, so an interrupted clone cannot leave a half-checkout behind that a later run would take for a good one. HECBENCH_CLONE=0 keeps the previous behaviour of requiring an existing checkout; HECBENCH_REPO and HECBENCH_BRANCH say what to clone. AI-assisted.
jplehr
left a comment
jplehr
left a comment
There was a problem hiding this comment.
Before doing any more meaningful review, one remark/question: This changes the way that we clone HeCBench to be part of this script.
I believe the original one would rely on clone_tests.sh to bring in the HeCBench sources. or am I missing something?
I understand that we currently have both ways in the repo and there is no strategy which one we use / should prefer. I had originally mentioned that should probably rely more on the clone_test.sh script as it would give us a single place for consistent handling of how things are done when we bring-in tests.
On the downside, this would, at least right now, also downloads truckloads of stuff we don't care about when we want to run only HeCBench.
What's your thoughts on that, @mhalk?
Discussed offline: For now this PR retains the clone mechanism, as multiple run scripts perform this action themselves. |
3 participants
Motivation
Prepare HeCBench's run-script for the feature addition: Dataset Provisioning
Technical Details
General canonicalization of script style, addition of repo cloning, etc.
Test Plan
Check behavior and make sure it performs correctly.
Test Result
Return codes and logged results as expected.
Submission Checklist