feature: Support glob patterns in --file

[gamma0987]

The bencher run --file parameter accepts only literal file paths. When a benchmark harness writes its output to files created during the run (with paths not known ahead of time), there is no way to point bencher run at them in a single invocation.

Motivation

Note for gungraun this requires #962. Tools like gungraun support --save-summary=json, which writes machine-readable JSON summaries to target/gungraun/** during the benchmark run. The exact paths include benchmark names, functions, ids, ..., so they cannot be predicted before the run starts.

The natural invocation would be:

bencher run --adapter rust_gungraun_json --file 'target/gungraun/**/summary.json' \
  "cargo bench -- --save-summary=json"

But this fails today because:

• Bencher does not expand glob patterns
• Shell glob expansion happens at argument parse time, before the benchmark command runs, so it cannot see files that do not exist yet.

Proposed behavior

Allow --file to accept glob patterns, expanded at read time (after the benchmark command completes). This would make the workflow above work as written and generalizes to any tool that writes reports to a directory tree.

Workaround

Run the benchmark outside of bencher run, then invoke bencher with the already-resolved paths:

cargo bench -- --save-summary=json
bencher run --adapter rust_gungraun_json \
  $(find target/gungraun -name 'summary.json' | sed 's/^/--file /')

This loses bencher's command wrapping and exit code propagation into the run result.

[epompeii]

The exact paths include benchmark names, functions, ids, ...

I'm not sure this is going to pair all that elegantly with how multiple --file options are currently handled by bencher:

The -⁠-file option can be used to read benchmark results from a file. You can specify the -⁠-file option multiple times to read from multiple files. If you do specify the -⁠-file option multiple times, then each file will be considered a separate iteration of the benchmark command. When displayed, these results will be shown in separate tables.

If you wish to combined these results into a single iteration, then you can set the -⁠-iter option to 1 and set the -⁠-fold option to either the min, max, or median aggregate functions. This will fold the results into a single iteration and display them in a single table. Do not set the -⁠-fold option to mean.

🐰 Tips for using the -⁠-file option

So it can work out of the box today, but it certainly isn't pretty. We should probably put some more thought into how to better handle this.

Is this how the JSON output support in #962 would need to operate?

[gamma0987]

So it can work out of the box today, but it certainly isn't pretty. We should probably put some more thought into how to better handle this.

Agreed. Maybe I misunderstood the --file argument. Would using a single --file argument with a glob work, where all glob matches are unioned into a single iteration. One combined result set with all the benchmarks, instead of separate iterations? --fold would still work as intended; it just doesn't make much sense for gungraun benchmarks, but maybe for other frameworks.

Is this how the JSON output support in #962 would need to operate?

No, that's why I created a separate issue. There are two different command-line options: --output-format=default|json|pretty-json controls the output to stdout. --save-summary=json|pretty-json tells gungraun to save the summary in json format to disk. This writes one summary file per benchmark. All files combined describe a single benchmark run with cargo bench. The rust_gungraun_json adapter can be used for both of them since the files have the same schema as the stdout json output.

I thought about your comment in #962:

I can see this being a footgun for Bencher users where they also want a somewhat human readable JSON output in a file that they then pass to bencher.

The optimal solution would be to keep the default output format for users and readability and use the stored summary.json files for more reliable bencher parsing.

[epompeii]

Would using a single --file argument with a glob work, where all glob matches are unioned into a single iteration. One combined result set with all the benchmarks, instead of separate iterations?

The issue here and why I settled on the less than perfect "each --file option is an iteration" is:

1. This breaks the separation between iterations and the adapters in BMF. That is, iteration on the backend would need to be come adapter aware.
2. How do we handle collisions? Error? First iteration wins? Last iteration wins? None of these felt good to me.

--fold would still work as intended; it just doesn't make much sense for gungraun benchmarks, but maybe for other frameworks.

Could you see a user wanting to run Gungraun multiple times (--iter) and then take the maximum for say cache hits (--fold max)?

There are two different command-line options: --output-format=default|json|pretty-json controls the output to stdout. --save-summary=json|pretty-json tells gungraun to save the summary in json format to disk.

Ah, I did not know about --save-summary. Hm... If Bencher says "we support Gungraun JSON output", I think it is understandable that users would assume we can handle all four combinations: (stdout || [single] file) x (minified || pretty)

This writes one summary file per benchmark. All files combined describe a single benchmark run with cargo bench.

Why the separate files? Would you consider creating an aggregate file per run?

The optimal solution would be to keep the default output format for users and readability and use the stored summary.json files for more reliable bencher parsing.

Yeah, I agree that's the ideal. I would prefer that Bencher would support all forms of "JSON" output. Currently that's four different permutations.
If you get rid of the --output-format and create a single file --save-summary for me to grab, that would drop it down to two. This would be a breaking change on the Gungraun side though.

[gamma0987]

Thanks for the detailed feedback. Let me address the BMF separation concern directly, because I think there's a mismatch between what I'm proposing and what you're worried about.

The glob expansion is purely a CLI-side file-reading concern. I'm not proposing that the backend merge anything, or that iterations become adapter-aware. The idea is:

1. After the benchmark command completes, expand the glob pattern (e.g. --file 'target/gungraun/**/summary.json').
2. Read all matched files and concatenate their contents into a single string (in memory or temp file if needed).
3. Feed that single string to the existing adapter pipeline as one result.

The backend sees one result string, produces one AdapterResults, counts as one iteration. No fold involved, no adapter awareness, no iteration logic changes. The glob expansion happens entirely in FilePath::get_results before any adapter or backend code runs.

On collisions: The behavior is whatever the adapter already does when it encounters duplicate benchmark names in a single result string. No new semantics are introduced. For gungraun, which keys by benchmark name in a HashMap, duplicates are last-wins per measure key, same as if a single summary file contained two entries with the same name. And to be clear, glob isn't limited to JSON adapters: concatenating two Criterion stdout files or two Rust built-in harness files works fine as long as the benchmark names are disjoint. The glob feature is general; someone running different sets of benchmarks with different command line arguments could merge all resulting files and treat them as if the benchmarks were executed in one go.

Why the separate files? Would you consider creating an aggregate file per run?

Totally, I love the idea of an aggregate file as an additional option but that wouldn't reduce the amount of permutations, although I don't think that this is a big concern. However, I understand your point that the ideal path for a bencher run is gungraun writing a single aggregate --save-summary file. That would eliminate the need for the glob, but that's a breaking change on the gungraun side. In the meantime, glob support in --file would let users work with the current per-benchmark output format.

On --iter + --fold: Those are orthogonal to this feature. The --iter loop wraps around the entire runner, and glob doesn't change that. For bencher run --file '...' -- cargo bench (command + --file), each iteration re-runs the command, re-expands the glob, and re-reads the files, then the iterations are folded by the existing mechanism. For bencher run --file '...' (just --file, no command), --iter just re-reads the same files, so --iter > 1 is pointless, but that's already the case today without glob. Either way, glob doesn't alter the iteration or fold semantics.

Could you see a user wanting to run Gungraun multiple times (--iter) and then take the maximum for say cache hits (--fold max)?

I think for the majority of cases it doesn't make sense even for cache metrics because the difference between two deterministic runs should be insignificant or in the best case around zero. But there are of course cases when it makes sense to run gungraun and specific benchmarks multiple times. For example for indeterministic benchmarks which use HashSet/HashMap accesses or similar and when these metrics have a substantial share on the total metrics. In general when randomness takes over you want and need to run benchmarks multiple times so --iter is a nice feature for gungraun users.

[epompeii]

2. Read all matched files and concatenate their contents into a single string (in memory or temp file if needed).

Having the bencher CLI concatenate the contents is the issue for me here. I want the bencher CLI to stay as simple as possible. All user-supplied data manipulation and parsing (which would include concatenation) should happen on the API server, not the bencher client. This has been a guiding design principle for Bencher for several years: keep the client as simple as possible.

but that's a breaking change on the gungraun side.

Could this be an additive change? It would ~double the size on disk. However, you could have individual per benchmark output files (today) and add an aggregate, single file. Other benchmark harnesses with JSON output to a file have had a single file to grab.

But there are of course cases when it makes sense to run gungraun and specific benchmarks multiple times.

Sorry I picked a bad example 😃 I just wanted to double check to see if multiple iterations could potentially be useful, and it seems like the answer is yes. Thank you for the better example!