Run CI "fire-and-forget" with --github-actions on Bencher Cloud

[jkaczman]

Hi! We're exploring adding Bencher to our GitHub CI (Pro plan): pgdogdev/pgdog#1617

We want to be able to run Bencher "fire-and-forget" instead of keeping a runner alive polling, so that we can terminate early (detach) from our Blacksmith runner where we run the Bencher CLI to start the task, and still allow Bencher to comment when it's done if there's any regressions. This doesn't seem possible right now with a GITHUB_TOKEN as it'll expire when the Action job ends (and, regardless, it doesn't look like that's sent to Bencher's servers)

Curious if we have any options around this? I don't mind contributing something up to address this, e.g., a per-project webhook that fires repository_dispatch to trigger a secondary workflow to add/update a comment.

Thanks!

[epompeii]

@jkaczman thank you for using Bencher, and I'm sorry things aren't working as expected!

This is definitely a use case that I want Bencher to support!
Currently, you have to keep a CI runner waiting for Bencher Bare Metal results to complete if you want to have the results posted. There's already the --detach flag however there's currently no mechanism for Bencher to push the results back to you.

My current plan is to implement a per-bencher run callback mechanism. If you use the --detach flag, you can also specify a callback: URL, headers, and JSON body template. When the bare metal job reaches a terminal job lifecycle state, Bencher sends the callback POST.

To make the GitHub Actions use case easier, if you specify the --detach flag + the --github-actions option a callback will be created for you that invokes repository_dispatch with the necessary client payload. The --github-actions token would be a GITHUB_TOKEN with at least permissions: { checks: write } to set the check at the start of the run. A new --ci-callback-token​ option would be added to pass in a fine-grained PAT for your repository with the contents: write permission to invoke the repository_dispatch in the callback.

A new --job option would also be added that allows you to reattach to a job. In a repository_dispatch workflow you would have a second bencher run invocation that uses the new --job option to reattach to the completed job and report the results (PR comment, GitHub Check, etc) using that workflow's GITHUB_TOKEN with permissions: { pull-requests: write, checks: write }.

Please, let me know what you think! cc @levkk