Repository navigation
CI tests with Selenium 2/3 use outdated browsers from 2016/2021 #72
Description
The problem
Our CI matrix uses selenium/standalone-<browser>:<2.53.1|3|4> docker images. SeleniumHQ rebuilds them when browsers/drivers update, but only for Selenium 4:
- Selenium 4 images: floating tags (
4,latest) move to the newest build, e.g.selenium/standalone-firefox:4gave Firefox 155.0.1 on 2026-09-05 and Firefox 156.0 on 2026-09-17. - Selenium 3 images: the last build is 3.141.59-20210929 with Firefox 92.0.1 / geckodriver 0.29.1, Chrome 94 / ChromeDriver 94.
- Selenium 2.53.1 images: Firefox 47.0.1 and Chrome 53-ish with ChromeDriver 2.24.
So Selenium 2/3 jobs test browsers from 2016/2021, while people running Selenium 2/3 locally have current browsers (browsers auto-update, the jar doesn't).
As a result, a failure on Selenium 4 and a pass on Selenium 3 looks like a Selenium issue, while it's a browser version difference. That's what happened in #69: the same 2 HoverTest tests fail on Selenium 3.141.59 + Firefox 156. Browser version isn't printed in CI logs either, only in the Selenium log in the artifacts.
What works with current browsers
- Selenium 3: works with current browsers, since it just proxies to geckodriver/ChromeDriver (checked locally with Firefox 156 and Brave).
- Selenium 2.53.1 + Chrome: works with current Chrome when
goog:chromeOptions.w3cisfalse(tested locally with Brave 154 / ChromeDriver 154: full test suite passes, with Brave Shields disabled viagoog:chromeOptions.prefs, as its fingerprinting protection fakes window sizes reported to JS). - Selenium 2.53.1 + Firefox: can't be helped, its FirefoxDriver doesn't support Firefox 48+, so the image is already at the latest possible version.
Possible solutions
- Print browser/driver versions in each job.
- For Selenium 3 jobs, run the Selenium jar directly and install the latest browser and driver on the runner (e.g.
browser-actions/setup-firefox,browser-actions/setup-chrome) instead of using docker images. MinkSelenium2Driver ran the jar this way until minkphp/MinkSelenium2Driver@a385124 switched it to docker images.
To avoid conditionals inci.yml, abin/start-selenium.sh <browser> <selenium>script could pick docker or jar, print browser/driver versions and exportWEB_FIXTURES_HOST/DRIVER_MACHINE_BASE_PATHto$GITHUB_ENV(these are<server>inphpunit.xml.distnow, so they'd have to become<env>to be overridable). Same script works locally.
Activity
uuf6429 commentedon Oct 11, 2026
Option 1 is a very nice to have, but isn't really a solution.
Option 2 is (IMO) not a good idea because it will make the build non-reproducible without a lot of manual effort (plus gh actions are effectively a block box).
I'm starting to wonder if it could maybe make sense to come up with our own docker images (perhaps heavily influenced from the official selenium one). The thing is we are really offering the mink part of the entire solution, but while that is reasonable, it's still a tad inconvenient to our end users.
We've been depending on the selenium images (at work and here) for such a long time that it maybe makes sense to take over that and provide a comprehensive solution (on the other hand I understand that it's a lot of effort and some maintenance cost). 🤔
aik099 commentedon Oct 11, 2026
The Option 1 I've sent in another PR.
As for Option 2 I also thought to fork SeleniumHQ repo and manually rebuild Selenium 3 docker images (we can store them on GitHub side as well like in "driver-testsuite" repo) to have latest browser version in them. Theory was good, but:
So custom docker image (unless it's selenium 4 based) is guaranteed to have outdated browser version.
As for reproducibility, yes if problem is with older browser version but not newer one, then yes only gh action run logs would help and no local debug could be done.
aik099 commentedon Oct 11, 2026
Right now GH Actions Runner is x86_64 and using arch-matching Docker images. This might (see below) create a problem for macOS-based developers using Apple Silicon (aarch64) Macs (like myself).
With the macOS 28.x update (coming in ~Oct 2027) the x86_64 emulation layer (Rosetta) would be dropped. This will prevent the ability to run x86_64 apps. What I'm uncertain about is how Docker itself is able to run
amd64images onaarch64Macs:aik099 commentedon Oct 11, 2026
I wanted to see myself how it goes on MinkSelenium2Driver repo using the minkphp/MinkSelenium2Driver#417. I've made 2 attempts:
setup-chromeandsetup-firefoxactions (Chrome for Testing 155.0.8059.39, Firefox 157.0.1)By the way here are browsers/selenium/selenium-related-drivers pre-installed on the GH Actions Runner:
https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md#browsers-and-drivers