improve device farm parallelism and reliability#72429
Draft
davidsbailey wants to merge 9 commits intostagingfrom
Draft
improve device farm parallelism and reliability#72429davidsbailey wants to merge 9 commits intostagingfrom
davidsbailey wants to merge 9 commits intostagingfrom
Conversation
Set OS_ACTIVITY_MODE=disable and OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES on Darwin at the top of runner.rb, before any require can initialize libsystem_trace or the ObjC runtime. Without this, Parallel.map's in_processes workers crash on macOS in libsystem_trace _os_log_preferences_refresh during their first AWS S3 log upload. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
create_mobile_session now also returns the picked Aws::DeviceFarm::Types::Device, so connect.rb can identify the provisioned device alongside the existing visual-log line in the cucumber HTML output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the runner's $options.device_farm and $options.force_db_access flags into the haml render context, and emit them in the rerun command only when set. Also switch the rerun-command construction to a flag array so non-Eyes builds no longer have a double-space artifact. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch the Aws::DeviceFarm::Client to retry_mode: 'adaptive' with retry_limit: 10 to absorb ThrottlingException bursts when many forked workers call create_test_grid_url near-simultaneously under high --parallel desktop runs. The token bucket is per-process post-fork, so this is per-worker absorption rather than cross-worker coordination. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
During the SauceLabs-to-Device-Farm migration we run both providers concurrently on the chef-managed test machine, where they collide on shared per-feature output filenames (Chrome_<feature>_output.html, the matching .rerun file, and the same-key S3 uploads). Append _df to the test_run_identifier when --device-farm is set, mirroring the existing _eyes pattern, and teach test_status.js to read a new haml-emitted hidden input so each provider's status page only matches its own keys. To remove once we settle on a single provider per browser, the change is the inverse of these three edits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A run with --parallel 50 still hit Aws::DeviceFarm::Errors::ThrottlingException on CreateTestGridUrl ~78s into the suite, where the existing 10-retry budget (worst-case ~150s of full-jitter backoff capped at 20s/attempt) isn't enough to ride out a multi-minute sustained throttle. 20 retries extends the worst-case window to ~350s without otherwise changing the retry policy. A separate AWS support ticket has been filed to raise the per-account TPS limit on CreateTestGridUrl. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
davidsbailey
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In preparation for moving real DTT traffic from saucelabs to device farm, this PR increases parallelism and makes some reliability and usability improvements:
runner.rb --parallel ...on mac--device-farmand--dbflags through to Copy Rerun Cmd button_df_output.htmlsuffix--magic_retrywith--retry_count 2--device-farmto Copy Rerun Cmd button for nowTesting story
To validate, I ran device farm rake tasks locally, with temporary modifications to make sure we (1) hit test-studio.code.org (2) skip tests which require
@dashboard_db_access.Device Farm Desktop
of the 25 successful reruns, 19 were due to
Aws::DeviceFarm::Errors::ThrottlingExceptionerrors which then passed on rerun.Device Farm Mobile
Deployment notes
@dashboard_db_accesstests are included.