I think these failures are benign -- the instances are actually starting -- but they seem to happen a bunch, so something still seems a bit wrong.
Command aws --region us-east-2 ec2 run-instances --client-token 99279408467-f63dc9960af60e05532c64c6c67e2ceb6ea2017e --image-id resolve:ssm:latest-gha-runner-ami --instance-type c8a.8xlarge --instance-initiated-shutdown-behavior terminate --launch-template LaunchTemplateName=gha-runner,Version=$Latest --tag-specifications ResourceType=instance,Tags=[{Key="Name",Value="rust-lang-rust-try - dist-x86_64-linux-quick-33319674725-99279408467-f63dc9960af60e05532c64c6c67e2ceb6ea2017e"},{Key="bors-terminate",Value="true"},{Key="bors-repo",Value="rust-lang/rust"},{Key="bors-job-id",Value="99279408467"},{Key="bors-job-name",Value="try - dist-x86_64-linux-quick"},{Key="bors-run-id",Value="33319674725"},{Key="bors-build-kind",Value="try-perf"},{Key="bors-pr-number",Value="161902"}] --user-data <REDACTED> ended with status exit status: 254.
aws: [ERROR]: An error occurred (IdempotentParameterMismatch) when calling the RunInstances operation: Arguments on this idempotent request are inconsistent with arguments used in previous request(s).
I think we had poked at this briefly and concluded it's because GitHub sends the webhook twice, which means that these are actually different calls to RunInstances with the same idempotency token but different user-data, due to different GitHub JIT tokens -- i.e., this is actually the right behavior, we don't want a second instance.
It seems worth adding some handling and hiding/treating differently the idempotency mismatch from this so the logs are less noisy, especially if we start using EC2 more.
I think these failures are benign -- the instances are actually starting -- but they seem to happen a bunch, so something still seems a bit wrong.
I think we had poked at this briefly and concluded it's because GitHub sends the webhook twice, which means that these are actually different calls to RunInstances with the same idempotency token but different user-data, due to different GitHub JIT tokens -- i.e., this is actually the right behavior, we don't want a second instance.
It seems worth adding some handling and hiding/treating differently the idempotency mismatch from this so the logs are less noisy, especially if we start using EC2 more.