slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1612989	dotnet/maui	Controls Shell	Log
1612942	dotnet/maui	Controls Shell	Log
1612809	dotnet/aspnetcore	Tests: macOS	Log
1612587	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#134666
1612412	dotnet/maui	Controls ScrollView	Log
1612380	dotnet/maui	Controls Shell	Log
1612218	dotnet/maui	Controls Layout	Log	dotnet/maui#38805
1612215	dotnet/maui	Controls Shell	Log	dotnet/maui#38888
1612197	dotnet/maui	Controls Shell	Log
1612148	dotnet/maui	Controls Shell	Log
1612146	dotnet/maui	Controls Shell	Log	dotnet/maui#38111
1612069	dotnet/maui	Controls Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#38856
1612037	dotnet/maui	Controls Shell	Log	dotnet/maui#38887
1612038	dotnet/maui	Controls Layout	Log	dotnet/maui#38843
1612079	dotnet/aspnetcore	Tests: macOS	Log
1611986	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1611911	dotnet/runtime	osx-arm64 Debug CoreCLR_AllSubsets	Log	dotnet/runtime#134040
1611877	dotnet/maui	Controls Layout	Log	dotnet/maui#38827
1611813	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1611800	dotnet/maui	Controls Page,Performance,Picker,ProgressBar	Log
1611791	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#134634
1611743	dotnet/maui	Controls Shell	Log	dotnet/maui#38861
1611699	dotnet/maui	Controls Layout	Log	dotnet/maui#38822
1611719	dotnet/maui	Controls Shell	Log	dotnet/maui#38858
1611680	dotnet/maui	Controls Shell	Log
1611641	dotnet/maui	Controls Shell	Log	dotnet/maui#38863
1611546	dotnet/maui	Controls Layout	Log	dotnet/maui#38859
1611456	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1611447	dotnet/maui	Controls Shell	Log	dotnet/maui#38855
1611439	dotnet/maui	Controls Shell	Log
1611347	dotnet/maui	Controls Shell	Log
1610845	dotnet/maui	Controls Shell	Log	dotnet/maui#38603
1611209	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#133304
1611087	dotnet/maui	Controls Shell	Log
1611030	dotnet/maui	Controls Layout	Log	dotnet/maui#38723
1610658	dotnet/maui	Controls Shell	Log
1610648	dotnet/maui	Controls Shell	Log	dotnet/maui#38843
1610842	dotnet/maui	Controls Shell	Log	dotnet/maui#38840
1610687	dotnet/maui	Controls Shell	Log	dotnet/maui#38659
1610657	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1610571	dotnet/maui	Controls Shell	Log	dotnet/maui#38842
1610569	dotnet/maui	Controls Shell	Log	dotnet/maui#38723
1610451	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130407
1610360	dotnet/maui	Controls Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#38840
1610342	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1610301	dotnet/maui	Controls Layout	Log	dotnet/maui#38805
1610256	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
1610263	dotnet/maui	Controls Layout	Log	dotnet/maui#38367
1610177	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1610190	dotnet/aspnetcore	Tests: macOS	Log
1610078	dotnet/maui	Controls Layout	Log
1609990	dotnet/maui	Controls Shell	Log	dotnet/maui#38550
1609988	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1609930	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#134078
1609924	dotnet/aspnetcore	Tests: macOS	Log
1609431	dotnet/maui	Controls Shell	Log
1609820	dotnet/maui	Controls Shell	Log	dotnet/maui#38067
1609762	dotnet/maui	Controls Shell	Log
1609689	dotnet/maui	Controls Shell	Log	dotnet/maui#38825
1609437	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Release	Log
1609584	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#134547
1609551	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1609093	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#134405
1609505	dotnet/runtime	osx-x64 Debug PALTests	Log
1609438	dotnet/maui	Controls Shell	Log	dotnet/maui#38822
1609441	dotnet/aspnetcore	Tests: macOS	Log
1609347	dotnet/maui	Controls Shell	Log
1609321	dotnet/maui	Controls Shell	Log
1609262	dotnet/aspnetcore	Tests: macOS	Log
1609173	dotnet/maui	Controls Shell	Log	dotnet/maui#38780
1609055	dotnet/maui	Controls Shell	Log	dotnet/maui#38808
1609102	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1608883	dotnet/maui	Controls Shell	Log
1608843	dotnet/maui	Controls Shell	Log	dotnet/maui#38805
1608707	dotnet/maui	Controls Shell	Log	dotnet/maui#38804
1608678	dotnet/maui	Controls Shell	Log
1608726	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#134075
1608616	dotnet/maui	Controls Shell	Log
1608494	dotnet/maui	Controls Shell	Log	dotnet/maui#38801
1608456	dotnet/runtime	osx-arm64 Debug CoreCLR_AllSubsets	Log	dotnet/runtime#134497
1608378	dotnet/maui	Controls Shell	Log	dotnet/maui#38798
1607982	dotnet/maui	Controls Layout	Log	dotnet/maui#38367
1608170	dotnet/maui	Controls Shell	Log
1608229	dotnet/maui	Controls Shell	Log
1608168	dotnet/maui	Controls Shell	Log	dotnet/maui#38786
1607955	dotnet/maui	Controls Shell	Log	dotnet/maui#38659
1608028	dotnet/maui	Controls ScrollView	Log
1608014	dotnet/maui	Controls Shell	Log	dotnet/maui#38067
1608011	dotnet/maui	Controls Shell	Log	dotnet/maui#38274
1607973	dotnet/maui	Controls Shell	Log	dotnet/maui#38533
1608008	dotnet/maui	Controls Shell	Log	dotnet/maui#38723
1607992	dotnet/maui	Controls Shell	Log	dotnet/maui#37952
1607995	dotnet/maui	Controls Shell	Log	dotnet/maui#37690
1607998	dotnet/maui	Controls Layout	Log	dotnet/maui#37337
1607963	dotnet/maui	Controls Shell	Log	dotnet/maui#38625
1607978	dotnet/maui	Controls Shell	Log	dotnet/maui#38445
1607960	dotnet/maui	Controls Shell	Log	dotnet/maui#38657
1607972	dotnet/maui	Controls Layout	Log	dotnet/maui#38534
1607968	dotnet/maui	Controls Shell	Log	dotnet/maui#38585
1607945	dotnet/maui	Controls Shell	Log	dotnet/maui#38744
Displaying 100 of 700 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
22	168	700

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.

[missymessa]

@jeffschwMSFT is this still an issue?

[lewing]

@jeffschwMSFT is this still an issue?

@missymessa does it make sense to ask this question about issues that have active hit from today/this week?

[missymessa]

@lewing given that these issues are over a year old, I'm checking to make sure they're still relevant for folks in the event an error happened with Build Analysis that would prevent the issue from updating on hit counts :)

[lewing]

@lewing given that these issues are over a year old, I'm checking to make sure they're still relevant for folks in the event an error happened with Build Analysis that would prevent the issue from updating on hit counts :)

Ah! Thanks for the explanation. One way to check if the issue is still happening is to see if the recent failed builds listed in the description are actually recent or alternatively, have already been deleted)