slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1559417	dotnet/aspnetcore	Tests: macOS	Log
1559361	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132502
1559310	dotnet/maui	Controls Shell	Log	dotnet/maui#37642
1559353	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132521
1559292	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132520
1559266	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132227
1559190	dotnet/aspnetcore	Tests: macOS	Log
1559111	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131866
1558980	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132440
1558955	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1558953	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1558819	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37645
1558829	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#132489
1558805	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131745
1558783	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1558768	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37111
1558757	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132306
1558743	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131145
1558721	dotnet/maui	Controls Shell	Log	dotnet/maui#37034
1558693	dotnet/maui	Controls CollectionView	Log
1558677	dotnet/aspnetcore	Tests: macOS	Log
1558663	dotnet/maui	Controls WebView	Log
1558652	dotnet/maui	Controls CollectionView	Log
1558646	dotnet/maui	Controls CollectionView	Log
1558643	dotnet/maui	Controls CollectionView	Log
1558640	dotnet/maui	Controls Shell	Log
1558639	dotnet/maui	Controls CollectionView	Log
1558522	dotnet/maui	Controls Shell	Log
1558360	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37424
1558366	dotnet/maui	Controls Shell	Log
1558478	dotnet/aspnetcore	Tests: macOS	Log
1558374	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#132348
1558272	dotnet/maui	Controls Shell	Log
1558287	dotnet/maui	Controls ScrollView	Log
1558290	dotnet/maui	Controls Shell	Log
1558284	dotnet/maui	Controls Shell	Log
1558275	dotnet/maui	Controls Shell	Log
1558278	dotnet/maui	Controls Shell	Log
1558281	dotnet/maui	Controls Layout	Log
1558293	dotnet/maui	Controls Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1558143	dotnet/maui	Controls CollectionView	Log
1558104	dotnet/maui	Controls Shell	Log
1557963	dotnet/aspnetcore	Tests: macOS	Log
1557882	dotnet/maui	Controls Shell	Log	dotnet/maui#37392
1557779	dotnet/maui	Controls Shell	Log
1557540	dotnet/maui	Controls Layout	Log	dotnet/maui#37426
1557559	dotnet/maui	Controls Shell	Log
1557508	dotnet/aspnetcore	Tests: macOS	Log
1557433	dotnet/aspnetcore	Tests: macOS	Log
1557306	dotnet/maui	Controls Shell	Log
1557344	dotnet/maui	Controls Shell	Log	dotnet/maui#37565
1551774	dotnet/maui	Controls Shell	Log	dotnet/maui#37111
1557310	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132339
1557212	dotnet/maui	Controls ScrollView	Log	dotnet/maui#37575
1557071	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132419
1556889	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132117
1556649	dotnet/maui	Controls Shell	Log	dotnet/maui#37438
1556617	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132450
1556261	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37546
1556214	dotnet/runtime	osx-arm64 Debug CoreCLR_AllSubsets	Log	dotnet/runtime#131655
1555888	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37424
1556081	dotnet/aspnetcore	Tests: macOS	Log
1555903	dotnet/aspnetcore	Tests: macOS	Log
1555865	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132306
1555776	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37392
1555767	dotnet/aspnetcore	Tests: macOS	Log
1555693	dotnet/maui	Controls Shell	Log
1555718	dotnet/aspnetcore	Tests: macOS	Log
1555508	dotnet/aspnetcore	Tests: macOS	Log
1555375	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132275
1555310	dotnet/maui	Controls CollectionView	Log
1555279	dotnet/aspnetcore	Tests: macOS	Log
1555056	dotnet/maui	Controls CollectionView	Log
1554878	dotnet/aspnetcore	Tests: macOS	Log
1554620	dotnet/aspnetcore	Tests: macOS	Log
1554582	dotnet/maui	Controls Layout	Log
1554548	dotnet/maui	Controls Shell	Log
1554509	dotnet/maui	Controls Shell	Log
1554500	dotnet/maui	Controls CollectionView	Log	dotnet/maui#37404
1554496	dotnet/maui	Controls Shell	Log
1554435	dotnet/aspnetcore	Tests: macOS	Log
1554285	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132353
1554278	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1554246	dotnet/maui	Controls Border,BoxView,Brush,Button	Log
1554269	dotnet/aspnetcore	Tests: macOS	Log
1554258	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129220
1554239	dotnet/aspnetcore	Tests: macOS	Log
1554194	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131507
1554097	dotnet/runtime	osx-arm64 Release Libraries_CheckedCoreCLR	Log
1554076	dotnet/sdk	Mac arm64	Log	dotnet/sdk#55785
1553914	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132326
1553863	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132319
1553795	dotnet/maui	Controls Layout	Log	dotnet/maui#37438
1553745	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#132098
1553731	dotnet/aspnetcore	Tests: macOS	Log
1553554	dotnet/maui	Controls CollectionView		dotnet/maui#37034
1553504	dotnet/aspnetcore	Tests: macOS	Log
1553489	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129853
1553382	dotnet/maui	Controls Layout	Log	dotnet/maui#37182
1553310	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130859
Displaying 100 of 985 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
45	153	985

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.