Repository navigation
TIME OUT - Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release #66083
Description
We couldn't find any obvious change that could be causing it between these commits:
git diff a82d92c..c8da2fd
Log which the timeout started:
https://dev.azure.com/dnceng/public/_build/results?buildId=1633880&view=results
Activity
thaystg commentedon Mar 2, 2022
@MattGal did we have any relevant infra change between friday and saturday?
simonrozsival commentedon Mar 2, 2022
The timeout seems to be related to my PR which was merged on Friday (#65128). I'll try reverting it and see if it helps.
MattGal commentedon Mar 2, 2022
Aside from serious "world-on-fire" hot-fixes we will not make infrastructure changes on a Friday evening or Saturday morning. If something's going to roll out and affect you, it'll almost always be on a Wednesday in the morning time for the Pacific time zone.
If your investigation does make you think there are infra issues, please feel free to also/instead tag @dotnet/dnceng as sometimes I take vacation days and there are other folks who might reply faster on those days. Including things like the Job GUID (sometimes called Correlation id), Work item friendly name, and why you think we caused your timeout can also speed investigation. Sounds like we're waiting for Simon to check it out first, though.
vargaz commentedon Mar 2, 2022
This seems to be caused by more assemblies getting AOTed.
In particular, every test project now seems to include TestLibrary.dll, so we end up AOTing it like 2000 extra times. The original lane already took about 2 hours, so these extra AOT steps might push it over the timeout.
vargaz commentedon Mar 2, 2022
The extra TestLibrary dll was added by
572405a#diff-13390079908954088047032dd93629fec2bcfd68749c2c7422c9ed60511ea108L73
Which adds TestLibrary to every test project.
lewing commentedon Mar 2, 2022
cc @trylek
naricc commentedon Mar 2, 2022
@trylek @jkoritzinsky Did you complete the test grouping change for this lane? That should have made it faster, since there would be less redundant AOT work.
jkoritzinsky commentedon Mar 2, 2022
We're part-way through the work. We added the
TestLibraryreference to each project to make it simpler than adding the reference to each project individually. If that's causing issues, then we can move to manually adding the project references to the required projects.9 remaining items
fanyang-mono commentedon Mar 4, 2022
With @trylek's PR, the
Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 releaselane still time out. And the time out currently was because the tests running on helix step took too long to finish (10min -> 1h). @simonrozsival Could you re-open your revert PR and merge main to it to see if the testing time would go down? (#66088) Since the time out issue is not fully fixed, I am reopening it now.SamMonoRT commentedon Mar 4, 2022
The aot time build issue seems to be fixed by the PR - but we still are seeing some tests running longer and exceeding the overall timeout of the lanes. Changing ownership of issue till we determine what is causing the tests to run longer than expected.
MattGal commentedon Mar 4, 2022
Please don't forget that ARM64 test machines are a limited, not-scalable resources and the runtime team is sending thousands of hours of work through them. Being more clever saves time but we can't forget that we have a static number of limited resources.
fanyang-mono commentedon Mar 7, 2022
The relevant CI lanes are back to normal now. Closing this issue.