Repository navigation
Slow "Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release" #65626
Description
Mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release takes around 2.5H to finish.
There are some interesting anomalies in the logs, e.g.:
(I checked various runs)
It says that prejitting of a single managed assembly Microsoft.Win32.SystemEvents.dll takes almost 10 minutes 😮 (mostly in LLVM's opt+llc)
I parsed the output into an excel table:
is it possible to move some libs/tests to the outerloop? e.g. JIT/Methodical/MDArray/GaussJordan/classarr_cs_do/classarr_cs_do test. And I guess we need to figure out what exactly makes Microsoft.Win32.SystemEvents.dll so long to prejit - there are not much stuff in it.
Reacted by SingleAccretion
Activity
steveisok commentedon Feb 20, 2022
Adding @SamMonoRT
vargaz commentedon Feb 20, 2022
Yes, these are very slow, they run opt+llc on unlinked assemblies.
agocke commentedon Mar 4, 2022
This test run is pretty regularly timing out -- can we get someone to investigate if the slowness is a bug, or if we need to adjust the timeout?
SamMonoRT commentedon Mar 7, 2022
This PR (#66157) should help ease the timeouts seen in last couple weeks. Even with that fix, the lane is 2.5+hrs long. Still discussing this, but we might possibly 1. want to exclude certain long running tests as part of PR runs in this lane, 2. Extend the timeout to stabilize CI in the short term
agocke commentedon Mar 7, 2022
@SamMonoRT which PR?
SamMonoRT commentedon Mar 7, 2022
#66157
agocke commentedon Mar 7, 2022
Looks like that resolved the problem. I'm going to close this out for now.
8 remaining items
SingleAccretion commentedon Jan 24, 2023
Wanted to mention that we should be careful to leave enough testing on PRs to reliably catch failures introduced by adding new Jit tests. In my experience these are not uncommon.
Move mono llvm fullaot runtime test leg to rolling build (#81125)
SamMonoRT commentedon Feb 8, 2024
@kotlarmilos @vitek-karas - not sure if this is something your team owns now and what more remains here? Please can you re-assign as appropriate.
steveisok commentedon Feb 8, 2024
I'll take this as it likely has to do w/ the aot compiler performance itself.
agocke commentedon Feb 8, 2024
My general philosophy is, "PR is for fast reliable tests" so I agree with the theory of moving everything out, then moving things back in that meet that criteria. Ideally we can find the sweet spot of fast + high confidence in finding bugs.