Repository navigation
investigate flaky sequential/test-benchmark-child-process on Windows #12560
Description
Activity
- added a commit that references this issue
on Apr 21, 2017 Guess it may be useful to loop in @nodejs/build too in case there's something relevant to know about the win2008r2 hosts....
- added a commit that references this issue
on Apr 21, 2017 Looking at https://ci.nodejs.org/computer/test-azure_msft-win2016-x64-6/builds , the job that run right after it killed a left-over
yes.exeprocess: https://ci.nodejs.org/job/node-test-binary-windows/RUN_SUBSET=2,VS_VERSION=vs2015,label=win2016/7860/consoleFullc:\workspace\node-test-binary-windows\RUN_SUBSET\2\VS_VERSION\vs2015\label\win2016>TASKKILL /F /IM yes.exe /T || TRUE SUCCESS: The process with PID 6256 (child process of PID 6652) has been terminated.Could that first test be leaving a
yes.exebehind draining CPU?Reacted by George AdamsI'm looking as well.
Could that first test be leaving a yes.exe behind draining CPU?
Sounds about right since the benchmarks use
yes.exe. I'm not sure whyyes.exeisn't terminating reliably, of course....@joaocgreis @refack Thanks for looking at this, by the way! I'm very grateful for that.
So, we need a way to make sure
yes.exeis reliably terminated after the benchmarks that use it are done. Has anyone else looked into this and gotten further than that? Just that info we already have is hugely helpful, but if there's more info to work with, I certainly want it. :-Dchild_process/child-process-exec-stdoutbenchmark usesexec('yes', ..)which will spawnyesinside a shell. Killing the shell does not always killyes, so we get a leftoveryesfrom time to time.We can try using
require('child_process').execSync(`taskkill /f /t /pid ${child.pid}`)
instead of
child.kill(). The/tswitch is for terminating entire process tree. From my tests it works, but I haven gone to testing if this solvestest-benchmark-child-processflakiness.Reacted by Rich Trott- added a commit that references this issue
on Apr 25, 2017 2 remaining items
- added a commit that references this issue
on Apr 28, 2017 - added a commit that references this issue
on May 16, 2017 - added a commit that references this issue
on May 18, 2017 - added a commit that references this issue
on Jul 19, 2017
sequential/test-benchmark-child-processis still failing sometimes flaky on Windows in CI. I'll open a PR to mark it as flaky. This issue is for trying to locate the problem and a solution.When the test succeeds, it seems to take just a few seconds.
https://ci.nodejs.org/job/node-test-binary-windows/RUN_SUBSET=3,VS_VERSION=vs2015,label=win2008r2/7865/console
When it fails, it's a timeout.
https://ci.nodejs.org/job/node-test-binary-windows/7867/RUN_SUBSET=3,VS_VERSION=vs2015,label=win2008r2/console
This would suggest a race condition or something else causing a child process to hang or something. And that might be the cause. But...
Interestingly, a stress test where the five benchmarks that this test calls were all split out into individual tests, succeeded but each test took around 30 seconds to run. Wha??!! I know! (Only other change in those tests is the
duroption for the benchmarks was increased from 0 to 0.1. Well, that, and that this test was run on win2016 so maybe the results are completely irrelevant? I don't know.)https://ci.nodejs.org/job/node-stress-single-test/1161/nodes=win2016/console:
So I'm not sure what's going on here. Maybe it can be worked out by someone more comfortable testing and debugging on Windows or someone more deeply familiar with child_process and/or our benchmarking code. @nodejs/platform-windows @nodejs/benchmarking @mscdex @cjihrig @bnoordhuis @nodejs/testing