Repository navigation
assertion failure in src/node_worker.cc with large number of workers #31614
Description
Activity
node: ../deps/uv/src/unix/core.c:556: uv__close_nocheckstdio: Assertion `fd > -1' failed.In the assertion failure list, I see this too.
/cc @nodejs/libuv
Does the process runs out of file descriptors? The first assertion is node's problem; it assumes that
uv_loop_init()never fails but it can.How did you trigger the second one? Can you get a backtrace?
(gdb) where #0 0x00007ffff6e04377 in raise () from /lib64/libc.so.6 #1 0x00007ffff6e05a68 in abort () from /lib64/libc.so.6 #2 0x0000000000a9dea1 in node::Abort() () #3 0x0000000000a9df17 in node::Assert(node::AssertionInfo const&) () #4 0x0000000000b3ad12 in node::worker::Worker::Run() () #5 0x0000000000b3b880 in node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::{lambda(void*)#1}::_FUN(void*) () #6 0x00007ffff71a3ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffff6ecc8cd in clone () from /lib64/libc.so.6
@bnoordhuis - thanks! the first assertion has this backtrace; not able to obtain for the second one. Looks like it is always triggered by a worker thread, while the main thread is already processing the first assertion failure, because the second one occurs only when first one is present (this is my guess, no proof)
running out of file descriptors looks like a possibility; I will debug from that angle.
not able to obtain for the second one.
If you turn on coredumps, you should be able to get a backtrace. You may need to select the right thread in gdb or just run
thread apply all backtrace.(gdb) where #0 0x00007f92514c7377 in raise () from /lib64/libc.so.6 #1 0x00007f92514c8a68 in abort () from /lib64/libc.so.6 #2 0x00007f92514c0196 in __assert_fail_base () from /lib64/libc.so.6 #3 0x00007f92514c0242 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000a03319 in uv__close_nocheckstdio (fd=-24) at ../deps/uv/src/unix/core.c:556 #5 0x00000000013cd271 in uv__close_nocheckstdio (fd=fd@entry=-24) at ../deps/uv/src/unix/core.c:563 #6 0x00000000013de2f7 in uv__read_proc_meminfo (what=what@entry=0x20be946 "MemTotal:") at ../deps/uv/src/unix/linux-core.c:1016 #7 0x00000000013df7f3 in uv_get_total_memory () at ../deps/uv/src/unix/linux-core.c:1043 #8 0x0000000000a0dc05 in node::SetIsolateCreateParamsForNode(v8::Isolate::CreateParams*) () #9 0x0000000000b3a3e7 in node::worker::Worker::Run() () #10 0x0000000000b3b930 in node::worker::Worker::StartThread(v8::FunctionCallbackInfo<v8::Value> const&)::{lambda(void*)#1}::_FUN(void*) () #11 0x00007f9251866ea5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f925158f8cd in clone () from /lib64/libc.so.6
this is the stack trace for the second assertion. Looks like the loop creation failed, but the worker creation sequence has progressed thus far, wrongly?
when we run out of descriptors, the error is
-24, which is returned byuv_loop_init. thefdat the assertion site is also the same. Is it by chance? or do we fill theuv_loop_twith the error codes?On the other hand, in its current form, #31621 will address this one too?
@gireeshpunathil No, #31621 is unrelated to that second failure, but libuv/libuv#2645 should have fixed that assertion (
fdshould not be negative anymore when being passed touv__close_nocheckstdio). That was landed in 05d350a – is your local copy of Node up to date?Reacted by Gireesh Punathil@addaleax - that is really promising! my copy does not have it; will check and confirm. [ since the recreate is consistent, the validation will be easy ]
- added a commit that references this issue
on Feb 17, 2020 - added a commit that references this issue
on Feb 27, 2020 - added 2 commits that reference this issue
on Mar 15, 2020 - added a commit that references this issue
on Mar 30, 2020 - added a commit that references this issue
on Jul 27, 2026
I was debugging #23277 and came across this:
$ cat bar.js
$ node --max-old-space-size=100000 bar
I guess this has to do with libuv failure due to lack of memory, but can this be better handled?
/cc @nodejs/workers