Repository navigation
Segmentation fault of node 10.8.0 x64 on Fedora 27 when running from PATH #22319
Description
Activity
Just tried
node-v10.8.1-nightly2018081382830a809b-linux-x64, and it works fine. So it seems this issue was already fixed somewhere along the way.Update, ok this is super-strange... never mind my last comment, the nightly has the same issue. It appears that the problem happens only when node is executed via a directory in the
PATH:Works fine when running
nodewith the full path specified:$ strace -e trace=file /opt/node-v10.8.0-linux-x64/bin/node execve("/opt/node-v10.8.0-linux-x64/bin/node", ["/opt/node-v10.8.0-linux-x64/bin/"...], 0x7ffcae816b30 /* 125 vars */) = 0 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/opt/node-v10.8.0-linux-x64/bin/node", O_RDONLY|O_CLOEXEC) = 9 openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 9 readlink("/proc/self/exe", "/opt/node-v10.8.0-linux-x64/bin/"..., 8191) = 36 readlink("/proc/self/fd/1", "/dev/pts/16", 255) = 11 stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0 openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 9 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 10 readlink("/proc/self/fd/2", "/dev/pts/16", 255) = 11 stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0 openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 11 getcwd("/home/raman/tmp", 4096) = 16 getcwd("/home/raman/tmp", 4096) = 16 readlink("/proc/self/fd/0", "/dev/pts/16", 255) = 11 stat("/dev/pts/16", {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 16), ...}) = 0 openat(AT_FDCWD, "/dev/pts/16", O_RDWR|O_CLOEXEC) = 12 > +++ exited with 0 +++But borks when running the same binary from the
PATH:$ strace -e trace=file node execve("/opt/node-v10.8.0-linux-x64/bin/node", ["node"], 0x7ffd6ef99d30 /* 125 vars */) = 0 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 9 openat(AT_FDCWD, "/lib64/libnode.so", O_RDONLY|O_CLOEXEC) = 9 openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 9 readlink("/proc/self/exe", "/opt/node-v10.8.0-linux-x64/bin/"..., 8191) = 36 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x25108dda45} --- +++ killed by SIGSEGV (core dumped) +++ fish: “strace -e trace=file node” terminated by signal SIGSEGV (Address boundary error)Also, it works fine as
root, but not as an unprivileged user. Selinux does not appear to be the culprit.- changed the title
[-]Segmentation fault of node 10.8.0 x64 on Fedora 27[/-][+]Segmentation fault of node 10.8.0 x64 on Fedora 27 when running from `PATH`[/+]on Aug 14, 2018 Stack trace of thread 91027: #0 0x00007ff3938b8c5a _ZN2v816FunctionTemplate3NewEPNS_7IsolateEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEENS_5LocalIS4_EENSA_INS_9SignatureEEEi (libnode.so)This is running
nodedirectly from the tar.xz download on nodejs.org (sha256 sum verified).The tar.xz download from nodejs.org doesn't contain nor link to
libnode.soso this looks like interference with another Node.js installation.Reacted by Refael AckermannThe tar.xz download from nodejs.org doesn't contain nor link to libnode.so so this looks like interference with another Node.js installation.
Ah, interesting. Do you have any idea why this might be happening? As you can see from the
stracethe rightnodebinary is being executed. I confirm it does not link tolibnode.so:# ldd /opt/node-v10.8.0-linux-x64/bin/node linux-vdso.so.1 (0x00007fff845e3000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f573a53d000) librt.so.1 => /lib64/librt.so.1 (0x00007f573a335000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5739fad000) libm.so.6 => /lib64/libm.so.6 (0x00007f5739c62000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5739a4b000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f573982d000) libc.so.6 => /lib64/libc.so.6 (0x00007f5739477000) /lib64/ld-linux-x86-64.so.2 (0x00007f573a741000)And this issue does not happen with either node 8 or 9, which are installed in exactly the same way as 10.
Attached is the output of
LD_DEBUG=all node. It contains the following:120607: file=libnode.so [0]; dynamically loaded by node [0] 120607: find library=libnode.so [0]; searching 120607: search cache=/etc/ld.so.cache 120607: trying file=/lib64/libnode.soI found out that
libnode.sowas owned byatom-libs, which appears to have been a leftover package since I uninstalled atom a while ago. Removing this package has fixed the problem, but I still find it odd that node 10 attempts to dynamically load this shared lib, which node 8 and 9 did not, and only in some cases such as not running it with an absolute path. While this is solved from my perspective, it certainly seems like this could be some bad logic in node 10.Reacted by JarykAt this point I'd cc @nodejs/platform-linux but we don't appear to have such a group 😞 .
Reacted by Refael AckermannReacted by Raman GuptaOn Fedora 28 I had the same issue. I have no clue where my libnode.so came from. But thanks for giving the full debug logs. Learnt something new about strace and coredumps. 👍
Reacted by Raman Gupta@hmt You can try
rpm -q --whatprovides /path/to/your/libnode.soThe path you can get from the output of
LD_DEBUG=all node.That might tell you something useful about which rpm it came from, and if it isn't a package on your system then it was probably installed manually at some point.
I still think that something is wrong with nodes internals though.
@nodejs/build Any thoughts/insights on this one?
nope, and I agree, Node shouldn't be trying to load libnode if it's a stand-alone build. I'm trying to find something in the commit list for 10 that are not in 8 that might point to a change that might be implicated but nothing is standing out. There has been a little bit of work to make compiling Node as a shared library so perhaps it's related to that. 🤷♂️ @bnoordhuis any thoughts on where to look for this one?
If It's specific to Fedora and/or other Red Hat OSes, maybe @danbev or @lance might have an idea?
Nothing I can think of without having a closer look I'm afraid. If I have time I'll try to take a look later this week but it might not be until next week as this is a short week for me (not working Friday).
#25733 looks similar and was reported on Ubuntu.
@rocketraman - is it still reproducible?
IMO this has something to do with dynamic linker / loader's symbol lookup being over-riden by some system settings.
excerpts from the dump:
120607: relocation processing: /lib64/libnode.so 120607: symbol=_register_async_wrap; lookup in file=node [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libdl.so.2 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/librt.so.1 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libstdc++.so.6 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libm.so.6 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libgcc_s.so.1 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libpthread.so.0 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libc.so.6 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/ld-linux-x86-64.so.2 [0] 120607: symbol=_register_async_wrap; lookup in file=/lib64/libnode.so [0] 120607: binding file /lib64/libnode.so [0] to /lib64/libnode.so [0]: normal symbol `_register_async_wrap'this would mean the symbol
register_async_wrap, despite being present (statically bound) innode, was superseded inlibnode.so!excerpts from strace:
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 9 openat(AT_FDCWD, "/lib64/libnode.so", O_RDONLY|O_CLOEXEC) = 9the library was loaded (soon?) after the ld cache was contacted, indicating the library being selected by some system (ld) config?
is there any output from
$ ldconfig -p | grep nodeis there any output from
$ env | grep LD_@rocketraman - is it still reproducible?
@gireeshpunathil Yes, easily reproducible.
is there any output from
$ ldconfig -p | grep node
$ env | grep LD_$ ldconfig -p | grep node libnode.so.64 (libc6,x86-64) => /lib64/libnode.so.64 libnode.so (libc6,x86-64) => /lib64/libnode.so $ env | grep LD_ <no output>As I noted in comment #22319 (comment) the latter of those two comes from atom-libs:
# rpm -q --whatprovides /lib64/libnode.so atom-libs-1.3.0-1.fc21.x86_64and when that is removed, the problem goes away.
As also mentioned previously, when node is run as
nodeit SIGSEGV, but if run with the absolute path, it works fine.@rocketraman - I tried the same steps, but could not reproduce this issue. At the same time, I am unable to mimic the setting that resulted in your ouput of
ldconfig -p | grep node- I installed atom usingdnf install. Do you know what system configuration resulted inlibnode.sobeing specially treated byldconfig?@gireeshpunathil I installed atom a long time ago, and I found the rpm for
atom-libswhich provides that extra libnode.so in my local dnf repo. Here it is:Give that a shot?
Reacted by Gireesh Punathilah! that did the trick! thanks.
[root@cd2735ca52b7 foo]# node Segmentation fault [root@cd2735ca52b7 foo]#[root@cd2735ca52b7 foo]# ldconfig -p | grep node libnode.so (libc6,x86-64) => /lib64/libnode.so [root@cd2735ca52b7 foo]#same as your case!
I will debug this!
ok, further insights:
[root@cd2735ca52b7 foo]# rpm -qp --scripts ./atom-libs-1.3.0-1.fc21.x86_64.rpm warning: ./atom-libs-1.3.0-1.fc21.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID f4928260: NOKEY postinstall program: /sbin/ldconfig postuninstall program: /sbin/ldconfig [root@cd2735ca52b7 foo]#this
rpmindeed updateslibnode.sofor dynamic linker's run-time bindings cache, effectively bringing the symbols of the library for lookup for executables built with runtime linking option (-lrt).If the object reference (node) does not contain slashes in it, the dynamic linker/loader searches the cache file /etc/ld.so.cache for libraries to be used for symbol lookup that is required in the said object. In this case, the cache was populated by
ldconfig.upto node v11.6.0 , node is built with
-lrtoption[root@cd2735ca52b7 foo]# ldd /foo/node-v11.6.0-linux-x64/bin/node linux-vdso.so.1 (0x00007ffe64b2d000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f9ba9764000) librt.so.1 => /lib64/librt.so.1 (0x00007f9ba955c000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f9ba91d4000) libm.so.6 => /lib64/libm.so.6 (0x00007f9ba8e89000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9ba8c72000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9ba8a54000) libc.so.6 => /lib64/libc.so.6 (0x00007f9ba869e000) /lib64/ld-linux-x86-64.so.2 (0x00007f9ba9968000) [root@cd2735ca52b7 foo]#pay special attention to
librt.so.1.So this explains:
- why the issue does not occur with node v11.7.0 and above (specifically with commit 03ec4cea30)
- why the issue does not occur with full path reference to node (slashes in the path)
Hope this helps.
@rocketraman - any concerns / observations on the above finding? I am closing this as resolved, feel free to re-open if you think there is something outstanding.
Thanks @gireeshpunathil -- note that node 8 and 9 also did not suffer from this issue, so it looks like the librt stuff was added in 10, and then removed again in 11.7? I can't pretend to understand all the intricacies of this, but it sounds like the behavior has been explained, and good workarounds are available, so I'm good with closing it. Thanks!
Reacted by Gireesh Punathilcan someone explain this issue in layman term? I'm new in nodejs.
@od2 Basically node 10 to 11.7 had an alternate library loading mechanism (via librt) which was broken in certain situations, which includes loading an external (and incorrect) libnode when running node from the PATH (i.e. having no slashes in the executable name).
To "solve" the problem do one of the following:
- use a node version before 10 or after 11.7,
- run node with an absolute path i.e.
/path/to/nodeinstead of justnode, or - delete the external libnode.so
I get a segfault just by running
node-- no arguments, just trying to get an interactive shell:Here is the output from
coredumpctl:This is running
nodedirectly from the tar.xz download on nodejs.org (sha256 sum verified).Node 8 and 9 work without any issues.