镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Intermittent hangs with 10.16.1 #28932

Description

@ofrobots

Over at Google Cloud we are noticing intermittent hangs on some virtual machines with the latest 10.16.1. This is likely due to the OpenSSL upgrade due to 1.1.1c (see openssl/openssl#9078).

Hangs seem to coincide with when the system has low-entropy and has the following stack:

(gdb) bt
#0  0x00007fb4b2d7403f in __GI___select (nfds=18, readfds=0x7ffdd2bf5c30, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00000000014d3c4c in rand_pool_acquire_entropy ()
#2  0x00000000014d2058 in rand_drbg_get_entropy ()
#3  0x00000000014cf5c4 in RAND_DRBG_instantiate ()
#4  0x00000000014cf8fe in drbg_setup ()
#5  0x00000000014cfbc7 in do_rand_drbg_init_ossl_ ()
#6  0x00007fb4b305d827 in __pthread_once_slow (once_control=0x2667838 <rand_drbg_init>, init_routine=0x14cfb80 <do_rand_drbg_init_ossl_>) at pthread_once.c:116
#7  0x00000000014febe9 in CRYPTO_THREAD_run_once ()
#8  0x00000000014ceb66 in drbg_status ()
#9  0x0000000000a104a0 in node::crypto::EntropySource(unsigned char*, unsigned long) ()
#10 0x00000000017aa753 in v8::base::RandomNumberGenerator::RandomNumberGenerator() ()
#11 0x00000000017a9901 in v8::base::CallOnceImpl(long*, std::function<void ()>) ()
#12 0x00000000017ac19f in v8::base::OS::GetRandomMmapAddr() ()
#13 0x0000000000af7947 in v8::internal::GetRandomMmapAddr() ()
#14 0x0000000000f037af in v8::internal::Heap::SetUp() ()
#15 0x0000000000fb6713 in v8::internal::Isolate::Init(v8::internal::StartupDeserializer*) ()
#16 0x00000000011ffd3a in v8::internal::Snapshot::Initialize(v8::internal::Isolate*) ()
#17 0x0000000000b14988 in v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) ()
#18 0x0000000000b149da in v8::Isolate::New(v8::Isolate::CreateParams const&) ()
#19 0x0000000000902954 in node::Start(int, char**) ()
#20 0x00007fb4b2c7eb97 in __libc_start_main (main=0x8bbd50 <main>, argc=2, argv=0x7ffdd2bf6788, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdd2bf6778) at ../csu/libc-start.c:310
#21 0x00000000008bbe85 in _start ()

Activity

  1. added
    opensslIssues and PRs related to the OpenSSL dependency.
    on Aug 2, 2019
  2. ofrobots commented on Aug 2, 2019

    @ofrobots
    ContributorAuthor

    Here's my experiment on some GCE virtual machines:

    for i in $(seq 1 50) ; do echo run $i; cat /proc/sys/kernel/random/entropy_avail ; time npm >/dev/null; sleep 1; done
    ...
    172
    
    real	0m0.257s
    user	0m0.222s
    sys	0m0.032s
    run 6
    173
    
    real	0m0.264s
    user	0m0.216s
    sys	0m0.044s
    run 7
    5
    ^C
    real	33m50.490s
    ...
    

    Note the entropy value of 5 at the time of the hang.

  3. addaleax commented on Aug 2, 2019

    @addaleax
    Member

    I think this is openssl/openssl#9078 / openssl/openssl#9084, and cherry-picking the latter would resolve this?

  4. ofrobots commented on Aug 2, 2019

    @ofrobots
    ContributorAuthor

    I believe so. I will try this out in the failing environment tomorrow.

  5. targos commented on Aug 2, 2019

    @targos
    Member

    I'm pretty sure I already saw an issue here about this a few days ago.

    Edit: forget it, it was before 10.16.1

  6. MylesBorins commented on Aug 5, 2019

    @MylesBorins
    Contributor

    Should we escalate a 10.16.2 to fix this? @ofrobots is there an easy way to test if I can get an rc together?

  7. MylesBorins commented on Aug 5, 2019

    @MylesBorins
    Contributor

    /cc @nodejs/lts

  8. ofrobots commented on Aug 5, 2019

    @ofrobots
    ContributorAuthor
  9. BethGriggs commented on Aug 7, 2019

    @BethGriggs
    Member

    This should be fixed with the v10.16.2 release, please reopen if not.

  10. bernd-edlinger commented on Aug 18, 2019

    @bernd-edlinger

    @ofrobots I am curious which linux version did you use where the available entropy
    dropped to zero again?

  11. ofrobots commented on Aug 19, 2019

    @ofrobots
    ContributorAuthor

    This was a Ubuntu instance IIRC.

  12. bernd-edlinger commented on Aug 19, 2019

    @bernd-edlinger

    I ask because the code with the code with the select should not be executed unless the
    linux kernel is older than v3.17, that is rather old, ubuntu 14.04 uses linux v3.13 for instance,
    but that is rather old, could you log in and try cat /proc/version, what does that look like?

  13. bernd-edlinger commented on Aug 19, 2019

    @bernd-edlinger

    The other way round I observed a similar behaviour that you described, when
    linux v4.8 or newer is used in an arm machine, and every 3-5 minutes when
    the CRNG gets reseeded it pulls all entroy from the input pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    opensslIssues and PRs related to the OpenSSL dependency.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions