镜像站点 · 本页由第三方 GitHub 只读镜像提供，非 GitHub 官方站点，不接受任何登录或凭据输入。前往 github.com
Skip to content

Investigate flaky test-dns #5554

Closed
Closed
Investigate flaky test-dns#5554
Labels
armIssues and PRs related to the ARM architecture.dnsIssues and PRs related to the dns subsystem.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@Trott
Trott
opened on Mar 3, 2016

Example failure:

Activity

  1. Trott
    added
    testIssues and PRs related to Node.js core tests and test infrastructure.
    armIssues and PRs related to the ARM architecture.
    on Mar 3, 2016
  2. mscdex
    added
    dnsIssues and PRs related to the dns subsystem.
    on Mar 4, 2016

  3. Trott commented on Mar 8, 2016

    @Trott
    Trott
    on Mar 8, 2016
    MemberAuthor

    Another failure:

  4. Trott commented on Mar 13, 2016

    @Trott
    Trott
    on Mar 13, 2016
    MemberAuthor

    And another:

  5. Trott commented on Mar 14, 2016

    @Trott
    Trott
    on Mar 14, 2016
    MemberAuthor

    It appears that this test is flaky due to timing out when the callback never fires on one of the two test cases:

    assert.doesNotThrow(function() {
  dns.lookup('www.google.com', {}, noop);
});

assert.doesNotThrow(function() {
  dns.lookup('www.google.com', {
    family: 4,
    hints: 0
  }, noop);
});

    I'm not sure what can be done (from within Node) to make sure those dns.lookup() calls always return. dns.getServers() returns an empty array so I'd imagine that these tests don't try to actually do the lookup. But run locally, if I replace noop with something that logs the found address, it is in fact finding the address. (Maybe c-ares is looking at my machine's DNS cache or something?)

    Anyway, if anyone has ideas how to make those tests more robust, I'd love to hear it. /cc @nodejs/testing @nodejs/build

  6. Trott commented on Mar 14, 2016

    @Trott
    Trott
    on Mar 14, 2016
    MemberAuthor

    Oh, I see now that I had it backwards in my mind: dns.lookup() always tries to resolve. So dns.getServers() is irrelevant. (It's relevant for dns.resolve(). I thought it was the other way around.)

    OK, so I'm thinking there's might be no foolproof way around this problem and maybe these test cases should be moved from parallel to internet. But I'd welcome a more informed opinion on this.

  7. Trott commented on Mar 17, 2016

    @Trott
    Trott
    on Mar 17, 2016
    MemberAuthor

    Maybe we can move those two tests into their own test file and find a way to replace cares.getaddrinfo() in lib/dns.js with a test double?

  8. Trott commented on Mar 25, 2016

    @Trott
    Trott
    on Mar 25, 2016
    MemberAuthor

    Another one: https://ci.nodejs.org/job/node-test-binary-arm/1489/RUN_SUBSET=0,nodes=pi2-raspbian-wheezy/console

  9. Trott commented on Mar 31, 2016

    @Trott
    Trott
    on Mar 31, 2016
    MemberAuthor

    Still happening: https://ci.nodejs.org/job/node-test-binary-arm/1540/RUN_SUBSET=1,nodes=pi2-raspbian-wheezy/console

    Any ideas, anyone? Pretty thin roster using git shortlog but maybe @bnoordhuis or @evanlucas might have some ideas what's going wrong here?

  10. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    Specific part of the test that's being problematic was added in 43067864. /cc @cjihrig although I'm really just kind of throwing darts at a wall to pick out names to cc at this point...

  11. cjihrig commented on Apr 1, 2016

    @cjihrig
    cjihrig
    on Apr 1, 2016
    Contributor

    Do those requests come back, given enough time? I don't think 4306786 would cause the flakey behavior.

  12. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    They timeout after 120 seconds, so I think the answer is "no, they don't come back".

    I imagine this is related to the general uptick in pi2-raspbian-wheezy flakiness we've seen lately but I don't know what the ultimate source is. It does seem to be primarily or exclusively in tests that involve network connections (to localhost in most/all cases, but that's the nature of our CI tests). I haven't dug into c-ares so I'm not sure if this test might actually touch the network for certain configurations or whatever. It passes without any network connectivity, so it doesn't need the network.

    Yeah, and I agree that it seems unlikely that 4306786 is the problem. It's more likely the victim here than the perpetrator.

  13. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    Good news. I tried putting all those hints tests in their own file to see if that fixed the flakiness on the theory that the tests were either interacting with other tests or else cumulatively hitting some sort of threshold that was triggering Pi2 to sometimes drop a ... packet or something.

    Anyway, looks like that fixes it, and I'm all for splitting out the really big honkin' test files into smaller honkin' test files. So, win-win, I guess.

    Stress test on current master confirming flakiness of current test: https://ci.nodejs.org/job/node-stress-single-test/577/nodes=pi2-raspbian-wheezy/console

    Stress test on my branch confirming non-flakiness of the test-dns stuff split across two files: https://ci.nodejs.org/job/node-stress-single-test/576/nodes=pi2-raspbian-wheezy/console (215 tests and counting... still time for it to go sideways, but I'm optimistic...) UPDATE: 1 failure in 500+ runs, which is a whole better than the 10 failures in 100+ runs that we get on current master, but still, there's an issue...

  14. Trott
    added a commit that references this issue on Apr 1, 2016

    test: split out hints from test-dns

    140eedc

  15. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    Stress test for further splitting up the test more in the hops of increased reliability: https://ci.nodejs.org/job/node-stress-single-test/578/nodes=pi2-raspbian-wheezy/console

  16. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    OK, so it does use the network despite setServers([]) (which I guess I should have known). But since all we're testing is throws vs. not throws, we can change these to look up an empty string rather than actual hostname, since all the throwing stuff happens during parameter validation. That will avoid the network lookup and probably result in a completely reliable test. PR coming soon...

  17. Trott
    added a commit that references this issue on Apr 1, 2016

    test: fix test-dns.js flakiness

    943e402

  18. Trott commented on Apr 1, 2016

    @Trott
    Trott
    on Apr 1, 2016
    MemberAuthor

    New PR: #5996

  19. Trott
    added a commit that references this issue on Apr 4, 2016

    test: fix test-dns.js flakiness

    8d96300
  20. Trott
    closed this as completedin 8d96300on Apr 4, 2016
  21. MylesBorins
    added 2 commits that reference this issue on Apr 5, 2016

    test: fix test-dns.js flakiness

    0127c2b

    test: fix test-dns.js flakiness

    ccf90b6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    armIssues and PRs related to the ARM architecture.dnsIssues and PRs related to the dns subsystem.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions