Repository navigation
Fast fail feature for the node:test context #42990
Description
Activity
@nodejs/test_runner
GitHub Actions does provide something like this - https://github.com/proxy/docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast. Not sure if this is something that's already present in other existing test runners.
I believe jest does as well. It’s typically present in CI runners with a matrix, but it’s very rare for a test runner to have it.
Usually when running your tests, you don’t want it to fail fast - you want the maximal information. Otherwise, you’re constantly running, fixing one problem, and rerunning - it’s a game of whack-a-mole.
Reacted by Julian Gruber, Darshan Sen and Moshe AtlowReacted by Leonardo RaeleBut even though it might be rare for it to exist on other existing test runners, it can be useful for it to fail and not continue further with the other tests until the one failed is also fixed
Reacted by Steven and IlyaYou could move the part you would expect to fail the rest of the tests outside the
testand that should work as expected, right?const test = require('node:test'); test("test for myModule", async (t) => { let module = require('node:does-not-exist'); let instance; await t.test("using module feature 1", () => { instance = new module.SomeExportedClass("arg1", { option: true }); }); await t.test("request to API", async () => { await instance.apiEndpoint({ ... }); }); });
@vierofernando the claim i'm making is that it's rare because it's not actually useful.
With no fast-fail, if you only want one failure, you can control-C, or mentally ignore everything after the first failure - both of which are easy.
With fast-fail, if you want more than one failure, you have to rerun your entire test suite - which is hard.
I assume this depends on the test suite size.
When you have a large test suite, you want all tests to always run, because test runs are expensive.
When you have a small test suite, running it is cheap so the main point becomes convenience - you don't want to have to scroll through / see test output after the first failure, because that's already all you want to know.
Reacted by Aranđel Šarenac and Ilya@juliangruber sure! but since running is cheap, if you want the first failure, you can quickly jump to the top of the scrollback (which you cleared prior to running the test suite) and it'll be right there :-)
Good point! You could also use Unix tools like less for this. Or have a test runner that offers this.
It's a question of dx vs simplicity, or docs I would say?
Reacted by Jordan HarbandI wouldn't be surprised if 80% of node users don't know how to clear their scrollback
Reacted by Aranđel Šarenacagreed, but i'd also be surprised if even 1% of that set of users wanted to see one failure at a time ;-)
Reacted by Julian Gruber and Alex YangI think your example is not a good idea.
testAPI shouldn't be related to each other.the correct API and usage should be like
let module; let instance; before("importing module", async () => { module = await import("..."); }); before("using module feature 1", () => { instance = new module.SomeExportedClass("arg1", { option: true }); }); test("request to API", async () => { await instance.apiEndpoint({ ... }); });
Reacted by Antoine du HamelThere has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.
For more information on how the project manages feature requests, please consult the feature request management document.
It's easily possible to bail out on the first failure by implementing your own entry using the run API if that's what you desire.
It seems like unnecessary complexity to support this feature in the default CLI, IMO.
I'd suggest closing this as "Won't do."
I use Fail Fast options in other test runners as a debugging tool in CI, specifically. When CI is flakey or just takes a long time to run, I want to know as soon as possible when there's a failure so that I can act. Watching fast moving logs for a single failure is annoying. Seeing the full test suite failed indicator light up is much easier. This reduces the feedback loop.
I consider this a temporary flag. I wouldn't run normal test suites with this on, because (as others have said) you usually want to know about all failing tests. However, as a tool I can use during debugging, then revert it, I find it very useful.
It's not a necessity, sure. Lots of test runners don't even have it. It is very handy when you need it, though.
Reacted by Aranđel Šarenac, Pascal Emond, Nicholas Kyriakides and DEBRIS APRONthis feature can be achieved very in a very straightforward way using reporters, so I will close now.
for convenience, I have created a user-land implementation of this: https://www.npmjs.com/package/@reporters/bail
even if you do not want to install an npm package for this, implementing this yourself is ±5 lines of codeReacted by Jordan HarbandReacted by Sean Massa, Golo Roden, John Smart, alex and Nicholas KyriakidesIt's a shame a trivial, but useful, feature like this doesn't exist natively inside the test runner as a CLI flag. For codebases that have many tests inside them, waiting for an entire test suite to run before being able to read an error stack trace can be incredibly annoying.
I use
mocha --bailorjest --bailall the time, if it needs hacking on some extra code to the project to get this functionality or requires installing some other 3P package, you may as well continue using mocha or jest.Reacted by Felix Becker, Pheromon, Barış Uşaklı, Erik Verheij, Nicholas Kyriakides, Aaron Vogler and IlyaIt's a shame a trivial, but useful, feature like this doesn't exist natively inside the test runner as a CLI flag. For codebases that have many tests inside them, waiting for an entire test suite to run before being able to read an error stack trace can be incredibly annoying.
PR's attempting to add this feature are welcome. just note adding cli flags to node.js isn't as easy as adding them for standalone test runners.
I recommend @cjihrig 's blog post https://cjihrig.com/test_runner_expectations explaining why not evert "trivial" feature is going to be implemented into node.js test runnerReacted by Colin IhrigWTH a bail() and timeout() function to migrate almost seamlessly from mocha would have been so great!
I'm gonna be harsh for a second.
Of all of the features we could have, this really should be prioritized.
Really takes away from the value of tests to the programmer when iterating on code. Comparing against Jest, I have to scour through the output to the terminal, which really messes up the natural flow of writing tests and writing new code.
Anyone in the same boat has to hack their way through this, it just feels like something that should have been a core feature.
Not to be a hater, but Bun has this already :/
Reacted by Pheromon, Daniel Cousens, Jeremy Walker and IlyaAmazing, thanks for the update 🙏. Appreciate everyone's hard work!
Reacted by Pietro Marchini
What is the problem this feature will solve?
A test's progress can sometimes be crucial for the next one, if one of them fails, say like this:
While yes, if the library has checks, this would be pointless, but it would be better to have an option for the the rest of the tests to fail if one had failed.
What is the feature you are proposing to solve the problem?
Something like this:
What alternatives have you considered?
No response