Repository navigation
module: ESM loader approach #36954
Description
Activity
@GeoffreyBooth I feel rather vindicated RE Concerns with Next() #3: When I was setting up the
resolvehook test fixture for #37468, I indeed did forget the default-casereturn next(…)at the end. Took me a few minutes to figure out why the heck suddenly all my cases were failing.I think if we opt for the
next()approach (as I was authoring #37468, I'm increasingly leading toward "shouldn't", not least of which because the implementation looks to be much more difficult—read: buggy, real or perceived), I think some extra fault detection will be needed. Without, I think this could be a table-flip moment for users (I was getting annoyed, and I wrote the darn thing—so if anybody should know/remember, it would be me (and you)).One point of concern about the Done method: when implementing a loader for mocking (as I did for testdouble's ESM support), you need to add cache busting to the final url, which means that you want to be the last in the chain. And yet, when loading the source, you want to be the first in the chain, because you know you're not really transforming the code, but rather replacing it.
This is why I prefer the Next() approach. While it means that loader developers need to be more aware of how the chaining works, it gives them more flexibility in how they implement the resolve and load functions of the loader, and does not "chain" them to the opinionated thinking of how things should happen, which is what the Done approach is trying to do.
Exhibit A 😊, the ESM mock loader for testdouble: https://github.com/testdouble/quibble/blob/main/lib/quibble.mjs
Reacted by Jacob Smith@giltayar sorry, I'm not following (but would like to understand your concern).
done()would ensure it's the last hook executed. If you mean out of all possible hooks, all other hooks need to run and a particular one dynamically needs be last in queue, yes,next()would facilitate that anddone()would not. Needing to do that dynamically is a critical differentiator but it sounds like very much an edge-case (please do correct me if I'm misunderstanding you).Regarding needing the particular
resolvehook's siblingloadhook to be called first, bothdone()andnext()could do that because ESMLoader will be aware of what file they both came from. But why do you need that?I looked through your code example (I've looked thru Quibble previously—slick stuff!), but the "why" didn't jump out at me.
At first read, it sounds like what you're describing meticulously strong-arms a process that would probably naturally result in what you want anyway.
If this is an edge-case, unless it's a very compelling edge-case, I would likely lean towards the option that does not ensure all developers have a much more complicated and gotcha-prone experience so that a small subset might use a small bit of extra functionality.
nextallows a single loader to:
a) pre-process the call to the next loader
b) post-processing the results from the next loader
c) skip/suppress the call to the next loaderdonesupports 2 out of 3?If I understand correctly, neither API allows awaiting an entirely different module resolution. For example if the format of
./transpiled-language.*depends on the contents ofpackage.jsonand/ortranspiled-language-config.json, and if the module is coming from the web through anunpkg,url-remapping, orhttp-to-httpsloader. However, node's own loader does not use the loaders chain to readpackage.json, does it? It assumes they exist on the local filesystem. I'm not sure if that will change, and I'm not sure whether loaders want to be a virtual filesystem of sorts.donewould not be able to supportbin your list because by the time the subsequent hook is invoked, the current will have finished (I'm not sure if it should dob). It can doaandc.Currently, node does read the package's package.json, checking its
"type"field.If you need to orchestrate the sequence of imports such that
foois available beforebar, a dynamic import might be what you're looking for:const { default: foo } = await import('foo'); const { default: bar } = await import('bar');
If you're saying that
fooneeds to happen during and be available tobar's load hook, the above may suit with some finagling, but this may be something to usecontext.conditions(which is a bit vaguely defined at the moment, but could potentially be extended to facilitate).all other hooks need to run and a particular one dynamically needs be last in queue, yes, next() would facilitate that and done() would not. Needing to do that dynamically is a critical differentiator
Yes, you understood!
but it sounds like very much an edge-case
Is a mocking library an edge case for loaders? I don't think so. I'm guessing there will be a few of those.
Regarding needing the particular resolve hook's sibling load hook to be called first, both done() and next() could do that
How does that happen in the Done method? If the
resolveMUST be called last, that means that the loader should be the last in the chain, which means that theloadwill also be last, no? Using thenextmethod, I can be the first in the chain, yet callnext()and transform the value I get fromnext(), thus acting like I'm last in the chain, whereas myload()can just not callnext()and thus be the first (and last!) in the chain.At first read, it sounds like what you're describing meticulously strong-arms a process that would probably naturally result in what you want anyway.
Maybe I'm missing something, but how would it naturally occur?
If this is an edge-case, unless it's a very compelling edge-case...
As I said above, I believe mocking libraries are a compelling case.
For sure supporting mocking is compelling.
I'm thinking mocking would be used only during automated testing (if not, please let me know of the other scenario(s)). If so, I would expect the mocking loader to need only a
load()hook (allowing the normal resolves to happen as they would). If so, just put the mocking loader at the front of the queue:"test": "NODE_OPTIONS='--loader mockLoader' …"
Then in mockLoader's load, read a static list of what to mock, check the url against that, and do on match (and opt-out on mismatch):
export async function load(input, url /* … */) { const mock = mapOfMocks.get(url); if (mock) return { format: 'module', source: mock, }; // omitting a return = opt-out }
If you wanted to bypass a mock, you could add some kind of query param flag to the specifier, like
'foo.mjs?noMock'. You could similarly have an adhoc mocking opt-in with a query param (and the easiest would be some kind of location convention, like adjacent, but the query param could have a value with a path to the mock):'foo.mjs?doMock'and the mockLoader looks for a file of the same name like'foo.mock.mjs':export async function load(input, inputUrl, context, done, defaultLoader) { const url = new URL(inputUrl); const params = new URLSearchParams(url.search); if (params.has('noMock')) return; if (params.has('doMock')) { const mockUrl = …; const mock = defaultLoader(mockUrl); if (mock) return { format: 'module', source: mock, }; } else { /* global mocks */ } // omitting a return → opt-out }
Perhaps don't call
done()in mockLoader because the mock file might be typescript or something that a subsequent loader would transform.The reason mocking loaders also need to hook the
resolveis for "cache busting": in any mocking library, you can mock an ESM one way, and then mock it another. To make this work, theresolveadds a "generation" query parameter to the URL to enable it to load the "latest" change to the mock. But for this to work, it needs to be the last in the chain.(in Quibble currently it's also used as a hacky way to figure out where the module file is (search for "dummyImportModuleToGetAtPath" in my blog post, but I expect this hack to go away once we have
import.meta.resolve)Reacted by Jacob SmithReacted by Jacob SmithDoes jest's mocking loader need
resolvefor an additional reason: to redirect imports into__mocks__directories?1 remaining item
@giltayar so actually I had read that blog post way back when! Having re-read it, now I remember the purpose of the "generation" query param.
To sum up the problem, it sounds like you're using the generation number in
resolveto ensure 2 things:- Instances of the mock are independent/isolated: the "global"
foo.mockused bybar.testandqux.testhas no cross-contamination (exbar.testdoes something tofoo.mockand that shouldn't bleed intoqux.test, like call counts) - A module can have multiple, concurrent mocks:
foois locally mocked inbar.testand also inqux.testand their local mocks are different (exbar's mock offoocontains exportsaandb, andqux's mock offoohas exportsbandc)
When running tests in parallel,
bar.testandqux.testcan be being processed at the same time (typically facilitated via multiple child processes); importantly, there is only 1 node process, and that contains the caches (so that is shared).Have I got that right?
If so, why does your resolve need to be last? I think all that matters is that your query params exist in the final value of
urlreturned from the resolve chain.Example
--loader=httpsLoader \ --loader=quibbleLoader \ --loader=babelLoader
httpsLoader→quibbleLoader→babelLoader// quibbleLoader export async function resolve(specifier, parentUrl) { if (!isQuibbly) return; // … const url = new URL(/* … */); url.searchParams.set('__quibble', global.__quibble.stubModuleGeneration); return { url: url.href, }; }
// babelLoader export async function resolve(specifier, parentUrl) { if (!isBabelly) return; // … const url = new URL(/* … */); url.searchParams.set('__babel', babelStuff); return { url: url.href, }; }
The result of the resolve chain would be something like
https://example.com/foo.mjs?__quibble=2&__babel=babelsConfig- Instances of the mock are independent/isolated: the "global"
Does jest's mocking loader need
resolvefor an additional reason: to redirect imports into__mocks__directories?@cspotcode yes for the same reason(s) Quibble does.
In your HTTPS loader example, what is
inputand why would it being defined mean we would opt out?@GeoffreyBooth
inputis the (interim) result of the previous loader, if any previous loader(s) provided anything yet (ex there could be 5 loaders that ran before HTTPS Loader, and they may have all opted out, in which caseinputwould be empty).In the case of HTTPS Loader, if interim source already exists, that indicates there's nothing for HTTPS Loader to do. My example may have over-simplified slightly, since
inputwould be empty or an object (so it would actually be checkinginput?.source). Perhapsinputshould be initialised as an object with emptyformatandsourceproperties (but that may make the node-land code a little more complicated).I use mocking of network calls during development to control API responses without needing live backends in particular states. That could be implemented at a low level (mock/intercept the networking machinery itself) or by mocking the function where the network fetch occurs.
Ah, true. In that case, just ensure that mocker occurs ahead of HTTPS Loader (and any other remote loaders) in the queue 😉 I'm thinking that they would not need to conditionally change sequence.
Application monitoring could also be considered another form of mocking. If you think of a use case like a live dashboard of an app’s traffic, getting the data for such a tool could be implemented by mocking/proxying lower level functions like core parts of Connect/Express/other framework or of Node.
I think in that case, it would just need to be first in the queue (which they already need, citing something like "ensure Foo is the first
require()in your application").The updates to ESMLoader.load() would also need to ensure individual properties are not inadvertently stomped (eg. loader4 returns only a
sourceproperty and omitsformat, that probably shouldn't overwrite an existingformatfrom a previous loader, but returningsourceand an emptyformatwould).A module can have multiple, concurrent mocks: foo is locally mocked in bar.test and also in qux.test and their local mocks are different (ex bar's mock of foo contains exports a and b, and qux's mock of foo has exports b and c)
Nope. There can be only one specific mock per-process at a specific time. But a test could mock the module in one way at a certain time, and serially after that mock it in another way. And that is the purpose of the generations: to allow the test to change the mocks serially.
If so, why does your resolve need to be last?
It needs to be last because I don't really care how the module is resolved. I just want to add my query parameter to it. Hence, it needs to be the last.
I think all that matters is that your query params exist in the final value of url returned from the resolve chain.
Exactly. That's why it needs to be last! Or did I misunderstand?
Nope. There can be only one specific mock per-process at a specific time. But a test could mock the module in one way at a certain time, and serially after that mock it in another way. And that is the purpose of the generations: to allow the test to change the mocks serially.
I think not important for
nextvsdone, buut: Possibly over-complicating things, might using the parentUrl be more robust and easier to troubleshoot (and also preclude collision)? Exfile://…/foo.mjs?__quibble=…/bar.mjs file://…/foo.mjs?__quibble=…/qux.mjsI think all that matters is that your query params exist in the final value of url returned from the resolve chain.
Exactly. That's why it needs to be last! Or did I misunderstand?
Yes, I'm pretty sure you misunderstood 🙂 If all you need is your query param to be present in the final value, that can happen anywhere in the chain (as long as other loaders behave themselves). Think of it like an assembly-line, where each loader does its own part: I make and attach the headlights, you make and attach the upholstery; neither of us does that to the exclusion of the other (hence why calling
donewould be exceptional).might using the parentUrl be more robust
Won't work, because same file might be importing the module multiple times (using
await import), yet wanting different mocking for each import.that can happen anywhere in the chain (as long as other loaders behave themselves)
- as long as other loaders behave themselves: I don't want to depend on them keeping existing query parameters (or hashes).
- I can't add query parameters to bare specifiers, so I first have to have somebody resolve it to a file, and then add the query parameters.
- as long as other loaders behave themselves: I don't want to depend on them keeping existing query parameters (or hashes).
That sounds like something everybody would want, which is mutually exclusive / impossible (regardless of
nextordone). If another loader is misbehaving, fix it or don't use it. I would ensure there's sufficient documentation and examples to understand how to get loaders to play nice together 😉 None of the concepts are novel, so someone with experience integrating multiple pieces would likely already be familiar.- I can't add query parameters to bare specifiers, so I first have to have somebody resolve it to a file, and then add the query parameters.
If quibble is before another loader that would supply that, I thiiink this is addressed by leveraging the
defaultResolverargument?export async function resolve(specifier, …, defaultResolver) { // … const fileUrl = defaultResolver(specifier, …); const url = new URL(fileUrl); // … }
That sounds like something everybody would want, which is mutually exclusive / impossible (regardless of next or done). If another loader is misbehaving, fix it or don't use it. I would ensure there's sufficient documentation and examples to understand how to get loaders to play nice together
The problem is that if the writer of the loader has not tested it with a loader that adds query parameters, they would not know that they need to preserve it. So we would have to document this. And this is precisely those things that people tend to ignore. A lot of the "Done" proposal's raison d'être is to ensure that loaders are NOT dependent on the behavior of one another, yet here we are enforcing something that if the writer of the loader broke, chaining would not work in some cases.
If quibble is before another loader that would supply that, I thiiink this is addressed by leveraging the defaultResolver argument?
This is what I am doing now. But I didn't know that in the "Done" proposal there is a
defaultResolverparameter. I would have thought that in any "chained loaders" proposal, the default would be somewhere in the chain, either first or last.Reacted by Jacob Smithwe are enforcing something that if the writer of the loader broke, chaining would not work in some cases.
I think that's just the nature of the beast for this. They can't be entirely independent. I think this is perhaps most easily noticeable in
resolvebecause it's simple and outputs ~1 small thing. Thedoneproposal's intention is indeed "principle of least knowledge", but that's mainly concerned with least knowledge of the tool (esm loader); ensuring one's loader doesn't clobber another loader's work is, I'm pretty sure, unavoidable (and would also exist withnext). We could add some diff detection and emit warnings ("hey, listen! Looks like you didn't preserve the previous loader's bits."), but I imagine those warnings could often be erroneous and thus very annoying.But I didn't know that in the "Done" proposal there is a defaultResolver parameter. I would have thought that in any "chained loaders" proposal, the default would be somewhere in the chain, either first or last.
Ah, yes, sorry. This was discussed verbally in the meeting, and it was suggested to provide code examples for
loadbut notresolvebecause resolve is simpler. I'll add bothresolveandloadcode examples in the top/first post here (to keep it all together) and ping you when it's done (since the edit won't send you a notification). I should have time this afternoon.@giltayar @GeoffreyBooth examples (at the top) updated. notably,
input→interimResultandinputUrl→resolvedUrl, and I changed the sequence of arguments:default*↔donesince default* is more likely to be used (allowingdoneto be omitted).moved to the nodejs/loaders team repo
There are 2 leading approach proposals for ESM loaders and chaining them.
Similarities
Both approaches:
resolve(): finding the source (equivalent to the current experimentalresolve()); returns {Object}format?: see esm → getFormaturl: same as current (see esm → resolve)load(): supplying the source (a combination of the current experimentalgetFormat(),getSource(), andtransformSource()); returns {Object}format: same as current (see esm → getFormat)source: same as current (see esm → getSource)resolveandload) are chained:resolve()s are executed (resolve1, resolve2, …resolveN)load()s are executed (load1, load2, …loadN)Differences
Next()
This approach is originally detailed in #36396.
Hooks are called in reverse order (last first): a hook's 3rd argument would be a
next()function, which is a reference to the previous loader hook. Ex there are 3 loaders:unpkg,http-to-https, andcache-buster(cache-busteris the final loader in the chain):cache-busterinvokeshttp-to-https, which in turn invokesunpkg(which itself invokes Node's default):cache-buster←http-to-https←unpkg← Node's defaultThe user must actively connect the chain or it (likely) fails: If a hook does not call
next, "the loader short-circuits the chain and no further loaders are called".Done()
This approach was also proposed in #36396 (in this comment).
The guiding principle of this approach is principal of least knowledge.
Hooks are called in the order they're declared/listed, and the return of the previous is fed as the input of the subsequent/next hook, and each hook is called automatically (unless short-circuited):
unpkg→http-to-https→cache-buster(if none of the supplied loaders output valid values, node's default loader/hook is invoked, enabling a hook to potentially handle only part and avoid re-implementing the native functionality node already provides via the default hook).Hooks have a
doneargument, used in rare circumstances to short-circuit the chain.Additionally, this proposal includes a polymorphic return:
done(validValue)validValueas final value (skipping any remaining loaders)falseExamples
Resulting in
https-loaderbeing invoked first,mock-loadersecond, etc, and node's internaldefaultLoaderlast.For illustrative purposes, I've separated
resolveandloadhooks into different code blocks, but they would actually appear in the same module IRL.Resolve hook chain
HTTPS Loader
Mock Loader
Load hook chain
HTTPS Loader
Mock Loader
CoffeeScript Loader
Updates to
ESMLoader.load()Concerns Raised
Next()
nextfunction does not behave as many current, well-known implementations behave (ex javascript's native generator'snextis the inverse order to this's, and not calling ExpressJS's route-handler'snextdoes not break the chain).nextis effectively required (not callingnextwill likely lead to adverse/undesirable behaviour, and in many cases, break in very confusing ways).Done()
This could potentially cause issue for APMs (does theAfter chatting with @bengl, it seems like this is not an issue as V8 exposes what they need.nextapproach also?)A hook that unintentionally does not return / returns nullish might be difficult to track downI believe this was resolved in the previous issue discussion?