Repository navigation
Provide some mechanism to conditionally and synchronously import modules (or just builtins) from ESM #52599
Description
Activity
Given that it doesn't depend on the context,
import.meta.builtinscould also beprocess.builtins.Reacted by Jake Bailey, Rob Palmer, Ashley Claymore and Jacob SmithThanks for the summary here, in terms of timelines I just want to summarize my own sense of things:
import.now- there is committee discussion and interest here, so it isn't unviable, but there's a lot of implementation edges, so we're likely looking at at least a year or so before a viable proposal could even reach Stage 2, and another year or two to get to implementation, if it happens at all.with { optional: true }- this would need to be tackled as a web or ecma spec, but could certainly be done in a timeline closer to within 4-8 months.import.meta.getBuiltin- Node.js could implement and ship this any day.
Perhaps in time we do hit all three, or at least two of the above.
Reacted by Jake Baileyprocess.builtinsSGTM. I don't see the reason to put extra stuff in theimport.metanamespace, which is shared across environments and runs the risk of collisions. Weak imports seem like a good medium-term improvement, but let's do those at the TC39 level (since this feature makes sense across environments).Reacted by Jacob SmithSomething on
processsounds totally fine to me; I also didn't realize when writing the above that you could also do:const require = process.getBuiltin("node:module").createRequire(import.meta.url)
And be no worse off than before for conditionally loading anything outside of the builtins, which is very handy.
I think the only concern I have for a
process-based approach is how bundlers won't recognize these as being "imports" per se, so may not be able to shim them. But I suppose that bundlers would need to change no matter what option is implemented, since it's all new besidesimport.meta.requirealready being implemented inbun.I think whether the access is module-based depends on #52575 (if policy is enabled, access control to builtins is different per module based on the policy. But then if policy is removed altogether...there is no need to worry about it. It's probably another reason to remove policy?)
Actually thinking through the implementation details of how to implement per-module access control for
import.meta.builtinsI realize thatglobalThis.process.builtinscan still do it too, in the end something can be stored in the script context similar to the host-defined options used to implementimport()(or at least that's what V8 plans to do), and in the native layer all the bindings can use this thing to determine resource control scopes for the module that invokes the builtin. That's going to be a lot more robust than doing it in the module loader anyway. So my conclusion is,globalThis.process.builtinswon't shut the door for per-module access control, it's still possible to have both.I'm working on a proof of concept; my quick hack for now (until I can get all of it as ESM) is to use a
--requirepreloaded script to patchprocessand a polyfill stuck on top of our bundle.$ cat patchProcess.cjs process.createRequire = require("node:module").createRequire; $ head ./built/local/tsc.mjs var require, __filename, __dirname; if (typeof process !== "undefined") { require = process.createRequire(import.meta.url); __filename = require("node:url").fileURLToPath(new URL(import.meta.url)); __dirname = require("node:path").dirname(__filename); } $ node --require ./patchProcess.cjs ./built/local/tsc.mjs -p ./src/compiler --diagnostics Files: 213 Lines: 245632 Identifiers: 412592 Symbols: 256043 Types: 103601 Instantiations: 189818 Memory used: 496843K I/O read: 0.01s I/O write: 0.00s Parse time: 0.90s Bind time: 0.38s Check time: 7.15s Emit time: 0.00s Total time: 8.44s
Just throwing on
createRequireis a good trick for now, given our code will just try and userequireglobally if it can. Rewriting everything to assume ESM is definitely possible, but will take a little more work than the few minutes I spent on this showing that it's possible to avoid TLA and yet still conditionally do things.So, that was a lot easier than I expected.
$ cat testRequireESM.cjs // Inject this into the process so that TS can synchronously require stuff. process.createRequire = require("node:module").createRequire; const ts = require("./built/local/typescript.js"); console.log(ts.version); $ node --experimental-require-module ./testRequireESM.cjs 5.5.0-dev (node:33109) ExperimentalWarning: Support for loading ES Module in require() is an experimental feature and might change at any time (Use `node --trace-warnings ...` to show where the warning was created)
The only change to TS I need to make is to change our import extensions and fix up our system implementation to require
process.createRequire. Other stuff is broken (in fixable ways), but this seems to show that a TLA-free ESM TypeScript API is possible without breaking downstream users.Code is at: https://github.com/jakebailey/TypeScript/tree/its-esm
Reacted by Jacob SmithOpened #52762
Reacted by Jake Bailey, Rob Palmer and Jacob Smith5 remaining items
- added a commit that references this issue
on May 28, 2024 - added a commit that references this issue
on May 28, 2024 - added 2 commits that reference this issue
on Jun 1, 2024 How does
process.getBuiltinModuleinteract with loaders? (specifically asking about theimport-in-the-middleandrequire-in-the-middleuse case). ref: open-telemetry/opentelemetry-js#4742I suppose packages like these can just do what they already do with the CJS loader…by monkey patching? (Not a great way to get it done but it gets the job done for now, just like until proper CJS loader hooks are implemented they will just keep monkey patching anyway…and arguably simply wrapping a method on process is less problematic than the magic they do with the CJS module loader).
Reacted by Abhijeet Prasad and ExE Boss- added 2 commits that reference this issue
on Jun 20, 2024
What is the problem this feature will solve?
Thanks to #51977, requiring ESM is looking to be a real possibility. As such, TypeScript is considering transitioning over to ESM in the near future (depending on when
require(ESM)is unflagged, hopefully in time for TS 6.0?), as that sort of change would no longer pose compatibility problems for the vast number of downstream CJS users. This has a number of benefits, mainly that we could finally share code betweentsc.jsand our public API without a startup perf regression, and that we wouldn't be duplicating code in our package (thankfully only two copies remain as of TS 5.5, down from six copies in TS 4.9).TypeScript's current public API bundle is intentionally "UMD-ish", detecting whether or not
module.exportsexists and using it (declaring a global otherwise), then later conditionally requiring built-in modules likefsif we believe to be running within Node. This allows us to ship one single bundle that works in Node, browsers, and bundlers alike.However, the code that relies on conditional
requireis executed at the top-level as it's constructing thets.sysobject, the default "system" implementation for most of our APIs. Within CJS, this is fine, but within ESM, the only way to conditionally import something is by either:createRequirefromnode:module.Using top-level await breaks
require(ESM), the whole reason we think we can use ESM in the first place, and TS is infamously not async and couldn't import it later. Importingnode:moduleis a moot choice, since if we could safely importnode:module, we could have just importednode:fsand so on.So, we need some mechanism to synchronously import modules, or at least the builtins.
Given
require(ESM)is now possible, it sure seems like there could be a way to safely synchronously import modules in ESM that are already require-able from CJS after #51977.What is the feature you are proposing to solve the problem?
After discussing this in the TC39 Module Harmony meeting (with @guybedford @joyeecheung @JakobJingleheimer, others), there seemed to be a number of different paths forward:
import fs from "node:fs" with { optional: true }) could be added; if the module fails to resolve, the imports are all undefined. This may require some sort of TC39 specing or proposal.import.meta.require. This was previously proposed at Pull request opened for import.meta.require on core modules#130, but unfortunately drags CJS into the ESM world (potentially no more thancreateRequire, I suppose).import.meta.importSyncorimport.now, which is effectively justawait import(...)that only works on sync-loadable ESM.import.meta.builtinsor similar (e.g. onprocess), which just provide access to Node's builtin modules. Largely, TS only needsfs,path,os, etc, so this would sidestep the "sync import" problem altogether. TS also conditionally importssource-map-support, so that would not work, though only in development. Thankfully, since one could getnode:modulethis way, you can also shimrequireviacreateRequire, which is pretty neat.What alternatives have you considered?
TS could also use
package.jsonimport maps to achieve "conditional" imports of Node-specific code, e.g. have an import like#systemwhich in the Node condition imports from Node, but is shims otherwise. This seems to have a number of downsides in my view, specifically:package.jsonimport maps. In the meeting, @JakobJingleheimer mentioned that one could remap imports likenode:fsto shims, even todata:...blobs, but I'm definitely not experienced enough in browser ESM to know how to do that.