Repository navigation
How could we support typescript without vendoring it? #43818
Description
Activity
- changed the title
[-]How could we support semi-native typescript without vendoring typescript?[/-][+]How could we support semi-native typescript without vendoring it?[/+]on Jul 13, 2022 cc @cspotcode
So basically you would like Node.js to automatically load something from a global predefined place when it starts?
Reacted by Moshe Atlow, Julian Gruber, Alex Yang, Matthias Langhard and Szilágyi KrisztiánSo basically you would like Node.js to automatically load something from a global predefined place when it starts?
Yes, but only when started with a
.tsfile.Wasn't that the main purpose of custom loaders? You should even be able to set that in a
NODE_OPTIONS.Reacted by Jacob Smith, Vaughan Rouesnel and lin72hPlease don't start issues related to loaders without tagging @nodejs/loaders
Reacted by Vaughan Rouesnel and lin72hThis is not related to loaders. This is about the developer experience of Node.js.
Loaders are custom, user-specified components that are started before the app.
I'm talking about shipping something that would provide a better user experience for TypeScript users without additional configuration.Please keep this issue about the user experience and not specific implementations.
Reacted by Benjamin Gruenbaum, Vaughan Rouesnel, Leo Dutra, Ruy Adorno, Scott Tolinski, Oleksandr Knyga, Franco RATOVOSON, Hoishin, Maksim Sinik, Moises Marquez and 10 moreReacted by Vidar EldøyI'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?
How I'm seeing it is: This is already easily achieved via Loaders with next to no effort (and that effort is basically set-and-forget). We have a simple working example of it in
node/loaders-tests/typescript-loader. For a more robust one,ts-nodeprovides such a loader (ts-node/esm). And what we currently have via loaders is super fast. We just switched to it at my work and saw like an 800% speed improvement.PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.
Reacted by Gürgün Dayıoğlu and Vidar EldøyI'm trying to see how this isn't reinventing the wheel, but I'm not getting it. Could you please explain how it's substantially different?
I would rather stay away from discussing a specific implementation of this. This could be loaders but it doesn't matter. I care about listening to our users. Once we agree on the user experience, then we figure out what's the best way to ship it. If it's already possible as you hinted, it would be terrific.
PS In case we veer into the territory: I would vehemently object to TypeScript support in Node itself.
That's what our users are asking for. We cannot provide TS directly into Node.js core for plenty of reasons (it's just not possible for the tsconfig mayhem), including the fact that the TS team think it's a bad idea. I propose that we implement the user experience described in #43818 (comment).
Reacted by Moshe Atlow, Ruben Bridgewater, Sebastien Dubois, Cory LaViska, Daniel Niccoli, Kirill Groshkov, Carlos Fuentes, François Best, Jody Hoon-Starr, Julian Gruber and 21 moreReacted by Vidar EldøyReacted by Simon Plenderleith, Kirill Groshkov, François Best, Julian Gruber, Scott Tolinski, Lucas Zamboni Orioli, Enzo Innocenzi, Maksim Sinik, Moises Marquez, Mike and 5 moreA few questions come to mind:
- Are we making an exception for TS only, or are we open to adding more file extensions custom error message in the future?
- How do we chose what tool to recommend?
- Is Node.js core the correct place to solve that kind of problem?
Currently, what happens when someone runs
node script.tsdepends on the env:- if we're inside a
{ "type": "module" }package, it will throw as.tsis not a recognized extension – in this case, it's quite easy to add a custom error instead as you're suggesting; - otherwise, it will parse
script.tsas a CJS module – in this case, we could try to parse the file as a CJS module, and if that fails display the error message you're suggesting?
We would then need a way to detect that
typescript-node-coreprogram is installed on the local env, and to defer to it to load that file. Maybe by making a$PATHlookup?Reacted by Julian Gruber, Amio Jin and AkkoIn an app with
"type": "module"in itspackage.json, you get this:node script.ts node:internal/errors:477 ErrorCaptureStackTrace(err); ^ TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /private/tmp/test/script.ts at new NodeError (node:internal/errors:388:5) at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:80:11) at defaultGetFormat (node:internal/modules/esm/get_format:122:38) at defaultLoad (node:internal/modules/esm/load:21:20) at ESMLoader.load (node:internal/modules/esm/loader:431:26) at ESMLoader.moduleProvider (node:internal/modules/esm/loader:350:22) at new ModuleJob (node:internal/modules/esm/module_job:66:26) at #createModuleJob (node:internal/modules/esm/loader:369:17) at ESMLoader.getModuleJob (node:internal/modules/esm/loader:328:34) at async Promise.all (index 0) { code: 'ERR_UNKNOWN_FILE_EXTENSION' }So to provide the UX you’re describing, we would have to add a special case within this error flow where if the unknown extension is
.ts, we print a special message. This will inevitably raise the question of what other unknown extensions do we want to print guides for; or we could print some message for all unknown extensions along the lines of “go to https://nodejs.org/guide-to-loading-non-javascript-file-types” and that documentation page could be a clearinghouse of instructions for various types.In an app without
"type": "module", however,script.tsis parsed and executed as CommonJS. If it happens to be runnable JavaScript, it runs. Otherwise it’ll error when V8 tries to parse TypeScript syntax. So to provide a similar experience in CommonJS, within that error path you’d have to add a check that the file that couldn’t be parsed had an unknown extension, and then print the message. Keep in mind that there’s a type annotations proposal that would make many TypeScript files into runnable JavaScript.Either way, the solution that we would be recommending to users would involve adding the TypeScript loader. And transpiling TypeScript is squarely one of the use cases for loaders. So I object to treating this as “not related to loaders” and removing that label.
@GeoffreyBooth let's imagine that the solution ends up looking like that (oversimplification):
if (entryPoint.endsWith('.ts')) { process.argv.unshift(entryPoint); entryPoint = '~/.node/typescript-node-core/bin.js'; }
Sure
typescript-node-corewill probably use loaders, but that's arguably an implementation detail. (FWIW I agree that a clean solution should involve loaders; if anyone is interested to work on this, that's where I would suggest them to explore)Reacted by Vaughan Rouesnel and Kirill GroshkovWhat if it were some kind of…plugin (idk what the official term is relative to Node.js), like crypto or intl, with which node could be compiled or not? We could then have however many and we aren't responsible for them. It would be very similar to loaders, but a bit different and i think perhaps a little closer to what Matteo is talking about.
Also, this would avoid incorporating typescript-specifics (eg their .ts vs .js file extension nonsense) into node core (which is my main concern).
Annnnd it wouldn't require the CLI args everyone laments about loaders.
75 remaining items
These days it is common to use services that do provisioning and management for you, AWS Lambda in particular. So they do not care that much about node.js runtime.
But developers do care. Developers are forced to transpile the code before deployment and have transpiling pipeline.Secondly, I can't believe external transpilation could do better than native execution.
Thirdly, bun and deno already have this feature. Their runtimes want to stand out and get some audience based on unsolved needs. Isn't it a good sign that market expects TypeScript to be there?
I think we can have pre-defined values of tsconfig and the developer who wants to use other resources create inside the folder manually.
Ok, I started to experiment with implementation and here are my findings/opinions for tonight:
- the UX suggested by @mcollina works pretty well
% ../node index.ts The TypeScript compiler is not available. Install it following documentation at https://nodejs.org/en/knowledge/getting-started/working-with-typescript/then
% ../node index.ts 1 numberif the
@node/tsc-adaptermodule is available in the path - it can be local or global- I don't have strong opinions on which transpiler to use. We can even allow multiple
- we might want to ship a very conservative default config (something that is expected to hopefully never change or break) and let users extend or override it
- we should probably prevent loading a
.tsfile fromnode_modulesor at least allow it behind a flag, this can't go well because of potential configurations missmatches - repl case will probably require a flag/entry in package.json if exist - that's probably ok - we probably might ignore TypeScript in the repl for the first iteration anyway?
% docker run -it --init denoland/deno:1.10.3 repl WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Deno 1.10.3 exit using ctrl+d or close() > const x: number = 1 Uncaught SyntaxError: Missing initializer in const declaration- we hit a similar limitation as Deno -> we need to have the relative modules have a
.tsextension. I guess that since for 99.9% of the cases, semantically,TypeScriptTranspile(jscode) === jscodewe could just ignore this problem?
I won't have coding time for a couple days, but should go back into writing stuff early next week
Reacted by Podaru Dragos, Ian Sutherland and Geoffrey BoothReacted by Toni Villena@vdeturckheim This looks good! A few thoughts:
- I don’t know what this proposed
@node/tsc-adapterwill contain, but I assume it’s a TypeScript loader likets-node? If so it’s tied to the Loaders API which is still experimental, but Move ESM loaders off-thread #44710 is the last PR before it could become stable. Also this TypeScript support will surely be experimental for a while too, and longer than the Loaders API is, so that shouldn’t be an issue. - Since the
@node/tsc-adapterpackage would be installed and versioned separately from the Node runtime, it would have to support a range of Node versions. I think this is a problem already solved byts-nodeand similar projects. - Regarding
package.jsonstuff, there was a prior discussion about including the loader settings (like what would be passed into--loader) in there, and I remember the consensus landing on including all ofNODE_OPTIONSin there, or as much as is feasible. I think the way forward is to create something like apackage.json"nodeOptions"field and add as many options as possible; it’s probably impractical to support all 100+ in a single PR, but we could at least come up with a design that can accommodate eventually supporting all of Node’s options and we can add them in batches. - Requiring the
.tsextensions isn’t so weird, they’re already required in ESM JavaScript generally. I know it’s unidiomatic for TypeScript, but support for explicit.tsextensions is coming in TypeScript 5.0 and we can provide an informative error on missing.tsextension like we already do for missing.jsextension.
Reacted by Vladimir de Turckheim, Carlos Fuentes and Yaroslav- I don’t know what this proposed
Thanks @GeoffreyBooth , these are very good points!
Yeah,@node/tsc-adapteris another name fortypescript-node-coresuggested in the first message of the thread and would be ts-node with node using it instead of having to plug into node. Basically, making the TS compiler live outside of node while allowingnode xx.tswithout any other argument. Right now, I made it work with cjs but for esm it will likely be a loader (we had experiments with @targos a few years ago on that too).Regarding node vs. package version, this is a very good point.
I think we are going to a direction where the typescript support can work very well in most cases and where we provide DIY solutions for when it does not (like overriding the typescript adapter for instance).For repl and
.tscases. I agree 💯 with you. In the end, these are best-effort and DX choices, we can take the decisions and document them. Also, this is what experimental support is for!I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼
I missed last summer's summit (and this thread until recently fwiw), so I hope I am still going in the direction people agreed upon 👼
Yes, I think nothing has changed from #43818 (comment) and what your recent experiments seem to assume. About the only “news” is that we’ve made so much progress with the Loaders API that I can see it applying to CommonJS too in the not-too-distant future; so I would prioritize getting TypeScript support via Loaders working first or at least ship that at the same time as CommonJS solution. @cspotcode and others have also had discussions about hooks in addition to those provided in the Loaders API, to do things like customize the REPL, for a more seamless DX beyond loading modules. Hopefully we can use the interest in better TypeScript support to build out some of those APIs.
It sounds like
package.jsonoptions aren’t needed for the core/MVP use case, so maybe we should get the initial to-do list done and then we can loop back to that and give it the attention it deserves.Reacted by Matteo Collina and Bert VerhelstThis sounds good, thanks for the highlights! Seems good to me! I'll resume work with that in mind!
Regarding the point in not treating
"typescript"differently (in #43818 (comment)), I agree, that's one of the advantage of having something like@node/tsc-adapterwhich we control and would only serve the goal of calling"typescript"from node under the hood on.tsfiles. But nothing prevents to also have@node/swc-adapteror@node/esbuild-adaptertooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.But let's focus on the MVP for now and add too much complexity later 👼
Reacted by Geoffrey Booth and Thomas OrlowskiBut nothing prevents to also have @node/swc-adapter or @node/esbuild-adapter tooto let the end user chose the transpiler in a list of provided alternatives (these packages are proxies to the other packages which also abstract breaking changes such packages could have). The end user will anyway be able to use a custom loader if they want to use something else in the end.
+1 on this ^^
But nothing prevents to also have
@node/swc-adapteror@node/esbuild-adaptertooto let the end user chose the transpiler in a list of provided alternativesI’m not sure Node needs to publish a transpiler at all. Whatever docs page our error message links to could include a list of ones known to work, and encourage the user to choose one to install. I would think we should at least start with that approach, and if we find a need for publishing something official we could do that after MVP.
Reacted by Tony Gorez and Igor SavinReacted by Carlos Fuentes, Steven and Matteo CollinaThere is a related proposal to add type syntax to JS, checkout:
Reacted by Jacob Smith, Nick, Albert Mañosa and silverwindWhat is the latest update on this? With both Deno and Bun improving their compatibility with Node.js and providing native support for TypeScript, there seem to be numerous examples and lessons to be learned from their advancements.
Since it hasn't been mentioned in this thread, I would like to inquire whether this proposal could enable the distribution of npm packages exclusively written in TypeScript. Such a capability would significantly enhance the user experience when working with Node.js and TypeScript, eliminating the need to juggle between .d.ts and .js files.
Reacted by Daniel Lenksjö, Mohamed Aziz Karoui, PierBusDev, Bluzzi, Arya Emami and Rafaell LycanWhat is the latest update on this?
We’re exploring this space via #49704 (cc @JakobJingleheimer). One idea we’ve been considering lately is that Node could possibly provide some kind of blessed or automated way to install plugins, one of which could be TypeScript. Part of the problem is that TypeScript itself doesn’t provide Node integration, and there are many third-party plugins for that purpose, and we would want to avoid picking a winner. There’s also no one correct way to set up TypeScript; it depends on your build tool and its configuration, your app and its framework and those expectations, and so on.
You might want to open an issue with the TypeScript team, if there isn’t one there already. They’re the ones who have the power to add official support for runtime transpilation and/or type-checking, and/or provide a blessed/recommended
tsconfig.jsonfor Node projects. TypeScript itself is somewhat coupled to Node, in that the extensionless resolution mimics Node’s CommonJS resolution and the conversion of ES modules to CommonJS modules was created to address former limitations of Node; the TypeScript team is perhaps the best source for how they advise integrating their tool with Node.Reacted by Shinebayar G, Nick, Steven, marziply, Kirill Groshkov, Daniel Lenksjö, Ido S., Brandon Bennett, Kai Sellgren, Bluzzi and 2 moreI think we can reasonably close this issue
Reacted by Pietro Marchini, leah, Momen, Matteo Collina and StevenReacted by Simon Plenderleith, Micael Levi L. Cavalcante, Steven, Daniel Lenksjö and marziply
I would like Node.js to support the following user experience
$ node script.ts Typescript support is missing, install it with: npm i --location=global typescript-node-core $ npm i --location=global typescript-node-core ... $ node script.ts "Hello World"(I picked the typescript-node-core name at random, I would be extremely happy if that could be ts-node)
Note that
script.tscould evaluate to either cjs or mjs depending on thetsconfig.jsonfile.Originally posted by @mcollina in #43408 (comment)
Why vendoring is not an option?
tsconfiggiven the 100+ options.