镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Infer project references from common monorepo patterns / tools #25376

Description

Genesis: see section "Repetitive configuration" in #3469 (comment)

Search Terms

monorepo infer project references automatically yarn lerna workspace package.json

Suggestion

For common monorepo managers, we should natively understand cross-project references declared in package.json as if they were declared in tsconfig.json

Open questions:

  • Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?
  • How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?
  • Would you need to opt in? Would there be a way to opt out? What should that look like?

Use Cases

  • Monorepos of all (supportable) flavors

Examples

https://github.com/RyanCavanaugh/learn-a

This repo has a fair bit of duplication where projects need to write down their dependencies in package.json and as references (with different syntax) in tsconfig.json.

Checklist

My suggestion meets these guidelines:

  • This wouldn't be a breaking change in existing TypeScript / JavaScript code
  • This wouldn't change the runtime behavior of existing JavaScript code
  • This could be implemented without emitting different JS based on the types of the expressions
  • This isn't a runtime feature (e.g. new expression-level syntax)

Activity

  1. weswigham commented on Jul 2, 2018

    @weswigham
    Member

    Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?

    All the tools you've listed use normal commonjs module resolution (and consequently use the normal package.json dependencies, optionalDependencies, and to a lesser degree devDependencies fields) - none of them change runtime behavior at at all in that regard. Where they differ is where they symlink things into/from (some install all packages in one dir, then link them into each other; some install everything in a top-level node_modules, since that makes everything available), which is what causes issues for us, usually - our symlink handling needs to be essentially perfect to handle all of these configurations correctly. All that'll solve that is hammering away at it with all the configurations we can find and fixing the bugs we find, IMO.

    One of the things I was running into in trying to setup an ui-fabric user suite test (where I symlinked things together in our harness rather than use rush) was that depending on where I symlinked stuff, TS would locate modules in ways I wasn't always prepared for. For example: If I symlink from packages/foo into packages/bar/node_modules, imports from packages/foo/index.d.ts is still going to resolve up from packages/foo, not packages/bar/node_modules/foo (which complicates where you need to place things a bit). These tools all usually handle all these cases right (they have to or the runtime wouldn't work); we just need to continue to be faithful to the commonjs resolver's behavior.

    How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

    I would assume package root, and if we don't find one there, we could read a tsconfig field in the package.json that points at it (cue people asking us to read the configuration straight from the package.json).

    Would you need to opt in? Would there be a way to opt out? What should that look like?

    The usecase is a (near) zero-config start for common monorepo setups, so opt-out, IMO. Could add something like a --no-pkg flag to tsc -b that stops it from walking package.json files.

  2. bajtos commented on Jul 3, 2018

    @bajtos

    How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

    In https://github.com/strongloop/loopback-next, we are using monorepo to develop a bunch of modules. Right now, we have one tsconfig.json file per each package. (Because of the way how we are working around missing support for project references, we call this file tsconfig.build.json.)

    IMO, it's important to allow each package to have its own tsconfig configuration, because this configuration often involves a list of files to include/exclude from compilation.

    Assuming the tsconfig.json file is located in the package root makes perfect sense to me 👍

    Initially, I was reading the proposal as to assume a single tsconfig.json file located in monorepo root, that would not work (at least for us).

    Would you need to opt in? Would there be a way to opt out? What should that look like?

    If we can find an elegant way how to support most of commonly-used monorepo layout (tools) that are setting up cross-package dependency tree using the information from standard npm/package.jsondependencies, optionalDependencies and devDependencies fields only, then I think TypeScript's build mode should automatically infer project references from package.json too.

    I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only? The package.json file does not provide any hints on that.

    The first solution that comes to my mind:

    • Find out where is the monorepo rooted, either by looking for lerna.json, yarn config file, etc. or by asking the user to explicitly provide that directory via configuration. I think the explicit config option is a better solution as it does not couple TypeScript with different monorepo solutions and provides a natural place for opting into auto-discovery of project references.
    • When parsing package.json dependencies, check out which dependencies are resolved as a symlink pointing to a different place inside monorepo - these dependencies should be added as project references. Dependencies symlinked to a different place (outside of monorepo, typically /user/local/lib/node_modules when using npm link manually) (*) or installed as a copy in node_modules should be consumed as read-only.

    I find this solution a bit too complex and involved, but don't have any better alternative right now :(

    (*) Handling of symlinks outside of monorepo is actually another interesting case to discuss. If we treat all symlinked dependencies as project references, then people using multiple single-repo projects with manual npm link could get benefits from the new build mode too.

    For example, loopback depends on strong-remoting, where both modules are maintained by the same team. Sometimes I need to make changes both in loopback and strong-remoting, where the change in loopback depends on the changes made in strong-remoting. To do that, I run npm link path-to-my-strong-remoting-clone in my loopback directory. If we were using TypeScript and the build mode was treating all symlinks as project references, then I could make cross-project renames and rebuild both loopback & strong-remoting in a single build step.

  3. weswigham commented on Jul 3, 2018

    @weswigham
    Member

    I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only?

    I was thinking we could update the -b flag to accept a list of globs (whereas today it takes a single folder or list thereof), so it works like the packages field in lerna or workspace fields in npm/yarn, so you can easily pass tsc -b packages/* apps/* to say "all the packages in these two folders are part of my build". I've found that to be the concise way to express how I've seen these repos laid out when I've been comparing rush, learna, and workspaces - of those, rush is the only one that I've seen be regularly much more explicit than that in its config. IMO, glob support in the input paths are probably worthwhile even if you'd not use package.json reading.

  4. mhegazy commented on Jul 3, 2018

    @mhegazy
    Contributor

    how would that work for tsserver?

  5. weswigham commented on Jul 3, 2018

    @weswigham
    Member

    Mohamed Hegazy (@mhegazy) AFAIK Nothing in tsserver yet cares about project references (or the dependency graph thereof), just the presence of declaration maps. 😉 The only one that requires it is probably a cross-project compile on save - which is a matter of integrating the entire -b flag into the server - arguments included in some way. Which, for that, we could choose to interpret a top-level tsconfig similar to

    {
      "references": [{ path: "packages/*" }]
    }

    to setup the context. (Which, hopefully, would also allow tsc -b in that directory to work without any arguments). This mimics a lerna.json, a rush,json, or the top-level package.json used for npm/yarn workspaces.

  6. weswigham commented on Jul 3, 2018

    @weswigham
    Member

    I mean, jamming it into a tsconfig is ultimately wrong (a .tsbuildrc.json with dedicated build-context wide settings would be more appropriate), since tsc -b's arguments don't correspond to tsc's normal arguments at all, so re-purposing tsc's config file is also wrong (since 99% of it would be meaningless and the remaining 1% would be repurposed overlap), but I've lost that fight already.

  7. weswigham commented on Jul 3, 2018

    @weswigham
    Member

    Or alternatively, we can just actually read a lerna.json, a rush.json (maybe), or a top-level package.json with a workspaces field; because why duplicate even that configuration when you don't need to.

  8. Cryrivers commented on Jul 5, 2018

    @Cryrivers

    I think cross repo reference + yarn workspace worked before TypeScript 2.9

    Say you want to import something from your monorepo folder packages/@common/library,the code fix feature in VSCode could correctly suggest the path @common/library in TypeScript 2.8.

    However, since 2.9, I believe it goes through symlinks or something, VSCode now suggests something like ../../../../../../node_modules/@common/library, resolving all the way back to the node_modules folder in monorepo root (with yarn workspace enabled).

  9. timfish commented on Jul 10, 2018

    @timfish

    Also bear in mind that the various tools can treat the workspace globs differently.

    I tried pnpm recursive install which goes looking through every subdirectory whereas other monorepo tools I've tried only look at the first level of directories. That meant it also went through a jspm_packages directory we have and tried to install and symlink all those too!

    The downside to trying to support specific monorepo types is that these are just the ones we're using this week. Next week we'll probably be using something completely different 😆

  10. friflo commented on Jul 13, 2018

    @friflo

    I am using a monorepo setup based on lerna. The problem I have in this scenario is, that the root folder contain all dependencies of all packages.
    This means that every *.ts file is able to import any module, which is in root/node_modules.
    Before I made this experience I expected that only the package.json dependencies are considered to be resolved. This is what a developer would expect.
    I think it is sufficient to apply this type of module resolution at compile time via a config setting in tsconfig.json (e.g. "dependencies": "./).
    The module resolution at runtime (via node) should be unchanged.

  11. dicarlo2 commented on Jul 15, 2018

    @dicarlo2

    Zhongliang Wang (@Cryrivers) actually, that bug was introduced in 2.9.2. There's an issue open for it, but I can't seem to find it now. 2.9.1 works as expected.

  12. akosyakov commented on Aug 15, 2018

    @akosyakov

    Just to share: for Theia I've added a script that converts yarn workspaces to ts project references: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/scripts/configure-references#L19

    It can be executed as prepare script to make sure that they stay in sync: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/package.json#L50

  13. 52 remaining items

  14. pleunv commented on May 25, 2021

    @pleunv

    I'm looking into converting a few existing repositories into a pnpm workspace monorepo as well (+ possibly add rush to the mix if I get pnpm to work) and I've been hitting my head against this for the past few days.

    There's various articles (and issues on SO) to be found covering monorepo setups with TypeScript and lerna or yarn. However, most of these are outdated, contain conflicting information or do things in different ways, in addition to having various DX shortcomings (i.e. no proper scope auto-import, no definition lookups, manual building of libs, and so on) that honestly make the whole monorepo setup a no-go for me.

    There doesn't really appear to be any official documentation anywhere merging all of these concepts together, never mind adding something like babel-preset-typescript into the mix. There's definitely parts of this setup that are package manager or bundler specific, but it has also been getting increasingly difficult to keep up with and figure out how various aspects of the TypeScript config interact with one another in a (nested) project structure like this. I'm thinking about things like project references, paths, incremental builds & composite, noEmit, declarations & maps, and so on. There's also VSCode's new multi-level root workspaces that I've seen mentioned here and there but that honestly doesn't seem to make much of a difference for me at all.

    My conclusion after these past few days is that the level of in-depth TS knowledge you seem to need to get a setup like this at least semi-functional just feels too high (read as: I couldn't figure it out and I'm rather frustrated by it). This is not a critique to the TS team, I'm just wondering if what I've been trying to set up is simply not feasible at the moment. I'm also not really sure where this belongs. Workspaces are currently implemented by package managers, but TypeScript seems to be moving more and more into a "holistic" role, adding various capabilities to link different (nested) projects together, so there's definitely quite a bit of overlap.

    Since it's seemingly going to take a while before package managers converge on a workspace approach, making native workspace support inside TS rather difficult, could it perhaps be an option to gather & document various approaches in use today, with their benefits and (DX) shortcomings? Or document a section with best practices somewhere?

  15. jaredpalmer commented on Dec 17, 2021

    @jaredpalmer

    Causing auto import to fail: vercel/turborepo#331

  16. pokey commented on Mar 3, 2023

    @pokey

    Probably not as good as a single source of truth, but fwiw meta-updater can be used to keep them in sync. See the config that pnpm uses in their own monorepo

  17. trikadin commented on Sep 4, 2024

    @trikadin

    Is it possible to implement something like this?

    tsconfig.json

    {
      "references": [
        {
          "path": "workspace:@namespace/package-name[/tsconfig.json]"
          // or "workspace": "@namespace/package-name"
        }
      ]
    }

    package.json of "@namespace/package-name"

    {
      "exports": {
        "tsconfig.json": "path/to/tsconfig.json"
      }
    }

    Advantages:

    1. Explicit Control: Developers can explicitly manage which references they want to include in the build.
    2. Minimal Changes: No additional fields in package.json and almost no new logic required.
    3. Ease of Implementation: Likely straightforward to implement.
  18. jtwebman commented on Nov 2, 2024

    @jtwebman

    I was thinking lets add a new compilerOptions called workspace. It defaults to none but supports npm, yarn, pnpm. All it does is add to the references based on the rules of that workspace tool. I am just now starting to read though the Typescript compiler and code so not sure what that would entail but it seems easy to me. The advantage here is it can look at the projects deps and dev deps and then see if those projects are workspaces in the current project. If so add them. If not don't. That way you don't have globs which will not work if you have circler references. Will fork and give it a try but what is the process to get this as a feature request and into the code? Does Microsoft except pull requests?

  19. samdenty commented on Apr 2, 2025

    @samdenty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions