Repository navigation
perf: Node.getChildren speed regression in ts 5.5 when used at very large scale #59101
Description
Activity
I didn't get that far on this today, but I believe this is from more time spent in the GC. If you try upping your
--max-old-space-sizeto something way higher, let me know if that alleviates things.As an aside, SO much time in your build is being spent on shims running within eslint-plugin-react.
While
findVariableByNamereally should be optimized to not create an intermediate array withvariablesInScope, is it possible that you are running on an undesirable input file?This is the opposite of expected! Which walk function is that? I'm not familiar with that "walk" function. Traces would be helpful, but walkerdb/sentry#1 is probably enough to give us a lead.
You may also consider running things via https://www.npmjs.com/package/pprof-it (what Daniel just used above) as it in my experience does a better job attributing time and memory allocations to specific lines of code.
Hm, my trace doesn't seem to show that particular code as to blame:
Walker Boyle (@walkerdb) What tool did you use to find your result?
I timed this, and in 5.5, it's 100s, but 5.4 is 91s; not 2-3x slower from what I can tell...
A second run of 5.4 was 100s too. Not sure what's going on here. What version of Node are you using?
I was using Node 22; switching down to Node 18 shows a worse regression:
// 5.4.5 total time: 160.75s user time: 212.16s system time: 4.96s CPU percent: 135% max memory: 3946 MB // 5.5.3 total time: 217.70s user time: 292.33s system time: 6.36s CPU percent: 137% max memory: 3995 MBNot 2-3x but very much farther apart than Node 22.
Unfortunately I'm now going to be gone until next week, so this will just have to eat me up inside until then...
Hi Jake Bailey (@jakebailey)! Sorry, I should have clarified a few things:
- the traces above were generated using 0x, but on our private monorepo, which has a 2.5x slowdown when running our typed lints under ts 5.5 instead of 5.4. I used sentry as a public repro but its slowdown was only ~1.3x on my machine.
- in our private monorepo we run lint with essentially unlimited max-old-space and also with higher max-semi-space since we run into gc issues with young gen / scavenge as well at our size.
- we are indeed running node 18! Likely will update to 22 by end of year but that's a major undertaking for us.
to Daniel Rosenwasser (@DanielRosenwasser) I'm not a part of sentry, I was just using that repo as an example repro since ours is private.
the types in Notion's private monorepo can be quite complex, and I suspect are close to a pathological case for some typescript perf behavior, which may help explain the 2.5x slowdown we were observing. I'll see if I can hack running the same checks under node 22 to see if the timings change there much.
Ran Node 18, and the results are very different from #59101 (comment):
That is an incredible amount of GC time, along with the badness Daniel noted above. Given the difference between Node 18 and Node 22 on this being in that shim code, I'm not 100% certain that this repro is representative of the repo you're testing internally, unfortunately...
Would you be able to run
pprof-iton your own codebase? Screenshots of flame graphs and such should not expose any details, but you can also run it withPPROF_SANITIZE=onto sanitize the output for sharing (as that will remove any paths from the profile).Jake Bailey (@jakebailey) I'll give pprof-it a try, ty!
here's my pprof result for typescript 5.5.2 on node 18.18. I'll also upload one for typescript 5.4 in a sec
pprof-time-typescript-55.pb.gz
pprof-heap-typescript-55.pb.gzpprof results for the exact same run but using typescript 5.4.5 on node 18.18 . This is significantly faster, and matches our expected run time from prior to the 5.5 update
pprof-time-typescript-54.pb.gz
pprof-heap-typescript-54.pb.gzlast run, with typescript 5.5.2 and node 22.3.0. Running under a newer node version doesn't seem to actually be much faster in this case?
pprof-time-typescript-55-node-22.pb.gz
pprof-heap-typescript-55-node-22.pb.gz16 remaining items
Can do, thank you for all your help! Is there any specific feedback that would help on the new version other than whether tsserver requests like getProgram, getSemanticDiagnostics, getCompletionsAtPosition etc still seem to work and are just as responsive as before? (we track perf metrics for each of them via a custom plugin so it should be pretty easy for us to tell whether there's been any major regression there)
we track perf metrics for each of them via a custom plugin so it should be pretty easy for us to tell whether there's been any major regression there
This sounds amazing, and I absolutely want to hear more about this. This is the first we've heard of someone tracking this to this extent so anything is awesome.
Had a huge
typescript-eslintregression just like this in our very large monolith. Had a similar idea of bumping--max-old-space-sizebut that didn't work:https://discord.com/channels/1026804805894672454/1254556633556713553/1255160613441769683
Setting my typescript dependency to
"typescript": "npm:@typescript-deploys/pr-build@5.6.0-pr-59154-2",fixed the issue, and actually seems a bit faster than baseline!Hey all, this is fixed in the
mainbranch but I want it to prove itself out for a bit. If you use the nightly build tomorrow (especially in the editor https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-typescript-next), that would go a long way.If you are actually able to measure, we are specifically looking for differences in memory usage, changes in editor operation delays, as well as changes in variance in delays.
If things feel good, we can cherry-pick the change back into 5.5.
Daniel Rosenwasser (@DanielRosenwasser) can do! I'll get a test group going internally tomorrow, will see if metrics move much. We do also track total tsserver memory usage, should be able to see if that moves much under the new version as well.
Jake Bailey (@jakebailey) happy to chat more about our tsserver observability setup! Could talk sync / show a few dashboards if you're interested?
and thank you both again for moving so fast on this, much appreciated!
Reacted by Jake Bailey and Daniel Rosenwasser+1 from the typescript-eslint side, we really appreciate it! Thanks so much for the fast follow and (very interesting) deep dive everyone!
following up to say we haven't noticed any serious perf regressions in IDE usage so far. We haven't had the participation numbers to say much more than that though.
Reacted by Jake BaileyLikewise on stability in my testing in very large monorepo (with a very large monolith). Curious if there are plans on a 5.5 backport? Or still targeting 5.6?
The backport is open here: #59211
Reacted by Max SchwenkReacted by Max SchwenkTypeScript 5.5.4 should contain the fix - thanks for reporting everyone!
Reacted by Josh Ghoulberg 👻- added a commit that references this issue
on Jul 30, 2024
🔎 Search Terms
typescript 5.5 performance, slowdown, typescript-eslint
🕗 Version & Regression Information
git bisecton a local typescript repo, it slows down specifically on this commit)⏯ Playground Link
No response
💻 Code
I've put together a repro in a forked version of the sentry monorepo here: walkerdb/sentry#1. It's using the sentry repo only because it's a large public repo that uses typescript-eslint.
🙁 Actual behavior
When running typescript-eslint's type-aware lint rules in a large monorepo with typescript 5.5, we observe lint times between 1.3x to 3x worse than when running the exact same rules and the same config under typescript 5.4. The slowdown seems to be entirely coming from the monomorphized objects change in the ts 5.5 release.
🙂 Expected behavior
There should be no performance impact when moving from typescript 5.4 to 5.5.
Additional information about the issue
We've submitted a bug report to typescript-eslint but we're also reporting here since the slowdown can be pinpointed to a specific typescript change.
Also attaching some before/after update perf traces via 0x that show the extra time is coming largely from typescript internals. The highlighted boxes show the specific part of the process that has ballooned in runtime. I'm happy to share the actual traces with maintainers if they'd be useful in debugging.