Repository navigation
Expand Perf Suite Codebases, Expand Perf Suite Operations #44033
Description
Activity
Ron Buckton (@rbuckton) we can turn on declaration emit, source maps, and declaration maps for most (if not all) perf projects (and remove
noEmitOnError), but doing so is going to invalidate historic relationships for the data, so if we want to, we'll have to regenerate some historical data - are we OK with doing that?Wesley Wigham (@weswigham) Perhaps you can do this additionally?
Reacted by pialinReacted by pialinMartin Johns (@MartinJohns) can you elaborate?
Daniel Rosenwasser (@DanielRosenwasser) Continue to run the perf suite as is, but additionally run the perf suite with declaration emit, source maps and declaration maps enabled. Basically produce two data sets.
In our current setup, we run benchmarks against a snapshot of a specific project, using the settings specific to that project. This is intended to cover different scenarios, such as:
- Single-file output vs Multi-file output vs no output.
- No source maps, vs external source maps, vs internal source maps.
- Declaration emit vs. no declaration emit.
- Emitting with and without comments.
- Exercising different parts of the compiler (i.e.,
Compiler-Unionsheavily uses union types to stress test those scenarios).
That way we aren't only focused on everything being a "kitchen sink" build, and we can catch performance regressions in non-"kitchen sink" areas (i.e., we don't want to slow down in code paths where we don't emit source maps). The current approach is somewhat randomized, but it covers most of the bases. I wouldn't want to have to run a benchmark against the same project for every permutation of every feature, as that doesn't scale.
Daniel Rosenwasser (@DanielRosenwasser) Continue to run the perf suite as is, but additionally run the perf suite with declaration emit, source maps and declaration maps enabled. Basically produce two data sets.
What we can do is add new scenarios that point to the same sources but with different settings. That way old runs wouldn't be invalidated. So in that case we might have a "TFS+SourceMaps" scenario, etc.
Reacted by Daniel RosenwasserRelated: #44251
24 remaining items
- assigned and unassigned
on May 11, 2023 - assigned and unassigned
on Aug 1, 2023 Drizzle? (see #54939)
As of today, our benchmark suite can now run
bun. For example: #52656 (comment)This is particularly interesting as it uses an entirely different engine (JSC) than Node/Deno (V8).
bun0.8.1 is currently broken when runningtsc(oven-sh/bun#4327), but one can run 0.7.3 and it works.tsservertests don't work, nor dotsserverstartup tests, but given I'm using an old release, hopefully it is fixed in a future release.I'm not yet sure if this should be a part of our main test suite; probably not until we increase our capacity a bit more.
Next up would be using VS Code's electron build via
ELECTRON_RUN_AS_NODE, which has different performance behaviors due to pointer compression and memory caging; we don't currently test it, yet there's probably more CPU cycles spent intsservervia VS Code than any other way to run TypeScript!Reacted by Jarred Sumnerbun0.8.1 is currently broken when runningtsc(oven-sh/bun#4327), but one can run 0.7.3 and it works.tsservertests don't work, nor dotsserverstartup tests, but given I'm using an old release, hopefully it is fixed in a future release.This is fixed in oven-sh/bun#4378 (currently the canary build) which will become the release build (v0.8.2) tomorrow or the next day. The error was in some code assuming
/dev/fd/${n}is mounted which it is not mounted by default in minimal Linux distributions like alpine. Our CI machine uses Ubuntu Server which mounts/dev/by default, which is why we didn't catch it sooner. We will add an integration test that runstsc --help.The last few lines of output of
strace tsc --helpwhentscis run in bun:Bun v0.8.1 Bun v0.8.2 
Note how it is no longer calling openat() on
/dev/fd/1in v0.8.2Thanks for the fix! There are other things that are broken (as noted above), but I have yet to go reproduce them. I'll send some bug reports once I get that sorted.
Hello from Prisma 👋🏼
Let us know if you're interested to hear from us about some complex types to test. It could be interesting to have some types used in Prisma Client Extensions, for example.The performance test suite is located at https://github.com/microsoft/typescript-benchmarking? but is currently private? Do you plan to make it public one day? (Of course you might have reasons to not make it public, I'm curious here)
That repo is currently only build infra, which I plan to make public in short order as there's nothing sensitive.
The actual tool that does the benchmarking isn't ready to be made public and that's where the benchmarks currently live. I'm hoping I can fix it up to pull benchmarks from the public repo but it's not really made for multiple places to store benchmarks, or benchmarks that have to be cloned from elsewhere. But that's just details and I'm working on it. The new hosts were just a lot easier to add!
Reacted by Joël Galeran, pierre and Søren Bramer Schmidtperf test vscodeis now live: #55267 (comment)Next up is adding some more benchmarks and making public what I can make public quickly, e.g. the infra itself (but not the tool, sorry).
Reacted by Joël GaleranReacted by Daniel Rosenwasser and SlurpTheoI'm happy to say that the new perf infra is now live and public: https://github.com/microsoft/typescript-benchmarking
I've been waiting to make this public as I did not yet have a good mechanism to support public benchmarks, but I got it all working in the past few days and now it's ready to go. The first public benchmarks are
vscode, as well as self tests of TypeScript compiling itself (bothtsc -p ./src/compileandtsc -b ./src).Note that the public benchmarks are still split off from our old internal benchmarks; we have a lot more perf capacity than we used to have but still not enough to start adding lots that always run.
For now, they can be run via the
perf test publicpreset. An example: #56987 (comment)This is the first time we've had new, actually error-free benchmarks in years, so I'm very excited.
Reacted by liuxingbaoyu and SlurpTheoI'm going to close this issue in favor of microsoft/typescript-benchmarking#32 on the benchmarking repo.
This thread was refocused toward benchmark suggestions, but now that the repo that holds them is public, discussion and PRing can happen there.
The perf suite code typically was added for specific scenarios, but if you're trying to get a sense of how sourcemap emit has changed, it's hard given that not all projects use that functionality.
I think we should