Repository navigation
Do compilation on GitHub-hosted runners, transfer as artifacts to self-hosted runners for benchmarking #446
Description
By using GitHub-hosted runners for compilation, we may get better overall throughput and shorter wait times for developers when a bunch of PRs are queued.
Another potential advantage here is that the setup of toolchains will only be required in the compiling action runner, hopefully making the benchmarking runners just "vanilla" installs of their respective OSes and therefore easier to spin up.
We should confirm whether this fits in our free allowance provided to private projects (2,000 min/month, 500MB of storage for artifacts).
We will also need to keep the "single Python command to do everything" refactor working for non-Github-Action-based environments.
Credit for the idea goes to @brandtbucher.
Metadata
Metadata
Assignees
Labels
No labels
Activity
mdboom commentedon Jun 12, 2025
Looking at the last full month, the back of the napkin math is that this would take 5,275 "adjusted minutes" which is a lot more than the 2,000 we get for free. Of course, last month isn't representative of what we will experience going forward (especially talking about what happens if we unleash this on a bunch of new core devs).
(Here "adjusted minutes" means Windows is 2x and macOS is 10x per GitHub's pricing).
It's still probably worth experimenting with this and then turning this on or off based on how the new system is utilized.