Repository navigation
Investigate ways to speed up venv creation #317
Description
According to the log timestamps, benchmark runs currently spend 3 minutes creating virtual environments.
We might be able to reduce that by caching the results of the sat solver and reusing that (that should only change when pyperformance/python_macrobenchmarks are updated).
An earlier experiment with uv was about 2x faster (from memory), but that adds another dependency.
Cc: @mpage, @colesbury
Activity
mpage commentedon Dec 11, 2024
Is that 3 minutes per benchmark run or 3 minutes in aggregate?
mdboom commentedon Dec 11, 2024
In aggregate.
mdboom commentedon Dec 16, 2024
For anyone seeing significantly longer venv creation times than ~3 minutes, @Fidget-Spinner had success by upgrading pip to a more recent version: python/pyperformance#375 (comment)
Fidget-Spinner commentedon Jan 8, 2025
I upgraded pip but it reduced the time by half. So it's still somewhere like 15-30 minutes. I know someone else who just set up pyperformance and it's doing the same too. So maybe this is really worth it to implement for machines that are somehow slower.
mdboom commentedon Jan 8, 2025
Isn't it the case that if you are running locally, the venvs are reused between runs, so this cost is only paid once?
For github actions (clean VMs), of course we pay it every time, but there the 3 min cost seems negligible.
Or am I missing something?
Fidget-Spinner commentedon Jan 8, 2025
Yeah that's the strange part. For my machine and the other person I mentioned, it takes longer than that. Something like 30 minutes to set up venvs in total. I upgraded pip to the latest version.
mdboom commentedon Jan 8, 2025
If this was run in Github Actions, the log should have a timestamp for each line, so maybe that would offer some clues as to where the time is going.
Fidget-Spinner commentedon Jan 8, 2025
This was manual
pyperformancesadly, so I don't have timestamps anymore, sorry! However, on my end I ran the benchmarks manually on my machine and watched them and the venv creation was definitely way over 3 minutes.mdboom commentedon Jan 8, 2025
Interesting. My main objection was to the approach was creating the venvs in parallel, since I don't think that's guaranteed to create the same results each time which is bad for reproducibility.
It might be fruitful to experiment with using uv (but still creating the venvs in order) rather than pip to see if that helps. (An alias from
pip installtouv pip installmay be enough). Even if we don't end up using that on Github Actions (where installing uv negates the benefit a bit), having that as an option might help people running pyperformance locally who are hitting this issue).Though I suspect there's just some environmental difference (maybe file system setup) that makes it slower in some cases -- but that's just a hunch.
Fidget-Spinner commentedon Mar 14, 2025
Ok I think I've narrowed down a key difference on why my venvs take so much longer: I'm using miniconda and its associated venvs for venv management on my system. That's the only difference I can see. Will try normal pip3 venv and see if there's any difference.
Edit: it made no difference.