Repository navigation
make altinstall for PGO is not parallel-safe #93584
Description
Bug report
setup.py has a helper method add_multiarch_paths. The method creates a temporary file build/lib.{platform}/multiarch and unlinks it at the end of the function call. This is not parallel-safe.
PGO builds of Python use $(MAKE), so called recursive make. Recursive makes are considered harmful because the main make process has no understanding what the child make process is doing. With heavy parallel makes this can cause race conditions.
The combination of unsafe add_multiarch_paths(), recursive make and loooots of CPU cores can lead to build issues like #84461 (comment).
Possible workarounds
- Rewrite our Makefile to not use
$(MAKE) - Move
CC -print-multiarchanddpkg-architecture ... -qDEB_HOST_MULTIARCHcalls toconfigure.ac - Fix
_bootsubprocess.check_output()andadd_multiarch_paths()to use a more safe tmp file
Option (3) is the simplest approach. Even tmpfile = os.path.join(self.build_temp, f'multiarch-{os.getpid()}') would be good enough to avoid file conflicts in parallel builds.
Activity
tiran commentedon Jun 7, 2022
My PR solved the race condition in
add_multiarch_paths()but revealed another race condition later.@pablogsal 's problem is caused by the fact that PGO builds use recursive make. The
sharedmodstarget and theprofile-opttarget run./python setup.py buildat the same time.setup.pyis not designed to be parallel-safe.emmatyping commentedon Jun 7, 2022
Weird, I never hit a compilation failure after 2 hours of trying on my 32 core machine, maybe this is an edge case you can only hit with the macOS scheduler.
tiran commentedon Jun 7, 2022
Did you create a build with
configure --enable-optimizations? The issue only occurs with PGO builds.emmatyping commentedon Jun 7, 2022
Yeah, I was running a script in a loop using the steps from Pablo's reproducer here: #84461 (comment)
After 2 hours it still hadn't failed
pablogsal commentedon Jun 7, 2022
That's odd. I have been able to reproduce in at least 3 systems, including Linux with arch Linux and RHEL7
tiran commentedon Jun 8, 2022
In comment #93589 (comment) @nascheme suggested to rename some of the targets to make the intention more clear. I like the idea, but I don't want to mix improvements with a release blocker fix.
[-]setup.py add_multiarch_paths() is not parallel-safe with recursive make[/-][+]make altinstall for PGO is not parallel-save[/+]gh-93584: Make all install+tests targets depends on all (GH-93589)
[3.11] pythongh-93584: Make all install+tests targets depends on all (p…
[3.11] gh-93584: Make all install+tests targets depends on all (GH-93589
[-]make altinstall for PGO is not parallel-save[/-][+]make altinstall for PGO is not parallel-safe[/+]tiran commentedon Jun 15, 2022
@pablogsal As far as I know this bug has been fixed by GH-93589.
Revert 30935 defer bpo45162 to 312 (#94285)