Repository navigation
Automate build and release to PyPI #7390
Description
Currently the release process has many steps, including preparing things in this repo, and then starting a build in https://github.com/python-pillow/pillow-wheels, before downloading the sdist + wheel artifacts to disk, and manually uploading them to PyPI using twine.
I'd like us to move towards more automated releases, including using the new Trusted Publishers to upload wheels directly from the CI to PyPI, which is more secure:
We can do this in a phased manner, not all of these things would need to be done at once, to ensure we can still make releases with just some of them completed.
We could potentially deal with the Travis CI wheels later, and tackle GitHub Actions first as they're a bigger win and we don't know when Travis CI will go away.
Big picture:
- Prepare release
- Push a tag
- The CI builds release artifacts
- The CI uploads artifacts to PyPI
I'd suggest something along these lines:
-
Move Linux and macOS https://github.com/python-pillow/pillow-wheels to this repo https://github.com/python-pillow/Pillow. We already have Windows wheels build here.
- We'll need to add
paths:/paths-ignoreto workflows so we only trigger the main test workflows for "normal" code changes, and only trigger the wheel workflow for things like release and modifying its config - Upload wheels as GitHub artifacts in this repo instead
- We'll need to add
-
Add automation to download artifacts and upload them to PyPI via Trusted Publishing
- This may need refactoring to move all the wheel building to the same workflow (but separate jobs), and then a final job that runs when the wheels have all been built, that builds sdist, checks and uploads them all.
- See https://github.com/ultrajson/ultrajson/blob/main/.github/workflows/deploy.yml and https://github.com/ultrajson/ultrajson/actions/workflows/deploy.yml for examples.
- We also have https://test.pypi.org/project/Pillow/ which we can use to verify the upload machinery is working smoothly before release. For example, we can upload for non-tagged wheel builds.
Thoughts?
Activity
wiredfool commentedon Sep 11, 2023
This sounds like a great step.
aclark4life commentedon Sep 11, 2023
Love it!!
radarhere commentedon Sep 24, 2023
I've created #7418 for 1.
hugovk commentedon Nov 12, 2023
I've opened #7544 to create the sdist during the release build.
Next, I'd like us to move the Windows wheel building from
test-windows.ymlinto the samewheels.ymlworkflow.So we get something like this:
And the Windows wheels should be part of the same
dist.zipartifact containing the Linux and macOS wheels, and the sdist.I think we also probably want to use a reusable workflow, similar to
wheels-linux.ymlandwheels-macos.yml.We don't need to build the wheels anymore in
test-windows.yml. If there's a lot of common steps needed for testing (intest-windows.yml) and building wheels, a reusable workflow could be useful here.This doesn't all need to be done in one go, incremental steps are fine, as long as they're atomic changes and we're still able to make releases during any intermediate step.
Any thoughts on how to approach this? cc @nulano
After all this is done, we have all* the release files created in one place on the CI, and do the upload to PyPI! But one step at a time :)
(* All the GHA files, not Travis CI, which will still need manual upload, but they should be ready before all the automation.)
nulano commentedon Nov 15, 2023
I've managed to use
cibuildwheelto build wheels locally, I should be able to make a PR on top of #7552 for Windows wheels once it is merged.Edit: I've got a WIP branch ready, including ARM64 wheels
(although I have no way to test them!).Edit2: I've created PR #7580.
hugovk commentedon Nov 16, 2023
Sounds good! When the time comes, we can ask around for someone to test the ARM64 wheels :)
nulano commentedon Nov 16, 2023
I've since realized that I can borrow a relative's M2 MacBook to test the wheels.
The Python 3.12 wheel from my WIP branch tested OK.
I then used
cibuildwheelmanually in the terminal to build and test wheels for the 10.1.0 tag for #7547 (comment)radarhere commentedon Dec 26, 2023
#7616 has now been merged.
@aclark4life could you go to https://pypi.org/manage/project/pillow/settings/publishing/ and add a publisher?
Owner: python-pillow
Repository name: Pillow
Workflow name: wheels.yml
Environment name: release-pypi
hugovk commentedon Dec 27, 2023
@aclark4life Please can you also set up an environment named
release-pypiin this repo, that all the release managers can access: at least @radarhere and me, probably also you and @wiredfool for backup.aclark4life commentedon Dec 27, 2023
@radarhere @hugovk Thanks and done, I think! The env
release-pypialready exists … need me to do anything to it?hugovk commentedon Dec 27, 2023
Thanks, the publisher looks good!
@aclark4life What settings does it have?
@radarhere What settings should it have?
aclark4life commentedon Dec 27, 2023
Most notably, no environment vars or secrets configured.
radarhere commentedon Dec 27, 2023
The publisher looks good to me, thanks. The details I mentioned were the only ones I added.
hugovk commentedon Dec 27, 2023
So when releasing, after pushing the tag, the CI will begin, build the sdist and all the wheels, and somehow put them in an environment? How does that look? What's the next step to upload to PyPI?
aclark4life commentedon Dec 27, 2023
Probably need a testpypi release first.
radarhere commentedon Dec 28, 2023
When I was using Test PyPI in #7616, once sdist and the wheels were built, the action uploaded them, and that was it.
https://github.com/radarhere/Pillow/actions/runs/7329629479/job/19960359962#step:4:1008
I presume that Test PyPI works the same way as real PyPI. I would have imagined that real PyPI releases don't rely on test PyPI to go first, since we've been ignoring test PyPI for every previous release we've done.
hugovk commentedon Dec 29, 2023
Okay, so the environment isn't doing anything. Anyway, I'm excited for the next release!
@radarhere Please will you update https://github.com/python-pillow/Pillow/blob/main/RELEASING.md? It can wait until after doing the release if you like.
radarhere commentedon Dec 29, 2023
Sure. I've created #7652
radarhere commentedon Jan 2, 2024
The tagged "Upload release to PyPI" ran smoothly in the Pillow 10.2.0 release.
With that, is this completed?
hugovk commentedon Jan 2, 2024
Great to hear it went smoothly!
Before closing this issue, we still have Travis to automate. Shall we try my suggestion at #7485 (comment) to upload to PyPI directly from Travis?
radarhere commentedon Jan 2, 2024
Sure, sounds good if it works.
hugovk commentedon Jan 3, 2024
Plan C: please see PR #7690 to replace Travis native builds with GHA emulated builds.
aclark4life commentedon Jan 3, 2024
Only loosely following, but I think I like Plan C!! Thanks all ❤️ (Losing travis in exchange for longer build times is my "hot take", sounds worth it to me.)