Repository navigation
fix(pypi): include platform-specific packages in unified hub - #4172
Open
hartikainen wants to merge 2 commits into
Open
hartikainen wants to merge 2 commits into
hartikainen wants to merge 2 commits into
Conversation
Use each hub's full wheel map when creating unified aliases so that platform-specific requirements resolve on supported platforms.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The news fragment must include the required link to issue #4171 before approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Fixes unified @pypi hub generation so platform-specific packages remain addressable while concrete hubs enforce platform compatibility.
Changes:
- Collect packages from each hub’s complete wheel map.
- Add Linux/Windows integration coverage.
- Add a PyPI news fragment.
| File | Description |
|---|---|
python/private/pypi/extension.bzl |
Includes all wheel-map packages in the unified hub. |
tests/integration/unified_pypi/MODULE.bazel |
Configures platform-specific requirements. |
tests/integration/unified_pypi/BUILD.bazel |
Defines test platforms. |
tests/integration/unified_pypi_test.py |
Adds platform resolution tests. |
tests/integration/unified_pypi/requirements_b_without_six.txt |
Adds non-Linux requirements. |
news/unified-platform-packages.fixed.md |
Documents the fix. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Clarified the inclusion of platform-specific packages in the unified @pypi hub for alias resolution on supported platforms. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
rickeylev
pushed a commit
to rickeylev/rules_python
that referenced
this pull request
Oct 7, 2026
…-contrib#4223) The unified `@pypi` hub (bazel-contrib#3837) does not generate a `requirements.bzl`. That makes renaming a hub away from the now-reserved `pypi` name an all-at-once change: the moment the concrete hub stops being `@pypi`, every `load("@pypi//:requirements.bzl", "requirement")` in the repo fails to load. In our monorepo that is ~940 BUILD files, and any BUILD file that lands on main mid-migration re-breaks it. The same thing would happen to every such repo if `RULES_PYTHON_PYPI_HUB_RESERVED` is flipped on by default, as discussed in bazel-contrib#3837. This adds an opt-in, `pip.default(unified_hub_requirements_bzl = True)` (root module only, off by default), that generates a `requirements.bzl` in the unified hub with the per-package macros of a concrete hub's. It's opt-in because the docs already steer users to `@pypi//<pkg>` labels over `requirement()`; this is a migration aid for repos whose hub used to be named `pypi`, not a new default surface. With it on: * `requirement()`, `whl_requirement()`, `data_requirement()` and `dist_info_requirement()` return labels in the unified hub (`@@<unified>//<pkg>:pkg` etc., the same canonical-name form the concrete hubs use), so they route through `--@rules_python//python/config_settings:venv` exactly like a plain `@pypi//<pkg>` label. All of those targets already exist in every unified package (`_STANDARD_ALIASES`). * The `all_*` lists (`all_requirements`, `all_whl_requirements(_by_package)`, `all_data_requirements`) are deliberately not generated. They are fixed at loading time, before the venv flag is known, so they could only list one hub's packages and would silently not follow the flag like the rest of the unified hub. Code that wants a whole lock loads them from that concrete hub, and a stale `load("@pypi//:requirements.bzl", "all_requirements")` fails loudly instead. Before: after renaming `hub_name = "pypi"` to e.g. `"pypi_main"` with `pip.default(default_hub = "pypi_main")`, `load("@pypi//:requirements.bzl", ...)` fails with "no such file". After, with the opt-in: it keeps working and resolves to the same wheels, and targets can migrate to `"@pypi//<pkg>"` labels incrementally (we're moving our own repo to labels and gating new `requirement()` calls in CI). Without the opt-in nothing changes. Tests: * `tests/pypi/extension`: the setting is off by default, on when the root module sets it, and ignored from a non-root module. * `tests/integration/unified_pypi` (which now opts in): `requirement()` with a non-normalized name on the default hub, `requirement()` under a venv transition, and loading `@pypi//:requirements.bzl` failing once the opt-in is removed. Passes on `bazel_self` and `bazel_7.7.0`. Context: we're carrying this as a local patch in a large monorepo, where renaming our main hub and loading `requirement` from the unified hub leaves the resolved wheel set unchanged. bazel-contrib#4172 also touches `tests/integration/unified_pypi`; the two should merge independently, but one of them may need a trivial rebase. (Done with help from an agent.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Done with help from an agent.)
The unified
@pypihub omits packages that appear only in some platform-specific requirements files. Dependencies such as@pypi//sixtherefore fail to resolve even on platforms where the selected concrete hub provides the package.This PR collects unified aliases from each hub’s complete wheel map so that platform-specific packages remain accessible. The concrete hub’s aliases continue to enforce platform compatibility.
Closes #4171.