Repository navigation
"files" is wrong when installed without "wheel" #115
Description
Activity
In GitLab by @jaraco on Mar 25, 2020, 15:55
Before adding support for
installed-files.txt, I'd like to know what command generates that file. It looks like it's generated in the legacy install.I'm leaning toward saying that importlib_metadata shouldn't support this use-case, that if someone wants to have metadata support, they should use the non-legacy features of the packaging ecosystem.
What is the impact of this issue? Are there real-world use-cases that experience issues as a result of this non-support?
I've recently run into this issue with the Qiskit (meta)-package:
- The package does not actually contain any (runtime) Python code itself, but serves as a mechanism to install its transitive dependencies (which populate the
qiskitpackage namespace). - The package is distributed as a source archive.
- When installed as an egg, it provides a
SOURCES.txtwhich is incorrect from a runtime POV: it references 3.pyfiles, asetup.pyand two files undertest/, none of which are actually installed. - The packages includes a
top_level.txtwhich is empty (contains a single newline). This is arguably correct given that it does not directly install any importable packages/modules.
IMHO, there are two issues with how
importlib_metadatahandles this package:- Because
SOURCES.txtdoes not reflect the installed files,importlib_metadata.files("qiskit")ends up returning paths that don't actually exist on disk. According to setuptools docs,SOURCES.txtshould not be relied upon at runtime (although I understand that it is likely used as a last resort in this case). packages_distributions()returns nonsense for this package:_top_level_declared()does not differentiate between an emptytop_level.txtand a missingtop_level.txt, which triggers a fallback to_top_level_inferred(). This fallback ultimately ends up looking for*.pypaths inSOURCES.txt, which finds the three non-existing paths mentioned above. The end result is thatpackages_distributions()associates the import namessetupandtestwith this package, neither of which exist, or make any sense.
Some possible ways to fix this:
- For (1), try
installed-files.txtbefore falling back toSOURCES.txt. This file is written bypipon installation, and is a more accurate source thanSOURCES.txt. (Ifinstalled-files.txtdoes not exist for any reason, then fall back toSOURCES.txtas last resort.) - For (1), if/when we need to use
SOURCES.txt, then at least make sure that only the paths inSOURCES.txtthat actually remain after installation are returned. - For (2), consider whether it's worthwhile to treat an empty
top_level.txtdifferent from a missingtop_level.txt. In the above case the emptytop_level.txtcorrectly communicates that this packages does not expose any importable names. Having said that, there might be good reasons for the current behavior.
- The package does not actually contain any (runtime) Python code itself, but serves as a mechanism to install its transitive dependencies (which populate the
When installed as an egg,
I believe the issue may be rooted in how qiskit is installed.
The preferred way to install Python packages is as a wheel using dist-info metadata. As eggs are deprecated and superseded by wheels, support for egg-info is incomplete and best-effort.
Consider this example where I've installed qiskit from the source archive as a wheel and am able to avoid the undesirable behavior:
~ $ pip-run --no-deps --no-binary qiskit --no-cache-dir --use-feature no-binary-enable-wheel-cache --use-pep517 qiskit Python 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import importlib.metadata >>> 'test' in importlib.metadata.packages_distributions() False >>> importlib.metadata.files('qiskit') [PackagePath('qiskit-0.41.1.dist-info/AUTHORS'), PackagePath('qiskit-0.41.1.dist-info/INSTALLER'), PackagePath('qiskit-0.41.1.dist-info/LICENSE.txt'), PackagePath('qiskit-0.41.1.dist-info/METADATA'), PackagePath('qiskit-0.41.1.dist-info/RECORD'), PackagePath('qiskit-0.41.1.dist-info/REQUESTED'), PackagePath('qiskit-0.41.1.dist-info/WHEEL'), PackagePath('qiskit-0.41.1.dist-info/top_level.txt')]The key here was to pass
--use-pep517, which caused the source archive to build as a wheel and produce dist-info metadata. Other parameters topip install(via pip run) are there to force the source dist install and bypass other warnings.This approach utilizes the preferred code path for metadata and avoids the undesirable behavior.
Honestly, if it weren't for the fact that editable installs still rely heavily on egg-info metadata, I'd deprecate that format outright.
When I wrote
files()to resolve fromSOURCES.txt, I did that knowing it was an imperfect approximation for editable installs. I was unaware thatinstalled-fileswas also a thing :(.I'm not confident that
installed-files.txtis properly supported either. I see now there was some discussion some years ago in pypa/setuptools#371 aboutinstalled-files.txt. Ideally, I'd like to rally around supported standards and not add support for inferred interfaces.Does the
--use-pep517method of installation provide a suitable approach for your use-case?Yes, installing it with dist-info metadata (either using
--use-pep517or by installing with modern pip in a venv that already haswheelavailable) does solve the issue (*.dist-info/RECORDis obviously more accurate than*.egg-info/SOURCES.txt).I 100% agree on using modern packaging practices. However, in this case, I'm not in the position to control the package installation process: I'm working on a tool to find undeclared and unused dependencies in the user's project, and as such I'm using
importlib_metadatainside the user's existing venv, and I do not control which packages are installed, or how they were installed.installed-files.txtseems to have been written by pip since version 0.3 (Jan 2009), and even if it has not been adopted by the rest of the packaging ecosystem, I think the current prevalence of pip makesinstalled-files.txtuseful as a better source thanSOURCES.txtwhen it exists. When it does not exist (e.g. it seems to not be created in the case of editable installs), falling back toSOURCES.txtis still available as a final - albeit imperfect - resort).Do you foresee problems with any of the three fixes I outlined above? In any case, I'm happy to whip up a PR.
Reacted by Jason R. Coombs and Filipe LaínsAll good points. Yes, I think I agree with the recommendation, and I'm delighted to hear you'd like to work on an implementation. Crucial will be to have tests that capture the relevant factors and expectations (including which scenarios apply to which logic branches). I'm slightly tempted to deprecate the use of SOURCES.txt altogether, but that probably should happen separately. Feel free to take a stab at it and let me know if you have any questions. I can be reached in Gitter (except I haven't enabled this project) and Discord.
Reacted by Johan HerlandI recently touched this code and also think looking at
installed-files.txtwhen available is a good approach 👍- added 8 commits that reference this issue
on Mar 11, 2023
In GitLab by @blueyed on Mar 17, 2020, 12:32
When "wheel" is not installed/used, installing a package will not contain "RECORD", and it falls back to reading "SOURCES.txt" (https://gitlab.com/python-devs/importlib_metadata/blob/3150ed4da9e1267d0787c6f4c1f8258a26a1dd93/importlib_metadata/__init__.py#L270), which then might result in paths not being
locate()able, when a src-based setup is used:This is similar to https://gitlab.com/python-devs/importlib_metadata/-/issues/112, but in this case here it could use
installed-files.txtfrom the egg-info.