You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Linking against libraries from meson on Windows #265
Assuming you mean shared libraries that are built within the project that meson-python is invoked on, that may be feasible. The copying itself isn't the problem, DLLs can be put into .pkgname.mesonpy.libs just like on other platforms. The question to answer is how then to ensure that the Python extension modules that depend on those DLLs start working (xref gh-259). I'm not sure that there is a better answer than preloading them with, e.g., ctypes.WinDLL (as done in https://github.com/scipy/scipy/blob/main/tools/openblas_support.py#L217 for SciPy). And doing that by default is tricky - at least the question should be answered how that is done, and also what happens if then delvewheel is run on top of that.
Thanks for detailed explanation.
The key is that how to make the packaged DLLs loadable from python (when they are not in PATH).
I think putting all generated DLLs of one package in the same directory will be feasible?
For example, we have the following directory structure (I build locally from https://github.com/mesonbuild/meson-python/tree/main/tests/packages/link-against-local-lib)
I think putting all generated DLLs of one package in the same directory will be feasible?
It fixes the issue, but that's way too much of a restriction. Your average Python package will have multiple directories, and it can have extensions in each one of them. The RUNPATH solution for non-Windows platforms works no matter where you put your extension modules. For Windows we can achieve the same probably, but it needs some preloading solution and it's going to be uglier than using RUNPATH. Or maybe a shim dll per shared library that you want to ship:
pypkg1
|.pkgname.mesonpy.libs
- example.dll
|pkgname
|subdir
- example.cp310-win_amd64.pyd
- example_mesonpy_dll_loader.dll # generated dll, which has the job of preloading example.dll
I understand your concerns. We need to make DLLs in .pkgname.mesonpy.libs loadable for all .pyd files in package files.
Maybe we can inject a one-line call to os.add_dll_directory to the top-level __init__.py of package?
For Windows we can achieve the same probably, but it needs some preloading solution and it's going to be uglier than using RUNPATH.
We could consider a solution similar than what we are doing for editable installs. But IMO this should be something provided by either Python, or the packaging ecosystem already.
But IMO this should be something provided by either Python, or the packaging ecosystem already.
I would not expect anyone in Python packaging land to care, except scikit-build which will have the same type of user profile and demand for non-Python shared library support. So I think if we coordinate with them, that should suffice.
Maybe we can inject a one-line call to os.add_dll_directory to the top-level __init__.py of package?
However, modifying a user's __init__.py is too magical. What if we just documented that users with shared libraries need to add this to their main __init__.py file, before importing any extension modules:
ifos.name=='nt': # note: add clause for sys.version >= 3.8 if project still support py37mesonpy_dlldir=`../.pkgname.mesonpy.libs`# fixme: use portable path specification hereifos.exists(mesonpy_dlldir): # directory may not exist, if we're building with `meson` directlyos.add_dll_directory(mesonpy_dlldir)
That should be simple enough to not get wrong, and this only affects a small amount of users so I'd be happy enough with a doc-only solution here.
It only leaves out the case of no Python package but a standalone Python extension module which depends on a DLL, but in that case the user can do what @FFY00 suggests above and install both files to the same directory. That can be documented as well.
One more note on this: on Python 3.8 installed with Conda there may still be problems with os.add_dll_directory that have to be worked around with CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 (see conda/conda#10897). This may be why SciPy is still preloading with ctypes.WinDLL.
One alternative solution I just thought of is to ad a setting to allow running delvewheel on the wheel produced by meson-python, after meson-python is done with what it currently does, but before returning control to the build frontend (so inside the build_wheel hook). If it's opt-in and we make the package author responsible for adding delvewheel to their build dependencies on Windows (or we use a patchelf-like dependency insertion rule), that seems like a quite nice capability to have, and easier to implement than reinventing what delvewheel does or preloading with ctypes.
I believe this may work for most "build on Windows from source with a shared library" needs. The exception being if you are trying to ship a shared library in a wheel for consumption outside of that package (because the symbol mangling applied by delvewheel will make that impossible).
So a complete solution may be:
include the shared library in the wheel just like on Linux/macOS
have a setting for the package author to opt in to their preferred way of using that:
os.add_dll_directory (and then they have to add the .mesonpy.libs snippet themselves)
Activity
rgommers commentedon Jan 17, 2023
Assuming you mean shared libraries that are built within the project that
meson-pythonis invoked on, that may be feasible. The copying itself isn't the problem, DLLs can be put into.pkgname.mesonpy.libsjust like on other platforms. The question to answer is how then to ensure that the Python extension modules that depend on those DLLs start working (xref gh-259). I'm not sure that there is a better answer than preloading them with, e.g.,ctypes.WinDLL(as done in https://github.com/scipy/scipy/blob/main/tools/openblas_support.py#L217 for SciPy). And doing that by default is tricky - at least the question should be answered how that is done, and also what happens if thendelvewheelis run on top of that.metab0t commentedon Jan 17, 2023
Thanks for detailed explanation.
The key is that how to make the packaged DLLs loadable from python (when they are not in PATH).
I think putting all generated DLLs of one package in the same directory will be feasible?
For example, we have the following directory structure (I build locally from https://github.com/mesonbuild/meson-python/tree/main/tests/packages/link-against-local-lib)
Then
import pypkg1.example as exampleruns without problem.rgommers commentedon Jan 17, 2023
It fixes the issue, but that's way too much of a restriction. Your average Python package will have multiple directories, and it can have extensions in each one of them. The RUNPATH solution for non-Windows platforms works no matter where you put your extension modules. For Windows we can achieve the same probably, but it needs some preloading solution and it's going to be uglier than using RUNPATH. Or maybe a shim dll per shared library that you want to ship:
metab0t commentedon Jan 17, 2023
I understand your concerns. We need to make DLLs in
.pkgname.mesonpy.libsloadable for all.pydfiles in package files.Maybe we can inject a one-line call to os.add_dll_directory to the top-level
__init__.pyof package?FFY00 commentedon Jan 18, 2023
The issue is that AFAIK there's no way to do this without custom code on Windows.
To install the libraries inside the module, I think you can pass
install: falseto them, and then use them in https://mesonbuild.com/Python-module.html#install_sources.We could consider a solution similar than what we are doing for editable installs. But IMO this should be something provided by either Python, or the packaging ecosystem already.
rgommers commentedon Jan 18, 2023
I would not expect anyone in Python packaging land to care, except scikit-build which will have the same type of user profile and demand for non-Python shared library support. So I think if we coordinate with them, that should suffice.
I think that will work in principle, for Python >= 3.8 (which is enough for this feature). See the notes in https://github.com/MacPython/scipy-wheels/blob/054a522e893f489cd5ccb206f620c51395b09649/_distributor_init.py#L28-L59.
However, modifying a user's
__init__.pyis too magical. What if we just documented that users with shared libraries need to add this to their main__init__.pyfile, before importing any extension modules:That should be simple enough to not get wrong, and this only affects a small amount of users so I'd be happy enough with a doc-only solution here.
It only leaves out the case of no Python package but a standalone Python extension module which depends on a DLL, but in that case the user can do what @FFY00 suggests above and install both files to the same directory. That can be documented as well.
metab0t commentedon Jan 18, 2023
The doc-only solution is OK, and meson-python should output warnings about this on Windows if DLLs are packaged.
rgommers commentedon Jan 27, 2023
One more note on this: on Python 3.8 installed with Conda there may still be problems with
os.add_dll_directorythat have to be worked around withCONDA_DLL_SEARCH_MODIFICATION_ENABLE=1(see conda/conda#10897). This may be why SciPy is still preloading withctypes.WinDLL.rgommers commentedon Nov 5, 2023
One alternative solution I just thought of is to ad a setting to allow running
delvewheelon the wheel produced bymeson-python, after meson-python is done with what it currently does, but before returning control to the build frontend (so inside thebuild_wheelhook). If it's opt-in and we make the package author responsible for addingdelvewheelto their build dependencies on Windows (or we use apatchelf-like dependency insertion rule), that seems like a quite nice capability to have, and easier to implement than reinventing what delvewheel does or preloading withctypes.I believe this may work for most "build on Windows from source with a shared library" needs. The exception being if you are trying to ship a shared library in a wheel for consumption outside of that package (because the symbol mangling applied by
delvewheelwill make that impossible).So a complete solution may be:
os.add_dll_directory(and then they have to add the.mesonpy.libssnippet themselves)delvewheelusagergommers commentedon May 7, 2025
This is now supported in the 0.18.0 release, after gh-551. There is detailed documentation of how to use shared libraries on Windows and how things are working under the hood at https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html.
I'll close this issue - please try the new feature and let us know if anything is still missing or unclear!