venv base path does not resolve symlinks using realpath()

It seems there is a bug in venv (and a similar one in virtualenv) where the "base" path in pyvenv.cfg is set incorrectly. If Python is installed in a non-standard folder, e.g. /usr/local/python-X.Y.Z and then symlinked into /usr/local/bin/python3, the venv package does not work correctly. I believe the source of the trouble is the setting of "home" variable. Specifically, this line:

dirname, exename = os.path.split(os.path.abspath(executable))

The dirname result is used to set "home". If the executable path is /usr/local/bin/python3 (actually a symlink to /usr/local/python-X.Y.Z/bin/python3), the "home" should not be set to /usr/local. Changing the above line (in the ensure_directories()) function to:

dirname, exename = os.path.split(os.path.realpath(executable))

This fixes the problem. I believe this is consistent with what the getpath.py module in Python does.

I noticed with problem when running the most recent Debian OS, version 12. It includes Python 3.11 and therefore /usr/lib/python3.11 exists. With the above bug, the venv tries to use /usr/lib/python3.11 as the sys.path. Importing the struct module fails with the mysterious error:

>>> import struct
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/struct.py", line 13, in <module>
    from _struct import *
ModuleNotFoundError: No module named '_struct'

Linked PRs

I've run into this as well:
Python built from source & installed in /opt/python-3.11, symlink in /usr/local/bin/python3.11.

In my case, it was on ubuntu 22.04, the venv could not be created with a failure to import subprocess because indeed it was trying to use /usr/lib/python3.11 for lib-dynload in sys.path.
Changing home manually or patching the venv module worked.

reproducer in GHA: https://github.com/mayeut/sandbox/actions/runs/7518304095/job/20465487706#step:3:81

pythongh-106045: Fix ``venv`` creation from a python executable symlink

Note that resolving symlinks will break Homebrew, see astral-sh/uv#1640

I noticed with problem when running the most recent Debian OS, version 12. It includes Python 3.11 and therefore /usr/lib/python3.11 exists. With the above bug, the venv tries to use /usr/lib/python3.11 as the sys.path. Importing the struct module fails with the mysterious error:

Do you have instructions how to reproduce this? We'd like to avoid this problem in uv.

From docker run --rm debian:12

$ apt update
$ apt install -y python3-venv
$ which python3
/usr/bin/python3
$ python3 --version
Python 3.11.2
$ ln -s /usr/bin/python3 /usr/local/bin/python3

Then venv/bin/python -c "import struct; import subprocess" works fine for me.

Python built from source & installed in /opt/python-3.11, symlink in /usr/local/bin/python3.11.

Same questions, do you have instructions how to reproduce this (outside the github actions one)? I tried this with a python checkout (4d3ee77):

./configure --prefix=/opt/cpython1 --enable-optimizations
make -s -j32
sudo make install

Then i tried:

mkdir a
cd a
ln -s /opt/cpython1/bin/python3 python3
cd ..
mkdir b
cd b
../a/python3 -m venv venv
venv/bin/python -c "import struct; import subprocess"
cd ..
mkdir c
cd c
virtualenv -p ../a/python3 venv
venv/bin/python -c "import struct; import subprocess"

But this all passed.

@konstin

@konstin,

On ubuntu 22.04, install sudo apt-get install -y python3-distutils, create a symlink to your python3.11 build in /usr/local/bin/python3.11.

I think the fact that the symlink is in /usr/local/bin folder matters. Just creating a symlink elsewhere does not necessarily reveals the issue.

Do you have more information about where the python3.11 came from? I still can't reproduce

FROM ubuntu:22.04
RUN apt update
RUN apt-get install -y python3-distutils
ENV DEBIAN_FRONTEND=noninteractive
RUN apt install -yy git build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
WORKDIR /root
RUN git clone https://github.com/python/cpython/
WORKDIR /root/cpython
RUN git checkout v3.11.8
RUN ./configure --prefix=/opt/cpython1
RUN make -s -j32
RUN make install
WORKDIR /root
RUN /opt/cpython1/bin/python3 --version
RUN ln -s /opt/cpython1/bin/python3 /usr/local/bin/python3
RUN /usr/local/bin/python3 --version
RUN /usr/local/bin/python3 -c "import struct; import subprocess"
RUN /usr/local/bin/python3 -m venv venv1
RUN venv1/bin/python -c "import struct; import subprocess"
RUN venv1/bin/python -m venv venv2
RUN venv2/bin/python -c "import struct; import subprocess"

I could reproduce it using your dockerfile with a slight modification.
The folder /usr/lib/python3.11/lib-dynload should exist for the venv creation to fail.
You can mkdir the folder or also add python3-gdbm to the list of packages to install.

To summarize, the error only happens (editing this to 3.13 to be actionable on main) when you have a custom built python that is linked to exactly /usr/local/bin/python3, but there's also a system python of the same version installed in /usr installed that created /usr/lib/python3.13/lib-dynload?

I've tried to trace the cause and it seems that installed_platbase gets switched to /usr on venv creation:

$ /opt/cpython1/bin/python3 -m sysconfig | grep -i installed_platbase
        installed_platbase = "/opt/cpython1"
$ venv1/bin/python -m sysconfig | grep -i installed_platbase
        installed_platbase = "/usr"

This in turn changes platinclude to the wrong python:

$ /opt/cpython1/bin/python3 -m sysconfig | head -n 14
Platform: "linux-x86_64"
Python version: "3.13"
Current installation scheme: "posix_prefix"

Paths: 
        data = "/opt/cpython1"
        include = "/opt/cpython1/include/python3.13"
        platinclude = "/opt/cpython1/include/python3.13"
        platlib = "/opt/cpython1/lib/python3.13/site-packages"
        platstdlib = "/opt/cpython1/lib/python3.13"
        purelib = "/opt/cpython1/lib/python3.13/site-packages"
        scripts = "/opt/cpython1/bin"
        stdlib = "/opt/cpython1/lib/python3.13"

$ venv1/bin/python3 -m sysconfig | head -n 14
Platform: "linux-x86_64"
Python version: "3.13"
Current installation scheme: "venv"

Paths: 
        data = "/root/venv1"
        include = "/opt/cpython1/include/python3.13"
        platinclude = "/usr/include/python3.13"
        platlib = "/root/venv1/lib/python3.13/site-packages"
        platstdlib = "/root/venv1/lib/python3.13"
        purelib = "/root/venv1/lib/python3.13/site-packages"
        scripts = "/root/venv1/bin"
        stdlib = "/opt/cpython1/lib/python3.13"

I unfortunately don't understand enough about the python sysconfig machinery to understand how/why installed_platbase is set.

Note that for the system (ubuntu apt install python3.11 python3.11-venv) python, all paths are (correctly) /usr based and stay this way:

$ /usr/bin/python3.11 -m sysconfig | head -n 14
Platform: "linux-x86_64"
Python version: "3.11"
Current installation scheme: "posix_local"

Paths: 
        data = "/usr/local"
        include = "/usr/include/python3.11"
        platinclude = "/usr/include/python3.11"
        platlib = "/usr/local/lib/python3.11/dist-packages"
        platstdlib = "/usr/lib/python3.11"
        purelib = "/usr/local/lib/python3.11/dist-packages"
        scripts = "/usr/local/bin"
        stdlib = "/usr/lib/python3.11"
$ venv-sys/bin/python -m sysconfig | head -n 14
Platform: "linux-x86_64"
Python version: "3.11"
Current installation scheme: "venv"

Paths: 
        data = "/root/venv-sys"
        include = "/usr/include/python3.11"
        platinclude = "/usr/include/python3.11"
        platlib = "/root/venv-sys/lib/python3.11/site-packages"
        platstdlib = "/root/venv-sys/lib/python3.11"
        purelib = "/root/venv-sys/lib/python3.11/site-packages"
        scripts = "/root/venv-sys/bin"
        stdlib = "/usr/lib/python3.11"

I haven't looked into this in detail, but might the real issue here be that you're effectively running the interpreter in a broken system-wide environment under /usr/local? In other words, don't just symlink the interpreter. (Disclaimer: I'm not an expert in the sysconfig machinery either.)

The actual command used to launch the interpreter determines what's considered the environment, even if it's a symbolic link into a valid installation. To my knowledge, sysconfig uses the nominal (unresolved) interpreter path to expand the location templates in installation schemes. (not sure about that, should double check)

I haven't looked into this in detail, but might the real issue here be that you're effectively running the interpreter in a broken system-wide environment under /usr/local?

I don't know what a "broken system-wide environment" means. As I said in the description, I think this is a bug in venv since it doesn't match what getpath.c does.

Rather than having venv have complicated logic to figure out what "home" should be, perhaps it should just run something like: python3 -c "import sysconfig; print(sysconfig.get_paths())".

I've worked around the issue by creating wrapper shell scripts for executables, rather than using symlinks, e.g. usr/local/bin/python3 contains:

#!/bin/sh
exec /usr/local/python-3.12.2/bin/python3.12 "$@"

pythongh-106045: Fix ``venv`` creation from a python executable symlink

on POSIX builds with a shared libpython, this should now be mitigated by looking at the libpython directory before resorting to the base interpreter, which should be much more reliable.

It does not seem to be helping in this case (on macOS):

mkdir -p ../temp-test/bin
mkdir -p ../temp-test/lib/python3.14/lib-dynload
touch ../temp-test/lib/python3.14/os.py
./configure --enable-shared --with-pydebug --prefix=$(pwd)/build/install  && make -j install
ln -sf $(pwd)/build/install/bin/python3 ../temp-test/bin/python
../temp-test/bin/python -m venv ../temp-test/venv   
Error: Command '['../temp-test/venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

With #115237 resolving home differently, the venv works with both shared & static builds.

@pelson

It turns out (after a lot of searching), that this is a two-line fix to getpath.py

@pelson, did you mean the following instead ?

diff --git a/Modules/getpath.py b/Modules/getpath.py
index be2210345a..6607aacb80 100644
--- a/Modules/getpath.py
+++ b/Modules/getpath.py
@@ -412,6 +412,9 @@ def search_up(prefix, *landmarks, test=isfile):
                             if isfile(candidate):
                                 base_executable = candidate
                                 break
+                if base_executable and isfile(base_executable):
+                    # Update the executable directory to be based on the resolved base executable
+                    executable_dir = real_executable_dir = dirname(base_executable)
             break
     else:
         # We didn't find a 'home' key in pyvenv.cfg (no break), reset venv_prefix.

I frequently encounter this problem because of how I've set up my multiple local builds of Python (used to test version compatibility for packages, and also to investigate differences between Python versions). Because of the wrong home path, the standard library can't be found at all, with more dramatic consequences:

$ py3.13 -m venv --without-pip test && test/bin/python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007464fa539740 (most recent call first):
  <no Python frame>

This seems to be a regression introduced in 3.11 (my py3.12 works because it sees the standard library of my system Python).

Can you run ldd py3.13? If libpython is linked, please also let me know if the linker is loading it from the correct place.

$ ldd `which py3.13`
	linux-vdso.so.1 (0x00007ffe127f4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007c9ffec28000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007c9ffe200000)
	/lib64/ld-linux-x86-64.so.2 (0x00007c9ffed2b000)

Not there. I just used all default compilation options for these builds.

But it looks the same for the ones where it works.

Right, if you did not build with --enable-shared, it will static link libpython. Are you moving or copying the python binary to a different location? Or rather, how exactly are you installing Python?

Python is installed by using configure to set a prefix and using make altinstall, then manually symlinking to the altinstalled binary from /usr/local/bin. At least, that's what my notes say to do; I'll try to make time this week to redo it from scratch and confirm.

I ran into this issue in python-build-standalone, astral-sh/python-build-standalone#380

Is there still interested in a change to getpath.py to address this? I'd be happy to create a PR for discussion.

@jjhelmus

@jjhelmus sure, let's see what a PR looks like.

Is there still interested in a change to getpath.py to address this? I'd be happy to create a PR for discussion.

It's unclear to me what would be changed, but feel free to open a PR. Just beware that touching getpath.py is always very delicate, and there's a big possibility that whatever you change will slightly change its behavior in ways other use-cases may not be expecting, and break them. This is why I am always so cautious when touching getpath 😅

Would it be possible to add all those untested use cases / branches to the unit test coverage before changing getpath?

I believe a lot of them are, IIRC the getpath tests are pretty extensive. If you change something, most likely some test will fail, and we'll have to carefully consider the behavior change.

I've come back to this issue after some time and put up a proposed fix with test cases in #156483.

Currently, getpath.py searches for landmark files relative to the home value in pyvenv.cfg. When that value refers to an symlink, discovery can fall back to the prefix value set during the build. If the install does not match this location, either from relocating after and install or installing into a different location, then sys.base_prefix, etc are set incorrectly. This is the case that occured n python-build-standalone as the distribution are almost never installed into the configured prefix, /install.

Additionally, if an landmark file exist near the symlink, perhaps from another valid python installation, these can result in the symlinked directory being identified as the base installation.

Both of these can cases be reproduced with Docker containers:

FROM python:3.14-slim

# Relocate Python
RUN cp -a /usr/local /opt/python \
    && rm -rf /usr/local/lib/python3.14

# Create a symlink
RUN mkdir -p /opt/public/bin \
    && ln -s /opt/python/bin/python3 /opt/public/bin/python

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Result:

❯ docker build -f Dockerfile.relocate -t relocate_example .
...
❯ docker run --rm relocate_example
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/opt/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.14'
  sys.path[0] = (not set)
  sys._base_executable = '/opt/python/bin/python3.14'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/venv/bin/python'
  sys.prefix = '/opt/venv'
  sys.exec_prefix = '/opt/venv'
  sys.path = [
    '/usr/local/lib/python314.zip',
    '/usr/local/lib/python3.14',
    '/usr/local/lib/python3.14/lib-dynload',
  ]
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffffa492d4c0 [python] (most recent call first):
  <no Python frame>

And the misleading landmark file:

FROM python:3.14-slim

# Symlink the interpreter
RUN mkdir -p /opt/public/bin \
    && ln -s /usr/local/bin/python3 /opt/public/bin/python

# Create misleading standard-library landmarks near the symlink.
RUN mkdir -p /opt/public/lib/python3.14/lib-dynload \
    && touch /opt/public/lib/python3.14/os.py

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Result:

❯ docker build -f Dockerfile.landmark -t landmark_example .
...
❯ docker run --rm landmark_example
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/opt/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/public/lib/python3.14'
  sys.path[0] = (not set)
  sys._base_executable = '/usr/local/bin/python3.14'
  sys.base_prefix = '/opt/public'
  sys.base_exec_prefix = '/opt/public'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/venv/bin/python'
  sys.prefix = '/opt/venv'
  sys.exec_prefix = '/opt/venv'
  sys.path = [
    '/opt/public/lib/python314.zip',
    '/opt/public/lib/python3.14',
    '/opt/public/lib/python3.14/lib-dynload',
  ]
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffffadbd84c0 [python] (most recent call first):
  <no Python frame>

Both of these layouts work as expected with the C-based path-discovery in Python 3.10:

Relocated Python

FROM python:3.10-slim

# Relocate Python
RUN cp -a /usr/local /opt/python \
    && rm -rf /usr/local/lib/python3.10

# Create a symlink
RUN mkdir -p /opt/public/bin \
    && ln -s /opt/python/bin/python3 /opt/public/bin/python

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

Results:

❯ docker build -f Dockerfile.relocate310 -t relocate_example310 .
...
❯ docker run --rm relocate_example310
/opt/python

Landmark

FROM python:3.10-slim

# Symlink the interpreter
RUN mkdir -p /opt/public/bin \
    && ln -s /usr/local/bin/python3 /opt/public/bin/python

# Create misleading standard-library landmarks near the symlink.
RUN mkdir -p /opt/public/lib/python3.10/lib-dynload \
    && touch /opt/public/lib/python3.10/os.py

# Creating venv
RUN /opt/public/bin/python -m venv --without-pip /opt/venv \
    && cat /opt/venv/pyvenv.cfg

CMD ["/opt/venv/bin/python", "-c", "import sys; print(sys.base_prefix)"]

❯ docker build -f Dockerfile.landmark310 -t landmark_example310 .
...
❯ docker run --rm landmark_example310
/usr/local

Navigation Menu

Repository navigation

venv base path does not resolve symlinks using realpath() #106045

Linked PRs

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

venv base path does not resolve symlinks using realpath() #106045

Description

Linked PRs

Activity

mayeut commented on Jan 14, 2024

cjolowicz commented on Mar 3, 2024

konstin commented on Mar 3, 2024

mayeut commented on Mar 4, 2024

konstin commented on Mar 4, 2024

mayeut commented on Mar 4, 2024

konstin commented on Mar 4, 2024

cjolowicz commented on Mar 4, 2024

nascheme commented on Mar 5, 2024

5 remaining items

mayeut commented on Jan 11, 2025

mayeut commented on Jan 16, 2025

zahlman commented on Mar 28, 2025

FFY00 commented on Mar 28, 2025

zahlman commented on Mar 29, 2025

FFY00 commented on Mar 30, 2025

zahlman commented on Mar 31, 2025

jjhelmus commented on Oct 31, 2025

brettcannon commented on Oct 31, 2025

FFY00 commented on Dec 10, 2025

reneleonhardt commented on Dec 10, 2025

FFY00 commented on Dec 10, 2025

jjhelmus commented on Aug 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions