Defining a support policy on overlapping installations

Core Development
1

During the downstream adoption of PEP 739, where we added a build-details.json file to the standard library directory, some issues came up regarding it conflicting on multiple installations.

We never defined a policy on what shape of installations we support, or what kind of configurations we support installed in parallel.

To make things simple, for now, I want to ask the following questions.

  1. Do we support installing debug and non-debug builds under the same prefix, libdir, and libdir subdirectory?

  2. Do we support installing regular and free-threaded builds under the same prefix, libdir, and libdir subdirectory? What about future different ABIs?

  3. Do we support installing the same build, but of multiple architectures under the same prefix, libdir?

  4. Are there any other considerations we should take into account?

These questions are in regard to shipping Python to users, not local installations for development.

Additionally, please consider the following:

  • Currently, the PyPI ecosystem is only prepared for same-ABI same-platform packages
    • Debug builds are the outlier here – they able to load extension modules for the same non-debug ABI.
    • Cases 2) and 3) currently fundamentally break the PyPI ecosystem
    • Currently, installations using the official Windows installer, but only because of a technical limitation with Windows installers, which is problematic
  • Even though installations might have the same prefix, libdir, and libdir subdirectory, currently, they all still need to have their own set of headers (this is something we could change, but is the current status-quo)

There’s probably something I missed, but I think this is enough to get the conversation started.

That said, whatever we decide, I acknowledge it would be difficult to enforce the policy, but I still believe it is something we should do. This sets expectations for downstream distributors, and is something we can reference when we receive bug reports from users due to the policy not being followed. With this, I am not saying we simply deny support to users, IMO we would consider situations on a case-by-case basis.

1 Like
PEP 803: Stable ABI for Free-Threaded Builds
2025 Python Steering Council Meeting Summaries
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
2

Here are my personal opinions.

I would greatly prefer to keep things as simple as possible and define that installations with different platforms and different ABI flags MUST NOT conflict with each other when sharing the same prefix and all other installation directories. This makes things much simpler to define, and makes everything consistent across the board.

EDIT: I realized that part was implied, and should not be part of the wording used for the definition.

My answer above would mean I would answer no to this question. This brings the question of the value of the use-case of sharing site-packages between debug and regular builds, which I think is valid. However, we already have officially supported mechanisms that make this possible in the site module, such as sitecustomize, for eg. that distributors can employ to achieve this — I don’t think it is something that needs to be supported at the installation level.

To clarify, I am proposing that the regular builds use /usr/lib/python3.15, and debug builds use /usr/lib/python3.15d, just like they do for the headers.

Distributions, such as Fedora, which provide the functionality described in the use-case above, could still implement it via a different mechanism.

Per my suggestion above, no. Each ABI gets its own install locations.

This presents the same issues as the multiple ABIs, so my opinion is no.

2 Likes
3

I’m sorry but I need some clarification first.

under the same prefix, libdir, and libdir subdirectory?

Are we talking about requiring a different prefix for every Python installation, or just one of the three being different?

4

Hello,
Let me describe how Fedora operates in terms of installations.

Do we support installing debug and non-debug builds under the same prefix, libdir, and libdir subdirectory?

We do that, which has multiple benefits.

The header files and built extension modules are shipped separately for the main Python and the debug build, but pure Python modules and bytecodes are shared from the main Python directory. python3-debug depends on the main Python package. They are built sequentially from the same source and their dependencies are versioned, meaning we’re sure don’t accidentally ship different main and debug Python.
This enables us seamless debugging issues in other Python packages - we just switch the interpreter to /usr/bin/python3.Xd.

Separating directories brings no visible benefits to us. The size of the debug package would greatly increase and the debugging process would get harder if not impossible at times.

There’s mentioned possible modification of site, but no details - is there any documentation how we could use it to achieve comparable results in our case? I’ll be happy to learn about it.

The additional question I’ve got is: what benefits would separating the directories bring to upstream Python and other use cases? What issue would that solve?

Do we support installing regular and free-threaded builds under the same prefix, libdir, and libdir subdirectory? What about future different ABIs?

The current solution, shipping them in a different libdir subdirectory (with a “t” suffix) works for us. We see no need for sharing the directory between them.

Do we support installing the same build, but of multiple architectures under the same prefix, libdir?

In Fedora, we don’t. We use platlibdir to differentiate between 32 and 64bit architectures. It’s impossible to install two incompatible architecture builds together (e.g. aarch64 and x86_64).

As for Fedora, the current status quo works and we don’t seek changes here.

Given build-details.json, which sparked the discussion (although it’s scoped more broadly than just that), the most preferable way for us would be to include the abiflag in its name, similarly to how _sysconfigdata exists. Then we could just ship it with every one of our builds (currently: optimized, debug, free-threaded and free-threaded debug Pythons) without worrying about the files getting overwritten and not actually corresponding to the given installation.

1 Like
5

FWIW, this relates to the old installer. The new install manager always uses a distinct prefix for each variation (but it also doesn’t do custom headers anymore, so the core problem - assuming Python.h contains a specific set of build-time options - also doesn’t exist).

This part basically comes down to the question of how will users discover/reference these files. The whole point of build-details.json is not merely to exist, but to be referenced by a user (e.g. if they’re saying “compile my extension for this configuration”). So unless we come up with an alternative mechanism, treat the entire path of the file as a user-visible API.[1]

  1. Originally, this “entire path” did not include the filename. Adding variations to the filename will mean users need to learn about the full path including filename. ↩︎

1 Like
6

I generally think that most merged installations are trouble and we shouldn’t support the complexity they add. It isn’t a supportable test matrix. And thus could become unbounded pain for packaging systems that’d now need to make sense of what was installed for which install within the same tree - let alone the installs themselves seeing modules as available when they aren’t actually because they may have only been installed for one version.

Maybe debug vs non-debug is the supportable exception? What test matrix on buildbots and CI are we missing for this scenario to define what we might declare as supported?

We should back up and look at the reasons why anyone thinks they want a merged installation for multiple ABIs.

Disk space? That is not meaningful for the generally larger unique to the build .so/pyd/dll files. Frankly, storage is cheap. Distros are free to use hardlinks for common data if they want to reduce it.

As an alternate aside, should build_details.json gain a build_details-$abitag.json named copy? Feels like it complicates matters.

2 Likes
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
7

I assume it is pinned to the main Python version, right? If the user, for example, explicitly updates the main Python package to a new version — even a patch version — python3-debug will also update, right?

The mechanism are documented in the site module docs. Your two main options would be using a .pth file, or shipping a sitecustomize module.

Having to take into account what would happen in every combination of overlapping installations adds a significant level of complexity, so I think we need to draw a line somewhere.

My proposal makes probably the simplest rule — all installations should be separated by their ABI flags, and architecture. This does happen to separate the site-packages directory for regular and debug builds. We could add an exception, but I personally don’t see enough reason for it given the same user experience provided by Fedora can be achieved using other mechanisms. I am happy to work with you on this, if you want.

Looking at this more objectively, as @gpshead mentioned, if we decide on adding the exception for debug builds, we should be testing it. I believe we aren’t, at the moment, so the CI is also something to take into account.

Could you elaborate on how exactly this would impossibilitate the debug process?

8

I admit that was too hard of a statement, since currently I have no example of such task.

We were discussing what could a separation of the debug installation bring in our case.
We need to share site-packages between both installations and that could be achieved with .pth files, adding the standard Python directories to the sys.path.
Roughly so:

$ cat /usr/lib64/python3.14d/site-packages/_rpm_optimized.pth
/usr/lib64/python3.14/site-packages

$ cat /usr/lib/python3.14d/site-packages/_rpm_optimized.pth
/usr/lib/python3.14/site-packages

$ python3.14d -c 'import sys; print(sys.path)'
['', '/usr/lib64/python314d.zip', '/usr/lib64/python3.14d', '/usr/lib64/python3.14d/lib-dynload', '/usr/lib64/python3.14d/site-packages', '/usr/lib64/python3.14/site-packages', '/usr/lib/python3.14d/site-packages', '/usr/lib/python3.14/site-packages']

In case Python moved to a support policy of separate installations, that could be the way to go (for us).

One thing with this approach: we don’t control all of the site-packages directories on the system. The above works for system libraries managed by rpm where we’re in full control. Other tools, like pip, create their own site-packages at custom locations, outside of our influence. This could be attempted to solve via parsing sys.path for all site-packages dirs loaded there, and adding a desired analog for each entry found. However, this makes the solution much more complicated and fragile. We don’t know what kind of surprises wait for us there.

We still prefer to keep to the existing practice. If Python chooses to redefine the support policy, we’ll aim to follow it as close as possible.

1 Like
9

FWIW Since this thread was linked to me!

I don’t believe it’s accurate to say that we don’t currently have a policy for overlapping installations (although probably it’s true that we don’t have an explicit policy written out). PEP3147 and PEP3149 were explicitly created to support overlapping installations and has been supported for 15 years now.

It would probably be good to actually write down what the actual promises we make for that (and if we want to abandon support for that and make no promises, that is also of course our prerogative as well :slight_smile:).

1 Like
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
10

In the form of automated tests.

11

There is also the special case of Debian and Debian derivatives (such as Ubuntu) which allow packages for multiple architectures to be co-installed simultaneously. It’s done by including the multiarch tuple of the architecture in their filename, e.g. cpython-314-x86_64-linux-gnu triplet in zlib.cpython-314-x86_64-linux-gnu.so. Stefano Rivera proposes doing the same for stable ABI extensions: see the issue and the pull request.

1 Like