Defining a support policy on overlapping installations

Core Development
1

During the downstream adoption of PEP 739, where we added a build-details.json file to the standard library directory, some issues came up regarding it conflicting on multiple installations.

We never defined a policy on what shape of installations we support, or what kind of configurations we support installed in parallel.

To make things simple, for now, I want to ask the following questions.

  1. Do we support installing debug and non-debug builds under the same prefix, libdir, and libdir subdirectory?

  2. Do we support installing regular and free-threaded builds under the same prefix, libdir, and libdir subdirectory? What about future different ABIs?

  3. Do we support installing the same build, but of multiple architectures under the same prefix, libdir?

  4. Are there any other considerations we should take into account?

These questions are in regard to shipping Python to users, not local installations for development.

Additionally, please consider the following:

  • Currently, the PyPI ecosystem is only prepared for same-ABI same-platform packages
    • Debug builds are the outlier here – they able to load extension modules for the same non-debug ABI.
    • Cases 2) and 3) currently fundamentally break the PyPI ecosystem
    • Currently, installations using the official Windows installer, but only because of a technical limitation with Windows installers, which is problematic
  • Even though installations might have the same prefix, libdir, and libdir subdirectory, currently, they all still need to have their own set of headers (this is something we could change, but is the current status-quo)

There’s probably something I missed, but I think this is enough to get the conversation started.

That said, whatever we decide, I acknowledge it would be difficult to enforce the policy, but I still believe it is something we should do. This sets expectations for downstream distributors, and is something we can reference when we receive bug reports from users due to the policy not being followed. With this, I am not saying we simply deny support to users, IMO we would consider situations on a case-by-case basis.

1 Like
2025 Python Steering Council Meeting Summaries
PEP 803: Stable ABI for Free-Threaded Builds
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
Can we consider an uncoditional change of `abi3t.so` to `abi3t-SOABI_PLATFORM.so` for 3.15 (last-minute)
2

Here are my personal opinions.

I would greatly prefer to keep things as simple as possible and define that installations with different platforms and different ABI flags MUST NOT conflict with each other when sharing the same prefix and all other installation directories. This makes things much simpler to define, and makes everything consistent across the board.

EDIT: I realized that part was implied, and should not be part of the wording used for the definition.

My answer above would mean I would answer no to this question. This brings the question of the value of the use-case of sharing site-packages between debug and regular builds, which I think is valid. However, we already have officially supported mechanisms that make this possible in the site module, such as sitecustomize, for eg. that distributors can employ to achieve this — I don’t think it is something that needs to be supported at the installation level.

To clarify, I am proposing that the regular builds use /usr/lib/python3.15, and debug builds use /usr/lib/python3.15d, just like they do for the headers.

Distributions, such as Fedora, which provide the functionality described in the use-case above, could still implement it via a different mechanism.

Per my suggestion above, no. Each ABI gets its own install locations.

This presents the same issues as the multiple ABIs, so my opinion is no.

2 Likes
3

I’m sorry but I need some clarification first.

under the same prefix, libdir, and libdir subdirectory?

Are we talking about requiring a different prefix for every Python installation, or just one of the three being different?

4

Hello,
Let me describe how Fedora operates in terms of installations.

Do we support installing debug and non-debug builds under the same prefix, libdir, and libdir subdirectory?

We do that, which has multiple benefits.

The header files and built extension modules are shipped separately for the main Python and the debug build, but pure Python modules and bytecodes are shared from the main Python directory. python3-debug depends on the main Python package. They are built sequentially from the same source and their dependencies are versioned, meaning we’re sure don’t accidentally ship different main and debug Python.
This enables us seamless debugging issues in other Python packages - we just switch the interpreter to /usr/bin/python3.Xd.

Separating directories brings no visible benefits to us. The size of the debug package would greatly increase and the debugging process would get harder if not impossible at times.

There’s mentioned possible modification of site, but no details - is there any documentation how we could use it to achieve comparable results in our case? I’ll be happy to learn about it.

The additional question I’ve got is: what benefits would separating the directories bring to upstream Python and other use cases? What issue would that solve?

Do we support installing regular and free-threaded builds under the same prefix, libdir, and libdir subdirectory? What about future different ABIs?

The current solution, shipping them in a different libdir subdirectory (with a “t” suffix) works for us. We see no need for sharing the directory between them.

Do we support installing the same build, but of multiple architectures under the same prefix, libdir?

In Fedora, we don’t. We use platlibdir to differentiate between 32 and 64bit architectures. It’s impossible to install two incompatible architecture builds together (e.g. aarch64 and x86_64).

As for Fedora, the current status quo works and we don’t seek changes here.

Given build-details.json, which sparked the discussion (although it’s scoped more broadly than just that), the most preferable way for us would be to include the abiflag in its name, similarly to how _sysconfigdata exists. Then we could just ship it with every one of our builds (currently: optimized, debug, free-threaded and free-threaded debug Pythons) without worrying about the files getting overwritten and not actually corresponding to the given installation.

1 Like
5

FWIW, this relates to the old installer. The new install manager always uses a distinct prefix for each variation (but it also doesn’t do custom headers anymore, so the core problem - assuming Python.h contains a specific set of build-time options - also doesn’t exist).

This part basically comes down to the question of how will users discover/reference these files. The whole point of build-details.json is not merely to exist, but to be referenced by a user (e.g. if they’re saying “compile my extension for this configuration”). So unless we come up with an alternative mechanism, treat the entire path of the file as a user-visible API.[1]

  1. Originally, this “entire path” did not include the filename. Adding variations to the filename will mean users need to learn about the full path including filename. ↩︎

1 Like
6

I generally think that most merged installations are trouble and we shouldn’t support the complexity they add. It isn’t a supportable test matrix. And thus could become unbounded pain for packaging systems that’d now need to make sense of what was installed for which install within the same tree - let alone the installs themselves seeing modules as available when they aren’t actually because they may have only been installed for one version.

Maybe debug vs non-debug is the supportable exception? What test matrix on buildbots and CI are we missing for this scenario to define what we might declare as supported?

We should back up and look at the reasons why anyone thinks they want a merged installation for multiple ABIs.

Disk space? That is not meaningful for the generally larger unique to the build .so/pyd/dll files. Frankly, storage is cheap. Distros are free to use hardlinks for common data if they want to reduce it.

As an alternate aside, should build_details.json gain a build_details-$abitag.json named copy? Feels like it complicates matters.

2 Likes
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
7

I assume it is pinned to the main Python version, right? If the user, for example, explicitly updates the main Python package to a new version — even a patch version — python3-debug will also update, right?

The mechanism are documented in the site module docs. Your two main options would be using a .pth file, or shipping a sitecustomize module.

Having to take into account what would happen in every combination of overlapping installations adds a significant level of complexity, so I think we need to draw a line somewhere.

My proposal makes probably the simplest rule — all installations should be separated by their ABI flags, and architecture. This does happen to separate the site-packages directory for regular and debug builds. We could add an exception, but I personally don’t see enough reason for it given the same user experience provided by Fedora can be achieved using other mechanisms. I am happy to work with you on this, if you want.

Looking at this more objectively, as @gpshead mentioned, if we decide on adding the exception for debug builds, we should be testing it. I believe we aren’t, at the moment, so the CI is also something to take into account.

Could you elaborate on how exactly this would impossibilitate the debug process?

8

I admit that was too hard of a statement, since currently I have no example of such task.

We were discussing what could a separation of the debug installation bring in our case.
We need to share site-packages between both installations and that could be achieved with .pth files, adding the standard Python directories to the sys.path.
Roughly so:

$ cat /usr/lib64/python3.14d/site-packages/_rpm_optimized.pth
/usr/lib64/python3.14/site-packages

$ cat /usr/lib/python3.14d/site-packages/_rpm_optimized.pth
/usr/lib/python3.14/site-packages

$ python3.14d -c 'import sys; print(sys.path)'
['', '/usr/lib64/python314d.zip', '/usr/lib64/python3.14d', '/usr/lib64/python3.14d/lib-dynload', '/usr/lib64/python3.14d/site-packages', '/usr/lib64/python3.14/site-packages', '/usr/lib/python3.14d/site-packages', '/usr/lib/python3.14/site-packages']

In case Python moved to a support policy of separate installations, that could be the way to go (for us).

One thing with this approach: we don’t control all of the site-packages directories on the system. The above works for system libraries managed by rpm where we’re in full control. Other tools, like pip, create their own site-packages at custom locations, outside of our influence. This could be attempted to solve via parsing sys.path for all site-packages dirs loaded there, and adding a desired analog for each entry found. However, this makes the solution much more complicated and fragile. We don’t know what kind of surprises wait for us there.

We still prefer to keep to the existing practice. If Python chooses to redefine the support policy, we’ll aim to follow it as close as possible.

1 Like
9

FWIW Since this thread was linked to me!

I don’t believe it’s accurate to say that we don’t currently have a policy for overlapping installations (although probably it’s true that we don’t have an explicit policy written out). PEP3147 and PEP3149 were explicitly created to support overlapping installations and has been supported for 15 years now.

It would probably be good to actually write down what the actual promises we make for that (and if we want to abandon support for that and make no promises, that is also of course our prerogative as well :slight_smile:).

1 Like
PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)
10

In the form of automated tests.

11

There is also the special case of Debian and Debian derivatives (such as Ubuntu) which allow packages for multiple architectures to be co-installed simultaneously. It’s done by including the multiarch tuple of the architecture in their filename, e.g. cpython-314-x86_64-linux-gnu triplet in zlib.cpython-314-x86_64-linux-gnu.so. Stefano Rivera proposes doing the same for stable ABI extensions: see the issue and the pull request.

1 Like
12

I would like to be able to unblock that PR, and not have to carry a Debian-specific patch to do this for stable ABI extensions. I’m sad that we have PEPs covering all the use-cases, but not clearly covering the stable ABI, and we’re stuck in a limbo where they are functionally incomplete.

It seems obvious to me that the intent of PEP-3149 was to apply to the stable ABI. But performance concerns have held up review of this change for years.

Right now, before Python 3.15 releases, we have an opportunity to make the abi3t stable ABI extensions multiarch co-installable, out of the box. Once the abi3t.so is a supported extension in 3.15, it probably has to be supported ~forever.

2 Likes
13

I think that this needs to be taken to the steering council because it requires revising PEP 803. I don’t see an open issue about this: Issues · python/steering-council · GitHub

14

Reading it again, I was wrong about it not “mentioning” the stable ABI. It absolutely mentions it, and my reading of it is that expresses intention to cover it. But the implementation of PEP 384 came later, and forgot to cover it.

15

Filed: Update PEP 3149 to explicitly include the stable ABI · Issue #355 · python/steering-council · GitHub

16

And Update PEP-803 to use multiarch in abi3t · Issue #356 · python/steering-council · GitHub which is really what you were asking about. My previous issue was a detour.

17

I think the ship may have sailed on supporting bare .abi3t.so given several build backends and build tools already have released versions that use it [1].

That said, I’m for doing things to help Debian package free-threaded Python in a nicer way for their maintainers if the runtime cost is negligible. I haven’t read through all the prior discussions about this though and I may be missing a subtlety.

  1. PyO3, Maturin, sciki-build-core, and cmake to name a few. ↩︎

18

We’d be supporting the bare suffix regardless, just like the bare non-stable ABI suffix. Adding a platform-specific suffix is a reasonable request, but it’s also an entirely reasonable patch - intentionally so. The list of suffixes is fixed at compile time (though you can probably add your own importer at runtime).

The subtleties are basically that if it goes in core, it becomes a global issue when it doesn’t work (e.g. someone decides that renaming a file is an exploit - which it may well be, we can’t prejudge that!), and we don’t necessarily control other changes made by downstream builders. Every choice we put in core basically has to become a restriction on downstreams, so that we can “force” users into the safe path that we control.

Alternatively, a downstream can patch it themselves however they like and then the buck stops with them. Hopefully the build backends are querying sysconfig or build-details.json to find the default suffix they should use, which reduces their burden for dealing with legitimate customisations.

1 Like
19

That is of course, fixable. If Python 3.15 final did not import these files, I think any accidental use would quickly be discovered and fixed. But yeah, there are probably going to be some people running into this.

The consensus from the people I discussed this with at PyCon US 2026 was that we didn’t necessarily need to do this, if there we could get people to agree that this was the right direction to move in. There are performance concerns, that are only improvable by reducing the number of supported suffixes.

Yes, we’ve been carrying this a patch for as long as stable ABI has been widely in use. But Debian has also got a lot of flack in the Python world for carrying intrusive patches like this.

If we agree that this is a supported layout, and the current PEP-3149 + C extension naming doesn't take bitness into account · Issue #67169 · python/cpython · GitHub implementation for .so is reasonable, then it absolutely should apply to the stable ABI too. Not applying it was simply an oversight.

I interpret keeping this as an out-of-tree patch as meaning: We don’t think we made the right decision in adding multiarch tags via PEP-3149, and would not like to repeat this mistake for the stable ABI. We have to support what we already merged, but don’t want to dig the hole deeper. That’s a theoretically defensible position, but I don’t think we got there out of an explicit decision so much as oversight.

I think we have to all declare the same default suffix, otherwise we create incompatible wheels. If this remains as an out-of-tree patch, it has to be non-default. And we have to do mangling in our buildsystems to inject the multiarch tags after the wheel build.

1 Like
20

From… three to two? Or four to three? There aren’t that many suffixes at runtime, and the list of files is already in memory.

From us? Or from people who are even further downstream than Debian?

That’s a reasonable interpretation, and I think I agree with you that it’s not really what we mean. (Though the multiarch tags were added post 3149, and rereading that decade-old discussion it seems likely that some people would say that we should avoid going any deeper down that hole…)

Hard to say what the right answer here is. I think the only reasonable choices are “do the same thing we currently do for others” or “don’t change it now” (IOW, we’re not making a new, better, naming system for abi3t modules, even though we can).

Publicly published wheels from Linux have to go through manylinux processing anyway, right? So there’s already a suitable spot to do the fixup.