Minimum supported Linux kernel and glibc versions

Core Development
1

Does CPython have a minimum supported Linux kernel version and/or a minimum supported glibc version? PEP 11 gives a Windows version support policy, but doesn’t say anything about Linux versions.

I have a particular reason for asking this that may not require y’all to decide on a general policy right now. (Though I do think a general policy, or the documented lack of one, would be a good idea.)

I’m working on a PR to use Linux statx(2) to implement os.stat to provide st_birthtime and new Linux-specific fields. My PR would be simpler if I could assume the statx system call exists at runtime and that libc provides a syscall wrapper function at compile time and run time. (I wrote about this in the issue, but so far only the author of an older PR responded.)

The statx system call was introduced in Linux 4.11, released April 2017. The oldest currently-supported upstream LTS branch is 5.4, released November 2019. If I am reading the RHEL life cycle and RHEL release dates correctly, the oldest supported release is RHEL8, based on kernel 4.18. It is likely that all mainstream Linux distributions ship kernels supporting statx, but also likely that some embedded environments are stuck on older kernels.

The code to fall back to the regular stat system call is not complex, but it is difficult to test. (Possible approaches include somehow poking the function-static variable that triggers the fallback, using an LD_PRELOAD hook to provide a new statx wrapper function that fails with ENOSYS, using a seccomp policy to fall the syscall with ENOSYS, or specifically testing on an older kernel or older glibc.)

glibc introduced a wrapper function for statx in version 2.28, released August 2018.

If I can’t depend on the wrapper being available at build time, I can provide my own definition for struct statx and use syscall(2) to make the call, falling back to stat on ENOSYS. I already have to provide a few constant definitions because of how statx is extensible, but this would add more. (I presume simply building a binary without statx support isn’t an option because explaining why some builds don’t have it would be difficult.)

If I can’t depend on the wrapper being available at run time, I can weakly link and fall back to stat if it isn’t available (or on ENOSYS).

My PR currently assumes the wrapper function is available at build time, but falls back to stat if the wrapper isn’t available at runtime or if it fails with ENOSYS. I don’t plan to implement tests for the fallback path unless someone else has a simpler idea of how to do that.

2 Likes
2

There are things out there that do not have a kernel that is modern. Within PEP-11 Linux that is probably only the insecure long tail of Android hardware. PEP 738 – Adding Android as a supported platform | peps.python.org

The configure.ac based HAVE_LINUX_STATX check is worth keeping. But it seems fair to assume at this point that it is unlikely anyone will be configuring against a modern libc and deploying to such an ancient kernel platform. So I think you can simplify and not worry about “libc has statx, but we’re running on a kernel that lacks the feature” situation and hope we don’t find anything otherwise. Android builds might just want to universally undef that flag.

3 Likes
3

FWIW, conda-forge still compiles against a CentOS 7 baseline, which corresponds to manylinux2014 == manylinux_2_17, which is also still in widespread use on the PyPI side:

image
image937×608 52.5 KB

In the case of manylinux, this is only for packages building on top of python (and with the exception of conda-forge, I don’t know how the python versions that people tend to install are built), so not directly impacted by how cpython handles this.

I’m the first to admit that download numbers have obvious issues (skew due to cloud providers, CI, caching), but still quite a significant amount of projects do not want to (or at least haven’t needed to) move to the manylinux_2_28 generation based on Alma8 yet.

Note that I’m generally advocating for keeping infrastructure baselines at least somewhat current, because I think that at some point the extreme end of the long tail of support does not balance out the effort necessary to support it. Still, it’s not exactly a popular task because people tend to think you’re taking something away from them.

So I provide this just as a datapoint, not as and “you need to support ancient Linux” admonition. If the reasons for moving on are strong enough, the case for moving the baseline can and should be made. But all other things being equal, if you can keep a working fallback that would also be cool :slight_smile:

FWIW, conda-forge only moved to 2.17 as a baseline a year ago (long story), and I don’t expect a move to 2.28 soon. That is, unless core packages like python start requiring 2.28, which would certainly provide a strong argument to reconsider this.

PS. This falls into a similar territory as the discussion for macOS, where there’s also no explicit lower bound mentioned in PEP 11, much less a policy how that lower bound would move up over time.

3 Likes
4

It feels reasonable to me to say that people building wheels on Python 3.15 in 2026 should raise their lowest common denominator standard to something less than 10 years old.

The important part of that chart is the now rapid decline of manylinux_2_17 and corresponding rise of 2_28. Python 3.15 won’t be released for another year at which point 2_28 should be the dominant legacy platform compatible wheel target.

The glibc readiness for Python 3.13 chart on Manylinux Timeline already confirms this is the case. 99.9% of 3.13 use is on glibc 2.28 or later. For Python 3.11 it is 99%. We should not concern ourselves with things requiring a glibc older than that when working on CPython main.

2 Likes
6

It’s hard for packagers to decide to do such a thing, especially without a strongly motivating reason (while the drawback of doing so is obvious: you risk breaking compatibility for some users). If the PyPA were to publish some general guidelines around when/why to bump the manylinux version they’re building against, perhaps there would be more movement towards doing that.

2 Likes
7

That sounds like a very reasonable idea. I think it would have to come from the people involved in managing the manylinux specifications, as I imagine they are the people with a good knowledge of what’s current, and what’s a reasonable “minimum supported” version. But I don’t know who those people are, or how the process is managed. For instance, is there a list of “currently supported” manylinux versions (or is that in fact what you’re asking for here)?

Edit: I realise I wasn’t clear here - I support this being an “official PyPA document” to give it the appropriate level of authority, but I think the decisions as to what the document should say need to be made by the appropriate experts.

1 Like
8

There’s a list of current images that are supported in the sense that they still get updates and fixes, which is essentially tied to the lifecycle of the upstream distribution that they’re based on (once upstream ends support, manylinux stops as well). However, old manylinux images don’t go away even when they aren’t explicitly supported anymore, and package authors can still use those to publish their libraries.

A recent snag in manylinux shows that the long tail is very long indeed; while the issue in this case is on the hardware- rather than the software-side, the overall shape of the problem is the same. Manylinux has no leverage over how long upstream distros provide updates (and so cannot realistically support old glibc versions beyond what distro vendors do), but I think the direction is clearly to stay as compatible as possible for as long as possible, which IMO makes sense for such a foundational piece of infrastructure. In other words, to get them to change direction would need a convincing reason, like a CPython- or even ecosystem-wide policy to point to.

Irrespective of what manylinux does though, I think it would be a good idea to come up with some policy or guideline for how long old OS versions should be supported by CPython, and by extension, the ecosystem. The scientific computing side of python has long had NEP 29 and SPEC 0 for this purpose (though it doesn’t cover requirements for the OS).

The lack of such a policy means piecemeal discussions here and there that get repeated anytime someone runs into an issue of this kind, and it has a tendency to turn acrimonious because people feel treated unfairly if their museum hardware/software ends up on the chopping block. Consequently, there’s a lot of fuzziness around the actual status (untested in CI, no responsible maintainer, but not explicitly unsupported either), and arguing in circles about this is just not a good use of anyone’s time and energy IMO.

I do think it’s a laudable goal to keep support going as long as possible, but we need to recognise that there’s a point where the maintenance effort is not worth the microscopic number of remaining users[1]. People on ancient hardware or software can always continue using less-than-latest python versions and libraries.

  1. in the macOS<10.13 case, we were talking about literally one-in-a-million ↩︎

4 Likes
9

That’s a good point - we’ve digressed from the OP’s original question and started talking about people who are building wheels, but what the OP wants is to know more precisely than PEP 11 states, what is the precise definition of what counts as a supported Linux platform. All the PEP says is x86_64-unknown-linux-gnu, which is (as far as I can see) almost unusably vague.

And that’s a fair question. If PEP 11 were sufficiently precise, I’d say that the answer for wheel builders could just be “Support whatever was tier 1[1] for the CPython versions you support”. But the problem is that PEP 11 isn’t that specific. And that wasn’t what OP was asking anyway.

  1. and maybe tiers 2 and 3 if you choose ↩︎

2 Likes
10

Most platforms in PEP 11 are vague on purpose. We’re volunteers, we have computers, we run whatever we feel like. We don’t want to get specific and pick distros and waste time continually updating the PEP as each of us change minor details of what computers and systems we bother to have available. That PEP 11 tier 1 entry also implies “on a distribution that is still receiving free security updates”. But we’re not going to specify that because it is again over specific and open to wrong interpretation.

So pick the release date for the .0 release of a given CPython and look at what major platforms in tier 1 that were still receiving security updates at that time. Our minimum is not going to be less than that unless explicitly stated otherwise.

Which is why I suggested “don’t worry about something with a libc sporting statx running on a kernel that lacks statx”… (probably a moot point anyways, glibc statx most likely falls back to stat internally when the kernel doesn’t support the statx syscall)

1 Like
11

There’s different groups within the group of old distro users:

  1. Some use an equally old version of Python, most likely provided by the system package manager, with packages provided by either:
    1.1 pip
    1.2 The system package manager
  2. Some may use a heavily statically linked build of the latest Python (e.g. python-build-standalone)
  3. Some may build the latest Python from source on the old target distribution

For packages, user groups 2 and 3 are affected by whatever minimum glibc version the wheels support and group 1.1 may be relevant if a package still supports that version of Python.

For CPython, only group 3 [1] is affected by the minimum glibc version that the latest CPython supports. [2]

So to answer to the OP’s question specifically about CPython, group 3 is the only group we should be sizing up and deciding how much effort it warrants. Packages affect a wider group of people and are typically a lot easier to keep compatible with older distros so may organically be suited to a more conservative glibc target than CPython. [3]

  1. which I hope is a minority ↩︎

  2. There is a nuance here in that building the manylinux images involves building CPython from source on an ancient distribution. AFAICT though that could be worked around by just using statically linked Python builds instead. ↩︎

  3. Personally, I would ignore any official PyPA-perscribed recommendation on glibc targets for my own projects in favour of PyPI statistics and whatever’s easy. ↩︎

2 Likes
12

That vagueness may once have been justified, but at the current size of the Python ecosystem, I think this deserves more clarity nowadays (especially to delineate where “Tier 1” support actually ends).

That doesn’t sounds that complicated to make unambiguous. An example:

  • “Tier 1 support on linux-64 requires at least a distribution that still receives freely available security updates, at the time of the release of a given 3.x python version”
  • “Tier 1 support for macOS ends 7 years after the release of a given macOS version”

Those statements can certainly be fixed, improved, clarified, etc., but it’d be unquestionably a benefit to the community to have this written down somewhere IMO. Because although it would be real work to hash out that policy, it would only have to be done once, while the benefit of removing all that doubt and circuitous discussion would continue to pay dividends indefinitely.

3 Likes
13

That leaves out a whole bunch of groups IMO. You can take an old distro and install current python in a variety of ways, and certainly not all of them are statically linked (it’s also harder than you might think to statically link glibc, so even statically compiled binaries will often still have an assumption about the required glibc baseline).

Not to beat a dead horse, but a big example here is conda-forge, which allows you to install current python (and libraries) on essentially any linux-{x86_64,aarch64,ppc64le} after ~2014, and most definitely that doesn’t just cover people building their own python from source. If you move the glibc baseline, you increment the lower bound at least to 2019[1].

Again, I think it’s fine to do that; what I’m talking about is the blast radius of incrementing the minimum glibc version in CPython, which reaches far wider than you outline. As a concrete example, slow-moving adopters (like institutions[2] in academia, government, supercomputing, etc.) will be cut off from being able to use current python versions[3], and soon after from current library versions. I’ll be the first to agree that those institutions need to plan and fund their infrastructure maintenance better, but on the flipside it’s good to be conscious of the reach of such a decision, because at least then it’s more likely that such a choice gets made for good reason, rather than just because.

  1. because the relevant infrastructure is effectively quantized by RHEL releases, and this is the gap from 7->8. ↩︎

  2. where users get to decide what packages get installed, but cannot modify the OS itself ↩︎

  3. even through places like conda-forge, unless the maintainers there manage to patch in some fallbacks for older glibc downstream of CPython ↩︎

4 Likes
14

In practice, Python usually works on kernels and glibc versions that are supported by the main Linux distributions.

It’s even more complicated, because most enterprise Linux distributions (I work for SUSE, but I am quite that the same situation is with Red Hat and Debian) have those kernels very heavily patched. You cannot go just by version numbers, because it could be very well true that your syscall statx(2) could be patched in by distributors.

3 Likes
15

As a distributor myself[1], my philosophy is “if you ~break~ patch it, you own it”. So I think it’d be 100% reasonable for CPython to rely on the feature sets of vanilla Linux versions, and whoever patched newer syscalls into their Linux or otherwise caused unaccounted divergences then gets to patch some more stuff. :upside_down_face:

  1. not the OS, but everything under the sun in user-space ↩︎

1 Like