CentOS 8 uses a version of g++/libstdc++ that’s more recent than the one available on Photon 3.0 (both distros are based on glibc 2.28). A build of a c++ python extension on CentOS 8 could thus require symbols that are not available on Photon 3.0, this is forbidden by PEP600.
This doesn’t mean CentOS 8 can’t be used to build manylinux_2_28 (or less) wheels. The risk of picking those symbols justifies not to use it as a manylinux_2_28 base image. Doing so would imply that building on that image will always build a wheel that works on any glibc 2.28+ system and that’s just not true.
2 Likes
PhotonOS 3.0 from VMWare is IMO not a “mainstream” linux distribution, just like Microsoft’s CBL-Mariner is not a “mainstream” linux distribution. I use the word “mainstream” as this is included in the PEP600 language. I would argue that the more mainstream distros like Debian, Ubuntu, OpenSUSE, Fedora, and RHEL follow the software ecosystem lifecycles and generally match gcc/glibc for C/C++ ABI deployments. Outliers used for cloud/embedded/edge are not going to fit the ABI model. My suggestion would be to base manylinux_2_28 on CentOS Stream 8, and handle PhotonOS 3.0 as an outlier where on a package-by-pakcage basis things can be resolved by avoiding specific C++ features as required.
2 Likes
Agreed that (in my words, not claiming you would phrase it the same) it’s not worth it to reach for theoretical purity here, but to provide a workable solution for the ecosystem that solves the largest number of problems for the largest number of users (but not necessarily all problems for all users).
With the death of CentOS (notwithstanding Stream, which does something different), there needs to be a choice of some foundation to build on. That might be UBI, AlmaLinux or RockyLinux (all claiming exact binary compatibility with CentOS), but compatibility with fringe distros should be a distant concern in this decision IMO.
Outliers used for cloud/embedded/edge
I’m not entirely sure we can say all of those are outliers given Amazon Linux popularity (500M+ pulls from docker hub).
PhotonOS 3.0 from VMWare is IMO not a “mainstream” linux distribution
Yet OpenSUSE is mentioned as “mainstream” but, according to my highly biased source of information which is Docker Hub, PhotonOS is roughly as popular as OpenSUSE with 10M+ pulls (no regards to the version).
Anyway, my current opinion is that we’re in the gray area of such a wording and that we should find a solution to go forward with manylinux_2_28 and allow package maintainers to choose which side of the gray area they want to be in (if necessary).
@h-vetinari, thanks for raising AlmaLinux & RockyLinux as alternatives to UBI.
AlmaLinux seems to be a great option for manylinux_2_28. UBI does not provide gcc-devtoolset-* which is I think the biggest issue for UBI (@tiran, maybe there’s something that can be done ?)
Finally, just for information, glibc 2.28+ compatible systems (not accounting for pip compatibility) accounts for roughly 45% of manylinux wheel downloads these days.
Also note that Amazon Linux is moving to be based on Fedora, instead of its own thing, which will make it more compatible with the rest of the EL8 ecosystem, not less 
Since UBI is intended for production of distributable images, it’s not surprising to me that the UBI repositories don’t make the compilers/toolchains available.
Amazon is a multi-billion USD company. Please don’t expect that unpaid volunteer implement workarounds so that paying AWS customers have an easier job. Matthew Wilson might be able to help here.
I’m not familiar with gcc-devtoolset. Do you mean Developer Toolset from Software Collections? SCL is a RHEL 7 thing. In RHEL 8 and 9 the SCL packages are in AppStream and build dependencies are in CRB (code ready builder).
UBI has GCC in the UBI appstream repository:
$ podman run --rm registry.access.redhat.com/ubi8/ubi:8.5 dnf list gcc
Available Packages
gcc.x86_64 8.5.0-4.el8_5 ubi-8-appstream
If any developer packages are missing from UBI then please let me know. I can forward request to the right team and they might be willing to include missing packages.
1 Like
I’m one of the unpaid volunteer here. There’s nothing to workaround and nothing broken here. It’s just an example of a mainstream linux distribution mostly used in cloud based environment. I think basing the fact that it’s “mainstream” or not depending if it’s backed by a multi-million USD company rather than how much users in the python user ecosystem would be affected by a breakage would not be the right way to approach the definition of “mainstream”. Probably just a misunderstanding. AWS did break with PEP600 at some point and to my knowledge, AWS did the work to conform to .
I’m not familiar with gcc-devtoolset. Do you mean Developer Toolset from Software Collections? SCL is a RHEL 7 thing. In RHEL 8 and 9 the SCL packages are in AppStream and build dependencies are in CRB (code ready builder).
@tiran, typo on my part, I meant gcc-toolset-* which are the AppStream names of the former SCL you mention for RHEL 7.
With ubi8:
manylinux % docker run -it --rm redhat/ubi8 dnf -y install gcc-toolset-11-gcc-c++
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered with an entitlement server. You can use subscription-manager to register.
Red Hat Universal Base Image 8 (RPMs) - BaseOS 239 kB/s | 796 kB 00:03
Red Hat Universal Base Image 8 (RPMs) - AppStream 745 kB/s | 2.6 MB 00:03
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 43 kB/s | 16 kB 00:00
No match for argument: gcc-toolset-11-gcc-c++
Error: Unable to find a match: gcc-toolset-11-gcc-c++
With almalinux:8:
manylinux % docker run -it --rm almalinux:8 dnf -y install gcc-toolset-11-gcc-c++
AlmaLinux 8 - BaseOS 977 kB/s | 5.9 MB 00:06
AlmaLinux 8 - AppStream 953 kB/s | 9.1 MB 00:09
AlmaLinux 8 - Extras 17 kB/s | 12 kB 00:00
Dependencies resolved.
=================================================================================================================================================================================================
Package Architecture Version Repository Size
=================================================================================================================================================================================================
Installing:
gcc-toolset-11-gcc-c++ x86_64 11.2.1-1.2.el8_5 appstream 13 M
Installing dependencies:
checkpolicy x86_64 2.9-1.el8 baseos 347 k
diffutils x86_64 3.6-6.el8 baseos 361 k
environment-modules x86_64 4.5.2-1.el8 baseos 421 k
gcc-toolset-11-binutils x86_64 2.36.1-1.el8_5.1 appstream 6.5 M
gcc-toolset-11-gcc x86_64 11.2.1-1.2.el8_5 appstream 33 M
gcc-toolset-11-libstdc++-devel x86_64 11.2.1-1.2.el8_5 appstream 3.2 M
gcc-toolset-11-runtime x86_64 11.0-1.el8 appstream 24 k
glibc-devel x86_64 2.28-164.el8 baseos 1.0 M
glibc-headers x86_64 2.28-164.el8 baseos 479 k
groff-base x86_64 1.22.3-18.el8 baseos 1.0 M
kernel-headers x86_64 4.18.0-348.12.2.el8_5 baseos 8.3 M
libgomp x86_64 8.5.0-4.el8_5.alma baseos 205 k
libmpc x86_64 1.1.0-9.1.el8 appstream 60 k
libpipeline x86_64 1.5.0-2.el8 baseos 54 k
libpkgconf x86_64 1.4.2-1.el8 baseos 35 k
libselinux-utils x86_64 2.9-5.el8 baseos 242 k
libxcrypt-devel x86_64 4.1.1-6.el8 baseos 24 k
make x86_64 1:4.2.1-10.el8 baseos 498 k
man-db x86_64 2.7.6.1-18.el8 baseos 886 k
pkgconf x86_64 1.4.2-1.el8 baseos 38 k
pkgconf-m4 noarch 1.4.2-1.el8 baseos 17 k
pkgconf-pkg-config x86_64 1.4.2-1.el8 baseos 15 k
policycoreutils x86_64 2.9-16.el8 baseos 372 k
policycoreutils-python-utils noarch 2.9-16.el8 baseos 251 k
procps-ng x86_64 3.3.15-6.el8 baseos 328 k
python3-audit x86_64 3.0-0.17.20191104git1c2f876.el8 baseos 86 k
python3-libselinux x86_64 2.9-5.el8 baseos 282 k
python3-libsemanage x86_64 2.9-6.el8 baseos 126 k
python3-policycoreutils noarch 2.9-16.el8 baseos 2.2 M
python3-setools x86_64 4.3.0-2.el8 baseos 625 k
scl-utils x86_64 1:2.0.2-14.el8 appstream 46 k
tcl x86_64 1:8.6.8-2.el8 baseos 1.1 M
Transaction Summary
=================================================================================================================================================================================================
Install 33 Packages
Total download size: 75 M
Installed size: 191 M
Downloading Packages:
AlmaLinux 8 - BaseOS 195% [==============================================================================================================(1/33): diffutils-3.6-6.el8.x86_64.rpm 524 kB/s | 361 kB 00:00
(2/33): glibc-devel-2.28-164.el8.x86_64.rpm 778 kB/s | 1.0 MB 00:01
(3/33): environment-modules-4.5.2-1.el8.x86_64.rpm 114 kB/s | 421 kB 00:03
(4/33): glibc-headers-2.28-164.el8.x86_64.rpm 146 kB/s | 479 kB 00:03
(5/33): checkpolicy-2.9-1.el8.x86_64.rpm 53 kB/s | 347 kB 00:06
(6/33): libgomp-8.5.0-4.el8_5.alma.x86_64.rpm 208 kB/s | 205 kB 00:00
(7/33): libpipeline-1.5.0-2.el8.x86_64.rpm 307 kB/s | 54 kB 00:00
(8/33): libpkgconf-1.4.2-1.el8.x86_64.rpm 275 kB/s | 35 kB 00:00
(9/33): libselinux-utils-2.9-5.el8.x86_64.rpm 231 kB/s | 242 kB 00:01
(10/33): libxcrypt-devel-4.1.1-6.el8.x86_64.rpm 373 kB/s | 24 kB 00:00
(11/33): groff-base-1.22.3-18.el8.x86_64.rpm 135 kB/s | 1.0 MB 00:07
(12/33): make-4.2.1-10.el8.x86_64.rpm 197 kB/s | 498 kB 00:02
(13/33): pkgconf-1.4.2-1.el8.x86_64.rpm 373 kB/s | 38 kB 00:00
(14/33): pkgconf-m4-1.4.2-1.el8.noarch.rpm 57 kB/s | 17 kB 00:00
(15/33): pkgconf-pkg-config-1.4.2-1.el8.x86_64.rpm 112 kB/s | 15 kB 00:00
(16/33): man-db-2.7.6.1-18.el8.x86_64.rpm 340 kB/s | 886 kB 00:02
(17/33): policycoreutils-2.9-16.el8.x86_64.rpm 175 kB/s | 372 kB 00:02
(18/33): policycoreutils-python-utils-2.9-16.el8.noarch.rpm 90 kB/s | 251 kB 00:02
(19/33): python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64.rpm 162 kB/s | 86 kB 00:00
(20/33): python3-libselinux-2.9-5.el8.x86_64.rpm 292 kB/s | 282 kB 00:00
(21/33): python3-libsemanage-2.9-6.el8.x86_64.rpm 493 kB/s | 126 kB 00:00
(22/33): procps-ng-3.3.15-6.el8.x86_64.rpm 72 kB/s | 328 kB 00:04
(23/33): python3-setools-4.3.0-2.el8.x86_64.rpm 346 kB/s | 625 kB 00:01
(24/33): tcl-8.6.8-2.el8.x86_64.rpm 366 kB/s | 1.1 MB 00:03
(25/33): python3-policycoreutils-2.9-16.el8.noarch.rpm 372 kB/s | 2.2 MB 00:06
(26/33): kernel-headers-4.18.0-348.12.2.el8_5.x86_64.rpm 303 kB/s | 8.3 MB 00:27
(27/33): gcc-toolset-11-binutils-2.36.1-1.el8_5.1.x86_64.rpm 446 kB/s | 6.5 MB 00:14
(28/33): gcc-toolset-11-libstdc++-devel-11.2.1-1.2.el8_5.x86_64.rpm 335 kB/s | 3.2 MB 00:09
(29/33): gcc-toolset-11-runtime-11.0-1.el8.x86_64.rpm 125 kB/s | 24 kB 00:00
(30/33): libmpc-1.1.0-9.1.el8.x86_64.rpm 361 kB/s | 60 kB 00:00
(31/33): scl-utils-2.0.2-14.el8.x86_64.rpm 333 kB/s | 46 kB 00:00
(32/33): gcc-toolset-11-gcc-c++-11.2.1-1.2.el8_5.x86_64.rpm 515 kB/s | 13 MB 00:25
(33/33): gcc-toolset-11-gcc-11.2.1-1.2.el8_5.x86_64.rpm 727 kB/s | 33 MB 00:46
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.0 MB/s | 75 MB 01:13
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : python3-libselinux-2.9-5.el8.x86_64 1/33
Installing : libmpc-1.1.0-9.1.el8.x86_64 2/33
Installing : python3-libsemanage-2.9-6.el8.x86_64 3/33
Installing : python3-setools-4.3.0-2.el8.x86_64 4/33
Installing : tcl-1:8.6.8-2.el8.x86_64 5/33
Running scriptlet: tcl-1:8.6.8-2.el8.x86_64 5/33
Installing : python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64 6/33
Installing : procps-ng-3.3.15-6.el8.x86_64 7/33
Installing : pkgconf-m4-1.4.2-1.el8.noarch 8/33
Installing : make-1:4.2.1-10.el8.x86_64 9/33
Running scriptlet: make-1:4.2.1-10.el8.x86_64 9/33
Installing : libselinux-utils-2.9-5.el8.x86_64 10/33
Installing : libpkgconf-1.4.2-1.el8.x86_64 11/33
Installing : pkgconf-1.4.2-1.el8.x86_64 12/33
Installing : pkgconf-pkg-config-1.4.2-1.el8.x86_64 13/33
Installing : libpipeline-1.5.0-2.el8.x86_64 14/33
Running scriptlet: libpipeline-1.5.0-2.el8.x86_64 14/33
Installing : libgomp-8.5.0-4.el8_5.alma.x86_64 15/33
Running scriptlet: libgomp-8.5.0-4.el8_5.alma.x86_64 15/33
Installing : kernel-headers-4.18.0-348.12.2.el8_5.x86_64 16/33
Running scriptlet: glibc-headers-2.28-164.el8.x86_64 17/33
Installing : glibc-headers-2.28-164.el8.x86_64 17/33
Installing : libxcrypt-devel-4.1.1-6.el8.x86_64 18/33
Installing : glibc-devel-2.28-164.el8.x86_64 19/33
Running scriptlet: glibc-devel-2.28-164.el8.x86_64 19/33
Installing : groff-base-1.22.3-18.el8.x86_64 20/33
Running scriptlet: man-db-2.7.6.1-18.el8.x86_64 21/33
Installing : man-db-2.7.6.1-18.el8.x86_64 21/33
Running scriptlet: man-db-2.7.6.1-18.el8.x86_64 21/33
Installing : environment-modules-4.5.2-1.el8.x86_64 22/33
Running scriptlet: environment-modules-4.5.2-1.el8.x86_64 22/33
Installing : scl-utils-1:2.0.2-14.el8.x86_64 23/33
Installing : diffutils-3.6-6.el8.x86_64 24/33
Running scriptlet: diffutils-3.6-6.el8.x86_64 24/33
Installing : policycoreutils-2.9-16.el8.x86_64 25/33
Running scriptlet: policycoreutils-2.9-16.el8.x86_64 25/33
Installing : checkpolicy-2.9-1.el8.x86_64 26/33
Installing : python3-policycoreutils-2.9-16.el8.noarch 27/33
Installing : policycoreutils-python-utils-2.9-16.el8.noarch 28/33
Installing : gcc-toolset-11-runtime-11.0-1.el8.x86_64 29/33
Running scriptlet: gcc-toolset-11-runtime-11.0-1.el8.x86_64 29/33
ValueError: SELinux policy is not managed or store cannot be accessed.
Installing : gcc-toolset-11-binutils-2.36.1-1.el8_5.1.x86_64 30/33
Running scriptlet: gcc-toolset-11-binutils-2.36.1-1.el8_5.1.x86_64 30/33
Installing : gcc-toolset-11-libstdc++-devel-11.2.1-1.2.el8_5.x86_64 31/33
Installing : gcc-toolset-11-gcc-11.2.1-1.2.el8_5.x86_64 32/33
Installing : gcc-toolset-11-gcc-c++-11.2.1-1.2.el8_5.x86_64 33/33
Running scriptlet: gcc-toolset-11-gcc-c++-11.2.1-1.2.el8_5.x86_64 33/33
Running scriptlet: man-db-2.7.6.1-18.el8.x86_64 33/33
Verifying : checkpolicy-2.9-1.el8.x86_64 1/33
Verifying : diffutils-3.6-6.el8.x86_64 2/33
Verifying : environment-modules-4.5.2-1.el8.x86_64 3/33
Verifying : glibc-devel-2.28-164.el8.x86_64 4/33
Verifying : glibc-headers-2.28-164.el8.x86_64 5/33
Verifying : groff-base-1.22.3-18.el8.x86_64 6/33
Verifying : kernel-headers-4.18.0-348.12.2.el8_5.x86_64 7/33
Verifying : libgomp-8.5.0-4.el8_5.alma.x86_64 8/33
Verifying : libpipeline-1.5.0-2.el8.x86_64 9/33
Verifying : libpkgconf-1.4.2-1.el8.x86_64 10/33
Verifying : libselinux-utils-2.9-5.el8.x86_64 11/33
Verifying : libxcrypt-devel-4.1.1-6.el8.x86_64 12/33
Verifying : make-1:4.2.1-10.el8.x86_64 13/33
Verifying : man-db-2.7.6.1-18.el8.x86_64 14/33
Verifying : pkgconf-1.4.2-1.el8.x86_64 15/33
Verifying : pkgconf-m4-1.4.2-1.el8.noarch 16/33
Verifying : pkgconf-pkg-config-1.4.2-1.el8.x86_64 17/33
Verifying : policycoreutils-2.9-16.el8.x86_64 18/33
Verifying : policycoreutils-python-utils-2.9-16.el8.noarch 19/33
Verifying : procps-ng-3.3.15-6.el8.x86_64 20/33
Verifying : python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64 21/33
Verifying : python3-libselinux-2.9-5.el8.x86_64 22/33
Verifying : python3-libsemanage-2.9-6.el8.x86_64 23/33
Verifying : python3-policycoreutils-2.9-16.el8.noarch 24/33
Verifying : python3-setools-4.3.0-2.el8.x86_64 25/33
Verifying : tcl-1:8.6.8-2.el8.x86_64 26/33
Verifying : gcc-toolset-11-binutils-2.36.1-1.el8_5.1.x86_64 27/33
Verifying : gcc-toolset-11-gcc-11.2.1-1.2.el8_5.x86_64 28/33
Verifying : gcc-toolset-11-gcc-c++-11.2.1-1.2.el8_5.x86_64 29/33
Verifying : gcc-toolset-11-libstdc++-devel-11.2.1-1.2.el8_5.x86_64 30/33
Verifying : gcc-toolset-11-runtime-11.0-1.el8.x86_64 31/33
Verifying : libmpc-1.1.0-9.1.el8.x86_64 32/33
Verifying : scl-utils-1:2.0.2-14.el8.x86_64 33/33
Installed:
checkpolicy-2.9-1.el8.x86_64 diffutils-3.6-6.el8.x86_64 environment-modules-4.5.2-1.el8.x86_64
gcc-toolset-11-binutils-2.36.1-1.el8_5.1.x86_64 gcc-toolset-11-gcc-11.2.1-1.2.el8_5.x86_64 gcc-toolset-11-gcc-c++-11.2.1-1.2.el8_5.x86_64
gcc-toolset-11-libstdc++-devel-11.2.1-1.2.el8_5.x86_64 gcc-toolset-11-runtime-11.0-1.el8.x86_64 glibc-devel-2.28-164.el8.x86_64
glibc-headers-2.28-164.el8.x86_64 groff-base-1.22.3-18.el8.x86_64 kernel-headers-4.18.0-348.12.2.el8_5.x86_64
libgomp-8.5.0-4.el8_5.alma.x86_64 libmpc-1.1.0-9.1.el8.x86_64 libpipeline-1.5.0-2.el8.x86_64
libpkgconf-1.4.2-1.el8.x86_64 libselinux-utils-2.9-5.el8.x86_64 libxcrypt-devel-4.1.1-6.el8.x86_64
make-1:4.2.1-10.el8.x86_64 man-db-2.7.6.1-18.el8.x86_64 pkgconf-1.4.2-1.el8.x86_64
pkgconf-m4-1.4.2-1.el8.noarch pkgconf-pkg-config-1.4.2-1.el8.x86_64 policycoreutils-2.9-16.el8.x86_64
policycoreutils-python-utils-2.9-16.el8.noarch procps-ng-3.3.15-6.el8.x86_64 python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64
python3-libselinux-2.9-5.el8.x86_64 python3-libsemanage-2.9-6.el8.x86_64 python3-policycoreutils-2.9-16.el8.noarch
python3-setools-4.3.0-2.el8.x86_64 scl-utils-1:2.0.2-14.el8.x86_64 tcl-1:8.6.8-2.el8.x86_64
Complete!
@mayeut It seems almalinux:8 has binaries for x86_64, aarch64, and ppc64le but no i686 and no s390x. Do you know if they have plans for other architectures?
They do have plan for s390x according to EL8 Distro Comparison | AlmaLinux Wiki but I could not find a timeline.
ppc64le docker image should land soon https://github.com/AlmaLinux/docker-images/issues/47 (in beta as I understand)
No plans for i686 as far as I see but these days, I think that amongst major distributions, only debian is still offering an i686 OS. This can probably be dropped for a manylinux_2_28 image. If really required, I think we could still use the x86_64 base image (dnf install gcc-toolset-11-libstdc++-devel.i686 works) but this would definitely require more work.
@mattip, do you have projects with a need for both i686 and manylinux_2_28 ? I see numpy dropped support for i686 (except windows python<3.10) so I’d expect the whole ecosystem that depends on it to probably do the same.
NumPy dropped support for i686, so my interest was less a hard requirement and more “keep our options open”. In any case almalinux seems like the most likely candidate for manylinux2_28 since it has good gcc support and seems to have some serious backing
I asked about PPC on both the AlmaLinux’ & Rockylinux’ discourse. In both cases were they quick to answer, and simply asking might be a better/faster alternative to trawling wikis etc.
As one more or less irrelevant metric, the discourse for RL seems to have a larger community, or at least higher message volume. OTOH, their PPC support has only been promised, but nothing has been announced yet.
Oh, and my 2cents about i686 is to please let it die. Dropping 32bit support was already discussed for manylinux2010(!), it’s time to ~move on~ stop wasting scarce resources on this IMO.
1 Like
(full disclosure: I am now also a Red Hat employee, making at least five who have participated in this topic)
I’ve just read through the thread again, and I don’t see any particular reason why UBI8 couldn’t be used as the base image to produce the ‘wheel builder’ image(s) for manylinux2_28. Certainly the image as distributed doesn’t contain all of the necessary tools, but it appears that they are available. I think it’s fair to say that Red Hat would be happy to see the Python community choose UBI8 as the base image, and we can certainly help build PoCs or examples if that would be useful.
What would be the most practical way to proceed here? The report linked above indicates that libnsl.so.1 may be an issue, but we can investigate that and find out whether it is actually available.
1 Like
I opened an issue on https://github.com/pypa/manylinux/issues/1282 to make manylinux2_28 a reality.
1 Like