Dropping 32-bit packages

Where would be an appropriate place to discuss dropping 32-bit packages across the python ecosystem? Neither the Redhat, the windows store nor conda-forge support any 32-bit variants, and producing packages doubles CI time for all library packagers. I realize each project could have its own support policy, but a python-wide statement would be more convincing. Specifically, I would like to drop windows 32-bit PyPI wheels from the scientific python stack, is that OK?

3 Likes

My main concern is that would leave Windows ARM64 users out in the cold, as they can run 32-bit binaries (emulated) but not 64-bit. And there’s a surprising amount of demand out there - literally everyone who’s asked me why there’s no 32-bit build on the Store (still only 5-6 people) has wanted it for ARM64.

Obviously native builds are better, and I’m working behind the scenes to get CI support for it (which is blocked on having native Python support :grin: but at least that’s internal from my POV so I can slip them a build) and there’ll need to be some tooling improvements to handle cross-compilation (no ARM64-native compilers), but eventually it’ll all pull together. I’ve been holding off releasing a native CPython build here until there’s a way for the ecosystem to match.

So I think dropping 32-bit will be fine once we’re ready to pick up ARM64 :wink:

1 Like

IIUC, I think by 32-bit, @mattip is referring to x86 in particular. @steve.dower has provided the answer for Windows, and macOS doesn’t support x86 anymore, so I just want to chime in that most distro supporting x86 (and other obscure architectures) have their own CI systems for downstream testing and building.

Ideally, it would be great to warn any embedders who are including them in 32-bit apps, though I know those are hard to find. Chances are they won’t easily be able to just switch their app to 64-bit, and if they’re dependent on a numpy stack it could come as a surprise. (That said, most embedders seem happy enough to be a few versions behind anyway to keep the stability. So likely not going to have any painful blowback if they find out with everyone else.)

Have you checked the download stats for the existing 32-bit wheels?

There’s also the fact that the default Windows download has been 32-bit for a long time, so the number of Windows users with 32-bit Python will be artificially inflated. 3.9 is 64-bit by default, so things are easier there. But for 3.8 and earlier, dropping 32-bit builds of new project releases would impact all of those 32-bit users, even though they might be perfectly able to run 64-bit Python.

3 Likes

That will hopefully change soon:

1 Like

Even so, plenty of people want Python for truly native things, and we want to offer it.

Though it would certainly be easier to tell people to use x64 if they want a Python data stack, and use ARM64 if they want a native Python runtime.

Reviving this topic, as building NumPy/SciPy wheels for 32-bit windows is getting more painful. I see that windows-arm64 can now use x86_64 emulation. Is there a way to check how “popular” 32-bit windows is across the python ecosystem?

1 Like

SciPy has dropped support since 1.9.2, released Oct’22. 1.9.1 was the last release to have 32-bit Windows wheels. SciPy received few complaints, nor am I aware of many complaints when Anaconda dropped 32-bit support. So I think it’s safe to say that it is very niche by now.

2 Likes

In Fedora, we build Python packages with extension modules on all supported architectures. This includes i686, which is not available as an installable Fedora Linux, but exists to support the infamous “multilib” scenario (so users can install glibc.i686 and friends on an x86_64 system to run e.g. Stream, Wine, or some legacy app that is shipped as an ix86 ELF).

While I don’t expect users need multilib SciPy or NumPy, the problem arises with (very deep transitive) build dependency chains. E.g. there is this dependency chain:

  • (everything) needs rpm needs
    • rust-rpm-sequoia build-requires
      • rust-packaging is written in Python and build-requires
        • pytest build-requires
          • hypothesis build-requires
            • pandas build-requires
              • scipy

(There are more such chains.)

While most of the chains are usually “noarch” (i.e. does not need to be built separately on each architecture, e.g. it is a pure Python package), such packages can no longer be built the same way on i686 – and our assumption is that “noarch” packages can be built anywhere.

I suppose with the trend we will need to figure out how to deal with this problem on our side[1], I am just mentioning this for completeness. We don’t require wheels built for 32bit, but we need the software to be buildable and testable on i686 (at least for now).


  1. I started the discussion in Fedora as well: How to drop 32bit support from the scientific Python stack - devel - Fedora Mailing-Lists ↩︎

3 Likes

Thanks for the context @hroncok. I agree that one needs to be able to build from source on i686. There’s also armv7 which is still quite relevant.

I suspect that this issue is mostly meant for dropping 32-bit Windows wheels, rather than anything else. @mattip if you agree, perhaps you can update the title of this thread?

Finally, I’ll also note that 32-bit Windows packages can be built from source just fine. The key problems are (a) no compiler toolchain for SciPy and other Fortran packages, and (b) 32-bit Windows is very niche and no longer worth supporting through wheels for any project where that is significant effort. To me that is a different question though than whether or not from-source builds work. It not doing so should be considered a bug, just like for any other niche platform that we don’t have wheels for.

2 Likes

Using pypinfo to get download numbers for NumPy files on PyPI over the last 7 days (max 1,000 files), and then filtering for Windows for the last release:

pypinfo --days 7 --limit 1000 numpy version file system > 1000.md
rg Windows 1000.md | rg 1.24.3

Gives:

File Downloads Percentage
numpy-1.24.3-cp310-cp310-win_amd64.whl 283,986 35.98%
numpy-1.24.3-cp311-cp311-win_amd64.whl 207,710 26.32%
numpy-1.24.3-cp39-cp39-win_amd64.whl 178,848 22.66%
numpy-1.24.3-cp38-cp38-win_amd64.whl 103,099 13.06%
numpy-1.24.3-cp38-cp38-win32.whl 5,903 0.75%
numpy-1.24.3-cp39-cp39-win32.whl 2,822 0.36%
numpy-1.24.3.tar.gz 2,684 0.34%
numpy-1.24.3-cp310-cp310-win32.whl 2,192 0.28%
numpy-1.24.3-cp311-cp311-win32.whl 1,506 0.19%
numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 174 0.02%
numpy-1.24.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 124 0.02%
numpy-1.24.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 105 0.01%
numpy-1.24.3-pp38-pypy38_pp73-win_amd64.whl 103 0.01%
789,256 100.00%

That’s 98.0% for 64-bit Windows and 1.6% for 32-bit Windows (ignoring the sdist and manylinux).

(Full output of pypinfo: Output of `pypinfo --days 7 --limit 1000 numpy version file system` · GitHub)

3 Likes

Looks like things will get even trickier in the future because we need the old 4.0 version of rtools installed in the workflow script, and since 4.3 is installed by default:

Chocolatey v1.3.1

30Installing the same package with multiple versions is deprecated and will be removed in v2.0.0.

Debian’s situation is similar to Fedora’s in that we try to build all our packages on all our supported architectures. That currently includes 3 32-bit architectures: i386 (32bit x86), armel (ARM EABI 5T+), armhf (ARMv7 with hard-float), and mipsel (32-bit little endian MIPS).

Unlike Fedora (if I understand correctly), arch-all packages (equivalent to Fedora’s noarch packages) are built in an amd64 environment, by default. So, dropping 32-bit support for a package doesn’t have as much distro-wide fallout.

It does get annoying when low-level tools that are widely used drop support for an architecture, and take down entire trees of packages with them. But that mostly only affects the users of the obscure architectures. And the package maintainers who have to look at failing builds/tests. In some cases we can just ignore an upstream’s desire not to support an architecture, and continue to build a tool (maybe without tests). That can be good enough, or a cause of even more headaches…

3 Likes

Thanks for all the replies. I would summarize that it is OK if NumPy (and the scientific python stack) stop testing 32-bit windows and stop providing 32-bit windows wheels. We (NumPy) will continue to test (but not provide wheels for) 32-bit linux.

2 Likes

FWIW for pyca/cryptography we had the opposite experience: We stopped testing on 32-bit Linux, and stopped building 32-bit Linux wheels several years ago, and that produced no problems.

However, of our Windows wheels, nearly 15% are still 32-bit, so we haven’t dropped support yet. (Though I would really like to.)

2 Likes

Maybe the download statistics represent some CI workflows that pick up cryptography as a transient dependency, and do not really care that they are getting the latest version.

It’s possible! One other factor is that for many years, python.org defaulted to 32-bit for downloading Windows installers, so many people were using a 32-bit binary despite having a 64-bit CPU and operating system.

2 Likes

Sharing some numbers for Pillow (from python-pillow/Pillow#6941 (comment)):

For Linux, the *i686.whl numbers are very low for all versions (0.01% - 0.09%):

Details
version file download_count percentage
9.5.2 Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl 219,628 91.71 %
9.5.2 Pillow-9.5.0-cp311-cp311-manylinux_2_28_aarch64.whl 7,407 3.09 %
9.5.2 Pillow-9.5.0-cp311-cp311-musllinux_1_1_x86_64.whl 5,810 2.43 %
9.5.2 Pillow-9.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 5,233 2.19 %
9.5.2 Pillow-9.5.0-cp311-cp311-musllinux_1_1_aarch64.whl 771 0.32 %
9.5.2 Pillow-9.5.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 405 0.17 %
9.5.2 Pillow-9.5.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 226 0.09 %
total 239,480 100 %
9.5.2 Pillow-9.5.0-cp310-cp310-manylinux_2_28_x86_64.whl 788,288 91.90 %
9.5.2 Pillow-9.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 46,996 5.48 %
9.5.2 Pillow-9.5.0-cp310-cp310-manylinux_2_28_aarch64.whl 13,842 1.61 %
9.5.2 Pillow-9.5.0-cp310-cp310-musllinux_1_1_x86_64.whl 6,597 0.77 %
9.5.2 Pillow-9.5.0-cp310-cp310-musllinux_1_1_aarch64.whl 934 0.11 %
9.5.2 Pillow-9.5.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 882 0.10 %
9.5.2 Pillow-9.5.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 224 0.03 %
total 857,763 100 %
9.5.2 Pillow-9.5.0-cp39-cp39-manylinux_2_28_x86_64.whl 910,193 84.64 %
9.5.2 Pillow-9.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 137,166 12.75 %
9.5.2 Pillow-9.5.0-cp39-cp39-manylinux_2_28_aarch64.whl 15,810 1.47 %
9.5.2 Pillow-9.5.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 7,066 0.66 %
9.5.2 Pillow-9.5.0-cp39-cp39-musllinux_1_1_x86_64.whl 4,184 0.39 %
9.5.2 Pillow-9.5.0-cp39-cp39-musllinux_1_1_aarch64.whl 542 0.05 %
9.5.2 Pillow-9.5.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 462 0.04 %
total 1,075,423 100 %
9.5.2 Pillow-9.5.0-cp38-cp38-manylinux_2_28_x86_64.whl 924,301 51.40 %
9.5.2 Pillow-9.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 856,048 47.60 %
9.5.2 Pillow-9.5.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 7,569 0.42 %
9.5.2 Pillow-9.5.0-cp38-cp38-manylinux_2_28_aarch64.whl 7,523 0.42 %
9.5.2 Pillow-9.5.0-cp38-cp38-musllinux_1_1_x86_64.whl 2,343 0.13 %
9.5.2 Pillow-9.5.0-cp38-cp38-musllinux_1_1_aarch64.whl 355 0.02 %
9.5.2 Pillow-9.5.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 241 0.01 %
total 1,798,380 100 %

For Windows, *win32.whl are a bigger proportion but still low (0.90% - 5.63%):

Details
version file download_count percentage
9.5.2 Pillow-9.5.0-cp311-cp311-win_amd64.whl 150,653 98.82 %
9.5.2 Pillow-9.5.0-cp311-cp311-win32.whl 1,580 1.04 %
9.5.2 Pillow-9.5.0-cp311-cp311-win_arm64.whl 225 0.15 %
total 152,458 100.00 %
9.5.2 Pillow-9.5.0-cp310-cp310-win_amd64.whl 198,304 99.10 %
9.5.2 Pillow-9.5.0-cp310-cp310-win32.whl 1,801 0.90 %
total 200,105 100.00 %
9.5.2 Pillow-9.5.0-cp39-cp39-win_amd64.whl 89,231 98.12 %
9.5.2 Pillow-9.5.0-cp39-cp39-win32.whl 1,710 1.88 %
total 90,941 100.00 %
9.5.2 Pillow-9.5.0-cp38-cp38-win_amd64.whl 71,998 94.37 %
9.5.2 Pillow-9.5.0-cp38-cp38-win32.whl 4,298 5.63 %
total 76,296 100 %

In next month’s Pillow 10.0 release, we’re going to skip 32-bit wheels for Python 3.12, and I think it might be a good time to drop 32-bit wheels and testing for all versions.

4 Likes