Packaging support for riscv64

Packaging
1

Hi all,

I’ve spent some time this year building riscv64 binary wheels for a select set of Python packages, e.g., numpy and scipy, and trying to understand what is preventing upstream projects from creating riscv64 binary wheels themselves and uploading those wheels to PyPI. I thought it might be useful to share my understanding of the current situation in the hope that this may be helpful to others interested in riscv64, in the hope that the Python packaging community can help me identify things I’ve got wrong or things that I’ve missed, and also, to start a discussion on what would need to be done to get full riscv64 support added to the pypa and other related projects. My understanding is based on the experience I’ve gained from building riscv64 wheels and also from reading the related issues and discussions in the various pypa github projects.

Current state of Packaging support for riscv64

This is the situation today as I understand it. I’m primarily interested in manylinux so my summary has a manylinux focus.

Python projects, like numpy, cannot currently create manylinux wheels for riscv64 as the infrastructure projects used to build manylinux wheels do not support riscv64. cibuildwheel, auditwheel and manylinux itself are all lacking riscv64 support.

The existing manylinux images for other architectures are based on AlmaLinux and, at the time of writing, AlmaLinux does not support riscv64. Ubuntu does support riscv64 (22.04 and 24.04 are available) but I understand that debian based distros are not a good basis for manylinux images due to their lack of GCC Toolset and because projects that use manylinux may assume a redhat based distro in their build/test scripts. It’s not clear to me whether the base image for the manylinux_2_34 image has been chosen yet but the candidates in the tracking issue (1585) all appear to be RHEL derivatives.

Auditwheel doesn’t support riscv64 but policy files with riscv64 support are available in the pep 600 compliance repository.

Cibuildwheel is also lacking support for riscv64.

A comment in the riscv64 auditwheel issue (408) suggests that PyPI might also need to be updated before it will accept riscv64 wheels.

There is a little bit of good news however. pip 24.1 is capable of installing riscv64 manylinux wheels (see below for why this is useful today).

Looking beyond the pypa projects, other tools like uv and maturin may need to be updated before riscv64 wheels can be created for projects that depend on these tools.

Needless to say, updating the pypa projects to support riscv64 is only a small piece of the overall work required. Python projects that build binary wheels would need to be individually updated to create wheels for riscv64, but these tasks I think, fall beyond the scope of this discussion.

Another potential issue projects may face is the lack of riscv64 runners. It is possible to build smaller projects with the default github runners using qemu-user but such an approach is unlikely to work for larger projects, including manylinux itself, that take a long time to build. Self-hosted runners may be necessary for these larger projects.

The main blocker to adding upstream support for manylinux riscv64 to the pypa projects is the lack of a RHEL derived distro for riscv64. Once such a distro becomes available, we’ll be able to create upstream manylinux images for riscv64 (assuming appropriate runners are available) and add riscv64 support to auditwheel and cibuildwheel.

I believe that adding linux-riscv64 platform support to the pypa projects would not require a new PEP.

What did I miss?

Does that seem like a reasonable summary of the status quo? What did I get wrong? Is there anything I’m obviously missing here?

Also, is there any useful work that can be done upstream for riscv64 in the pypa projects before an appropriate riscv64 base image for manylinux becomes available? I’d be happy to contribute/participate in any such efforts.

riscv64 wheels built by RISE

One final comment. Until riscv64 is supported by the pypa packaging projects and PyPI, Python projects, such as numpy or scipy cannot build and upload riscv64 binary packages to PyPI. Users of riscv64 devices are obliged to build these packages locally, and doing so is time-consuming and hard to get right, making it difficult to use Python for things like AI and numerical analysis on riscv64. For this reason, the RISE project is currently building and publishing manylinux_2_35_riscv64 packages for a select set of projects on RISE’s gitlab. These wheels are built using patched versions of manylinux (patched to use Ubuntu 22.04), cibuildwheel and auditwheel. They can be installed using upstream pip 24.1.

1 Like
2

I understand that debian based distros are not a good basis for
manylinux images due to their lack of GCC Toolset

Can you elaborate? All my systems run pure Debian and have gcc
installed directly from the distro’s official packaging. Are you
maybe confusing it with the (historically brief) temporary eglibc
fork of glibc? If so, that hasn’t existed for the past decade now,
so shouldn’t be a concern either.

3

The existing manylinux images use AlmaLinux 8 in conjunction with RedHat’s GCC Toolset. My understanding is that GCC Toolset provides modern versions of gcc which are patched to work with the base libraries provided by the distribution (e.g., glibc and libstdc++). The use of GCC Toolset then allows the manylinux_2_28 images to build packages with gcc 12 that are linked against glibc 2.28. The newer compiler ensures that packages using modern features will still build and the old glibc ensures that the built packages will run on older distributions as far back as Ubuntu 18.10.

As I understand it, there isn’t a GCC Toolset equivalent for debian based distros, making it difficult to create a debian based manylinux image that achieves the same trick of combining an old glibc with a new gcc. Reading through the various threads on the manylinux github repo it seems that there was previously a debian based manylinux image, manylinux_2_24, which was removed in part due to the lack of a GCC Toolset equivalent.

4

Thanks, that makes more sense. As a non-RH-user (at least not since
the late '90s), I was wholly unaware GCC Toolset existed and thought
you were trying to say “GNU toolchain.” Apologies for the confusion!

5

The auditwheel patch in RISE should not be needed anymore as of auditwheel 6.1.0 (manylinux policy updated from the pep600_compliance repo / musllinux not supported, see also RISC-V support · Issue #408 · pypa/auditwheel · GitHub)

PyPI does not accept riscv64 wheels, warehouse/warehouse/forklift/legacy.py at main · pypi/warehouse · GitHub. PEP600 does not restrict the architecture and leaves that up to index maintainers PEP 600 – Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions | peps.python.org. That being said, given PEP425 tags just uses uname -m, it might be a good thing to check what’s acceptable if riscv64 is too broad (e.g. different ABIs / different base of required ISA extensions, …). armv7l for example was too broad (no way to discern between the soft-float ABI and hard-float ABI). Given it was introduced in PEP599, it was assumed to be hard-float and leaded to a specific implementation in packaging in order to check the ABI.

cibuildwheel currently does not support building for architectures other than the ones for which pypa manylinux/musllinux images exists. Speaking personally - as opposed to having my cibuildwheel hat on -, that does not mean that a pypa/manylinux image must exist for it to be added. If the RISE project has the resources to provide & maintain an image that’s suitable to build riscv64 manylinux wheels, then it might very well be considered as an addition using said image as a default (maybe pending checks & PyPI update).

Finally, manylinux:
The whole gcc toolset stuff mentioned here & there is almost a must have but that’s not a blocking issue as mentioned in RISCV64 support · pypa/manylinux · Discussion #1426 · GitHub. I wouldn’t add ubuntu based images for architectures where an RHEL-derivative with gcc-toolset image exists as it caused more pain than gain in the past.
Building images using the same process as the other ones is a no-go as there are no appropriate runners to do this.
That being said, there are other ways of building images when using ubuntu that rely mostly on ppa (although deadsnakes currently does not build python for riscv64). Such a PoC can be seen there GitHub - mayeut/manylinux-ubuntu.

1 Like
6

This is good news. It seems that the newer policy files were also pulled into maturin v1.7.1, so maturin now supports riscv64 as well.

That being said, given PEP425 tags just uses uname -m , it might be a good thing to check what’s acceptable if riscv64 is too broad

The binaries we’re building at RISE target rv64gc with lp64d ABI (the default ABI for rv64gc systems), which seems like a sensible interpretation of the riscv64 tag. GCC and clang both generate rv64gc code by default, QEMU riscv64 VMs are rv64gc by default, and rv64gc is the minimum requirement for existing riscv64 distros, e.g., Debian and Fedora. rv64gc is also required by the base application profile, rva20u64. In addition, as far as I’m aware, all riscv64 devices on the market support rv64gc. So IMHO, rv64gc is a good base, it’s how most riscv64 software is currently compiled, and there’s no point in supporting anything less.

Regarding manylinux, reading the linked forum post it seems like a riscv64 only debian/ubuntu based manylinux image might be acceptable upstream, and that the blocker isn’t so much the lack of a suitable Fedora based distro but rather the lack of a suitable runner to build the image. Is this the case? If so, and if the manylinux project were interested, it may be possible for RISE to supply a cloud based runner to the manylinux project. This would probably be an x86 machine similar to the one used on the RISE gitlab builds. The RISE riscv64 manylinux images take about 3 hours to build under qemu-user on such a machine. If this were an acceptable path forward, I would of course be happy to open an issue to discuss what the riscv64 image should look like and to contribute/review/test patches.