Setuptools-rust/maturin: fetching rust deps when building wheel

Packaging
1

In Nixpkgs we’ve seen several packages now using the PyO3 ecosystem of packages. Two of these, maturin and setuptools-rust, allow one to build extension modules and/or executables written in Rust but distributed as a wheel.

Seeing rust being used this way is interesting, however, there is one issue that I think is problematic. Unfortunately, cargo is used not only to just build the rust code, but also to fetch dependencies. Indeed, as part of a build, dependencies are fetched. Cryptography is a common package that now also uses setuptools-rust and indeed fetches dependencies.

How do other distributors intend to handle this? Does this follow the spirit of PEP 517/518? I know they don’t mention anything regarding fetching during a build. Would it make sense for a new PEP to require that? Given pip does some sandboxing, are there any ideas/plans to block network access as well (as we do in Nixpkgs)?

cc @konstin as author of maturin.

2 Likes
2

I’m the package maintainer of PyCA cryptography in Fedora and RHEL.

Due to security, compliance and legal requirements, the Fedora process is more complex than your typical Rust workflow. In Fedora all packages must be build offline and build dependencies should not be vendored. The Fedora Rust SIG has helped me to package all Rust crates as RPMs, e.g. 1907087 – Review Request: rust-pyo3 - Bindings to Python interpreter . You can find more information about packaging rules in the documents Fedora Packaging Guidelines :: Fedora Docs and Rust Packaging Guidelines :: Fedora Docs .

The RPM specfile for python-cryptography uses build macros to generate a special .cargo/config.toml and to auto-discover build dependencies from the project’s Cargo.toml. The macro %cargo_prep creates the config file and %cargo_generate_buildrequires generates dynamic BuildRequires from Cargo.toml.

If you don’t need to follow a similar set of strict rules, then you can use cargo vendor to vendor your dependencies for offline building/

$ mkdir -p .cargo
$ cargo vendor --manifest-path src/rust/Cargo.toml > .cargo/config.toml
$ tar cvj vendor .cargo -f vendor.tar.bz2
4 Likes
3

PS: AFAIK pip doesn’t do sandboxing yet. I have done some experience with ctypes bindings for libseccomp to block SYS_socket syscall except for AF_UNIX for IPC.

1 Like
4

Thank you Christian. Can Fedora handle different versions of rust crates or will it be limited to single versions as is often the case with other types of packages?

We indeed use cargo vendor already in Nixpkgs for packaging of rust-based programs. Turns out a community member had already packaged some maturin/setuptools-rust packages. There is now a PR to simplify this for other Python packages requiring maturin/setuptools-rust. In the Python package builder we point to a directory with the vendored dependencies which have been fetched before with cargo.

1 Like
5

I don’t know yet – guess I’ll figure this out soon enough. The on-disk structure supports multiple versions of a crate.

1 Like
6

I can’t speak authoratively, but IMO this is well within PEP 517’s spirit. Rust crates are all build-time dependencies, and a backend has all the freedom to handle them it considers best. There’s no need of a PEP as long as it does not affect other packages’ build process (and not necessarily even if it does).

1 Like
7

I agree. PEP 517/518 allow packages to specify what packages the installer must ensure are present in order to build the target package. That does not mean that these are the only build time dependencies - only that “if you install this stuff and call the build hook, you’ll get a wheel”.

If the build backend installs extra stuff in order to complete the build, that’s fine.

2 Likes
8

As a quality-of-life thing, I can see an argument that setuptools-rust might want to have an option to run cargo-vendor as part of the sdist generation process. Not sure if it’s actually a good idea, but it’s at least straightforward in concept :slight_smile:

2 Likes
9

In essence this is no different from telling the user that they have to install a Fortran compiler to build numpy; that’s not reflected in the PEP 517/518 metadata either.

2 Likes
10

The difference is that this is now fetching and building those components for you. Its like setuptools all over, where people run all kind of horrible things during build time (panel, sphinx_rtd_theme).

This sometimes happens as well and it seems like a fair trade-off to me. The maintainer of sphinx_rtd_theme basically suggested the same.

2 Likes
11

That’s a different question. PEP 517/518 let the installer determine what is needed to do the build. The build backend can do anything it wants to generate the wheel. If the community has views on what backends should and should not do, the expectation is that “consumer pressure” will handle that, with backends that don’t behave the way people want not getting used.

The difference between the current situation and the original distutils/setuptools situation is that before PEP 517, setuptools was in a privileged position of being the only backend pip supported, so people didn’t have the option to “vote with their feet”.

1 Like
12

I have created feature request Add option to cargo vendor crates into sdist · Issue #112 · PyO3/setuptools-rust · GitHub to add vendoring of crates to setuptools_rust.

1 Like
13

It can, but each version needs to be packaged under a separate RPM package name (e.g. rust-tokio0.1 vs rust-tokio0.2). The package name is an implementation detail for the mechanism, because the package provides crate(tokio) = 0.1.22 / crate(tokio) = 0.2.24 and that is the information used by %cargo_generate_buildrequires (e.g. python-cryptography buildrequires (crate(pyo3/default) >= 0.13.1 with crate(pyo3/default) < 0.14.0~)).

1 Like
14

setuptools-rust sdist vendoring support merged.

2 Likes