PEP 725: Specifying external dependencies in pyproject.toml (round 2)

[jamestwebber (James Webber)]

Regardless of the exact form, IMO “valid TOML” is a hard requirement.

[tobiasdiez]

Thanks, this makes a lot of sense and I’m okay with not being able to formalize such compiler constraints. (As a small suggestion: maybe reformulate the corresponding sentence in the PEP a little bit to make it clear that version constraints on virtual packages only make sense when/if all implementation of that virtual package share the same versioning).

[rgommers (Ralf Gommers)]

@ncoghlan

I’d put it in the same category as using direct URLs in Python dependency declarations, so applying the same restriction we do to those would make sense to me

I like the analogy - and the restriction you suggest sounds about right to me, thanks. I’ll plan to include it in a next update of the PEP.

@mikeshardmind

Personally, I’d prefer something that looked more like:

...
[external.dependencies."dep:generic/ffmpeg"]

Thanks! That looks like a nicer way of doing distro-specific selections than what I sketched out. I’d like to include this version in a next PEP update under Rejected Ideas (because starting more restrictive and not covering every corner case), with an “if we’d want to allow overrides in the future, this preferred-alternatives is a clean way to extend [external] without breaking backwards compatibility”.

@tobiasdiez

As a small suggestion: maybe reformulate the corresponding sentence in the PEP a little bit to make it clear that version constraints on virtual packages only make sense when/if all implementation of that virtual package share the same versioning

thanks, will include that.

[mcepl (Matěj Cepl)]

This sounds like it can get out of hands quite quickly: do you really want to include a list of all possible Linux distributions (and don’t forget about *BSD, Homebrew et al for Mac OS, and Chocolatey for Window, and I am certain I forgot something)? Could we use some mapping, https://release-monitoring.org/ comes to my mind, but it has limited set of packages (e.g., there isn’t ffmpeg-libs only full ffmpeg).

[bwoodsend (Brénainn Woodsend)]

Yes I do. I’m aware that there are lots of platforms but under the use of the registry, it’s then more platforms’ registries that I then have to get my definition^[1] of each dependency into rather than platforms I can just write into a toml file so the friction goes up rather than down. I have Docker. I have VMs. I have CI. I fully expect to be able to get a higher platform coverage through those than the registry will ever have. I already have to test against a decent number of these platforms for distro-specific patches or gotchas like rsvg-convert’s configuration on OpenSUSE. With the registry, I can be less vigilant for packages being renamed but have to be more vigilant for unacceptable dependency growth^[2] or Void Linux misassuming that two packages are interchangeable or a registry adding a dependency on a package that didn’t exist on my oldest supported version of $some_distro… neither are fun but at least the renames I can fix.

But having the ability to specify all platforms manually doesn’t mean that it has to be used to that extreme to be useful. Being able to mainly use the registry with the option for platform specific overrides would be the difference between all edge cases and non registry providing distributions are toast and PEP 725 being tolerant of both.

---

1. assuming I’m even allowed my definition ↩︎

2. the strategy for dealing with conflicting requirements will no doubt err on the side of installing more packages than needed ↩︎

[tiangolo (Sebastián Ramírez)]

Thanks for the PEPs!

I don’t have experience from the point of view of distro repackaging, so no opinions there.

But from the point of view of a team building a cloud product for Python apps (FastAPI Cloud), this sounds great.

We would like to be able to let users define system dependencies they need (or even better, defined by the packages they use), and if it’s in a standard way (as proposed in these PEPs) we could probably support that standard rather than trying to build our own custom setup.

[stefanor (Stefano Rivera)]

Hi, sorry I’m pretty late to the game here. I’ve had an open tab to read this for a month…

I see this not the first time it’s been raised, but I also don’t see a clear conclusion:

I’m confused by the vague host-requires, which is both a build and runtime dependency. Differentiating between built-time and run-time dependencies makes sense, of course. And differentiating between cross-compilation build and host architectures in dependencies makes sense. This gives us a 2x2 grid:

	Build Architecture	Host (target) Architecture
Build-time	build-requires	host-requires
Run-time	N/A	dependencies + host-requires

host-requires awkwardly occupies 2 squares. I would strongly suggest removing the runtime component of it. Things can always be listed in dependencies if necessary for both.

[stefanor (Stefano Rivera)]

I would replace the word “run” with “used” in the context of dependencies. Not all dependencies are tools that are executed.

[pf_moore (Paul Moore)]

Surely the problem is that dependencies is python dependencies (and changing that would be a significant compatibility issue) whereas we are talking here about external dependencies?

[stefanor (Stefano Rivera)]

I’m talking about dependencies inside the [external] table.

[oscarbenjamin]

@stefanor

host-requires awkwardly occupies 2 squares. I would strongly suggest removing the runtime component of it. Things can always be listed in dependencies if necessary for both.

It partially occupies the second square since the PEP says:

host dependency : package needed during the build and often also at runtime

The main answer above is that if the host dependency was built against then it cannot be satisfied by a nameable package afterwards i.e. the dependency is just whatever was there at build time. The reason it seems awkward to me to combine them like this is that there is no way to specify whether or not it actually is a runtime dependency. For example if the host dependency is statically linked or bundled then it is not a separate runtime requirement but if it was dynamically linked and not bundled it is.

I can see arguments for not listing this independently as both a build-time and run-time requirement since when it is both the two requirements cannot usually be satisfied by taking the dependency from different sources at build-time and at run-time and it would not make sense to have e.g. separate version constraints. Perhaps though there could be two separate lists for host-requires where one is only needed at build time and one is needed both at build time and at runtime.

One example I know of where the runtime dependency is satisfied by separately sourcing a nameable package is in the case of piwheels. You can see the numpy piwheels page here and the installation instructions are

sudo apt install libgfortran5 libopenblas0-pthread
pip3 install numpy

My understanding is that the way this works is that piwheels builds wheels against Debian packages and then those same packages need to be installed as runtime dependencies when installing the wheels.

[stefanor (Stefano Rivera)]

@oscarbenjamin

The main answer above is that if the host dependency was built against then it cannot be satisfied by a nameable package afterwards i.e. the dependency is just whatever was there at build time.

My understanding is that this was dealt with by auditwheel:

and (2) these runtime dependencies can be detected and handled automatically by tools like auditwheel.

[oscarbenjamin]

Well auditwheel is used for preparing wheels for PyPI but not in other situations like conda, brew, distro packages, piwheels and so on. I thought that the purpose of this external dependency metadata is really for all those other situations where you are not consuming wheels from PyPI.

[stefanor (Stefano Rivera)]

Surely the distros all already have tooling for this? In those non-PyPI situations, this metadata isn’t needed, because the native package manager is responsible for it.

[oscarbenjamin]

The distros do have tooling for this but that tooling needs the metadata. The primary purpose of this PEP is for Python projects to provide that metadata to the distro packagers. I just checked again the abstract and motivation sections of the PEP and it is quite explicit about this.

[stefanor (Stefano Rivera)]

So what is auditwheel’s role in determining these runtime dependencies, outlined in the build an host dependencies section?

[rgommers (Ralf Gommers)]

Thanks for the detailed feedback here and on the PEP 804 thread @stefanor

@stefanor

So what is auditwheel’s role in determining these runtime dependencies

auditwheel itself is irrelevant to this PEP, its purpose is limited to wheels distributed on PyPI (or another PyPI-like index), while external metadata is broader - it’s meant to cover dependency info for any build from source, not just when producing wheels for uploading to PyPI. That external metadata in pyproject.toml does not change when the distributing method changes.

@stefanor

Surely the distros all already have tooling for this?

Yes, and by all means use that. Note that the host dependency sentence you quoted is from the Terminology section, it’s explaining what a host dependency is. It doesn’t say that anything needs to be added to runtime dependencies in the METADATA file of a wheel. And it doesn’t say that exactly because distros already have tooling for this, so there isn’t anything extra that needs doing by any tool. Whatever method a distro has to make a package redistributable (and that includes auditwheel) is the thing that ensures that the “and often also at runtime” is satisfied. That distro tooling relies (at least in all cases I know of) on analyzing the built extension modules for shared libraries they need EDIT: or encoding that info in the metadata of the dependency itself, e.g. the run-exports feature from conda-forge, and then handles that appropriately. Which is the only reliable thing to do.

I think that should answer your question/concern?

[rgommers (Ralf Gommers)]

@tiangolo

But from the point of view of a team building a cloud product for Python apps (FastAPI Cloud), this sounds great.

Thanks for sharing @tiangolo, that’s an interesting use case. There are probably more such products and platforms that need this - your example made me think of ReadTheDocs, and they now have a custom scheme as well to handle external dependencies, see build.tools, build.tools.nodejs|rust|golang, build.apt-packages in their v2 config file docs.

I’d like to add this use case to the Motivation section.

[stefanor (Stefano Rivera)]

@rgommers

auditwheel itself is irrelevant to this PEP, its purpose is limited to wheels distributed on PyPI (or another PyPI-like index), while external metadata is broader - it’s meant to cover dependency info for any build from source, not just when producing wheels for uploading to PyPI. That external metadata in pyproject.toml does not change when the distributing method changes.

OK, I think the PEP text should change to make that clearer, because this section tells me that runtime dependencies on things like libgfortran should not be explicitly added as they can be automatically inferred by tools like auditwheel, after a wheel is built, and added to it. If we require this metadata to be complete in the source package, then it should be explicitly listed.

@rgommers

Note that the host dependency sentence you quoted is from the Terminology section

Maybe use a level 4 heading to clarify this? At the moment it’s at the same level as Terminology.

I was reading this to understand what exactly we were meaning by a host dependency in this PEP (see my question above for context, before we went down the auditwheel rabbithole). What it took me several readings (and this discussion) to understand is that the host dependencies DO NOT resolve to the same set of external packages in both the build and run contexts. This could be clearer earlier on, the vague language (like “often also at runtime”) speaks to imprecision rather than context-dependant resolution.

It would really help to specify that context-dependent resolution is happening in the abstract, as it’s core to understanding PEP-804 too.

I was trying to get clarity on what goes into each of the three categories of dependencies: host-requires, build-requires and dependencies.

I think I’d describe it as:

• build-requires: Build-time external dependencies, native to the builder system. e.g. compilers and command-line utilities.
• host-requires: Build & run-time external dependencies, native to the target system. e.g. library packages providing headers (build time), shared libraries (build & run time), and/or static libraries (build time).
• dependencies: Run-time external dependencies, native to the target system. e.g. command-line utilities executed by the package.

The same dependency in each of these will be interpreted differently depending on context. e.g. a library in host-requires should install headers at build time, but is not required to at runtime.

Would you be willing to accept text like this for the PEP?

This implicit interpretation leaves us with some potential holes. e.g. if you really require a header at runtime, you either need to list the library in dependencies or define a new generic PURL for this case.

[mgorny (Michał Górny)]

@bwoodsend

[external]
dependencies = [
    "pkg:apk/alpine/ffmpeg-libs",
    "pkg:deb/debian/ffmpeg",
    "pkg:rpm/fedora/ffmpeg-free",
    "pkg:deb/ubuntu/ffmpeg",
]

@ncoghlan

Edit: it does occur to me that this is duplicating info that is present in the concrete dep URLs already (the target ecosystem and platform IDs). So perhaps the simpler convention is to say that converters should ignore concrete external dependency declarations for unrelated ecosystems, unless they have an applicable non-Python-specific PURL translation registry they can use to make them applicable.

While this is an interesting idea, I don’t think it would work well. The “applicable […] translation registry” part would effectively mean that you’d end up applying the “broader” pkg:deb/debian/ffmpeg package on Alpine where already a “narrower” pkg:apk/alpine/ffmpeg-libs is usable, if a mapping for Debian packages was provided. And you’d want to provide one, if enough packages didn’t list Alpine dependencies.