PEP 725: Specifying external dependencies in pyproject.toml (round 2)

Packaging Standards
1

Hi all, I’m happy to share, also on behalf of my co-authors @pradyunsg and @jaimergp, an update of PEP 725 - Specifying external dependencies in pyproject.toml (see also the round 1 discussion).

Please use this new thread for any additional feedback of this updated version.

Here are the significant changes since the previous version:

  1. Add support for dependency groups (PEP 735).
  2. Add support for version ranges in external dependency specifiers.
  3. Define and use DepURLs (Dependency URLs), a derivative of PURLs, rather than use PURLs directly.
    • Motivation: this allows using version ranges and virtual packages with an ergonomic syntax, which turned out to be impossible with PURLs directly. The PURL spec did merge support for version ranges at long last, but the syntax didn’t seem acceptable.
  4. Drop the attempt to reuse the Requires-External Core Metadata field with tightened semantics. Instead, deprecate that field, and introduce two new fields. This causes a minor version bump to Core Metadata to 2.6.
  5. Specify clearly how environment markers are used in external dependency specifiers.
  6. Reference PEP 804 (new) for a central registry of external dependency specifiers with canonical names and the ability to address new questions around versioning of virtuals and split packages.
    • Motivation: this address a lot of the feedback about how this was (loosely) handled in previous versions. And with the actual central registry and maintenance/processes around it, makes it possible to keep answering new such questions as they come up.
  7. Significant copy-edits to make the text clearer to understand, and in particular make the Specification section more complete.

This update closes all open issues and hopefully addresses feedback on previous versions. In addition, the prototype code from the previous version was improved significantly, in support of this PEP as well as PEP 804. pyproject-external is now a proper package including tests and docs.

Please see the full text below the fold, as well as:

Full text

PEP: 725
Title: Specifying external dependencies in pyproject.toml
Author: Pradyun Gedam pradyunsg@gmail.com,
Jaime Rodríguez-Guerra jaime.rogue@gmail.com,
Ralf Gommers ralf.gommers@gmail.com
Discussions-To: PEP 725: Specifying external dependencies in pyproject.toml
Status: Draft
Type: Standards Track
Topic: Packaging
Created: 17-Aug-2023
Post-History: 18-Aug-2023 <https://discuss.python.org/t/31888>__

Abstract

This PEP specifies how to write a project’s external, or non-PyPI, build and
runtime dependencies in a pyproject.toml file for packaging-related tools
to consume.

This PEP proposes to add an [external] table to pyproject.toml with
seven keys. “build-requires”, “host-requires” and “dependencies”
are for specifying three types of required dependencies:

  1. build-requires, build tools to run on the build machine
  2. host-requires, build dependencies needed for the host machine but also needed at build time.
  3. dependencies, needed at runtime on the host machine but not needed at build time.

These three keys also have their optional external counterparts (optional-build-requires,
optional-host-requires, optional-dependencies), which have the same role that
project.optional-dependencies plays for project.dependencies. Finally,
dependency-groups offers the same functionality as :pep:735 but for external
dependencies.

Cross compilation is taken into account by distinguishing between build and host dependencies.
Optional build-time and runtime dependencies are supported too, in a manner analogous
to how that is supported in the [project] table.

Motivation

Python packages may have dependencies on build tools, libraries, command-line
tools, or other software that is not present on PyPI. Currently there is no way
to express those dependencies in standardized metadata
[#singular-vision-native-deps], [#pypacking-native-deps]. Key motivators for
this PEP are to:

  • Enable tools to automatically map external dependencies to packages in other
    packaging repositories,
  • Make it possible to include needed dependencies in error messages emitted by
    Python package installers and build frontends,
  • Provide a canonical place for package authors to record this dependency
    information.

Packaging ecosystems like Linux distros, conda, Homebrew, Spack, and Nix need
full sets of dependencies for Python packages, and have tools like pyp2spec_
(Fedora), Grayskull_ (conda), and dh_python_ (Debian) which attempt to
automatically generate dependency metadata for their own package managers from the metadata in
upstream Python packages. External dependencies are currently handled manually,
because there is no metadata for this in pyproject.toml or any other
standard location. Other tools resort to extracting dependencies from extension
modules and shared libraries inside Python packages, like elfdeps_ (Fedora).
Enabling automating this type of conversion by only using explicitly annotated metadata
is a key benefit of this PEP, making packaging Python packages for distros easier
and more reliable. In addition, the authors envision other types of tools
making use of this information, e.g., dependency analysis tools like Repology_,
Dependabot_ and libraries.io_.

Software bill of materials (SBOM) generation tools may also be able to use this
information, e.g. for flagging that external dependencies listed in
pyproject.toml but not contained in wheel metadata are likely vendored
within the wheel. :pep:770, which standardizes how SBOMs are included in
wheels, contains an instructive section on how that PEP differs from this one.

Packages with external dependencies are typically hard to build from source,
and error messages from build failures tend to be hard to decipher for end
users. Missing external dependencies on the end user’s system are the most
likely cause of build failures. If installers can show the required external
dependencies as part of their error message, this may save users a lot of time.

At the moment, information on external dependencies is only captured in
installation documentation of individual packages. It is hard to maintain for
package authors and tends to go out of date. It’s also hard for users and
distro packagers to find it. Having a canonical place to record this dependency
information will improve this situation.

This PEP is not trying to specify how the external dependencies should be used,
nor a mechanism to implement a name mapping from names of individual packages
that are canonical for Python projects published on PyPI to those of other
packaging ecosystems. Canonical names and a name mapping mechanism are addressed
in :pep:804.

Rationale

Types of external dependencies

Multiple types of external dependencies can be distinguished:

  • Concrete packages that can be identified by name and have a canonical
    location in another language-specific package repository. E.g., Rust
    packages on crates.io <https://crates.io/>, R packages on
    CRAN <https://cran.r-project.org/>    , JavaScript packages on the
    npm registry <https://www.npmjs.com/>__.
  • Concrete packages that can be identified by name but do not have a clear
    canonical location. This is typically the case for libraries and tools
    written in C, C++, Fortran, CUDA and other low-level languages. E.g.,
    Boost, OpenSSL, Protobuf, Intel MKL, GCC.
  • “Virtual” packages, which are names for concepts, types of tools or
    interfaces. These typically have multiple implementations, which are
    concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI.

Concrete packages are straightforward to understand, and are a concept present
in every package management system. Virtual packages are a concept
also present in a number of packaging systems – but not always, and the
details of their implementation vary.

Cross compilation

Cross compilation is not yet (as of September 2025) well-supported by stdlib
modules and pyproject.toml metadata. It is however important when
translating external dependencies to those of other packaging systems (with
tools like pyp2spec). Introducing support for cross compilation immediately
in this PEP is much easier than extending [external] in the future, hence
the authors choose to include this now.

Terminology
‘’‘’‘’‘’‘’’

This PEP uses the following terminology:

  • build machine: the machine on which the package build process is being
    executed.
  • host machine: the machine on which the produced artifact will be installed
    and run.
  • build dependency: package required only during the build process. It must
    be available at build time and is built for the build machine’s OS and
    architecture. Typical examples include compilers, code generators, and
    build tools.
  • host dependency: package needed during the build and often also at runtime.
    It must be available during the build and is built for the host machine’s OS
    and architecture. These are usually libraries the project links against.
  • runtime dependency: package required only when the package is used after
    installation. It is not required at build time but must be available on
    the host machine at runtime.

Note that this terminology is not consistent across build and packaging tools,
so care must be taken when comparing build/host dependencies in
pyproject.toml to dependencies from other package managers.

Note that “target machine” or “target dependency” are not used in this PEP. That
is typically only relevant for cross-compiling a compiler or other such advanced
scenarios [#gcc-cross-terminology], [#meson-cross] – this is out of scope for
this PEP.

Finally, note that while “dependency” is the term most widely used for packages
needed at build time, the existing key in pyproject.toml for PyPI
build-time dependencies is build-requires. Hence this PEP uses the keys
build-requires and host-requires under [external] for consistency.

Build and host dependencies
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’’

Clear separation of metadata associated with the definition of build and host
platforms, rather than assuming that build and host platform will always be
the same, is important [#pypackaging-native-cross]_.

Build dependencies are typically run during the build process - they may be
compilers, code generators, or other such tools. In case the use of a build
dependency implies a runtime dependency, that runtime dependency does not have
to be declared explicitly. For example, when compiling Fortran code with
gfortran into a Python extension module, the package likely incurs a
dependency on the libgfortran runtime library. The rationale for not
explicitly listing such runtime dependencies is two-fold: (1) it may depend on
compiler/linker flags or details of the build environment whether the
dependency is present, and (2) these runtime dependencies can be detected and
handled automatically by tools like auditwheel.

Host dependencies are typically not run during the build process, but only used
for linking against. This is not a rule though – it may be possible or
necessary to run a host dependency under an emulator, or through a custom tool
like crossenv_. When host dependencies imply a runtime dependency, that runtime
dependency also does not have to be declared, just like for build dependencies.

When host dependencies are declared and a tool which is executing an action
unrelated to cross-compiling, it may decide to merge the host-requires list
into build-requires - whether this is useful is context-dependent.

Specifying external dependencies

Concrete package specification
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

A “Package URL” or PURL_ is a widely used URL string for identifying packages
that is meant to be portable across packaging ecosystems. Its design is::

scheme:type/namespace/name@version?qualifiers#subpath

The scheme component is a fixed string, pkg, and of the other
components only type and name are required.

Since external dependencies are likely to be typed by hand, we propose a PURL
derivative that, in the name of ergonomics and user-friendliness, introduces a
number of changes (further discussed below):

  • Support for virtual packages via a new virtual type.
  • Allow version ranges (and not just literals) in the version field.

In this derivative, we replace the pkg scheme with dep. Hence,
we will refer to them as DepURLs.

As an example, a DepURL for the requests package on PyPI would be::

dep:pypi/requests
# equivalent to pkg:pypi/requests

Adopting PURL-compatible strings to specify external dependencies in
pyproject.toml solves a number of problems at once, and there are already
implementations of the specification in Python and multiple other languages. PURL is
also already supported by dependency-related tooling like SPDX (see
External Repository Identifiers in the SPDX 2.3 spec <https://spdx.github.io/spdx-spec/v2.3/external-repository-identifiers/#f35-purl>),
the Open Source Vulnerability format <https://ossf.github.io/osv-schema/#affectedpackage-field>,
and the Sonatype OSS Index <https://ossindex.sonatype.org/doc/coordinates>__;
not having to wait years before support in such tooling arrives is valuable.
DepURLs are very easily transformed into PURLs, with the exception of
dep:virtual which doesn’t have an equivalent in PURL_.

For concrete packages without a canonical package manager to refer to, either
dep:generic/dep-name can be used, or a direct reference to the VCS system
that the package is maintained in (e.g.,
dep:github/user-or-org-name/dep-name). Which of these is more appropriate
is situation-dependent. This PEP recommends using dep:generic when the
package name is unambiguous and well-known (e.g., dep:generic/git or
dep:generic/openblas), and using the VCS as the type otherwise. Which name
is chosen as canonical for any given package, as well as the process to make
and record such choices, is the topic of :pep:804.

Virtual package specification
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

PURL does not offer support for virtual or virtual dependency specification yet.
A proposal to add a virtual type <https://github.com/package-url/purl-spec/pull/450>__
is being discussed for revision 1.1.

In the meantime, we propose adding a new type to our dep: derivative, the virtual
type, which can take two namespaces (extensible through the process given in
:pep:804):

  • interface: for components such as BLAS or MPI.
  • compiler: for compiled languages like C or Rust.

The name should be the most common name for the interface or language, lowercased.
Some examples include::

dep:virtual/compiler/c
dep:virtual/compiler/cxx
dep:virtual/compiler/rust
dep:virtual/interface/blas
dep:virtual/interface/lapack

Since there are a limited number of such dependencies, it seems like it will be
understandable and map well to Linux distros with virtual packages and to the
likes of conda and Spack.

Versioning
‘’‘’‘’‘’‘’

PURLs support fixed versions via the @ component of the URL. For example,
numpy===2.0 can be expressed as pkg:pypi/numpy@2.0.

Support in PURL for version expressions and ranges beyond a fixed version is
available via vers URIs (see specification <vers>__)::

vers:type/version-constraint|version-constraint|…

Users are supposed to couple a pkg: URL with a vers: URL. For example,
to express numpy>=2.0, the PURL equivalent would be pkg:pypi/numpy plus
vers:pypi/>=2.0. This can be done with:

  • A two-item list: ["pkg:pypi/numpy", "vers:pypi/>=2.0"].
  • A percent-encoded <https://github.com/package-url/purl-spec/blob/main/PURL-SPECIFICATION.rst#character-encoding>__
    URL qualifier: pkg:pypi/numpy?vers=vers:pypi%2F%3E%3D2.0.

Since none of these options are very ergonomic, we chose instead for DepURLs
to accept version range specifiers too with semantics that are a subset of
:pep:440 semantics. The allowed operators are those that are widely available
across package managers (e.g., ==, > and >= are common, while
~= isn’t).

Some examples:

  • dep:pypi/numpy@2.0: numpy pinned at exactly version 2.0.
  • dep:pypi/numpy@>=2.0: numpy with version greater or equal than 2.0.
  • dep:virtual/interface/lapack@>=3.7.1: any package implementing the
    LAPACK interface for version greater or equal than 3.7.1.

The versioning scheme for particular virtual packages, in case that isn’t
unambiguously defined by an upstream project or standard, will be defined in
the Central Registry (see :pep:804).

Environment markers
‘’‘’‘’‘’‘’‘’‘’‘’‘’’

Regular environment markers (as originally defined in :pep:508) may
be used behind DepURLs. PURL qualifiers, which use ? followed by a package
type-specific dependency specifier component, should not be used for the
purposes for which environment markers suffice. The reason for this is
pragmatic: environment markers are already used for other metadata in
pyproject.toml, hence any tooling that is used with pyproject.toml is
likely to already have a robust implementation to parse it. And we do not
expect to need the extra possibilities that PURL qualifiers provide (e.g., to
specify a Conan or conda channel, or a RubyGems platform).

We name the combination of a DepURL and environment markers as “external
dependency specifiers”, analogously to the existing dependency specifiers_.

Canonical names of dependencies and -dev(el) split packages
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’’

It is fairly common, but far from universal, for distros to split a package
into two or more packages. In particular, runtime components are often
separately installable from development components (headers, pkg-config and
CMake files, etc.). The latter then typically has a name with -dev or
-devel appended to the project/library name. Also, larger packages are
sometimes split into multiple separate packages to keep install sizes
manageable. More often than not, such package splits are not defined or
recognized by the maintainers of a package, and it’s therefore ambiguous what
any split would mean. Hence, such splits should not be reflected in the
[external] table. It is not possible to specify this in a reasonable way
that works across distros, hence only the canonical name should be used in
[external].

The intended meaning of using a DepURL is “the full package with the name
specified”. I.e., including all installable artifacts that are part of the
package. It will depend on the context in which the metadata is used whether a
package split is relevant. For example, if libffi is a host
dependency and a tool wants to prepare an environment for building a wheel,
then if a distro has split off the headers for libffi into a
libffi-devel package then the tool has to install both libffi and
libffi-devel.

For defining what canonical package names are and how package splits are
handled in practice when tools attempt to use [external] for installation
purposes, we refer to :pep:804.

Python development headers
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

Python headers and other build support files may also be split. This is the
same situation as in the section above (because Python is simply a regular
package in distros). However, a python-dev|devel dependency is special because
in pyproject.toml Python itself is an implicit rather than an explicit
dependency. Hence a choice needs to be made here - add python-dev implicitly,
or make each package author add it explicitly under [external]. For
consistency between Python dependencies and external dependencies, we choose to
add it implicitly. Python development headers must be assumed to be necessary
when an [external] table contains one or more compiler packages.

New Core Metadata fields

Two new Core Metadata fields are proposed:

  • Requires-External-Dep. An external requirement. Mimics the transition
    from Requires to Requires-Dist. We chose the -Dep suffix to
    emphasize that the value is not a regular Python specifier (distribution),
    but an external dependency specifier containing a DepURL.
  • Provides-External-Extra. An extra group that carries external dependencies
    (as found in Requires-External-Dep) only.

Since the Core Metadata specification does not contain fields for any metadata in
pyproject.toml’s [build-system] table, the build-requires
and host-requires content do not need to be reflected in existing core
metadata fields.

Additionally, this PEP also proposes to deprecate the Requires-External field.
The reasons being:

  • Avoiding confusion with the newly proposed fields.

  • Avoiding potential incompatibilities with existing usage (even if limited).

  • Low penetration in the ecosystem:

    • There is no direct correspondence to a field in the pyproject.toml metadata.
    • Mainstream build backends like setuptools (see pypa/setuptools#4220),
      hatch (see pypa/hatch#1712      ), flit (see pypa/flit#353), or poetry
      do not offer ways to specify it or require a plugin (e.g. poetry-external-dependencies      ).
      maturin does seem to support it since 0.7.0 (see PyO3/maturin@5b0e4808_),
      but it’s not directly documented. Other backends like scikit-build-core or
      meson-python returned no results for External-Requires.
    • The field is not included in the PyPI JSON API responses_.

Effect of vendoring shared libraries on wheel metadata
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

A wheel may vendor its external dependencies. This happens in particular when
distributing wheels on PyPI or other Python package indexes – and tools like
auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a
Requires-External-Dep entry in an sdist may disappear from a wheel built from
that sdist with a tool like cibuildwheel. It is also possible that a
Requires-External-Dep entry remains in a wheel, either unchanged or with
narrower constraints. auditwheel does not vendor certain allow-listed
dependencies, such as OpenGL, by default. In addition, auditwheel and
delvewheel allow a user to manually exclude dependencies via a
--exclude or --no-dll command-line flag. This is used to avoid
vendoring large shared libraries, for example those from CUDA.

Requires-External-Dep entries generated from external dependencies in
pyproject.toml can therefore differ between an sdist and its corresponding
wheel(s) depending on the build/distribution process.

Note that this does not imply that the field must be marked as Dynamic, since
this distinction only applies to wheels built from an sdist by a build backend.
In particular, wheels built from other wheels do not need to satisfy this
constraint.

Dependency groups

This PEP has chosen to include the :pep:735 key dependency-groups under
the [external] table too. This decision is motivated by the need of having
similar functionality for external metadata. The top-level table cannot be used
for external dependencies because it’s expected to have PEP 508 strings (and tables
for group includes), while we have chosen to rely on dep: URLs for the external
dependencies. Conflating both would raise significant backwards compatibility
issues with existing usage.

Strictly speaking, the dependency-groups schema allows us to define external
dependencies in per-group sub-tables::

[dependency-groups]
dev = [
“pytest”,
{ external = [“dep:cargo/ripgrep”] },
]

However, this has the same problem: we are mixing different types of dependency
specifiers in the same data structure. We believe it’s cleaner to separate concerns
in different top-level tables, hence why we still prefer to have
external.dependency-groups.

Optional dependencies versus dependency groups
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

The rationale for having external.dependency-groups is identical for the
rationale given in :pep:735 for introducing [dependency-groups]. The
intended usage and semantics of inclusion/exclusion into Core Metadata
is thus identical to [dependency-groups].

external.optional-dependencies will show up in Core Metadata.
external.dependency-groups will not.

Specification

If metadata is improperly specified then tools MUST raise an error to notify
the user about their mistake.

DepURL

A DepURL implements a scheme for identifying packages that is meant to be
portable across packaging ecosystems. Its design is::

dep:type/namespace/name@version?qualifiers#subpath

dep: is a fixed string, and always present. type and name are
required, other components are optional. All components apply for both PURL
and virtual type’s, and have these requirements:

  • type (required): MUST be either a PURL_ type, or virtual.
  • namespace (optional): MUST be a PURL_ namespace, or a namespace in
    the DepURL central registry (see :pep:804).
  • name (required): MUST be a name that parses as a valid PURL_ name.
    Tools MAY warn or error if a name is not present in the DepURL central
    registry (see :pep:804).
  • version (optional): MUST be a regular version specifier_ (PEP 440
    semantics) as a single version or version range, with the restriction that
    only the following operators may be used: >=, >, <, <=,
    ==, ,.
  • qualifiers (optional): MUST parse as a valid PURL_ qualifier.
  • subpath (optional): MUST parse as a valid PURL_ subpath.

External dependency specifiers

External dependency specifiers MUST contain a DepURL, and MAY contain
environment markers with the same syntax as used in regular dependency specifiers_ (as originally specified in :pep:508).

Changes in Core Metadata

Deprecations
‘’‘’‘’‘’‘’‘’

The External-Requires Core Metadata field will be marked as obsolete and its
usage will be discouraged.

Additions
‘’‘’‘’‘’’

Two new fields are added to Core Metadata:

  • Requires-External-Dep. An external requirement expressed as an external
    dependency specifier string.
  • Provides-External-Extra. An extra group that carries external dependencies
    (as found in Requires-External-Dep) only.

Version bump
‘’‘’‘’‘’‘’‘’

Given that the proposed changes are purely additive, the Core Metadata
version will be bumped to 2.6.

This will only impact PyPI and tools that want to support external runtime dependencies,
and require no changes otherwise.

Changes in pyproject.toml

Note that pyproject.toml content is in the same format as in :pep:621.

Table name
‘’‘’‘’‘’‘’

Tools MUST specify fields defined by this PEP in a table named [external].
No tools may add fields to this table which are not defined by this PEP or
subsequent PEPs. The lack of an [external] table means the package either
does not have any external dependencies, or the ones it does have are assumed
to be present on the system already.

build-requires/optional-build-requires
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

  • Format: Array of external dependency specifiers (build-requires) and a
    table with values of arrays of external dependency specifiers
    (optional-build-requires)
  • Core metadata_: N/A

The (optional) external build requirements needed to build the project.

For build-requires, it is a key whose value is an array of strings. Each
string represents a build requirement of the project and MUST be formatted as
a valid external dependency specifier.

For optional-build-requires, it is a table where each key specifies an
extra set of build requirements and whose value is an array of strings. The
strings of the arrays MUST be valid external dependency specifiers.

host-requires/optional-host-requires
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

  • Format: Array of external dependency specifiers (host-requires) and a
    table with values of arrays of external dependency specifiers
    (optional-host-requires) -
    Core metadata_: N/A

The (optional) external host requirements needed to build the project.

For host-requires, it is a key whose value is an array of strings. Each
string represents a host requirement of the project and MUST be formatted as
a valid external dependency specifier.

For optional-host-requires, it is a table where each key specifies an
extra set of host requirements and whose value is an array of strings. The
strings of the arrays MUST be valid external dependency specifiers.

dependencies/optional-dependencies
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

  • Format: Array of external dependency specifiers (dependencies) and a
    table with values of arrays of external dependency specifiers
    (optional-dependencies)
  • Core metadata_: Requires-External-Dep, Provides-External-Extra

The (optional) runtime dependencies of the project.

For dependencies, it is a key whose value is an array of strings. Each
string represents a dependency of the project and MUST be formatted as a valid
external dependency specifier. Each string must be added to Core Metadata_ as
a Requires-External-Dep field.

For optional-dependencies, it is a table where each key specifies an extra
and whose value is an array of strings. The strings of the arrays MUST be valid
external dependency specifiers. For each optional-dependencies group:

  • The name of the group MUST be added to Core Metadata_ as a
    Provides-External-Extra field.
  • The external dependency specifiers in that group MUST be added to Core Metadata_ as a Requires-External-Dep field, with the corresponding ; extra == 'name' environment marker.

dependency-groups
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’’

  • Format: A table where each key is the name of the group, and the values are
    arrays of external dependency specifiers, tables, or a mix of both.
  • Core metadata_: N/A

PEP 735 -style dependency groups, but using external dependency specifiers
instead of PEP 508 strings. Every other detail (e.g. group inclusion, name
normalization) follows the official dependency groups specification_.

Examples

These examples show what the [external] table content for a number of
packages, and the corresponding PKG-INFO/METADATA content (if any) is
expected to be.

cryptography 39.0
‘’‘’‘’‘’‘’‘’‘’‘’’

pyproject.toml content:

.. code:: toml

[external]
build-requires = [
  "dep:virtual/compiler/c",
  "dep:virtual/compiler/rust",
  "dep:generic/pkg-config",
]
host-requires = [
  "dep:generic/openssl",
  "dep:generic/libffi",
]

PKG-INFO / METADATA content: N/A.

SciPy 1.10
‘’‘’‘’‘’‘’

pyproject.toml content:

.. code:: toml

[external]
build-requires = [
  "dep:virtual/compiler/c",
  "dep:virtual/compiler/cpp",
  "dep:virtual/compiler/fortran",
  "dep:generic/ninja",
  "dep:generic/pkg-config",
]
host-requires = [
  "dep:virtual/interface/blas",
  "dep:virtual/interface/lapack@>=3.7.1",
]

PKG-INFO / METADATA content: N/A.

Pillow 10.1.0
‘’‘’‘’‘’‘’‘’’

pyproject.toml content:

.. code:: toml

[external]
build-requires = [
  "dep:virtual/compiler/c",
]
host-requires = [
  "dep:generic/libjpeg",
  "dep:generic/zlib",
]

[external.optional-host-requires]
extra = [
  "dep:generic/lcms2",
  "dep:generic/freetype",
  "dep:generic/libimagequant",
  "dep:generic/libraqm",
  "dep:generic/libtiff",
  "dep:generic/libxcb",
  "dep:generic/libwebp",
  "dep:generic/openjpeg@>=2.0",
  "dep:generic/tk",
]

PKG-INFO / METADATA content: N/A.

NAVis 1.4.0
‘’‘’‘’‘’‘’’

pyproject.toml content:

.. code:: toml

[project.optional-dependencies]
r = ["rpy2"]

[external]
build-requires = [
  "dep:generic/XCB; platform_system=='Linux'",
]

[external.optional-dependencies]
nat = [
  "dep:cran/nat",
  "dep:cran/nat.nblast",
]

PKG-INFO / METADATA content:

.. code::

Provides-External-Extra: nat
Requires-External-Dep: dep:cran/nat; extra == ‘nat’
Requires-External-Dep: dep:cran/nat.nblast; extra == ‘nat’

Spyder 6.0
‘’‘’‘’‘’‘’

pyproject.toml content:

.. code:: toml

[external]
dependencies = [
  "dep:cargo/ripgrep",
  "dep:cargo/tree-sitter-cli",
  "dep:golang/github.com/junegunn/fzf",
]

PKG-INFO / METADATA content:

.. code::

Requires-External-Dep: dep:cargo/ripgrep
Requires-External-Dep: dep:cargo/tree-sitter-cli
Requires-External-Dep: dep:golang/github.com/junegunn/fzf

jupyterlab-git 0.41.0
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’’

pyproject.toml content:

.. code:: toml

[external]
dependencies = [
  "dep:generic/git",
]

[external.optional-build-requires]
dev = [
  "dep:generic/nodejs",
]

PKG-INFO / METADATA content:

.. code::

Requires-External-Dep: dep:generic/git

PyEnchant 3.2.2
‘’‘’‘’‘’‘’‘’‘’’

pyproject.toml content:

.. code:: toml

[external]
dependencies = [
  # libenchant is needed on all platforms but vendored into wheels
  # distributed on PyPI for Windows. Hence choose to encode that in
  # the metadata. Note: there is no completely unambiguous way to do
  # this; another choice is to leave out the environment marker in the
  # source distribution and either live with the unnecessary ``METADATA``
  # entry in the distributed Windows wheels, or to apply a patch to this
  # metadata when building those wheels.
  "dep:github/AbiWord/enchant; platform_system!='Windows'",
]

PKG-INFO / METADATA content:

.. code::

Requires-External-Dep: dep:github/AbiWord/enchant; platform_system!=“Windows”

With dependency groups
‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’‘’

pyproject.toml content:

.. code:: toml

[external.dependency-groups]
dev = [
  "dep:generic/catch2",
  "dep:generic/valgrind",
]

PKG-INFO / METADATA content: N/A.

Backwards Compatibility

There is no impact on backwards compatibility, as this PEP only adds new,
optional metadata. In the absence of such metadata, nothing changes for package
authors or packaging tooling.

The only change introduced in this PEP that has impact on existing projects is the
deprecation of the External-Requires Core Metadata field. We estimate the impact
of this deprecation to be negligible, given the its low penetration in the ecosystem
(see Rationale).

The field will still be recognized by existing tools such as setuptools-ext_
but its usage will be discouraged in the Python Packaging User Guide_, similar to
what is done for obsolete fields like Requires (deprecated in favor of
Requires-Dist).

Security Implications

There are no direct security concerns as this PEP covers how to statically
define metadata for external dependencies. Any security issues would stem from
how tools consume the metadata and choose to act upon it.

How to Teach This

External dependencies and if and how those external dependencies are vendored
are topics that are typically not understood in detail by Python package
authors. We intend to start from how an external dependency is defined, the
different ways it can be depended on—from runtime-only with ctypes or a
subprocess call to it being a build dependency that’s linked against—
before going into how to declare external dependencies in metadata. The
documentation should make explicit what is relevant for package authors, and
what for distro packagers.

Material on this topic will be added to the most relevant packaging tutorials,
primarily the Python Packaging User Guide_. In addition, we expect that any
build backend that adds support for external dependencies metadata will include
information about that in its documentation, as will tools like auditwheel.

Reference Implementation

This PEP contains a metadata specification, rather that a code feature - hence
there will not be code implementing the metadata spec as a whole. However,
there are parts that do have a reference implementation:

  1. The [external] table has to be valid TOML and therefore can be loaded
    with tomllib. This table can be further processed with the
    pyproject-external_ package, demonstrated below.
  2. The PURL specification, as a key part of this spec, has a Python package
    with a reference implementation for constructing and parsing PURLs:
    packageurl-python. This package is wrapped in pyproject-external
    to provide DepURL-specific validation and handling.

There are multiple possible consumers and use cases of this metadata, once
that metadata gets added to Python packages. Tested metadata for all of the
top 150 most-downloaded packages from PyPI with published platform-specific
wheels can be found in rgommers/external-deps-build_. This metadata has
been validated by using it to build wheels from sdists patched with that
metadata in clean Docker containers.

Example

Given a pyproject.toml with this [external] table:

.. code-block:: toml

[external]
build-requires = [
“dep:virtual/compiler/c”,
“dep:virtual/compiler/rust”,
“dep:generic/pkg-config”,
]
host-requires = [
“dep:generic/openssl”,
“dep:generic/libffi”,
]

You can use pyproject_external.External to parse it and manipulate it:

.. code-block:: python

from pyproject_external import External
external = External.from_pyproject_path(“./pyproject.toml”)
external.validate()
external.to_dict()
{‘external’: {‘build_requires’: [‘dep:virtual/compiler/c’, ‘dep:virtual/compiler/rust’, ‘dep:generic/pkg-config’], ‘host_requires’: [‘dep:generic/openssl’, ‘dep:generic/libffi’]}}
external.build_requires
[DepURL(type=‘virtual’, namespace=‘compiler’, name=‘c’, version=None, qualifiers={}, subpath=None), DepURL(type=‘virtual’, namespace=‘compiler’, name=‘rust’, version=None, qualifiers={}, subpath=None), DepURL(type=‘generic’, namespace=None, name=‘pkg-config’, version=None, qualifiers={}, subpath=None)]
external.build_requires[0]
DepURL(type=‘virtual’, namespace=‘compiler’, name=‘c’, version=None, qualifiers={}, subpath=None)

Note the proposed [external] table was well-formed. With invalid contents such as:

.. code-block:: toml

[external]
build-requires = [
“dep:this-is-missing-the-type”,
“pkg:not-a-dep-url”
]

You would fail the validation:

.. code-block:: python

external = External.from_pyproject_data(
{
“external”: {
“build_requires”: [
“dep:this-is-missing-the-type”,
“pkg:not-a-dep-url”
]
}
}
)
ValueError: purl is missing the required type component: ‘dep:this-is-missing-the-type’.

Rejected Ideas

Specific syntax for external dependencies which are also packaged on PyPI

There are non-Python packages which are packaged on PyPI, such as Ninja,
patchelf and CMake. What is typically desired is to use the system version of
those, and if it’s not present on the system then install the PyPI package for
it. The authors believe that specific support for this scenario is not
necessary (or at least, too complex to justify such support); a dependency
provider for external dependencies can treat PyPI as one possible source for
obtaining the package. An example mapping for this use case is proposed in
:pep:804.

Using library and header names as external dependencies

A previous draft PEP ("External dependencies" (2015) <https://github.com/pypa/interoperability-peps/pull/30>__)
proposed using specific library and header names as external dependencies. This
is both too granular, and insufficient (e.g., headers are often unversioned;
multiple packages may provide the same header or library). Using package names
is a well-established pattern across packaging ecosystems and should be
preferred.

Splitting host dependencies with explicit -dev or -devel suffixes

This convention is not consistent across packaging ecosystems, nor commonly
accepted by upstream package authors. Since the need for explicit control
(e.g., installing headers when a package is used as a runtime rather than a
build-time dependency) is quite niche and we don’t want to add design
complexity without enough clear use cases, we have chosen to rely solely on the
build, host and run category split, with tools being in charge of
which category applies to each case in a context-dependent way.

If this proves to be insufficient, a future PEP could use the URL qualifier
features present in the PURL schema (?key=value) to implement the necessary
adjustments. This can be done in a backwards compatible fashion.

Identifier indirections

Some ecosystems exhibit methods to select packages based on parametrized
functions like cmake("dependency") or compiler("language"), which
return package names based on some additional context or configuration. This
feature is arguably not very common and, even when present, rarely used.
Additionally, its dynamic nature makes it prone to changing meaning over time,
and relying on specific build systems for the name resolution is in general not
a good idea.

The authors prefer static identifiers that can be mapped explicitly via well
known metadata (e.g., as proposed in :pep:804).

Ecosystems that do implement these indirections can use them to support the
infrastructure designed to generate the mappings proposed in :pep:804.

Adding a host-requires key under [build-system]

Adding host-requires for host dependencies that are on PyPI in order to
better support name mapping to other packaging systems with support for
cross-compiling seems useful in principle, for the same reasons as this PEP
adds a host-requires under the [external] table. However, it isn’t
necessary to include in this PEP, and hence the authors prefer to keep the
scope of this PEP limited - a future PEP on cross compilation may want to
tackle this. This issue <https://github.com/rgommers/peps/issues/6>__
contains more arguments in favor and against adding host-requires under
[build-system] as part of this PEP.

Reusing the Requires-External field in Core Metadata

The Core Metadata_ specification contains one relevant field, namely
Requires-External. While at first sight it would be a good candidate to
record the external.dependencies table, the authors have decided to not
re-use this field to propagate the external runtime dependencies metadata.

The Requires-External field has very loosely defined semantics as of
version 2.4. Essentially: name [(version)][; environment marker] (with
square brackets denoting optional fields). It is not defined what valid strings
for name are; the example in the specification uses both “C” as a language
name, and “libpng” as a package name. Tightening up the semantics would be
backwards incompatible, and leaving it as is seems unsatisfactory. DepURLs
would need to be decomposed to fit in this syntax.

Allowing use of ecosystem-specific version comparison semantics

There are cases, in particular when dealing with pre-releases, where PEP 440
semantics for version comparisons don’t quite work. For example, 1.2.3a may
indicate a release subsequent to 1.2.3 rather than an alpha version. To
handle such cases correctly, it would be necessary to allow arbitrary
versioning schemes. The authors of this PEP consider the added value of
allowing that is not justified by the additional complexity. If desired, a
package author can use either a code comment or the qualifier field of a
DepURL (see the Versioning section under Rationale) to capture this level of
detail.

Open Issues

None at this time.

References

.. [#singular-vision-native-deps] The “define native requirements metadata”
part of the “Wanting a singular packaging vision” thread (2022, Discourse):
Wanting a singular packaging tool/vision - #92 by steve.dower

.. [#pypacking-native-deps] pypackaging-native: “Native dependencies”
Native dependencies - pypackaging-native

.. [#gcc-cross-terminology] GCC documentation - Configure Terms and History,
Configure Terms (GNU Compiler Collection (GCC) Internals)

.. [#meson-cross] Meson documentation - Cross compilation
Cross compilation

.. [#pypackaging-native-cross] pypackaging-native: “Cross compilation”
Cross compilation - pypackaging-native

.. [#pkgconfig-and-ctypes-findlibrary] The “pkgconfig specification as an
alternative to ctypes.util.find_library” thread (2023, Discourse):
`pkgconfig` specification as an alternative to `ctypes.util.find_library`

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

.. _PyPI: https://pypi.org
.. _Core Metadata: Core metadata specifications - Python Packaging User Guide
.. _setuptools: https://setuptools.readthedocs.io/
.. _setuptools metadata: Building and Distributing Packages with Setuptools - setuptools 80.9.0 documentation
.. _SPDX: https://spdx.dev/
.. _PURL: GitHub - package-url/purl-spec: A minimal specification for purl aka. a package "mostly universal" URL, join the discussion at https://gitter.im/package-url/Lobby
.. _version specifier: Version specifiers - Python Packaging User Guide
.. _dependency specifiers: Dependency specifiers - Python Packaging User Guide
.. _dependency groups specification: Dependency Groups - Python Packaging User Guide
.. _packageurl-python: Client Challenge
.. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst
.. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139
.. _pyp2spec: GitHub - befeleme/pyp2spec: Generate Fedora RPM spec files for Python packages
.. _Grayskull: GitHub - conda/grayskull: Grayskull - Recipe generator for Conda
.. _dh_python: Debian Python Policy 0.12.0.0 documentation
.. _Repology: https://repology.org/
.. _Dependabot: Dependabot · GitHub
.. _libraries.io: https://libraries.io/
.. _crossenv: GitHub - robotpy/crossenv: Cross-compiling virtualenv for Python
.. _Python Packaging User Guide: https://packaging.python.org
.. _auditwheel: GitHub - pypa/auditwheel: Auditing and relabeling cross-distribution Linux wheels.
.. _delocate: GitHub - matthew-brett/delocate: Find and copy needed dynamic libraries into python wheels
.. _delvewheel: GitHub - adang1345/delvewheel: Self-contained Python wheels for Windows
.. _verspurl: How to use `vers:` with `pkg:`? · Issue #386 · package-url/purl-spec · GitHub
.. _rgommers/external-deps-build: GitHub - rgommers/external-deps-build
.. _pyproject-external: GitHub - jaimergp/pyproject-external: Utilities to work with PEP 725 `[external]` metadata
.. _Reference LAPACK: GitHub - Reference-LAPACK/lapack: LAPACK development repository
.. _setuptools-ext: setuptools-ext · PyPI
.. _PyPI JSON API responses: JSON API - PyPI Docs
.. _pypa/hatch#1712: Specifying external dependencies · Issue #1712 · pypa/hatch · GitHub
.. _pypa/flit#353: Ability to write Requires-External metadata · Issue #353 · pypa/flit · GitHub
.. _pypa/setuptools#4220: use of setup_keywords and pyproject.toml · pypa/setuptools · Discussion #4220 · GitHub
.. _poetry-external-dependencies: Client Challenge
.. _PyO3/maturin@5b0e4808: Add full python core metadata support; fixes #159 · PyO3/maturin@5b0e480 · GitHub
.. _elfdeps: GitHub - python-wheel-build/elfdeps: Python implementation of RPM elfdeps

12 Likes
2

I’ve only skimmed the new PEP so far, but can I just confirm one point? If I’m reading this right, the proposal is to add purely informational metadata - there’s no requirement on any tools around consuming this data. Is that correct?

In particular, I’m assuming that installers like pip are not being required to check that external dependencies are present in any way.

2 Likes
3

I suppose one possible strategy for pip is to catch any build errors and then display the external dependencies in the synthesized error message.

4

I would say that up to the installers / users to configure.

I have an interest in installers supporting installing external depenencies, see also: Support for PEP 725 - external (runtime) dependencies with user defineable helper · Issue #14889 · astral-sh/uv · GitHub

I would say that installing is most likely impossible to be done by installers like pip.

But they could maybe inform users about the requirements.

For sdist installations they could be only outputted in case of an error:

I.e. Failed to install pkg-abc, are you sure, you installed all required pkgs: buildkit-c…

When installing wheel they could summarize that the installed packages assume the following pkgs to be installed.

Check whenever they are really installed will be most probably too hard for tools like pip without user configuration.

5

Sorry, I didn’t mention that, but yes of course it would be useful for pip to use the external dependency information to improve the message the user gets when a build fails. We might even be able to do something like add a flag that says “only use sdists if they don’t have any external dependencies”.

My point wasn’t whether tools like pip could use the new information, but rather that there’s nothing in the PEP that requires us to do so. Or, to put it another way, if the PEP gets accepted, there’s no change needed in pip to be compliant with the new standard[1].

  1. I’m looking at it from a pip maintainer’s point of view - “how much work does this PEP require of us?” ↩︎

1 Like
6

Most of the features here read as entirely opt-in by tools and packages, so I don’t have major concerns there.

The scope and expression of this looks at initial read to be a lot better than prior attempts.

One thing I noticed is that there’s a potential future conflict in how depurls are specified. Namely, if virtual ever becomes a valid PURL type for some reason[1]. It may be worth precluding this in some way by having a top level type be purl or virtual, and then have the remaining parts be based on which of these the type is.

  1. Much as it is needed for the python ecosystem, I could see PURL also wanting to support virtual abis and arbitrary providers of them in the future, and this ending up a type for the same reason. If implemented with identical or close enough semantics, it might not be a problem, but I’d rather be cautious here in terms of future proofing this. ↩︎

1 Like
7

IME there’s strong pushback whenever we’ve tried to use the PEP process to pin down any specifics around installer’s (default?) UX. Please don’t tell me that not doing that will also get pushback. :upside_down_face:

Yes. I really hope that’s not what we end up doing, since this information is useful when there’s a failure. It’s an explicit motivation of this PEP to enable installers to provide more informative error messages:

Make it possible to include needed dependencies in error messages emitted by Python package installers and build frontends

6 Likes
8

Not at all. But there is a legitimate case for mandating tool behaviours when it’s appropriate (for interoperability, for example) and I was just wanting to confirm that there were no such cases here (i.e., that I hadn’t missed any as a result of only skimming the PEP quickly).

I agree. But given pip’s chronic lack of maintainer resource, I can see it taking a while before we get something like that.

With my PEP delegate hat on, my concern is that the PEP doesn’t get approved and then end up “in limbo” because key tools don’t support it yet. But the “key tools” here are build backends, so I’ll let them speak on their own behalf.

2 Likes
9

The PEP discusses the auditwheel-style vendoring case and says:

Requires-External-Dep entries generated from external dependencies in pyproject.toml can therefore differ between an sdist and its corresponding wheel(s) depending on the build/distribution process.

Note that this does not imply that the field must be marked as Dynamic, since this distinction only applies to wheels built from an sdist by a build backend. In particular, wheels built from other wheels do not need to satisfy this constraint.

How would this work in cases where the build backend does vendor/link things into the wheels (perhaps conditionally)?

10

In that case, because the wheel in question is being built from the sdist, it would have to be marked as dynamic (in both the sdist and in pyproject.toml).

11

But what is the “it” here that should be marked as dynamic?

Let’s say the package has a build-time requirement for (a host build of) some shared library and then the build backend bundles that library in.

To me it seems like that would be a host-requires situation so you would have e.g.

[external]
build-requires = [
  "dep:virtual/compiler/c",
]
host-requires = [
  "dep:generic/libgmp",
]

The PEP says that host-requires may or may not be a runtime requirement:

When host dependencies imply a runtime dependency, that runtime dependency also does not have to be declared, just like for build dependencies.

This leaves us in an ambiguous situation where we don’t know if there is an external requirement for the library that may or may not have been bundled by the build backend.

What would you mark as dynamic in this case?

The host-requires and build-requires keys are both describing build-time requirements that are irrelevant after you have built a wheel so what does it mean for those to be dynamic?

1 Like
12

Ah, I see your point. Basically, host-requires is (may be?) irrelevant in a wheel, because it’s potentially just a build time requirement.

I’m not sure I like the idea of having this sort of “build or runtime or maybe both” requirement. I’ll have to read the PEP in detail to understand the justification for this, because like you say this feels like an ambiguous situation.

I think this is a whole new area. We’ve never[1] yet had metadata that’s build time only. So there’s no precedent here, and I don’t think either the dynamic field in pyproject.toml or the core metadata Dynamic field support this idea.

IMO, if host-requires and build-requires only affect the build step, then they should never be present in a wheel. Which means (based on current definitions) they must always be marked as Dynamic in the sdist core metadata (because that’s the only way they can be present in the sdist but not in the wheel). On the other hand, making them dynamic in pyproject.toml makes no sense, as the build backend can’t calculate a value to put into the sdist, because it needs to know the value to do the build in the first place. Doesn’t it? I’m getting very confused now.

PEP authors, can you please provide an example of how you intend the build-requires key would be used in practice? Something simple like “the build needs a C compiler” should be sufficient.

  1. as far as I can recall ↩︎

1 Like
13

Is there not metadata for the non-external build requirements?

14

(Apologies in advance, I have my sceptical hat on)

How accurate do you expect these to be? How far have you stress tested this across distributions and packages?

I’ve been doing $distro equivalent of $dependency mappings for some years now and I’m always seeing just how numerous and how much deeper the asymmetries between distributions go beyond just different naming conventions. So I’m automatically very sceptical of portable across packaging ecosystems claims or indeed anything less explicit than each Python project providing an exact {"distro": ["dependency1", "dependency2"]} mapping actually working – even if the virtual → concrete package mapping is hand curated.

For example, what does dep:virtual/compiler/c mean on Alpine? Literally, it would mean gcc but if you take that as-is then that’s even a hello world C extension failing to build on Alpine because you also need the libc’s development headers (musl-dev).

And there’s plenty worse out there! I’ve just popped open the last couple of projects I did and I’m reading them out:

  • Is netcat the BSD or GNU implementation of netcat?
  • Is 7zip just 7zip or all or some of its plugins?
  • Is rsvg-convert (SVG support for ImageMagik) the deprecated convert command, the modern but incompatible and not widely avaiable magick command, the SVG support plugin for magick or an alias for whatever SVG-support-providing library magick is compiled against but not shipped with?[1] Also, did you know that OpenSUSE’s flavour of ImageMagick configures SVG support disabled to reduce security surface area and requires that you tweak said config before it’ll let you past a very confusing error message? Getting the right dependency doesn’t really get you away from cross distribution testing.
  • Have you tried Void Linux? Almost anything you can think of has multiple implementations available there.
  • Don’t talk to me about ffmpeg codecs. [2]

With that in mind, I expect virtual packages to be more of a trapping. Truly getting them right will still require a developer to iterate through a list of Docker images and if they’re going to do that, they might as well stick to concrete dependencies where at least fixing one distribution can’t break another.

But, correct me if I’m wrong, we can’t use the concrete packages because we have no distribution markers? i.e. I can’t do pkg:apk/alpine/muls-dev; platform_linux_distro == 'alpine'? Or are consumers already expected to ignore dependencies that aren’t meant for them?

  1. Getting that one wrong will likely leave you with only half the dependency. ↩︎

  2. There’s legal baggage involved ↩︎

2 Likes
15

I’ve been doing $distro equivalent of $dependency mappings for some years now and I’m always seeing just how numerous and how much deeper the asymmetries between distributions go beyond just different naming conventions. So I’m automatically very sceptical of portable across packaging ecosystems claims or indeed anything less explicit than each Python project providing an exact {"distro": ["dependency1", "dependency2"]} mapping actually working – even if the virtual → concrete package mapping is hand curated.

I’m one of the maintainers of bindep (mentioned in the prior art section of either this or 804, also for diskimage-builder’s pkg-map), and can attest that this is even a challenge just between versions of the same Linux distribution. Packages get renamed or replaced over time, and so maintaining equivalencies between them when you want to be able to build on, e.g., multiple Ubuntu LTS releases is a challenge. But yes, doing it between different distros or distro types gets harder still.

That said, if this gets solidified and ends up covering some of the same use cases, I’ll eagerly implement support in bindep to read pyproject.toml files or package metadata as an additional source of dependency info.

4 Likes
16

I would consider build-system.build-backend and build-system.requires (etc) as build time only metadata. They’re stored in a different file, so perhaps that’s the solution.

1 Like
17

From the PEP, this is a cross-compilation distinction.

build-requires and host-requires are both build requirements.

While I know host is the conventional term for cross compilation build targets, it has always seemed hopelessly ambiguous to me, since it could mean either “build host” or “runtime host”.

My suggestion would be that we rename the host-requires field as build-target-requires.

For native builds, the two fields are equivalent, for cross compilation, build-requires would follow the build machine architecture, while build-target-requires would follow the runtime machine architecture.

I don’t think either field should imply runtime external dependencies. If the external dep is always needed, list it in dependencies. If it may or may not be needed based on the build options, mark dependencies as dynamic.

Edit: having finished reading the full PEP, the “host” naming is my only real concern. It’s the most annoying part of cross compilation naming conventions, and I really hope we can find a reasonable way to avoid inflicting it on a wider audience.

6 Likes
18

Yes, that is it indeed - with that “different file” being only pyproject.toml.

Apologies for the confusion, that motivation section that mentions auditwheel is misleading. I tried to write that as similarly as possible to the recent decision on auditwheel-like tools for PEP 639 (ref), but it was meant to apply only to external.dependencies|optional-dependencies. build-requires/host-requires are not included in Core Metadata at all. Please see:

In practice there are, AFAIK, no commonly used tools that are able to vendor pure runtime-only dependencies. So the need for Dynamic usage in core metadata should be extremely rare. Hence, we did not include dynamic as a top-level key under [external]; there’s no normal way to achieve this with a build backend aside from some custom scripting, so why ask every build backend to deal with the possibility of this?

To further clarify: the requirements here as understood for wheels today do NOT change. You must use auditwheel et al. to create redistributable wheels, and that will not alter Core Metedata content in any way. This should also be clear from reading the examples, which show pyproject.toml as well as resuting PKG-INFO/METADATA content.

1 Like
19

Can I answer this by linking to my previous answer from round 1 on building the 37 out of the 150 most downloaded packages from PyPI that need compilers? The updated results, which we regularly run as CI jobs, together with a detailed explanation of methodology, is maintained at github.com/rgommerrs/external-deps-build.

I expect the results of going from 8/37 to 35/37 packages successfully built from source in clean containers on 3 distros to be at least representative of real-world success percentages. Some of those packages are really hard; there are harder ones around (like PyTorch) but I think the long tail of packages is mostly significantly simpler than this list of packages.

Agreed, you need that detailed per-distro mapping somewhere. Hence, PEP 804 creates a generic mechanism for that.

[quote=“Brénainn Woodsend, post:14, topic:103890, username:bwoodsend”]
With that in mind, I expect virtual packages to be more of a trapping. Truly getting them right will still require a developer to iterate through a list of Docker images
[/quote]

It shouldn’t. The PEP 725 metadata for these examples in pyproject.toml should not change at all. Something like compiler/c needing musl-dev is handled in the PEP 804 mapping for Alpine, and that’s probably one of the first thing every Alpine user needs so it’ll be implemented as soon as there is an Alpine mapping - a Python package author is very unlikely to see this problem.

Whether that means that, in a future where a fully automated build tool that understands all this, everything is always 100% correctly implemented - probably not for niche distros. But that should not affect Python package authors, at least not more than today. The idea is that the distro maintainer + users on that distro can improve their distro-specific mapping without the package maintainers having to be in the loop.

Could you please read PEP 804 and comment on its discussion thread? I think it provides a clear way to answer all this, possibly with the exception of package renames in a distro with the old package not remaining an alias to the new package.

5 Likes
20

I a bit on the fence about not using standard cross compilation terminology but instead inventing some Python packaging specific term. It could indeed be a bit clearer for Python devs not familiar with cross compilation yet, but on the other hand it’s a lost opportunity to educate them. As well as make things more familiar to distro packagers who aren’t deep into Python packaging.

That said, it’s easy enough to change so if enough people like the terminology you propose, let’s do that.

4 Likes