PEP 804: An external dependency registry and name mapping mechanism

jaimergp · September 22, 2025, 2:53pm

Hello everyone!

On behalf of my co-authors @pradyunsg, @rgommers, @mgorny and @msarahan, I’d like to share our work on this new PEP 804 “An external dependency registry and name mapping mechanism”.

This PEP is meant to complement PEP 725 by adding a central registry of known identifiers and a mapping mechanism to translate the ecosystem-agnostic identifiers to names known to target package managers.

From previous rounds of PEP 725 community reviews, it was clear that a significant number of people wanted a way to ensure external dependency specifiers have an agreed-upon canonical form and recommended (possibly enforceable) ways of spelling them for each package. This PEP will allow implementing that.

To that end, we propose a series of schemas and a few examples in external-metadata-mappings, which can also be browsed in the accompanying Streamlit app.

A CLI and Python API is provided by the package pyproject-external, which allows you to perform tasks like “Print and validate the [external] table in this PEP-725-ready package” or “give me the install command to provide the external dependencies I need to build this project”. You can see it in action in external-deps-build’s CI.

Please feel free to check the rendered version.

Full PEP text:

PEP: 804
Title: An external dependency registry and name mapping mechanism
Author: Pradyun Gedam pradyunsg@gmail.com,
Ralf Gommers ralf.gommers@gmail.com,
Michał Górny mgorny@quansight.com,
Jaime Rodríguez-Guerra jaime.rogue@gmail.com,
Michael Sarahan msarahan@gmail.com
Discussions-To: Pending
Status: Draft
Type: Standards Track
Topic: Packaging
Requires: 725
Created: 03-Sep-2025
Post-History: 03-Sep-2025,

Abstract

This PEP specifies a name mapping mechanism that allows packaging tools to map
external dependency identifiers (as introduced in :pep:725) to their
counterparts in other package repositories.

Motivation

Packages on PyPI often require build-time and runtime dependencies that are not
present on PyPI. :pep:725 introduced metadata to express
such dependencies. Using concrete external dependency metadata for
a Python package requires mapping the given dependency identifiers to the specifiers
used in other ecosystems, which would allow to:

Enabling tools to automatically map external dependencies to packages in other
packaging repositories/ecosystems,
Including the needed external dependencies with the package
names used by the relevant system package manager on the user’s system in
error messages emitted by Python package installers and build frontends,
as well as allowing the user to query for those names directly to obtain install
instructions.

Packaging ecosystems like Linux distros, conda, Homebrew, Spack, and Nix need
full sets of dependencies for Python packages, and have tools like pyp2rpm_
(Fedora), Grayskull_ (conda), and dh_python_ (Debian) which attempt to
automatically generate dependency information from the metadata available in
upstream Python packages. Before PEP 725, external dependencies were handled manually,
because there was no metadata for this in pyproject.toml or any other
standard metadata file. Enabling its automatic conversion is a key benefit of
this PEP, making Python packaging easier and more reliable. In addition, the
authors envision other types of tools making use of this information; e.g.,
dependency analysis tools like Repology_, Dependabot_ and libraries.io_.

Rationale

Prior art

The R language has a System Requirements for R packages < GitHub - rstudio/r-system-requirements: System requirements for R packages >__ with a central
registry that knows how to translate external dependency metadata to install
commands for package managers like apt-get. This registry centralises the
mappings for a series of Linux distributions, and also Windows. macOS is not
present. The "Rule Coverage" of its README < r-system-requirements/README.md at 7314012a48d38854c19f439e1c2d2e4b383fe7ea · rstudio/r-system-requirements · GitHub >__
used to show that this system improves the chance of success of building packages
from CRAN from source. Across all CRAN packages,
Ubuntu 18 improved from 78.1% to 95.8%, CentOS 7 from 77.8% to 93.7% and openSUSE
15.0 from 78.2% to 89.7%. The chance of success depends on how well the registry
is maintained, but the gain is significant: ~4x fewer packages fail to build on
Ubuntu and CentOS in a Docker container.

RPM-based distributions, like Fedora, can use a rule-based implementation < Wanting a singular packaging tool/vision - #117 by encukou >__
(NameConvertor) in pyp2rpm_. The main rule is that the RPM name for a PyPI package is
f"python-{pypi_package_name}". This seems to work quite well; there are a
few variants like Python version specific names, where the prefix contains the
Python major and minor version numbers (e.g. python311- instead of
python-).

Gentoo follows a similar approach to naming Python packages, using the dev-python/
category and some well-specified rules <``https://projects.gentoo.org/python/guide/package-maintenance.html``>__.

Conda-forge has a more explicit name mapping, because the base names are the
same in conda-forge as on PyPI (e.g., numpy maps to numpy), but there
are many exceptions because of both name collisions and renames (e.g., the PyPI
name for PyTorch is torch while in conda-forge it’s pytorch). There are
several name mappings efforts maintained by different teams. Conda-forge’s infrastructure
generates one in regro/cf-graph-countyfair < cf-graph-countyfair/mappings/pypi at master · regro/cf-graph-countyfair · GitHub >.
Grayskull maintains its own curated mapping < grayskull/grayskull/strategy/config.yaml at main · conda/grayskull · GitHub >.
Prefix.dev created the parselmouth mappings < GitHub - prefix-dev/parselmouth >__
to support conda and PyPI integrations in their tooling. A more complete overview of
their approaches, strengths and weaknesses can be found in
conda/grayskull#564 < Tame the PyPI / Conda mapping chaos · Issue #564 · conda/grayskull · GitHub >__.

The OpenStack <``https://www.openstack.org/``>__ ecosystem also needs to deal with
some mapping efforts. All of them focus on Linux distributions, exclusively.
pkg-map < pkg-map — diskimage-builder 3.39.1.dev22 documentation >__
accompanies diskimage-builder and provides a file format where the user defines
arbitrary variable names and their corresponding names in the target distro
(Red Hat, Debian, OpenSUSE, etc). See example for PyYAML < diskimage-builder/diskimage_builder/elements/svc-map/pkg-map at 5bc5f8aff3b40b1918ce72660f1dba38c3c4f27a · stbenjam/diskimage-builder · GitHub >.
bindep <https://opendev.org/opendev/bindep> defines a file bindep.txt
(see example < bindep/bindep.txt at master - bindep - OpenDev: Free Software Needs Free Tools >__)
where users can write down dependencies that are not installable from PyPI. The format is
line-based, with each line containing a dependency as found in the Debian ecosystem.
For other distributions, it offers a “filters” syntax between square brackets where users
can indicate other target platforms, optional dependencies and extras.

The need for mappings is also found in other ecosystems like SageMath < `sage --package create --pypi`: Fill `distros/` · Issue #36356 · sagemath/sage · GitHub >,
but also by end-users themselves who want to install PyPI packages with their system
package manager of choice (example StackOverflow question <``https://unix.stackexchange.com/q/761371``>).

Governance and maintenance costs of name mappings

The maintenance cost of external dependency mappings to a large number of packaging
ecosystems is potentially high. We choose to define the registry in such
a way that:

A central authority maintains the list of recognized DepURLs and the
known ecosystem mappings.
The mappings themselves are maintained by the target packaging ecosystems.

Hence this system is opt-in for a given ecosystem, and the associated
maintenance costs are distributed.

Generating package manager-specific install commands

Python package authors with external dependencies usually have installation
instructions for those external dependencies in their documentation. These
instructions are difficult to write and keep up-to-date, and are usually only
covering one or at most a handful of platforms. As an example, here are SciPy’s
instructions for its external build dependencies (C/C++/Fortran compilers,
OpenBLAS, pkg-config):

Debian/Ubuntu: sudo apt install -y gcc g++ gfortran libopenblas-dev liblapack-dev pkg-config python3-pip python3-dev
Fedora: sudo dnf install gcc-gfortran python3-devel openblas-devel lapack-devel pkgconfig
CentOS/RHEL: sudo yum install gcc-gfortran python3-devel openblas-devel lapack-devel pkgconfig
Arch Linux: sudo pacman -S gcc-fortran openblas pkgconf
Homebrew on macOS: brew install gfortran openblas pkg-config

The package names vary a lot, and there are differences like some distros
splitting off headers and other build-time dependencies in a separate
-dev/-devel package while others do not. With the registry in this PEP,
this could be made both more comprehensive and easier to maintain through a tool
command with semantics of “show this ecosystem’s preferred package manager
install command for all external dependencies”. This may be done as a
standalone tool, or as a new subcommand in any Python development workflow tool
(e.g., Pip, Poetry, Hatch, PDM, uv).

To this end, each ecosystem mapping can provide a list of package managers
known to be compatible, with templated instructions on how to install and query
packages. The provided install command templates are paired with query command templates
so those tools can check whether the needed packages are already present without
having to attempt an install operation (which might be expensive and have unintended
side effects like version upgrades).

Registry design

The mapping infrastructure has been designed to present the following components and properties:

A central registry of PEP 725 identifiers (DepURLs), including at least the
well-known generic and virtual identifiers considered canonical.
A list of known ecosystems, where ecosystem maintainers can register their name mapping(s).
A standardized schema that defines how mappings should be structured. Each mapping can
also provide programmatic details about how their supported package manager(s) work.

The above documents are provided as JSON files validated by accompanying JSON schemas.
A Python library and CLI is provided to query and utilize these resources. The user can
configure which system package manager they prefer to use for the default package mappings
and command generation (e.g. a user on Ubuntu may prefer conda, brew or spack
instead of apt as their package manager of choice to provide external dependencies).

Specification

Three schemas are proposed:

A central registry of known DepURLs, as introduced in PEP 725.
A list of known ecosystems and the canonical URL for their mappings.
The ecosystem-specific mappings of DepURLs to their
corresponding ecosystem specifiers, plus details of their package manager(s).

Central registry

The central registry defines which identifiers are recognized as canonical,
plus known aliases. Each entry MUST provide a valid DepURL in the
field id, with an optional free form description text. Additionally
an entry MAY refer to another entry via its provides field, which takes
a string or a list of strings already defined as id in the registry. This is useful
for both aliases (e.g. dep:generic/arrow and dep:github/apache/arrow) and
concrete implementations of a dep:virtual/ entry (e.g. dep:generic/gcc
would provide dep:virtual/compiler/c). Entries without provides content
or, if populated, only with dep:virtual/ identifiers, are considered
canonical. The provides field MUST NOT be present in dep:virtual/ definitions.

Having a central registry enables the validation of the [external] table.
All involved tools MUST check that the provided identifiers are well formed.
Additionally, some tools MAY check whether the identifiers in use are recognized as
canonical. More specifically:

Build backends, build frontends, and installers SHOULD NOT do any validation
of identifiers being canonical by default.
Uploaders like twine SHOULD validate if the identifiers are canonical
and warn or report an error to the user, with opt-out mechanisms. They
SHOULD suggest a canonical replacement, if available.
Index servers like PyPI MAY perform the same validation as the uploaders and
reject the artifact if necessary.

This registry SHOULD also centralize authoritative decisions about its
contents, such as which entry of a collection of aliases is preferred as
canonical, or which versioning scheme applies to virtual DepURLs (see Appendix
B). The corresponding answers are not given in this PEP; instead we delegate
that responsibility to the central registry maintainers.

Mappings

The mappings specify which ecosystem-specific identifiers provide the canonical
entries available in the central registry. A mapping mainly consists of a list
of dictionaries, in which each entry consists of:

an id field with the canonical DepURL.
an optional free form description text.
a specs field whose value MUST be one of:
- a dictionary with three keys (build, host, run). The values
  MUST be a string or list of strings representing the ecosystem-specific package
  identifiers as needed as build-, host- and runtime dependencies (see PEP 725 for
  details on these definitions).
- for convenience, a string or a list of strings are also accepted as a
  shorthand form. In this case, the identifier(s) will be used to populate
  the three categories mentioned in the item above.
- an empty list, which is understood as the ecosystem not having packages to
  provide such dependency.
a specs_from field whose value is a DepURL from which the specs
field will be imported. Either specs or specs_from MUST be present.
an optional urls field whose value MUST be a URL, a list of URLs, or a
dictionary that maps a string to a URL. This is useful to link to external
resources that provide more information about the mapped packages.

The mappings SHOULD also specify another section package_managers, reporting
which package managers are available in the ecosystem and how to use them. This field MUST
take a list of dictionaries, with each of them reporting the following fields:

name (string), unique identifier for this package manager. Usually, the executable name.
commands (list of dictionaries), the commands to run to install the mapped package(s) and
check whether they are already installed.
specifier_syntax: instructions on how to map a subset of PEP 440 specifiers to
the target package manager. Three levels of support are offered: name-only, exact-version-only,
and version-range compatibility (with per-operator translations).

Each mapping MUST have a canonical URL for online retrieval. These mappings
MAY also be packaged for offline distribution in each platform. The authors
recommend placing in the standard location for data artifacts in each operating
system; e.g. $XDG_DATA_DIRS on Linux and others, ~/Library/Application Support on
macOS, and %LOCALAPPDATA% for Windows. The subdirectory identifier MUST
be external-packaging-metadata-mappings. This data directory SHOULD only
contain mapping documents named {ecosystem-identifier}.mapping.json. The central
registry and known ecosystem documents MAY also be distributed in this directory,
as registry.json and known-ecosystems.json, respectively.

Known ecosystems

The list of known ecosystems has two roles:

Reporting the canonical URL for its mapping.
Assigning a short identifier to each ecosystem. This is the identifier
that MUST be used in the mapping filenames mentioned above so they can be
found in the local filesystem.

For ecosystems corresponding to Linux distributions, the identifier MUST be the
one reported by their os-release < os-release >__
ID parameter. For other ecosystems, it MUST be decided during the submission to
the list of known ecosystems document. It MUST only use the characters allowed in
os-release’s ID field, as per this regex [a-z0-9\-_.]+.

Schema details

Three JSON Schema documents are provided to fully standardize the registries and mappings.

Central registry schema
^^^^^^^^^^^^^^^^^^^^^^^

The central registry is specified by the following
JSON schema < external-metadata-mappings/schemas/central-registry.schema.json at main · jaimergp/external-metadata-mappings · GitHub >__:

$schema


.. list-table::
    :widths: 25 75

    * - Type
      - ``string``
    * - Description
      - URL of the definition list schema in use for the document.
    * - Required
      - False

``schema_version``