PEP 804: An external dependency registry and name mapping mechanism

[mhsmith (Malcolm Smith)]

package_managers comments:

Exactly one of the command items MUST include a {} placeholder, which will be replaced by the mapped package identifier(s). The install command SHOULD support the placeholder being replaced by multiple identifiers, query MUST only receive a single identifier per command.

For consistency, I think this should say “specifier” rather than “identifier”.

Since the placeholder may expand into multiple tokens, it wouldn’t make sense for the item to contain any other text, so suggest changing “include a {} placeholder” to “be a {} placeholder”. Or maybe, for future extensibility, the placeholder should be given a name like {specifiers}.

exact_version MUST be None or a list of strings that describe the syntax used for specifiers that only express exact version constraints

What’s the distinction between exact_version and equal, and how does this relate to the DepURL? The conda example uses == for exact_version and = for equal, but conda defines = as something which doesn’t correspond directly to any PEP 440 operator.

the keys equal, greater_than, greater_than_equal, less_than, less_than_equal, and not_equal

not_equal is not one of the operators allowed in PEP 725; I guess this is an oversight.

the key {and} takes a string

This shouldn’t have braces.

[jaimergp (Jaime Rodríguez-Guerra)]

Thanks for the feedback, @mhsmith!

For consistency, I think this should say “specifier” rather than “identifier”.

Noted, will update.

Since the placeholder may expand into multiple tokens, it wouldn’t make sense for the item to contain any other text, so suggest changing “include a {} placeholder” to “be a {} placeholder”. Or maybe, for future extensibility, the placeholder should be given a name like {specifiers} .

The choice of “include” there is intentional. In some package managers you may need some boilerplate around the placeholder of the specifier. More specifically, cases like --package={}. That said, it feels like this can be done in the specifier_syntax dict too, so I’ll double check whether we can remove it.

What’s the distinction between exact_version and equal , and how does this relate to the DepURL? The conda example uses == for exact_version and = for equal , but conda defines = as something which doesn’t correspond directly to any PEP 440 operator.

• exact_version is a literal string equality, just like PEP 440 ===. However, some package managers don’t have the notion of operators, so we need to provide this subset of functionality separately too.
• equal is PEP 440 ==, which has dual behaviour: literal version matching, and fuzzy matching (if requested with a .* suffix). The closest mapping between PEP440 and conda is the one described in the example (conda’s = is always PEP440 ==x.y.*).

not_equal is not one of the operators allowed in PEP 725; I guess this is an oversight.

It is not included in PEP 725 because some package managers do not support it, and we want to avoid the presence of incompatible DepURLs for now. However the specifier_syntax dict does specify all operators in case that changes in the future. I understand this is confusing though, so I’d be happy to clarify further if needed.

This shouldn’t have braces.

Good catch! Will fix.

[thorwhalen1 (Thor Whalen)]

I’m also wondering how much of this discussion is taking into account the fact that we’re rapidly moving toward AI-assisted development, where agents will be reading, writing, and acting on this metadata. What should this new context inform our design choices here? Things like clear check/install commands, rationale fields, predictable schemas, explicit platform distinctions, idempotent actions—and perhaps even more precise ways of describing the full environment a package needs, including system settings or environment variables, so agents can reconstruct it safely and reliably.

[pombredanne (Philippe Ombredanne)]

@jaimergp @rgommers Late to the party, but as it happens there is an effort to actually create a registry of PURLs, supported in part by NLnet NLnet; C/C++ Package Registry with a focus on C/C++ code that would likely mesh up well with this PEP. I think this is going to morph into something a bit more generic than strictly C/C++, to address the problem of identifying packages that are not in a registry, like GNU libc, the GNU GCC libstdc++, OpenSSL, Boost, zlib, and similar.

Eventually (and even more so since CVE.org has merged PURL support in their schema 5.2) there is a need to have PURLs for packages that do not live in a registry, and this is a discussion I entertained specifically with Linux maintainers.

I will read this PEP in details and report, including reviewing PEP 725: Specifying external dependencies in pyproject.toml (round 2)

[jaimergp (Jaime Rodríguez-Guerra)]

Hello everyone!

Just wanted to mention that we have submitted an update to PEP 804 at PEP 804: version 2, address all feedback received so far by jaimergp · Pull Request #4749 · python/peps · GitHub. Hopefully this revision addresses the main feedback points received in this thread and other channels

I would like to take the opportunity to thank everyone that took the time to take a look, read through the text and provide incredibly insightful comments that have helped improved this proposal so much. I’m beyond grateful!

[mhsmith (Malcolm Smith)]

@jaimergp

What’s the distinction between exact_version and equal , and how does this relate to the DepURL? The conda example uses == for exact_version and = for equal , but conda defines = as something which doesn’t correspond directly to any PEP 440 operator.

• exact_version is a literal string equality, just like PEP 440 ===. However, some package managers don’t have the notion of operators, so we need to provide this subset of functionality separately too.
• equal is PEP 440 ==, which has dual behaviour: literal version matching, and fuzzy matching (if requested with a .* suffix).

Thanks, I wasn’t aware of the === operator. But it isn’t listed in PEP 725, and the current spec describes it as “heavily discouraged”. And if its main distinction from == is whether it supports the .* suffix, that suffix isn’t mentioned in PEP 725 either, and I don’t suppose it would be easy to map to different package managers if it was.

@jaimergp

• The closest mapping between PEP440 and conda is the one described in the example (conda’s = is always PEP440 ==x.y.*).

But the actual mapping we’re doing is in the opposite direction. If we convert a PEP 440 dependency of ==x.y into conda’s =x.y, then that would allow the installation of x.y.99, which isn’t correct according to PEP 440 semantics. Conda’s ==x.y seems like a closer match.

In which case, there wouldn’t be any need for an exact_version key, because package managers that don’t support operators, but do support exact versions, can express that using a version_ranges which only contains equals.

[mhsmith (Malcolm Smith)]

There’s some inconsistency between the structure of the top-level tables:

• definitions and mappings are arrays within which each item has an id key.
• package_managers is also an array, but each item has a name key.
• ecosystems is a dict instead.

I’d suggest regularizing all of these to dicts. This would enforce uniqueness of keys, and allow for faster startup time when reading what could be some quite large files, since the data would come out of the JSON decoder already in a form suitable for lookup.

[mhsmith (Malcolm Smith)]

Each schema has a schema_version, but this isn’t defined anywhere. It would seem reasonable to follow the same rules as Python’s Metadata-Version, with major and minor versions so the consumer can detect whether the format has changed in an incompatible way.

[mhsmith (Malcolm Smith)]

@jaimergp

We really wanted to use the original pkg: implementation, and indeed we did open a virtual proposal for pkg: URLs at Support for “virtual packages” / dependencies on interfaces? · Issue #222 · package-url/purl-spec · GitHub and Add virtual type by jaimergp · Pull Request #450 · package-url/purl-spec · GitHub, unfortunately with limited feedback. However, even if this went through, it wouldn’t be until the next revision of the standard (according to the milestones), and we would still miss an ergonomic way to express version constraints (the PURL solution is to append a ?vers=... qualifier, %-encoded).

It does seem a shame to diverge from the PURL ecosystem when it’s just getting started. I wonder if there are other ways to achieve these two goals. For example:

• Virtual packages could use namespaces of the generic type. The PURL spec doesn’t define any generic namespaces, but it doesn’t forbid them as it does with conda. So we’re free to use them however we want.
• Version constraints could be written immediately after the PURL, just as we’ve already allowed environment markers. This would effectively give an external dependency specifier the same syntax as a Python dependency specifier, with the PURL taking the place of the package name.
  • Because = is allowed in PURLs, a space would probably be required between the PURL and the constraints.
  • We could allow a single version in the PURL itself, or a set of PEP 440 constraints after the PURL, but not both.

Taking this example from PEP 725:

dep:virtual/interface/lapack@>=3.7.1

It would instead be written as:

pkg:generic/interface/lapack >=3.7.1

[jaimergp (Jaime Rodríguez-Guerra)]

Thanks @mhsmith, I’ll take this into consideration for the current revision in the coming days! I’ll reply here to the last points for now.

@mhsmith

• Virtual packages could use namespaces of the generic type. The PURL spec doesn’t define any generic namespaces, but it doesn’t forbid them as it does with conda. So we’re free to use them however we want.
• Version constraints could be written immediately after the PURL, just as we’ve already allowed environment markers. This would effectively give an external dependency specifier the same syntax as a Python dependency specifier, with the PURL taking the place of the package name.
  • Because = is allowed in PURLs, a space would probably be required between the PURL and the constraints.
  • We could allow a single version in the PURL itself, or a set of PEP 440 constraints after the PURL, but not both.

I agree that the need for dep:-variants is a pity. I am willing to explore ways around the limitations we found in PURL. The generic namespace can work. We would “claim” the compiler and interface namespaces (but only as namespaces, not as names). Perhaps @pombredanne is also willing to bless these two generic namespaces.

About the space separators, though, I don’t dislike it, but I see two potential problems:

• The space might be easy to miss and some folks may be tempted to not use it, especially given how the space in Python requirements is kind of optional (e.g. numpy>=2 and numpy >=2 are both valid).
• The percent encoding rules disallow spaces in PURL, so it has to be percent-encoded. However, they are allowed as input, and the packageurl-python will happily parse it as part of the name. Not sure if this introduces ambiguity as to what constitutes the PURL part and our version specifier.

>>> packageurl.PackageURL.from_string("pkg:generic/interface/lapack >=3.7.1")
PackageURL(type='generic', namespace='interface', name='lapack >=3.7.1', version=None, qualifiers={}, subpath=None)
>>> str(packageurl.PackageURL.from_string("pkg:generic/interface/lapack >=3.7.1"))
'pkg:generic/interface/lapack%20%3E%3D3.7.1'

[mhsmith (Malcolm Smith)]

@jaimergp

The percent encoding rules disallow spaces in PURL, so it has to be percent-encoded. However, they are allowed as input, and the packageurl-python will happily parse it as part of the name. Not sure if this introduces ambiguity as to what constitutes the PURL part and our version specifier.

The same is true of environment markers with a semicolon, so I guess some kind of pre-parsing to separate the PURL from the rest was always going to be required.

[jaimergp (Jaime Rodríguez-Guerra)]

@mhsmith

The same is true of environment markers with a semicolon, so I guess some kind of pre-parsing to separate the PURL from the rest was always going to be required.

That’s true, but visually the semicolon is more obvious. With spaces, I find it tricky to spot the diff, it’s too subtle:

[external]
build-requires = [
  "pkg:generic/compiler/cxx >=13",
  "pkg:generic/compiler/cxx>=13",
]

Instead, what if we break the need for string-only identifiers, and use structured objects? This is valid TOML:

[external]
build-requires = [
  # simple PURLs, no version, can use a string
  "pkg:generic/compiler/c",
  # for deps with versions, use an object
  {purl = "pkg:generic/compiler/cxx", version = ">=21"}, 
  # add environment markers too, should these two be equivalent? 
  "pkg:generic/compiler/go; sys_platform == 'darwin'";
  {purl = "pkg:generic/compiler/go", environment_marker = "sys_platform == 'darwin'"},
]

I like structured objects like this a lot, it removes all the ambiguity, but at the cost of verbosity. It also challenges the familiarity of strings in the dependencies table.

Let me know what you think.

[mhsmith (Malcolm Smith)]

I’ve had another look through the PURL spec, and I think it’s rigorous enough that we can make the space optional and still keep the parsing simple. Specifically:

• The syntax of an external dependency specifier is:
  • purl wsp* versionspec? wsp* quoted_marker?
• The purl is terminated by any of the following strings, none of which are valid in a PURL:
  • ==, >, <, ;
• The remainder of the specifier is then parsed for versionspec and quoted_marker according to the same syntax as a Python dependency specifier, except for the limited set of version operators. If the purl already contains a version, then versionspec is not allowed.

[jaimergp (Jaime Rodríguez-Guerra)]

@mhsmith

There’s some inconsistency between the structure of the top-level tables:

• definitions and mappings are arrays within which each item has an id key.
• package_managers is also an array, but each item has a name key.
• ecosystems is a dict instead.

I’d suggest regularizing all of these to dicts. This would enforce uniqueness of keys, and allow for faster startup time when reading what could be some quite large files, since the data would come out of the JSON decoder already in a form suitable for lookup.

mappings are arrays because there can be more than one possible way to map a DepURL to package names/specifiers. If we want a dict there, then we need to make the value an array of dictionaries, nesting the schema even further. I’m ok with that but I found the list of dicts approach simpler.

[jaimergp (Jaime Rodríguez-Guerra)]

I agree that this is technically parsable without ambiguity, but it will also look like a malformed PURL, which may be confusing to users:

[external]
build-requires = [
  "pkg:generic/ninja>=2",
  "pkg:generic/ninja@2.0",
  "pkg:generic/ninja@>=2.0",
  "pkg:generic/ninja==2.0",
]

I’d like to hear more points of view about this to assess whether the ergonomics are worth the potential confusion.

[mhsmith (Malcolm Smith)]

Thanks, I’d also like to hear other opinions on this question. For clarity, my proposal is that all of these examples would be valid except for pkg:generic/ninja@>=2.0, because you can use either the PURL version syntax or the Python syntax, but not a mixture.

[mhsmith (Malcolm Smith)]

@jaimergp

mappings are arrays because there can be more than one possible way to map a DepURL to package names/specifiers. If we want a dict there, then we need to make the value an array of dictionaries, nesting the schema even further. I’m ok with that but I found the list of dicts approach simpler.

OK, my main concern here is performance, since these lists could contain 10,000 items or more. I guess Python’s JSON reader is optimized enough that actually reading the file wouldn’t be a problem, but I’d like to see some numbers for how long it would take to do a typical lookup with a linear search in Python code.

[jaimergp (Jaime Rodríguez-Guerra)]

One could even argue that it is indeed valid, because the version may be empty. The spec is not super clear (it hints and it should not be empty but doesn’t forbid it in my opinion). The Python parser doesn’t complain either:

>>> import packageurl as purl
>>> purl.PackageURL.from_string("pkg:generic/package@")
PackageURL(type='generic', namespace=None, name='package', version=None, qualifiers={}, subpath=None)

But, yes point stands. Let’s see if there are more comments about it.

[jaimergp (Jaime Rodríguez-Guerra)]

Hello @ksurma, thanks for the feedback in the PEP 725 thread! I’ll reply here to the items that concern PEP 804. Let me know if you need further clarification and I’ll be happy to elaborate.

[…] adding an indicator that package contains compiled extensions (which would mean adding gcc to the rpm buildroot on our side), would solve the majority of packages for us.

That can be inferred by the presence of compiler definitions in the build table for almost all cases. You could even got one step further and assume that the presence of a build table almost always guarantees that a compiler will be needed. But you don’t need that, since the proposal includes mappings that would convert the necessary definitions into Fedora’s decision to use gcc as the package that provides a C compiler in that distribution.

Mapping: do I understand correctly that if Fedora wants to make use of the PEPs, we have to volunteer to maintain the respective mappings, as they grow?

Yes, that’s what the current proposal suggests: federated maintenance of the mappings. The target communities usually know better (and earlier) about the necessary changes or updates. But even without mappings, the presence of the external tables alone is useful ans it can be the source for better error messaging. For example, upon a setuptools (or any other build backend, really) build error, pip could add a small message saying “this project has an external table with the following dependencies: [list of PURLs]. Their unavailability may be behind the cause of this build error”.

Regarding naming: will there be a semantic hierarchy of mappings (fedora{version}.mapping.json if version is specified, otherwise use fedora.mapping.json?) or are you planning on flat structure that would entail always creating a new versioned file (less preferred by us)? Would we then be able to add more mappings if we need them, e.g. for CentOS Stream, or RHEL?

Yes, see this paragraph:

Each mapping MUST have a canonical URL for online retrieval. Its complete filename MUST be
`{ecosystem-identifier}.mapping.json`, where “ecosystem identifier” MUST conform to this regex: 
`[a-z0-9\-_.]+(\+[a-z0-9\-_.]+)?`. In other words, a first field optionally followed by a second,
separated by a + symbol.

The second field is meant to be the version, but a unversion fallback name is recommended. You can add as many mappings as you need, e.g. centos-stream.mapping.json and rhel.json.

looking at dep:virtual/interface/blas in fedora.mapping.json, the key appears 3 times, once for blas, openblas and flexiblas. dep:virtual/interface/lapack can be realized by lapack or flexiblas. Is there any recomendation how tools should interpret multiple entries per id?
For example, adding a priority key to the entry could solve the issue of Fedora defining a preferred/canonical tool for the job.

The current convention followed by the prototype tooling is that the first occurrence wins. The other options are fallback options. But another tool could choose to always prompt the user for interactive workflows. We decided against making this decision on the mapping maintainers side and leave that to the tooling or end user. Having a priority key would mean another field to maintain, keep in sync, and discuss about.

the mappings look arbitrary at some entries (examples mentioned above) and overly specific at others: for example, what are tools supposed to do with a GH repository? Should installers suggest an installation of build dependencies from arbitrary repositories? For rpm builds, this is no-op, as we can only use official rpm packages. How to canonically handle a non-existent mapping?

The Github repositories mentioned in the registry can be considered alternative identities of the same package. Mostly useful for packages that don’t belong in a language-specific registry. e.g. Rust crates in crates.io or JavaScript packages in Npmjs.com are easy to identify because these sites offer a flat namespace. Projects written in C, C++ or Fortran haven’t had such a central location to claim an identifier, so there’s no unambiguous way to refer to them in abstract ways. In those cases, the Github repository URL can serve of additional clarification when a simple generic/<name> is not well known.

It’s unclear to me how the initial registry was compiled, whether it contains only real life use cases or attempts to cover the “whole world”. In the latter case, I’d suggest starting with a minimal set of ids coming from the real life packages, and adding new on demand.

It has been populated by the requirements of real life packages. More specifically, those with external dependencies in the top 150 downloaded packages. More details at GitHub - rgommers/external-deps-build · GitHub.

Technically, the mapping is not specific for the Python land. Should there rather be a overarching registry for all ecosystems dealing with external dependencies? (Not implying that you should be tasked with creating one; just thinking over the best place for the registry).

The text discusses existing options and their unsuitability. We have also discussed the overlap with the PURL maintainers and are monitoring the progress on the prototype purl-registry, but it’s far from ready and is unclear whether it could fit our purposes here in Python land.