As a procedural point, I’m a bit concerned if this PEP doesn’t stand up to scrutiny without needing another PEP to support it. What if this PEP gets approved but PEP 804 gets rejected? Does approving this PEP imply a commitment to approve that one?
2 Likes
I think they need to be considered and accepted or rejected together (so they should probably declare a circular dependency)
2 Likes
This was discussed on the PR for this PEP, see this comment thread. PEP 804 does declare a dependency on PEP 725, but the reverse dependency was taken out after I had initially added it.
In case PEP 804 gets rejected, PEP 725 would need some changes to adjust, but only minor textual ones. It only means that any DepURL that parses as valid is accepted, and there is no central registry with a canonical list. It would still be a significant benefit to be able to declare in metadata which compilers and non-python dependencies you need - there just wouldn’t be a standardized way for a tool to translate that to concrete package names for a given distro.
5 Likes
The reason this question was raised is because your response to a question about the usability of an aspect of PEP 725 was “that’s covered in PEP 804”. So I don’t think “minor textual changes” would be enough - you’d need an alternative response to that point, at least.
I’m inclined to agree with @ncoghlan that the two PEPs need to be considered together, in which case I’d like to understand why it’s worth having them as separate PEPs in the first place - it’s easier to review and discuss if all of the functionality is in one document.
1 Like
I’m not sure what you mean by asking the backend to “deal with” this. If a backend does not do this vendoring then why would it need to deal with this possibility?
My questions above are not really asking why the PEP is designed the way it is but just trying to understand how it is expected to be applied. Here is an example that I don’t understand:
SciPy 1.10
pyproject.toml content:
[external]
build-requires = [
"dep:virtual/compiler/c",
"dep:virtual/compiler/cpp",
"dep:virtual/compiler/fortran",
"dep:generic/ninja",
"dep:generic/pkg-config",
]
host-requires = [
"dep:virtual/interface/blas",
"dep:virtual/interface/lapack@>=3.7.1",
]
PKG-INFO / METADATA content: N/A.
The metadata is apparently N/A but wouldn’t the built wheel have an external runtime dependency on BLAS/Lapack if the build backend linked against them but did not vendor them?
It seems that this is implicit in pyproject.toml that host-requires implies a runtime dependency but it is not shown at all in the metadata. I would have expected the wheel to have some metadata:
Requires-External-Dep: dep:virtual/interface/blas
Requires-External-Dep: dep:virtual/interface/lapack@>=3.7.1
In the wheel though it would not be a virtual dependency any more. After building the requirement would be for the concrete BLAS/Lapack that were used at build time. Why is that not listed in the wheel?
1 Like
I think the split makes sense, as the two PEPs have different primary audiences:
- the primary audience for PEP 725 is Python packaging tooling authors. They need some awareness of PEP 804 to be able to read the mapping info correctly, but maintaining the mappings isn’t directly their responsibility.
- the primary audience for PEP 804 is platform providers & contributors. They need some awareness of PEP 725 to understand how the mapping information will be used, and may want to consume it in their own tooling, but the parts specific to Python project publishing tools won’t be their problem.
The boundaries between the two are fuzzy (hence @rgommers earlier answer), but having the “tool development” piece and the “mapping metadata maintenance” piece in the same doc would make the division of responsibilities even murkier rather than making it clearer.
6 Likes
Yeah, I’m on the fence as well. It’s just thoroughly unfortunate that gcc (understandably) standardised on terminology that only makes sense for compilers (“build” the compiler on machine type A, “host” the compiler on machine type B, “target” the compiler at machine type C), and then used the first two terms for cross-compilation of non-compiler binaries instead of the first and last (understandable in terms of the mechanics of configuring things, but not particularly clear from a usage point of view).
Perhaps rather than changing the host term, we can just make it clearer that those are still build requirements in the context of building and deploying Python packages:
build-requires(no change)build-host-requires(just adds a prefix tohost-requires)dependencies(no change)
And then make the semantic clarification that build-host-requires is still solely a build time dependency declaration for our purposes, and if something is needed at both build time and runtime, it needs to be listed in both build-host-requires and dependencies.
I also think the PEP itself would benefit from explaining the “build”, “host” and “target” terminology a bit differently:
- add a forward reference from the abstract down to the terminology section
- mention up front in the terminology section that the terminology is chosen because it’s what gcc uses and we perceive that as the most common approach across the fractured cross-compilation tooling landscape (even though it’s still far from universal)
- describe the compiler use case first, as it’s the one situation where the terms actually make sense
The way it’s currently worded suggests to me that we made up our own terminology because there was no common established convention, which isn’t the case.
(Configure Terms (GNU Compiler Collection (GCC) Internals) does get referenced from that section already, but it’s not clear that it’s the origin of the proposed terminology)
1 Like
Yes, but it’s the answer I expected rather than the answer I was hoping to hear. Arch and Fedora give much less coverage than you might expect (I can’t speak for Conda).
- Arch makes a point of serving packages as-is, not chopping them up into weird bits and its rolling build model combined with its replaces $package mechanism allows it to easily shake off legacy quirks and fix prior mistakes. As a result, Arch is arguably the least nuanced distribution to find the right packages on.
- Fedora is very much built around its own attempts to automate absolutely everything it can in repackaging. The mappings from libraries to packages are already there and are kept in shape from heavy usage throughout Fedora’s build system.
A more meaningful test would be Void Linux or OpenSUSE[1], Alma[2] and Debian/Ubuntu[3].
PEP 804 creates a single mapping that must fit universally for all projects. A project maintainer has no control if the mapping is incomplete, wrong or the virtual package definitions just don’t fit.
What if it’s not as universally needed as musl-dev?
I have read it. It only talks about how virtual packages are defined and nothing about how those or concrete packages are used. I still don’t have an answer to the question of what does a package author do if the virtual packages don’t work for them? The obvious answer would be to switch to concrete packages but it doesn’t look like that’s an option under the current PEPs?
Is the only goal of these PEPs then just to aid repackagers? Have I misassumed a secondary aim to make packages with system dependencies generally easier to install?
1 Like
Maybe it would just raise a clear exception when it sees dynamic = ["dependencies"] then, and silently accept dynamic = []. It has to at least do something like that. That’s not that much work though - the more compelling argument is that there is, as of now, no clear use case. So we’d rather leave this out; when a need does show up it can always be done in a backend-specific way first, and if there’s enough need it can be standardized in the future.
Not quite, more like its extension modules now have dependencies on specific shared libraries, not just on a concrete or virtual package. Remember that when the wheel is built by a build backend, it is not redistributable but needs the exact libraries already installed on the system. This information is encoded in the extension modules, as you can see with a tool like ldd:
Partial output of `ldd` for the `scipy` example
scipy/interpolate/interpnd.cpython-39-x86_64-linux-gnu.so:
linux-vdso.so.1 (0x00007ffcec98a000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f15edcd8000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f15edf26000)
...
scipy/linalg/_decomp_update.cpython-39-x86_64-linux-gnu.so:
linux-vdso.so.1 (0x00007ffe6c890000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f77d9570000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f77d97b2000)
scipy/linalg/_fblas.cpython-39-x86_64-linux-gnu.so:
linux-vdso.so.1 (0x00007ffd00320000)
libopenblas.so.0 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so.0 (0x00007f48fd671000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f48fd47e000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f48fd33a000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f48fd319000)
libgfortran.so.5 => /home/rgommers/anaconda3/envs/scipy-dev/lib/./libgfortran.so.5 (0x00007f48fd170000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f48ff740000)
libquadmath.so.0 => /home/rgommers/anaconda3/envs/scipy-dev/lib/././libquadmath.so.0 (0x00007f48fd134000)
libgcc_s.so.1 => /home/rgommers/anaconda3/envs/scipy-dev/lib/././libgcc_s.so.1 (0x00007f48fd11f000)
You can call that a “runtime dependency”, but it’s not a package just specific libraries. On the build machine those libraries are already installed, and when making the wheel distributable, those libraries are vendored. So there is no real user for this kind of “shared library dependency” information [1].
Equally importantly, it isn’t even possible for a build backend to produce such metadata. What it knows is (1) the build succeeded so apparently all dependencies were present, (2) the extension modules it produced point at a bunch of shared libraries. There is no way to map the host-dependencies metadata to those shared libraries for the build backend - trying to build a way to disentangle this would be very complex because it’s now needing info at a much more granular level, and it also includes transitive dependencies (e.g., in the SciPy example, libquadmath.so.o just “showed up”).
At best maybe for debugging purposes, but you can also use
ldddirectly for that ↩︎
2 Likes
Thanks, I do like that idea better! I’d be fine with using build-host-requires in the way you suggest, as well as make the textual changes to explain “build”, “host” and “target” differently. Unless there are more comments, we’ll plan to make that change in the next update of the PEP.[1]
It may take a couple of weeks for that update, since (a) I’m traveling for two weeks, and (b) we will also need to update all the packages and CI in the same way - it’s straightforward to do but a fair bit of churn. ↩︎
3 Likes
The PEP says
The versioning scheme for particular virtual packages, in case that isn’t unambiguously defined by an upstream project or standard, will be defined in the Central Registry (see PEP 804).
Could you give an example on how this may look? Say, I’m requiring a c-compiler but know that gcc 14 will not work. How do I encode this?
2 Likes
I’m fairly puzzled why you’d not consider going from ~20% to ~95% success rate a good/hoped-for result. Especially since 100% is never going to be realistic. Also, in the linked answer I pointed out “The few remaining failures could all be understood, and none were due to the external metadata design. Causes included: …”
Re Conda: it’s more different from Linux distros than distros tend to be from each other I’d say. It has inconsistent package names[1] and inconsistent package splits with few clear rules.
I’d be willing to add one more if that would convince you. Probably Ubuntu since it’s the least work for us due to familiarily and ease of looking up package names. That said, I’m quite confident that the success percentage will be similar[2], and will not require any changes to the PEP 725 metadata for those 37 packages, but only adding a PEP 804 mapping for Ubuntu.
It’s a one-to-many mapping: one on the Python packaging side, many on the distro side. For the “one” side, if something is missing, it can simply be added through a PR or feature request. If it’s “wrong” I don’t know what to tell you here beyond a standard “if it’s a bug let’s fix it, if it’s a request for a breaking change then it’s up to the maintainers of the central registry to assess it case by case”.
Re “virtual packages definitions don’t fit”, you may have the wrong idea about how many there will be. They are not a hammer to whack every package variation or fork of something with, they’re only compilers and interfaces at this point. Of the examples you gave, only a C compiler is clearly virtual, and maybe netcat (I’m not familiar enough with it to know, there’s a PEP 804 process to hash that out for concrete cases like these as part of adding them to external-metadata-mappings). 7zip and rsvg-convert clearly look like concrete packages only.[3]
Note that neither PEP will prescribe installer UX, so if that’s what you’re asking then no, that’s left up to installer authors. For a clear example of how it could work though, https://peps.python.org/pep-0804/#pyproject-external-cli has a long-form example going all the way from PEP 725 metadata to a pip output of:
The following external dependencies are needed to install the package
mentioned above. You may need to install them with `apt`:
g++
zlib1g
zlib1g-dev
You can imagine variations of that, like an installer having an interactive dialog to actually run those apt install commands if the user has configured it wants external dependencies from Ubuntu and they seem to be missing on the system the user is building on.
That’s pretty explicit, so if that’s not what you want to see then can you please rephrase the “nothing about how those or concrete packages are used”?
You have not misassumed, that is definitely a goal.
many popular packages match PyPI names, and others are renamed without a single rule like “prepend
py-” ↩︎We understand all failures after all, and half of them were due to outdated packaging practices (
lxml,grpcio) and the other half have more complex package-specific or distro-specific causes ↩︎Again, this really is PEP 804, if you want to delve into concrete cases like this, please do it there ↩︎
2 Likes
That is something that won’t work. Main reason: there is no OR operator in Python packaging’s dependency specifiers. It was discussed a little bit in the round 1 discussion thread for this PEP, but we decided not to try adding one.
Distros typically don’t handle this case either if they support multiple compilers via a generic “needs C compiler” type virtual. Spack is a distro that does handle this[1], it looks something like this (from its scipy recipe (omitted more conflicts declarations):
depends_on("c", type="build")
conflicts("%gcc@:9.0", when="@1.14:", msg="SciPy 1.14: requires GCC >= 9.1")
conflicts("%apple-clang@:9", when="@1.10:", msg="SciPy requires Apple Clang >= 10")
conflicts(
"%msvc@:19.19",
when="@1.10:",
msg="SciPy requires at least vc142 (default with Visual Studio 2019) "
"when building with MSVC",
)
...
# Unknown build error, version ranges may be incorrect
conflicts("%oneapi@2024:", when="@:1.8")
Even then though, there are no upper bounds for compilers usually. If a newer version of a compiler doesn’t work, the package author can either fix that problem (preferred) or raise an informative error at build time.
This level of detail is very maintenance-intensive, and I think if we tried adding something like this:
[external]
build-requires = [
"dep:virtual/compiler/c", [
# An OR list to say that if the `compiler/c` virtual is provided by
# one of these concrete packages, that concrete package will need
# to use the given version ranges and respect the environment markers
# (e.g., in this example, `oneapi` isn't supported on Windows)
"dep:generic/gcc@>=9,<14",
"dep:generic/msvc@>=19.19",
"dep:generic/clang@>=16",
"dep:generic/oneapi@>=2024.1.0; platform_system!='Windows'",
],
]
that would be a large amount of extra complexity, with semantics that aren’t even available for regular Python dependency specifiers.
It’s feasible in principle, but the complexity/benefit tradeoff doesn’t look great to me. Very few Python package authors bother to check compiler versions at build time at all[2], so I wouldn’t expect this level of detail to be used regularly in practice.
4 Likes
It is a good success rate but like I said, it’s on two very easy distributions. Your test has been exposed to very little of what makes this hard such as the per-version inconsistencies and/or legacy quirks that LTS distributions bring or the ambiguity and subjectiveness that’s brought in by distros with decentralised repositories or heavy use of virtual/variadic dependencies. (And presumably, you’ve been in the privileged position of being able to define the virtual packages to work for whatever you’re building whereas everyone else will have to do it the other way around.)
If I have a dependency on a system package (let’s say ffmpeg) and I can’t use a dep:generic/dep:virtual form[1] then what does PEP 725 say can I do? I would like to be able to list the concrete packages myself using:
[external]
dependencies = [
"pkg:apk/alpine/ffmpeg-libs",
"pkg:deb/debian/ffmpeg",
"pkg:rpm/fedora/ffmpeg-free",
"pkg:deb/ubuntu/ffmpeg",
]
But it looks I can’t do that because only dep: PURLs are allowed. Is my only option to request a new PEP 804 package name, hope it’s approved, hope that I can get all the distributions to agree with me on exactly what that name should imply, then jettison support for the distributions that don’t provide their own name mappings? Really, all I want is the ability to specify my own exact dependencies – if I can have that then it doesn’t matter if all my other concerns aren’t addressed because there will always the just specify exact dependencies yourself escape hatch.
it doesn’t exist, the definition doesn’t match my definition w.r.t. plugins or full CLI vs libraries-only, a distribution I want to support isn’t interested in providing a PEP 804 mapping ↩︎
3 Likes
This would be a good example to include, as the limitation here isn’t the dep URLs (s/pkg/dep gives a valid concrete dep URL), it’s the lack of “A or B or C” dependency declarations outside virtual package definitions.
Without adding new syntax or semantics, we could potentially define a conventional approach to that situation via external extras (using the same target platform IDs as are defined for dep URLs):
[external.optional-dependencies]
ext-alpine = [
"dep:apk/alpine/ffmpeg-libs",
]
ext-debian = [
"dep:deb/debian/ffmpeg",
]
ext-fedora = [
"dep:rpm/fedora/ffmpeg-free",
]
ext-ubuntu = [
"dep:deb/ubuntu/ffmpeg",
]
Scoping by packaging manager may also be a reasonable idea for package names that are consistent across a distribution family (with the tools replacing the platform ID if necessary):
[external.optional-dependencies]
ext-apk = [
"dep:apk/alpine/ffmpeg-libs",
]
ext-deb = [
"dep:deb/debian/ffmpeg",
]
ext-rpm = [
"dep:rpm/fedora/ffmpeg-free",
]
As long as the platform specific converters agree on the name of the extra (or extras) to use for that platform, this would work (since package manager specific dep URLs use the same paths as pkg URLs, just different version specification syntax)
Edit: it does occur to me that this is duplicating info that is present in the concrete dep URLs already (the target ecosystem and platform IDs). So perhaps the simpler convention is to say that converters should ignore concrete external dependency declarations for unrelated ecosystems, unless they have an applicable non-Python-specific PURL translation registry they can use to make them applicable.
Potential wording: “Package conversion tools for specific platforms SHOULD ignore concrete external dependencies for other ecosystems and platforms that they do not recognise. Package conversion tools MAY map such external dependencies to the relevant external dependency declarations for their target platform if mapping data is available (while, at time of writing, we are not aware of any such mappings being generally available, it’s possible that tools may choose to develop such mappings if concrete external dependency declarations are used in popular packages).”
3 Likes
The central registry proposed in PEP 804 provides a schema where different DepURLs can “provide” other DepURLs. So all the ecosystem-specific PURLs could refer to dep:generic/ffmpeg. But this is duplicative of the mappings, maintained by the target community.
We are trying to prevent every package maintainer from shipping their own concrete dependencies, which can end up becoming stale (name changed) or incomplete (how to build that wheel on Windows if there are no instructions for ffmpeg in that platform). These can be tackled in the evolving mappings, but IMO the [external] metadata should be as agnostic as possible. Hence the emphasis on dep:generic/*. I assume tools would allow concrete mappings to be passed so folks can override the default mappings for their platform in given situations.
We technically support markers, but they are not granular enough to distinguish distros or, of course, external package managers.
2 Likes
PEP 804 proposes that mappings can define install and query commands for their supported package managers. These can then be used to run programmatic checks on the target ecosystems.
pyproject-external implements this in the query subcommand:
python -m pyproject_external query ./cryptography-45.0.7.tar.gz --package-manager brew [09:37:36] INFO Detected ecosystem 'homebrew' for package manager 'brew'
[09:37:38] ERROR Could not find ['dep:virtual/compiler/c'] as ['gcc']!
[09:37:39] ERROR Could not find ['dep:virtual/compiler/rust'] as ['rust']!
ERROR Could not find ['dep:generic/pkg-config'] as ['pkgconf']!
[09:37:41] INFO Found ['dep:generic/openssl'] as ['openssl']
[09:37:42] ERROR Could not find ['dep:generic/libffi'] as ['libffi']!
ERROR Could not find ['dep:generic/python'] as ['python']!
So I’d say that it’s technically possible without too much user configuration.
3 Likes
It was three distros and I’ve offered to add a fourth, under the constraint of zero changes to [external] metadata. I don’t think that will convince you either though, even if that works with a 100% success rate. Because really your feedback is “I know there will be some corner cases[1], and I want an escape hatch to handle those”. Which in a way is quite nice feedback to see, because we’re discussing complexity/performance tradeoffs at the edges of the design, rather than for the earlier versions of this PEP where the feedback was about the fundamentals of the design (e.g., how version ranges and compilers are handled) - so we moved a lot in the right direction. 
Also for context on why we chose these distros:
- I use Arch Linux myself, so it’s easy to check things there. Hence why I started there.
- Conda was a natural next choice, because it’s quite different conceptually from any Linux distro.
- Then I added Fedora because I wanted a second Linux distro, and it’s probably the most popular one aside from Debian/Ubuntu. I on purpose avoided the latter, because that’s what almost every project uses in their CI and I expected it to be the least challenging for that reason.
And, every distro has its quirks. For BLAS/LAPACK, Arch and Fedora were actually the most difficult two mainstream distros, because of non-default build options (Arch) or deliberately shipping an incompete package (Fedora).
All I can say is that (a) in the current naming choices and mappings there are zero things I’d consider questionable and would have done very differently without a particular package build included, and (b) I started with the top 100 most downloaded packages (23 with compiled code) and a single distro, then to two distros, then to the top 150 packages (37 with compiled code), then to three distros, and at no point did we go back and change choices made for already present mappings when doing that expansion.
@ncoghlan already explained it well (it’s the missing OR operator), but just to emphasize: DepURLs are a superset of PURLs, so anything that has a PURL type is valid.
Re an OR operator itself and this “use optional extras” alternative to it: this reminded me of PEP 771. This use case is very close to one of the two main use cases of PEP 771. For this PEP, we want to closely mirror for [external] what is allowed under [build-system] and [project] for dependency specifiers, so if optional extras get accepted then I think they make sense here as well. And that would cover a gap in your proposed solution/workaround here, which is that things don’t work on any distro not explicitly handled. So it’d look like:
[external]
default-optional-dependency-keys = [
"ext-ffmpeg-generic",
]
[external.optional-dependencies]
ext-ffmpeg-generic = [
"dep:generic/ffmpeg",
]
ext-apk = [
"dep:apk/alpine/ffmpeg-libs",
]
ext-deb = [
"dep:deb/debian/ffmpeg",
]
ext-rpm = [
"dep:rpm/fedora/ffmpeg-free",
]
That will then be valid and cover this type of corner case. Whether it’s actually desirable to do this when uploading to PyPI[2] is another question. I’d say that while it’s valid it should be discouraged, since distro-specific links on PyPI seem bad practice.
Of course, it’s a bit clunky and having an actual OR operator would behave better. Rewriting the example assuming we have that plus a second dependency (say ripgrep), it’d look something like this:
[external]
dependencies = [
# Installers should evaluate OR in left-to-right order,
# stopping at the first one that works
"dep:apk/alpine/ffmpeg-libs" OR
"dep:deb/debian/ffmpeg" OR
"dep:rpm/fedora/ffmpeg-free" OR
"dep:deb/ubuntu/ffmpeg" OR
"dep:generic/ffmpeg",
"dep:cargo/ripgrep",
]
This would perhaps be quite nice. However, the exact same argument can be made for Python dependencies. It’d be quite useful to be able to write this:
[project]
# Try the performance-optimized friendly fork first, if not available
# use the default package
dependencies = ["pillow-simd" OR "pillow"]
```toml
[project]
# Project needs one of a number of dataframe libraries installed.
dependencies = ["polars" OR "duckdb" OR "pandas"]
and it’d be cleaner and in ways more generic than default extras (it’d work for build-system.requires as well). It doesn’t look like this option was discussed for PEP 771. For this PEP, I think we should only consider something that has an analog for regular dependency specifiers.
That would be quite problematic, because now an installer needs a semantic understanding of every PURL type to know what an “unrelated ecosystem” is. E.g., generic, github, cargo, golang cannot be ignored while apk, rpm, conda-forge etc. can be ignored if and only if a machine/platform it is running on doesn’t support that package manager (or doesn’t have it installed?). That seems unworkable.
My take on all this is that:
- We are talking about corner cases. It’s O(1%) of cases that may be solved by a different package mapping than the one that the PEP 804 mappings maintainers think is the right one. If default extras solve some of those cases, great. If not, I think it’s a poor idea to add extra complexity to this PEP for custom escape hatches. I’m perfectly comfortable with some corner cases not working, such is life. That’s still not a regression for those cases, since they also don’t work today.
- Adding an easy way to say “on distro X use exact package name Y” in metadata will probably do more harm than good. I fully expect in that case that a lot of packages will be uploaded to PyPI by authors that see packaging as a chore that only handle the one distro that they care about because they use it themselves. That’s fine to do on one’s own index server, but not great on PyPI.
- Also please keep in mind that one of the main messages from the previous review round was that a canonical way to spell a dependency was needed. A number of people were not happy with leaving that up to common sense of package authors completely and letting, e.g.,
dep:generic/ffmpeganddep:github/ffmpeg/ffmpegbe used interchangeably with no preference between them from PyPI & Python packaging’s point of view.
- Also please keep in mind that one of the main messages from the previous review round was that a canonical way to spell a dependency was needed. A number of people were not happy with leaving that up to common sense of package authors completely and letting, e.g.,
- This PEP can always be extended in the future, after we do have more experience with it in the real world. Maybe adding an
ORoperator for both normal and external dependency specifiers is the way to go, and someone will go do that. If so, great!
6 Likes
I’d put it in the same category as using direct URLs in Python dependency declarations, so applying the same restriction we do to those would make sense to me (that is, permitting PyPI to disallow them, even though they’re supported in the specification for private distribution use).
I’d personally be OK with allowing them in external extras declarations (since referencing those will be opt-in at installation time), but getting more permissive later would be easier than trying to tighten up an overly permissive initial state (and in the early stages of a rollout, we want early adopters to be reporting back to the central registry maintainers when generic and virtual names are missing from the initially defined set).
4 Likes
Personally, I’d prefer something that looked more like:
[external]
dependencies = [
"dep:generic/ffmpeg",
"dep:cargo/ripgrep"
]
[external.dependencies."dep:generic/ffmpeg"]
preferred_alternatives = [
"dep:apk/alpine/ffmpeg-libs",
"dep:deb/debian/ffmpeg",
"dep:rpm/fedora/ffmpeg-free",
"dep:deb/ubuntu/ffmpeg"
]
The alternatives here are all inherently mutually exclusive, but preferred over the generic “lowest common denominator” when available. This avoids the pitfall with “only properly packaged for one distribution, but placed on pypi” issue because it still has the lowest common denominator version expressed but then lists per-distribution preferred alternatives.
Obviously, someone intent on not supporting specific distributions still could like this, but this paired with all examples and recommendations for doing it, placing the generic version in dependencies would at least be a strong indication that it isn’t designed for that, and give people a good default way of doing it that they will find first.
5 Likes