Partially dynamic project metadata proposal (pre-PEP)

[notatallshaw (Damian Shaw)]

@henryiii

Allowing the backend to decide how it wants to apply this would allow it to select the best possible variation (which I assume is ["torch>=1.2"], but ["torch", "torch>=1.2"] and ["torch>=1.2", "torch"] would both be valid options for the backend), while forcing the backend to emit ["torch", "torch>=1.2"] is less flexible and doesn’t represent the user’s wishes better.

While that simplification technically follows the spec:

1. Such simplifications aren’t well defined, in general, due to handling of pre-releases (see packaging/776#2848198645 and related discussion in Proposal: Intersect and Disjoint Operations for Python Version Specifiers)
2. Even that simplification is problematic because Packaging does not implement handling of pre-releases the same for SpecifierSet.filter with or without a specifier (which I have a PR to fix), e.g.

>>> list(SpecifierSet("").filter(["1.0a1"]))
['1.0a1']
>>> list(SpecifierSet("<=2").filter(["1.0a1"]))
[]

And regardless, this simplification necessitates choosing an order if there is another requirement in the middle, e.g. if the static part is ["torch", "pandas"] and the dynamic part is ["torch>=1.2"] then ["torch>=1.2", "pandas"] vs. ["pandas", "torch>=1.2"] can have a significance to the consuming tool.

@henryiii

while forcing the backend to emit ["torch", "torch>=1.2"] is less flexible and doesn’t represent the user’s wishes better.

If the user defined ["torch"] statically and ["torch>=1.2"] dynamically, then ["torch", "torch>=1.2"] is literally what the user requested.

I’m worried that trying to infer the intent of the user, in the spec, will lead to underspecified scenarios and create situations where different backends will produce different requirements, and different Python package installers will interpret them differently, and we will end up with users wondering why seemingly innocuous changes create different installed package.

@henryiii

The point of the rule on static dependencies is allowing tools (like dependency tree detection) to tell this package depends on at least torch. You can’t know what’s in the dynamic part, so you can’t statically compute the exact resolution anyway. We could require that the backend not reorder the static portion, though.

In anything but the most trivial cases if you’re simplifying or only deduplicating how can you not change the order?

@henryiii

So I’m in favor of a) allowing prepends, appends, insertions, or simplifications by the backend. But b) allowing prepends, appends and c) only allowing appends are also fine, I just think a) is best and allows the most flexibility to do the best thing on the backend while keeping the most important feature of the static list.

If a’s “simplifications” were to be accepted I would want to see this PEP include:

1. A well defined algorithm for producing a stable order of deduplication
2. Rigorously define what “simplify” means to packaging specifiers, including correctly following handling of pre-releases.

Largely I would be strongly in favor of c, unless there was a strictly defined behavior for backends to follow.