Handling of Pre-Releases when backtracking?

Packaging
1

Currently the document on handling pre-releases states the following (Version specifiers - Python Packaging User Guide):

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

Currently my understanding is Pip follows this behavior when initially collecting packages, but not when backtracking through transitive dependencies (as came up here: Dependency resolution with pre-releases · Issue #12049 · pypa/pip · GitHub).

My question is Pip’s behavior expected from the specification?

Adding this behavior to backtracking I think might require careful design decisions in the resolution process, as it could be implemented in different ways and possibly any solution could have significant impact on performance.

2

Perhaps @pradyunsg or @pf_moore might want to comment on this, as Pip maintainers?

3

I’m trying to understand if there’s an agreement on the spec more than whether Pip made this as a deliberate choice or not.

Specifically as I found that rip does not handle pre-releases correctly (rip does not follow spec for handling of pre-releases · Issue #118 · prefix-dev/rip · GitHub) and I wasn’t sure from reading the spec and observing what pip does what is the “correct behavior”.

4

There’s one specific behaviour that pip has that could lead to looking like what you’re describing:

If a requirement specifier includes a pre-release or development version (e.g. >=0.0.dev0) then pip will allow pre-release and development versions for that requirement.

Going off of memory, pip uses pre-release versions for a package if:

  1. There are only pre-release versions for a project.
  2. The version specifier is a pre-release specifier.
  3. --pre was passed.
5

This sounds right to me as well.

I’m not honestly convinced that pip’s behaviour is as described in the linked issue comment. As far as I know, pip should choose what releases to consider based purely on the conditions @pradyunsg noted, and not on what it’s doing (installing or backtracking). I’d need more evidence of why you think pip’s behaviour is as the comment describes before I’d be willing to accept that.

As far as the spec is concerned, I believe the behaviour described by @pradyunsg is PEP-compliant and is what pip implements.

6

Hmmm, well unfortunately I don’t have a device with that platform, otherwise it would be possible for me to confirm it easily with pypi-timemachine. I’ll see if modifying the platform specifier is enough to reproduce it.

Fair enough, if I get time I will set up a local index with constructed packages to demonstrate the two different scenarios and see how Pip behaves.

7

FYI, I’ve started filing bugs against Pip where I do not see it match the “Handling of pre-releases” documentation, e.g.

I am writing up a reproducible test case for the example I gave above, but it’s a little more involved and will require building at least 5 package versions to demonstrate. I will post here once ready.

2 Likes
8

I’ll note that item (1) is slightly different from the spec - it’s when there are only pre-releases, not when only pre-releases are selected by the specifier under consideration.

It’s quite possible that changing to “dynamically” deciding whether to use pre-releases depending on the specifier could be difficult or have backward compatibility issues (or user-unfriendly consequences). The existing behaviour is built into the “finder” level of pip, which is pretty early in the processing.

My question here is how important is this specific detail of the spec? While pip’s behaviour has caused questions in the past, my impression is that the behaviour in the spec would probably be no less confusing, and I’m not aware that anyone has ever found pip’s behaviour to be an actual problem (once they understand how it works). Does any tool (such as poetry) implement the rule as written in the spec? And if so, how does it compare to pip’s approach?

I guess what I’m asking is whether we should be looking at changing pip or changing the spec here.

Having said all of this, if someone were to submit a PR to pip that changes the behaviour to match the spec, I wouldn’t object. I just don’t personally feel that it’s the most worthwhile area of pip to spend time on.

9

It’s other tools that led me to start writting these issues against Pip. For example RIP intends to be a Pip replacement, in some sense, but does not currently handle pre-releases the same way as Pip: rip does not follow spec for handling of pre-releases · Issue #118 · prefix-dev/rip · GitHub, which made me realize Pip does not follow the spec here.

Such tools now need to decide whether to follow Pip, follow the spec, or implement their own behavior as best fit their architecture.

2 Likes
10

Agreed. The point of having a spec is to ensure that users get the same results regardless of tool. So I’d say everyone should follow the spec.

What’s not clear to me is that the current wording of the spec is actually the right solution. It looks like it’s never been proven in a real-world tool, which suggests to me that implementing the spec as is could be a worse solution than changing the spec (either to reflect pip’s behaviour, or to specify something better than what pip does).

And just to further complicate the matter, we’re very close to the question of whether standards should be allowed to dictate tool UI. The various SHOULD statements in the spec are basically tool UI recommendations, and I think they should probably mostly be omitted, or put into some sort of non-normative “UI discussion” section. Right now they mostly read like “we aren’t sure the mandated behaviour is right, so we think tools should offer a bunch of options to work around that”.

2 Likes
11

RIP developers have chosen, for now, to follow Pip.

As best as I can tell, Poetry seems to follow the spec. When I give the requirement opentelemetry-exporter-prometheus = ">0.1" I get 0.43b0, pandas = ">2" I get 2.1.4, and pandas = ">2.1.4" I get 2.2.0rc0.

Poetry does seem to struggle with the consequences of this spec when resolving transitive dependencies: poetry won't check for pre-releases during dependency resolving of non-direct dependencies · Issue #4405 · python-poetry/poetry · GitHub. The same thing I highlighted for Pip in the original post here (but now I think is covered by Pip not following the spec in general).

12

Found two further issues with Pip inconsitently handeling pre-releases, even against the rules that @pradyunsg outlined:

13

As I noted on one of the issues, I think this discussion would be better served by consolidating everything into one larger item on the pip tracker, “Pip’s pre-release version handling doesn’t follow the spec”, which might well need to be a funded project, rather than simply a bunch of bugs to be fixed.

The discussion here should focus on whether there is anything in the specification that needs to be clarified, corrected, or rewritten. If pip’s behaviour is problematic, but doesn’t violate the spec, then that’s a spec issue to be fixed first, before worrying about what might need to change in pip.

14

Well, I have never written a specification for tools before. But as I understand your viewpoint in other threads, currently the spec doesn’t mandate almost anything of the handeling of pre-releases for tools because it only uses SHOULD and MAY.

I would say, if that’s the case, the problem at the moment is that:

  1. Two different tools can “follow” the spec
  2. Two different tools can receive the same requirements
  3. One tool can tell the user that the requirements has no resolution according to the spec, and the other tool can provide a valid solution to the requirements according to the spec

IMO I think any specification on requirments MUST be well enough defined that two different implementations can agree on if a requirements is impossible or not.

Do you think that’s a reasonable take?

1 Like
15

I think it’s a reasonable position for a proposal to take.

I think it’s also reasonable for a proposal to say that certain cases are undefined, and tools can give different results in those situations.

I don’t personally have a view over which position is the more useful, in a practical sense. I very rarely[1] work with pre-release versions in a context where it matters if two tools do different things. So I’m happy with either situation.

However, as far as the current spec goes, I’ve said this before, but I consider the following paragraph, taken directly from the spec, to be the only normative statement that’s made about pre-releases:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

Note that the filter method of specifiers in the packaging library implements this spec, and the way it does so is (IMO) neither complex nor subject to much debate. So I think it’s reasonable to claim that the existing spec is fine, in terms of being something that different tools can follow in a consistent way.

I think you’re reading too much here into pip’s handling of pre-releases. Instead of seeing it as an “alternative interpretation”, you should probably just accept that it’s not compliant with the spec, but it’s burdened with a lot of legacy behaviour that tries too hard to “do what I mean” and in doing so makes things look harder than they really are.

  1. if at all ↩︎

16

I don’t have any strong opinions on the the packaging library filter method implementation or how well it follows that spec, although I have been recently testing boundry conditions around version specifiers and am finding and reporting bugs.

But, to me, the spec is a lot less clear for the use case package installers face than the packaging library faces. As package installers must resolve multiple requirements of multiple packages. Off the top of my head here are questions I would have if I was writing my own package installer:

  • If one of these conditions: “are already present on the system”, or "explicitly requested by the user, “or if the only available version that satisfies the version specifier is a pre-release” is met:
    • Does that mean pre-releases will count for just that package or for all packages?
    • If a requirement is specified on an optional extra and that requirement can only be satisfied with a pre-release, but without the optional extra it could be be satisfied with a final release, should the pre-release or final release be the solution?
  • For the requirement “explicitly requested by the user”:
    • Does that mean the user has to pass some flag to the installer? Or is it good enough to pass a version that includes a pre-release number?
    • What if the pre-release specifier is included in a dependency or transitivty dependency? Does that count as “explicitly requested by the user”?
    • What if the user has a requirement on a package that doesn’t include a pre-release but a dependency has a requirement that does include a pre-release?
    • What if the user requires a pre-release in one requirement but not in another on the same package?
  • For the statement “only available version that satisfies the version specifier is a pre-release”?
    • Should this be applied when two transitive dependencies can only be resolved by a pre-release?
    • If so, when backtracking and two transitive dependencies can only be resolved by a pre-release should the package installer fully exhaust the dependency graph and make sure there are no solutions that do not involve a pre-release? Or is it good enough to present the found solution where the found transitive dependencies can only be resolved by a pre-release?

I don’t think the spec you’ve quoted unambigiously answers any of these questions.

1 Like
17

Cool. If you think these questions must all be answered unambiguously, you can propose improvements to the spec.

If you want my (very much off the top of my head, I reserve the right to contradict myself without prior notice the next time you ask me!) thoughts:

  • The “version specifiers” spec is about comparing versions to a spec. Everything is at the “which versions from this list match the spec” level. So the answer to your “all packages or just this one” questions is always “just this one”.
  • Same with questions about extras - the spec is about “does X.Y match >=W.Z”, extras are not relevant at that level.
  • Explicitly requested is (IMO) a matter for the tool to decide, and as such, tools may well differ.
  • I can’t see any justification in the spec for the idea that “if the specifier includes a pre-release identifier, pre-releases should be included”. Both pip and the packaging library seems to implement this, but I don’t see any explanation of why. I can only assume it’s considered as “explicitly requested”. Personally, I think this is too subtle, and too much a case of “do what I mean”, not least because it triggers all the questions you asked…
  • I’ve already mentioned on the pip issue that the spec doesn’t cover how things work when multiple specifiers need to be satisfied at once. That’s a spec issue.
  • Backtracking is out of scope for the spec, so it’s a tool issue. Of course tools can document their backtracking behaviour, and even agree to follow the same rules, but it’s not something that this spec covers. (Not all resolvers even backtrack - pip’s legacy resolver didn’t, for example).

But in reality, as I said, I mostly don’t care what the answers are - for me, this has no practical impact, it’s just a theoretical exercise. Someone who is actually affected by how all this works should think about it and make a proposal.

Edit: One further thought on backtracking. Given that this thread is titled “Handling of pre-releases when backtracking”, talk of the spec is somewhat off-topic, as the spec doesn’t cover backtracking. And pip’s backtracking behaviour is an implementation detail that I’d be reluctant to document in any depth as we might change it at any point (possibly even to a resolver that doesn’t backtrack at all). I think we need to be careful to be precise about what we’re discussing, or things will end up even more confused than they currently are… :slightly_smiling_face:

18

Well, I think most of these questions are required for my proposition:

And assuming that proposition I think extras and dependency resolution (whether it’s covered by a backtracking algorithm or something else) would be a required part of the specification.

Yeah, I think based off your replies there’s a few small tweaks that could be proposed to make it less ambigious and not need to get involved the entire community of stakeholders. What is the process to do that?

I do think, however, most changes would ideally involve maintainers of multiple package installers. And therefore it’s outside my capability, so all I can do is report that it doesn’t appear any package installer follows the spec, and the spec itself isn’t clear in many cases.

1 Like
19

We don’t have a standard for dependency resolution. We have a standard for version specifiers that explains their behaviour in terms of a list of available versions, but that’s very different and only a tiny part of dependency resolution.

Should we have a standard for dependency resolution? I don’t know. I feel like the answer is probably no, we shouldn’t. There’s just so many areas that would need to be agreed on, and if we pin them all down, we stifle innovation and experimentation. As a pip maintainer, would I support a standard for dependency resolution? I don’t think so - I can’t see how it would benefit us, and yet I can see many ways it would make our life harder.

Speaking as one of the people who implemented pip’s new resolver, I can pretty much guarantee that if you wanted to write a standard for dependency resolution, I could throw many questions at you that you wouldn’t be able to answer. Many of them are on the pip tracker in some form or another, and for a lot of them, we’ve never found a good solution.

I’m not sure how true that is (fitting anything about extras into the version specifier spec seems like it would be a non-starter, for example) but if you think so, by all means have a go.

Basically you’d need to write a PEP. If you think the change is small enough to count as a “clarification” rather than a “change”, you can post it to Discourse as a proposal, and if there’s consensus that it’s sufficiently non-controversial, I can approve it as a “clarification only” change. Be warned, though, that I’ve been bitten in the past by allowing changes that turned out to cause issues, so I’m generally reluctant to do this for any but the most obviously trivial changes.

1 Like
20

As the one of the original author of PEP 440, I can say pretty definitively that the reason that the PEP says this:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

and not this

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system or explicitly requested by the user.

Is because there were widely used packages at the time that had versions which didn’t technically comply with PEP 440 (because they predated PEP440) and got interpreted as prerelease versions, and we were trying to prevent breakages (a lot of effort in PEP 440 and a lot of weird edge cases came from trying to gain another 1%-2% of compatibility with existing versions on PyPI). One such package at the time was pytz which used versions like 2005a.

There were also a number of packages that were only available as pre-releases (for one reason or another) and we wanted to avoid additional breakages.

At the time of PEP 440, none of the Python installers had any sort of real dependency resolving or backtracking or anything of that nature, so the implications of that statement, wrt to how it impacted backtracking/resolving simply wasn’t a thing that existed to be considered. I suspect this is why (1) from above is built into the Finder instead of into the resolver as well.

I don’t have nearly enough time to do it (though I’d be willing to offer opinions where folks cared) but I suspect that a follow up to PEP 440 may be warranted to clean up some of the edge cases or “weird corners” that we put in explicitly for compatibility. It’s been 10 years since PEP 440 and I suspect a lot of those cases simply no longer apply to anything that has been released recently, and we’ve learned things as an ecosystem that weren’t obvious a decade ago.

Some possible things (would need to see how widely used each of these are, so these aren’t concrete suggestions):

  • I would probably drop most of the information about prerelease handling, and just put in a recommendation that tools should exclude prereleases by default unless the end user provides some configuration or flag to tell it to use prereleases. I think this matches what most ecosystems that treat prereleases specially at all do now.
  • Figure out if anyone ever actually used Epoch, and if not does it make sense to keep supporting it.
  • Disallow combinations of pre-release, post release, and dev releases.
  • Add specifiers that are easier to use (The biggest one being ^=, which is a semver compatible update is probably the major one of these).
  • Ditch the === specifier, which I suspect might never have been used?
  • Maybe support grouping and and and or? Currently and is supported through ,, but more complex expressions may mean that some complicated version specifiers are easier to specify using or and/or ( ) for grouping (though this also extends to PEP 508, and it might be better to do this there).

I dunno though, I do think there’s been enough pain points come up over the years, and if someone is going to tackle a Versioning v2 (or well… I guess v5?, distutils, setuptools, distuils2, pep 440, and hypothetical new thing), I think it would be very useful to try and go through past discussions and issue trackers to find out where they are.

As far as specifying dependency resolution itself goes, I don’t think that is something we should do. In theory any version that matches the specifier should be acceptable, and how the resolver works shouldn’t matter, except to users of that tool.

7 Likes