How to address formerly unknown incompatibilities

Ah, I incorrectly assumed from previous discussion and testing I’d seen that pip would ignore earlier post releases when backtracking, but that’s evidently not the case, which @pf_moore 's idea (assuming I’m understanding correctly what specifically he’s proposing) would solve.

Yeah, or more precisely prune all but the latest post release for any given release version that has them from the set of matching versions to check/backtrack over for a version specification.

The express purpose of the post-release segment per the spec and in practice is to correct errors/issues in the packaging/release artifact (i.e. metadata, release notes, missing files, etc), and there are multiple stern warnings that post-releases should not be used for changes to the code itself.

Therefore, skipping such earlier post-releases when backtracking skips over potentially erronious metadata (if the post release corrects/updates packaging metadata), and also speeds up the solve (by skipping versions with either erroneous or unchanged metadata). The only theoretical scenario in which backtracking to an earlier post release might actually find a valid version is in the unlikely event a dependency/Python version/etc. compatibility was erroneously dropped in the latest post release, but that’s a rather pathological case and is a packaging error that can be corrected by simply issuing another post-release, and at most means pip might install a slightly older patch version rather than a potentially broken one.

You could also change the meaning of strict equality matching (==) as specified in the spec to include post-releases even without .*, but that would be a much bigger change (to the spec and in practice) that has non-trivial downsides to consider, unlike just tweaking the backtracking strategy.

That’s what I previously thought too, but as mentioned in my post above, and confirmed both from the spec and from testing it with pip, pkg == 1.0.0 will always select 1.0.0 exactly, not any post release.

Per the spec:

By default, the version matching operator is based on a strict equality comparison: the specified version must be exactly the same as the requested version. The only substitution performed is the zero padding of the release segment to ensure the release segments are compared with the same length.

For example, given the version 1.1.post1, the following clauses would match or not as shown:

== 1.1        # Not equal, so 1.1.post1 does not match clause
== 1.1.post1  # Equal, so 1.1.post1 matches clause
== 1.1.*      # Same prefix, so 1.1.post1 matches clause

And indeed, for a random example package I found, edlib (which has no deps and a release with not one but two post releases), tested on the latest pip 24.0:

$ pip install --dry-run edlib==1.3.8
Collecting edlib==1.3.8
  Downloading edlib-1.3.8.tar.gz (93 kB)
     ---------------------------------------- 93.5/93.5 kB 895.8 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Would install edlib-1.3.8
$ pip install --dry-run edlib==1.3.8.*
Collecting edlib==1.3.8.*
  Downloading edlib-1.3.8.post2.tar.gz (93 kB)
     ---------------------------------------- 93.1/93.1 kB 663.1 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Would install edlib-1.3.8.post2
$ pip --version
pip 24.0 from C:\Miniconda3\envs\py311-env\Lib\site-packages\pip (python 3.11)
1 Like