Install prerelease of a project but not of its dependencies

Packaging
1

For context see these sympy and pytorch issues.

The pytorch issue is that people want to install nightly releases of pytorch but pytorch depends on sympy which depends on mpmath. Using pip install --pre torch results in installing mpmath 1.4.0 alpha 0 which is incompatible with the current release sympy 1.12. As a temporary fix sympy will release 1.12.1 alpha 1 with a fix for compatibility with mpmath 1.4.0 alpha 0. This is only a temporary fix because future prereleases of any of pytorch’s dependencies could break pip install --pre torch again.

My question here is basically the same as this SO question:

What is a good way with pip or other tools to install a prerelease of a package without installing prereleases of its dependencies?

If A depends on B and both A and B have recent prereleases I would like to install the prerelease of A but I want to get the normal release of B. Neither of these is what I want:

pip install A        # full releases of both A and B
pip install --pre A  # prereleases of both A and B

One solution is:

pip install B        # normal release of B
pip install --pre A  # prerelease of A

The downside of this is that I need to list all of A’s dependencies which might be a long list and also if A adds new dependencies in future then I need to update this. Project A could make a requirements.txt for this so that it is:

pip install -r requirements-A.txt
pip install --pre A

The problem with this is where do you put the requirements-A.txt file? You could put it in A’s codebase but then that’s no use for someone who is trying to install A from PyPI since they do not have A’s code. Also it seems unnecessary to make a redundant requirements.txt when all of the necessary information is already in A’s packaging metadata.

The solution suggested on SO is:

pip install 'A >= 1.0.dev1'

As I understand it including something that looks like a prerelease version in A’s version constraint will cause pip to look for prereleases of A but not of A’s dependencies which is what we want. This solution looks cryptic though and including a version 1.0 in the constraint suggests that 1.0 is somehow significant even though we would want to use this command to install new versions of A long after version 1.0. Probably this is as good as can be done if we need to encode this in a PEP 440 version constraint but I think that there should be a better way to spell this both for version constraints and for command line options.

Am I missing something obvious here or is A >= 1.0.dev1 the way to do this?

There were recent discussions I think about having something like:

pip install --only-dependencies A
pip install --pre A

It seems to me though that it is quite likely that someone using --pre actually might not want prereleases of all transitive dependencies. Whenever I have personally used --pre it was just because I wanted prereleases of one particular project.

2 Likes
2

The accepted solution is from StackOverflow, but don’t forget that you are still constraining the version there, so you could have:

pip install 'A >= 1.2.3b4, < 2.0a0.dev0'

Of course, it’s a very unique scenario to always install the dynamically-determined latest pre-release version, while not knowing the precise version you want to install:

pip install A==1.3.5a7
3

I think it is a fairly common thing to do e.g. for a CI job. If you want to track latest upstream changes then one thing to do is to have a CI job that runs off of upstream git. If the project is difficult to build though then you might want to use nightly wheels rather than build from git. If you don’t quite want the instability of nightly builds then tracking alpha, beta releases might be better. The purpose of prereleases is to test downstream breakage so it makes sense for downstream to install latest prereleases always in a CI job.

SymPy’s CI has separate jobs that test SymPy’s master branch with:

  • Latest prereleases of CPython.
  • Nightly wheels of numpy and scipy
  • mpmath’s master branch

I don’t want any of these to make the whole dependency stack nightly though e.g. I don’t need CPython to be built with the latest gcc master branch or anything like that.

1 Like
4

What about:

pip install A
pip install --upgrade --pre A
5

If the prerelease of A has a new dependency C or a new lower version cap like B > 1.0 would this install new versions of B and C and could prereleases of B and C end up being installed?

1 Like
6

As long as the installed B still satisfies the updated requirement, then it should not attempt to install a pre-release of B. If there is a new dependency C that would not have been installed otherwise, then yes, pip would look for a pre-release there.

I’m not sure you’ll get something completely bulletproof, but that C case does seem like it’s going to be rarer and can be most easily handled by temporary updates to CI and/or a dependency list.

I use tox to handle these cases where a global --pre is too brittle:

1 Like
7

Actually this version doesn’t work either if the prerelease of A has different dependencies from A. We want the dependencies of the prerelease of A but not prereleases of those dependencies.

8

Just for your and others’ reference, here’s the canonical issue for this feature, which is something I and many others would find quite useful and which Conda has had since forever:

2 Likes
9

--no-deps exists. Does this help?

pip install A
pip install --upgrade --no-deps --pre A
1 Like
10

… But if you’re already running CI tools, wouldn’t you already have a tool that dynamically determines the latest pre-release version for you?

11

Yeah, pip doesn’t really have a good option to do that. And the spec isn’t particularly clear on how prereleases should be handled.

pip install 'A >= 1.0.dev1'

(or probably better than an arbitrary 1.0.dev1, use the latest prerelease you’re aware of) is the best solution right now, for pip at least. I don’t know if uv will treat it the same way as pip - it’s very much implementation-defined behaviour.

2 Likes
12

That tool is pip. It just doesn’t provide the dependency resolution configuration that I want. I now realise that although I have used pip install --pre many times it has never really meant what I wanted it to mean.

2 Likes
13

Like other spellings above this will probably work most of the time but i think it is still not right. What if the dependencies or dependency constraints of the prerelease of A are different from those of A?

For me the purpose of doing e.g. pip install --pre sympy is not that I want pip to install a prerelease of mpmath or anything else. Rather I want to test what the next release of sympy is going to be like by testing sympy’s prerelease (say 1.13rc1) now. The dependency versions of that release are a major part of the release itself. I want pip install --pre sympy to install sympy 1.13rc1 along with the exact version of mpmath that we expect would be installed if I were to run pip install sympy in the future after sympy 1.13 is released.

I realise now that that is not what pip’s --pre feature does. I can also see situations where the actual behaviour of --pre makes sense but it has never been what I wanted to do when asking pip for a prerelease of something.

2 Likes
14

Thanks Paul. I haven’t used uv yet but at least from the README A >= 1.0.dev1 looks like the way to configure this with uv as well.

Apparently uv has a --prerelease=allow option that applies to all dependencies but it is not clear to me if that is effectively the same as pip’s --pre option. Allowing prereleases for satisfiability is not the same as preferring the latest ones (as pip does).

15

You may be interested in Handling of Pre-Releases when backtracking? which has a fairly extensive discussion of prerelease handling.

16

To confirm though, many users use something like A >= x.y.dev0 to get their installer to select a prerelease for that package and not anything else. As this feature is implied by the spec, even if not intended, it seems like installers don’t feel the need to add a per package option through their UI.

If you want a package to be selected for a prerelease if it is collected as part of the dependency chain, but you don’t want to make it an explicit requirement, you can pass this same prerelease specifier in via a constraint, this works for both pip and uv.

In general installers have a global prerelease option, where prereleases are considered for everything, the spec implies that this should exist for installers.

The main difference, at least currently, between pip and uv is that uv must know upfront whether to search on a prerelease, i.e. you must provide it in your requirements or only prereleases are available for the package. Whereas pip, at least currently, will select a prerelease when backtracking if it encounters a transitive dependency that has a prerelease marker in the version of the specifier. Poetry will go further, if there exist final and prereleases but your requirements would only be solved by prereleases it will select prereleases, even if no prerelease markers are included in the versions of any specifiers during the resolution, this is suggested to be the correct behavior by the spec.

The other difference is that uv accepts exclusionary ordered prerelease specifiers, e.g. A < x.y.dev10 where as pip doesn’t because pip inherits this behavior from packaging and packaging isn’t following the spec here: Specifier `<` that contains prerelease version does not match on valid prerelease versions · Issue #788 · pypa/packaging · GitHub

1 Like
17

Ah thanks. I remember reading that thread at the time but I guess it seemed like it was discussing esoteric situations that would not concern me. Now that I have a concrete situation where I understand what the problem is that thread makes more sense and any question I would have asked is answered there as well as any suggestion I would have made.

The conclusion then is that A >= 0a0 is the current way to do this at least with pip. I don’t see that this behaviour is mandated in the spec but it sounds like other tools use it too.

It was suggested in the other discussion to have a per-distribution flag like --allow-prereleases-for=numpy. This sort of sounds like what I would want but I’m not sure if I understand all the implications. I don’t just want to allow prereleases but rather I want to force choosing the most recent prerelease of one particular package.

If the most recent prerelease of sympy is 1.13rc1 then I would want pip install --new-pre sympy to be almost equivalent to pip install sympy==1.13rc1. I say almost because it is possible that the requirements of sympy 1.13rc1 might be unsatisfiable and there are different ways to handle that. Probably the best default behaviour is to find the most recent prerelease for which the dependencies are satisfiable. I want the “most recent prerelease” objective to take priority over everything else except satisfiability during dependency resolution like:

for version in reversed(all_releases_including_prereleases):
    requirements = get_requirements(version)
    if satisfiable(requirements):
        return version, requirements

I think every time I have used --pre this is basically what I wanted it to do.

I can see that using this with more than one distribution simultaneously would get complicated but I’m not sure that I have ever actually wanted that and I don’t know how much it matters.

1 Like
18

If we just consider one prerelease it would non-trivial to efficently come up with a resolution algorithm to get this, basically for each prerelease newer than the latest final release you would need to exhaust all possible resolutions and confirm that in fact you need to backtrack to a final release.

It’s a little easier if your resolution algorithm supports the concept of ranges, and that you can validate the dependency metadata between prereleases doesn’t change, for example this can be done with pubgrub (what uv uses) but not resolvelib (what pip uses).

If we consider multiple prereleases I think this idea quickly becomes complex and easily produce solutions the user wasn’t expecting. What if you specify this for both package A and package B and the latest prerelease of A is not compatible with the latest prerelease of B? Now you have two solutions and it’s ambigious which one to choose. What if there is some combination of prereleases that are combatible with each other but not the latest ones, do we exhaust every combination, and if so, out of valid combinations which ones do we prefer?

2 Likes
19

One thing I think would be easier to implement and I’ve seen user demand for, and would mostly support this scenario, is foo==latest, and by extension something that included prereleases foo==latest-prerelease. (Would also have to consider if latest includes post releases?)

This would fail if the resolution couldn’t be constrainted to the latest version of that package, but many users have said they would prefer that.

I suppose someone would have to engage in the PEP process to look at getting that though.

2 Likes
20

One other possibility here - although I acknowledge it involves more moving parts than you might want - is to use something like simpleindex to proxy PyPI and selectively make visible only the versions you want pip to consider.

1 Like