That’s a great point. It won’t be able to avoid that, because the standard for recording installed packages doesn’t have any way to record what extras were specified when a package was installed.
So without a change to this specification, it’s not possible to know what extras are installed, and freeze/reinstall will always install the default extras for any installed package. This doesn’t just apply to pip install package[minimal], it also applies to pip install package[added_functionality], and in the worst case could result in a broken environment (if the default extra dependencies conflict with what’s in added_functionality).
Just to make sure I understand, is there any obvious reason why one couldn’t propose in the PEP an update to this specification to record extras that were actually included at install time? Or would it need to be done via a different PEP or via a different process?
No, it would be fine to do that. However, what extras were specified when installing should probably be considered as “provenance data” for the installed package, and as such, be recorded in the provenance_url.json file defined by PEP 710 (which is not yet accepted). So you would probably have to wait until that PEP was approved and implemented before adding this.
It also wouldn’t help for existing environments without the new metadata, which wouldn’t be copiable using freeze/reinstall. That’s a question that you’d have to discuss in the PEP - is it OK to break copying those environments via pip freeze?
The problem here is that the “Backward Compatibility” statements in the current PEP are wrong. There is a backward incompatibility introduced by this proposal. Currently, pip install pkg[xtra] has identical behaviour to pip install pkg x1 x2 x3 where x1…x3 are the packages defined in xtra. This proposal breaks that equivalence, and hence the workflow of anyone relying on it. The behaviour of pip freeze is simply one example of code that relies on that equivalence.
I believe pip install --no-deps -r requirements.txt would be the way to recreate an environment without attempting to re-resolve. I think lock files are probably the better long-term solution, but this seems to be manageable at the moment. It might make sense to conventionally use something like environment.txt to signify that it’s not just requirements but a resolved environment.
The ambiguity here comes from pip using requirements files for two different things rather than having a distinct lock file. When installing from a lock file --no-deps should be the expected behaviour.
Yes, having had a chance to think about it more, there are issues with treating the output of pip freeze as a requirements file even now - for instance:
pip install pandas # installs numpy as a dependency
pip uninstall numpy
pip freeze > requirements.txt
pip install -r requirements.txt # will install numpy
Here numpy will get installed even though it was uninstalled before running freeze - just to illustrate that if what you want is to replicate the environment output by pip freeze then you do need to use --no-deps.
We can’t keep track of extras that were requested in the pip freeze output (i.e. we can’t make it so the output of pip freeze includes e.g. package[extra]) since there’s no guarantee that none of the packages originally pulled in as part of those extras are still present - they might have been removed subsequently.
So I’m not sure this is something that needs to be fixed as part of the PEP - people writing requirement files by hand can include [minimum] in the requirement file if they want a minimal install, and people using pip freeze and then wanting to recreate that environment can use --no-deps when installing.
I remember there were similar discussions on this forum, but I only found this one.
At first, I guessed that ; extra != "lite" could work, but I tried and found that pip could not correctly exclude this dependency when [lite] is requested.
Is this result expected, and do we have any plans to support this kind of environment markers in pip? I planned to implement it in PDM but I think it’s better to ask for pip maintainers before doing that.
The “extra” variable is special. It is used by wheels to signal which specifications apply to a given extra in the wheel METADATA file, but since the METADATA file is based on a draft version of PEP 426, there is no current specification for this. Regardless, outside of a context where this special handling is taking place, the “extra” variable should result in an error like all other unknown variables.
The reference to PEP 426 makes no sense to me, as (1) PEP 426 was withdrawn, and (2) I don’t understand how what it says about extras relates anyway.
I’d say that pip is unlikely to change its current behaviour without the marker spec being modified to properly define how an “extra” marker works.
In effect, the requirements are evaluated with req.marker.evaluate({"extra": ""}) in case of no extras and any(req.marker.evaluate({"extra": extra}) for extra in extras) in case there’s one or more extras being used.
Do you think we should start a PEP process to make this improvement? Since i’ve seem people complaining this many times and this can solve the problem without introducing new metadata, but rather clarify and fix the existing extra marker.