Python Packaging Strategy Discussion - Part 1

[dstufft (Donald Stufft)]

@pf_moore

We already have something like that with pip --python, which runs a second copy of pip using the target environment’s interpreter (basically re-using the isolated build environment code). So installing pip in every environment is no longer needed. It’s still done because (a) inertia and legacy expectations from people, and (b) tools that expect to be able to run pip in a subprocess don’t have a reliable way to find pip if it’s not installed in the environment.

Hmm, is it just running the whole pip inside that target environment? That feels more fragile that just interrogating the environment and running Pip outside of that target environment… but in any case, as long as it doesn’t depend on pip being installed in that environment then that’s excellent. I had missed that feature had already been implemented.

@pf_moore

For a long time now, I’ve wanted to do something like this. I wasn’t aware that it had previously existed. I certainly think this is a useful direction for pip to take, although I’m not sure how close that gets us to what people mean when they say “environment management” - as usual, the difficulty is likely to be in agreeing what we actually want, in sufficient detail to deliver it.

I don’t think the -E flag, which had a UX similar to that of --python ^[1] is what people mean when they say environment management. I think that the ability to install into a Python you’re not running on is, or well was, the main technical blocker to doing so. With that ability, the problem then shifts to defining what it is we want pip to actually do.

Like just as an example, if we decide that something like PEP 704 is the way forward, we could actually modify that such that instead of erroring out for people. pip locates where the environment should be, creates it, and then runs as if --python .venv had been run.

Given the existance of --python, I think the hardest parts then become:

• Getting agreement that evolving pip to be that unified tool is a good path forward, or at least could be a good path forward.
• Getting agreement on what our desired end state actually looks like.

For the first of those, I think we can make a rough consensus decision on this thread. It wouldn’t block the ability of other tools to continue to exist, iterate, and compete. It would just be declaring that we view a future where pip was the primary tool for interacting with Python packaging for the 80% use case as a reasonable outcome. It wouldn’t be any sort of mandate, it would just really be keeping pip where it is now, but extending it so that people don’t have to also learn twine, virtualenv, etc.

If/Once we had that rough consensus, the for the second of those, I think maybe we’d be best served by taking proposals for what exactly we think our endstate goal should look like, what commands exist, what do they do, etc. Then weigh between them and figure out what path we can take to get from where we are now, to where we want to go.

Then we would just need someone(s) to look at those proposals, and pick one as the roadmap for unification. That could be through the PEP process, or it could just be treated as a pip issue and let the pip maintainers select one. Then we “just” work towards that end goal ^[2].

---

1. Except implemented in a bad way, it executed the pip that was installed in the target environment. ↩︎

2. Of course, we can mutate that end goal as needed if we change our mind or something becomes more obvious as implementation happens. ↩︎