Pip plans to introduce an alternative (zipapp) deployment method

Hatch is also global, like tox, Poetry, etc.

edit: Basically, I don’t want users to have to install pip separately by default to get this feature.

The hatch documentation suggests pip install hatch (as well as pipx, but pip install is first). So I hadn’t appreciated that. Installing pip in a pipx-managed environment would by default manage that environment, so I now understand what you meant by wanting a --python option.

I’d strongly suggest you wait. Presumably at the moment hatch just runs subprocess.run([target_python_exe, "-m", "pip"]). That’s going to continue working for a long while yet. I’m hoping that before that becomes a problem, we’ll have something that you can depend on[1] which will let you do what you want without requiring your users to manage the pip installation that gets installed with hatch. In my experience, it’s not trivial to upgrade dependencies installed as part of a pipx-managed application. Do you really want to tell your users that when they get the “you should upgrade pip” message, they need to run pipx runpip hatch install -U pip?

Also, if you want to install pip in a pipx-managed venv, you should probably test if it works as you expect. pipx uses a “shared pip” installation to avoid having pip installed in individual environments, and I have no idea which pip would take priority, or what issues might arise, if there was also a pip in the environment.

I think this makes me feel that adding a --python option to pip might be a bad idea until we’ve worked out a clearer long-term strategy on how we want pip to be installed/managed. It’ll just result in people experimenting in ways that we might subsequently have to break, which won’t be a good experience for packaging end users.


  1. Making sure of that is basically the whole point of this thread :slightly_smiling_face: ↩︎

1 Like

Just to add another data point, I’ve used the some/path/to/python -m pip technique to install a package in a specified environment in Flit (e.g. here and here). I may have used it in other projects too, but that’s the only concrete example I can quickly recall.

I’ve also recommended that many times as the way to be sure which environment a package will be installed in - it’s not hard to get in a muddle about what running plain pip will actually do. Obviously recommendations that people have taken up are much harder to go back and change than mere code. :slightly_smiling_face:

So far this has never come up, so I’d be inclined to say it’s an unsupported corner case. If it became much more common that python -m pip didn’t work, I guess I’d have to find some way of dealing with it. But it would be with a degree of frustration, because I’m used to relying on that. :wink:

3 Likes

I see a lot of novice users having difficulty with pip installing into the wrong environment or in general not understanding what Python installations they have in their computer. If this proposal can help with that then that would be great. There is a risk though that everything just becomes even more confusing: with the proposal here even if you’re in the right environment you might not be using the right pip depending on whether your environment does or does not have pip and whether or not there is also a separate global install of pip that may or may not have been added to PATH etc.

One thing that would help a lot would be if the standalone pip itself had a way to show what Python installations and/or environments exist to help selecting the right one when choosing to install something. I’m not sure if that’s something that’s even possible though.

A natural next step if pip itself exists outside of any particular user Python installation would be to want pip to be able to install Python itself which would actually be very useful although I can imagine it is probably deemed out of scope.

I suspect it won’t - if you use the zipapp you’ll always install into the currently active environment, which is good, but the downside is that it’s a nuisance to install into an explicit environment (that isn’t activated). And forgetting to activate the environment you want is certainly something I do a lot :wink: I definitely wouldn’t recommend a zipapp for a novice at this point.

Things are already confusing, and adding more options will always make that worse, unfortunately. We’re still a long way from a simple, easily understandable solution, I’m afraid. Maybe making pip’s output better could help here. One option would be to clearly state at the top what environment we were installing into. That wouldn’t prevent errors, but it would help people work out what happened after the fact, I guess.

But I suck at UI design, so don’t trust anything I say on that matter :wink:

I’m pretty sure it’s not. You can create an environment anywhere and it’s not recorded in any sort of registry.

Very much so, for pip at least. I could imagine a “Python manager” tool along the lines of something like rustup, that handled installing Python, managing environments, and installing packages (for which I’d assume it would use pip under the hood). But isn’t that what conda does (for people comfortable with the conda ecosystem)? I’m not sure there’s much incentive to write “conda for people who don’t like conda” - but maybe someone might want to have a go.

1 Like

Yup, this is something we’ve constantly had to deal with with Spyder, as well as Python users in general, and can be tricky to troubleshoot remotely. Even though we don’t normally recommend it for the same reasons @mbussonn mentioned, IPython’s %pip magic can come in handy here in a pinch as a quick stopgap for beginners, since sometimes explaining how to find, activate and use pip in the correct environment is non-trivial—in fact, it just came up a few days ago in a user question on this very forum.

We generally guide people toward our videos, tutorials and guides on the subject, of which we’ve put a substantial amount of effort into given how common this issue is. Our desired long term solution is integrating a lot more proper package management functionality into Spyder itself, as some other more software engineering oriented IDEs do, but it is fairly complex to do this properly and not just make things work.

And of course, obligatory xkcd:

This isn’t really something within pip’s domain of responsibility, as opposed to something like conda; in Spyder we have our own routines to find, display and select conda and virtualenvwrapper environments, though there’s apparently an effort to standardize this that we’re really looking forward to if it happens:

You might want to look into something like conda instead—it basically does everything you ask for here:

  • Displays environment and packages to be installed before installing
  • Allows listing and selecting all environments
  • Allows installing Python itself (and other binary packages)

Now that pip has a proper resolver and—finally added in 22.2—dry run support, IMO displaying what will be installed into where and prompting for confirmation by default (configurable) would be a major boon to avoiding mistakes and making it clearer what pip is going to do before it does it. Conda has done this since forever when run in an interactive terminal and I personally find it utterly invaluable.

For the first time in forever, I’m actually somewhat cautiously optimistic that we might be able to have something like this in the future—at SciPy there was a fair amount of high-level talk on “both sides of the aisle” so to speak of potentially working toward new standards that would achieve a much greater measure of cooperation and interoperability between PyPA and Conda metadata and distribution formats. Not sure exactly what will come out of it, but given all that’s changed over the years it seems like something at least worth trying again, provided people’s expectations don’t get ahead of themselves…but we’ll see.

Pip as a stand-alone tool sounds exciting!

Correct me if I’m wrong, but It doesn’t decouple Python version or other environment markers right? Pip still would have a hard dependency on the Python environment in which it is executed (due to the nature of python packaging not being statically resolvable or universally cross-compilable)?

I don’t understand what you mean here. The zipapp literally just adds pip to sys.path and then runs it, so it will work the same as an installed pip.

Got it, thanks. That makes it easier to understand.

Was just imagining one reason why I might like a stand-alone zipapp would be so that I could resolve and/or install dependencies for multiple different python versions by just providing arguments to a pip.pyz

Now you’ve clarified how it works, it’s clear to me how that can’t work without running pip.pyz with the desired python interpreter/environment.

Pip has --python-version, --platform, --implementation and --abi flags for install that should let you specify the characteristics of a target system. Add that to the new --dry-run and --report options (along with --ignore-installed) and you could probably get what you want, without needing a zipapp version.

There are limitations though, since these pip options are currently ignored in environment markers evaluation.

1 Like

My workflow is that I have many virtual environments, and also I sometimes install packages with —user, though I keep that to a minimum.

I have never once activated a virtual environment. I always use /full/path/to/venv/bin/python -m pip install …. And that’s what I teach to groups I mentor.

As long as pip has some way of specifying which venv to use on its command line, I suppose that’s the same functionality I have now.

2 Likes

Hi Paul,

This piqued my curiosity so I made a proof of concept that adds a --find-script option, calling shutil.which from C code to get the absolute path. Demo (with some scripts that need their site-packages, I haven’t made zipapps yet, but at least shows two different matches):

$ ./python --find-script pip list
Traceback (most recent call last):
  File "/usr/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
$ ./python --find-script black
Traceback (most recent call last):
  File "/home/<user>/.local/bin/black", line 5, in <module>
    from black import patched_main
ModuleNotFoundError: No module named 'black'

I have to handle invalid cases like python --find-script or python --find-script does-not-exist, but if useful I could publish my branch. Cheers

2 Likes

I opened Provide a `py run` that searches for `.pyz` files · Discussion #221 · brettcannon/python-launcher · GitHub as an idea of having such a mechanism work via py run.

1 Like

FYI I opened Allow uploading `.pyz`/zipapp files to PyPI? to discuss the idea of allowing .pyz files on PyPI. The reason I bring this is up is discoverability of pip.pyz is going to be pip-specific and thus have to be hard-coded into any CI/CD system that wants to integrate this copy of pip. For instance, if VS Code were to ship this, how do we know when there’s a new .pyz copy of pip to start shipping?

And more importantly, to know where to get the zipapp file from (I assume I could use the simple API from PyPI to notice when a new version was released).

It will be at bootstrap.pypa.io alongside get-pip.py.

Great! I see there’s a https://bootstrap.pypa.io/pip/pip-22.2.2.pyz . Thanks to that version number being in the name that will make it easy to correlate back to the versions listed on PyPI.

1 Like

Note that we’re only going to offically announce the zipapp method in 22.3, so use 22.2.2 at your own risk. (The zipapp will initially be experimental, so it’ll still be at your own risk in 22.3, but it’ll be formally at your own risk, which is a bit better, I guess?)

The reason we’re making it experimental is that we’re somewhat at the mercy of our dependencies being zip-safe. We now have CI testing that pip works from a zipapp, but only for Python 3.10, and we’ve already hit one bug in 3.9 (which we’ve now worked around - importlib.resources doesn’t handle a subdirectory of a zipapp being on sys.path). So there could still be rough edges to address.

Totally understood! I just wanted to make sure an idea I have on how to use the Python Launcher for Unix to only ever have a single copy of pip installed that is always kept up-to-date and still installs to the proper interpreter/environment via the zipapp install was going to be feasible.