Yup, this is something we’ve constantly had to deal with with Spyder, as well as Python users in general, and can be tricky to troubleshoot remotely. Even though we don’t normally recommend it for the same reasons @mbussonn mentioned, IPython’s %pip magic can come in handy here in a pinch as a quick stopgap for beginners, since sometimes explaining how to find, activate and use pip in the correct environment is non-trivial—in fact, it just came up a few days ago in a user question on this very forum.
We generally guide people toward our videos, tutorials and guides on the subject, of which we’ve put a substantial amount of effort into given how common this issue is. Our desired long term solution is integrating a lot more proper package management functionality into Spyder itself, as some other more software engineering oriented IDEs do, but it is fairly complex to do this properly and not just make things work.
And of course, obligatory xkcd:
This isn’t really something within pip’s domain of responsibility, as opposed to something like conda; in Spyder we have our own routines to find, display and select conda and virtualenvwrapper environments, though there’s apparently an effort to standardize this that we’re really looking forward to if it happens:
You might want to look into something like conda instead—it basically does everything you ask for here:
Displays environment and packages to be installed before installing
Allows listing and selecting all environments
Allows installing Python itself (and other binary packages)
Now that pip has a proper resolver and—finally added in 22.2—dry run support, IMO displaying what will be installed into where and prompting for confirmation by default (configurable) would be a major boon to avoiding mistakes and making it clearer what pip is going to do before it does it. Conda has done this since forever when run in an interactive terminal and I personally find it utterly invaluable.
For the first time in forever, I’m actually somewhat cautiously optimistic that we might be able to have something like this in the future—at SciPy there was a fair amount of high-level talk on “both sides of the aisle” so to speak of potentially working toward new standards that would achieve a much greater measure of cooperation and interoperability between PyPA and Conda metadata and distribution formats. Not sure exactly what will come out of it, but given all that’s changed over the years it seems like something at least worth trying again, provided people’s expectations don’t get ahead of themselves…but we’ll see.
Correct me if I’m wrong, but It doesn’t decouple Python version or other environment markers right? Pip still would have a hard dependency on the Python environment in which it is executed (due to the nature of python packaging not being statically resolvable or universally cross-compilable)?
Got it, thanks. That makes it easier to understand.
Was just imagining one reason why I might like a stand-alone zipapp would be so that I could resolve and/or install dependencies for multiple different python versions by just providing arguments to a pip.pyz
Now you’ve clarified how it works, it’s clear to me how that can’t work without running pip.pyz with the desired python interpreter/environment.
Pip has --python-version, --platform, --implementation and --abi flags for install that should let you specify the characteristics of a target system. Add that to the new --dry-run and --report options (along with --ignore-installed) and you could probably get what you want, without needing a zipapp version.
My workflow is that I have many virtual environments, and also I sometimes install packages with —user, though I keep that to a minimum.
I have never once activated a virtual environment. I always use /full/path/to/venv/bin/python -m pip install …. And that’s what I teach to groups I mentor.
As long as pip has some way of specifying which venv to use on its command line, I suppose that’s the same functionality I have now.
This piqued my curiosity so I made a proof of concept that adds a --find-script option, calling shutil.which from C code to get the absolute path. Demo (with some scripts that need their site-packages, I haven’t made zipapps yet, but at least shows two different matches):
$ ./python --find-script pip list
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
$ ./python --find-script black
Traceback (most recent call last):
File "/home/<user>/.local/bin/black", line 5, in <module>
from black import patched_main
ModuleNotFoundError: No module named 'black'
I have to handle invalid cases like python --find-script or python --find-script does-not-exist, but if useful I could publish my branch. Cheers
FYI I opened Allow uploading `.pyz`/zipapp files to PyPI? to discuss the idea of allowing .pyz files on PyPI. The reason I bring this is up is discoverability of pip.pyz is going to be pip-specific and thus have to be hard-coded into any CI/CD system that wants to integrate this copy of pip. For instance, if VS Code were to ship this, how do we know when there’s a new .pyz copy of pip to start shipping?
And more importantly, to know where to get the zipapp file from (I assume I could use the simple API from PyPI to notice when a new version was released).
Great! I see there’s a https://bootstrap.pypa.io/pip/pip-22.2.2.pyz . Thanks to that version number being in the name that will make it easy to correlate back to the versions listed on PyPI.
Note that we’re only going to offically announce the zipapp method in 22.3, so use 22.2.2 at your own risk. (The zipapp will initially be experimental, so it’ll still be at your own risk in 22.3, but it’ll be formally at your own risk, which is a bit better, I guess?)
The reason we’re making it experimental is that we’re somewhat at the mercy of our dependencies being zip-safe. We now have CI testing that pip works from a zipapp, but only for Python 3.10, and we’ve already hit one bug in 3.9 (which we’ve now worked around - importlib.resources doesn’t handle a subdirectory of a zipapp being on sys.path). So there could still be rough edges to address.
Totally understood! I just wanted to make sure an idea I have on how to use the Python Launcher for Unix to only ever have a single copy of pip installed that is always kept up-to-date and still installs to the proper interpreter/environment via the zipapp install was going to be feasible.