Yup, this is something we’ve constantly had to deal with with Spyder, as well as Python users in general, and can be tricky to troubleshoot remotely. Even though we don’t normally recommend it for the same reasons @mbussonn mentioned, IPython’s %pip magic can come in handy here in a pinch as a quick stopgap for beginners, since sometimes explaining how to find, activate and use pip in the correct environment is non-trivial—in fact, it just came up a few days ago in a user question on this very forum.
We generally guide people toward our videos, tutorials and guides on the subject, of which we’ve put a substantial amount of effort into given how common this issue is. Our desired long term solution is integrating a lot more proper package management functionality into Spyder itself, as some other more software engineering oriented IDEs do, but it is fairly complex to do this properly and not just make things work.
And of course, obligatory xkcd:
This isn’t really something within pip’s domain of responsibility, as opposed to something like conda; in Spyder we have our own routines to find, display and select conda and virtualenvwrapper environments, though there’s apparently an effort to standardize this that we’re really looking forward to if it happens:
You might want to look into something like conda instead—it basically does everything you ask for here:
Displays environment and packages to be installed before installing
Allows listing and selecting all environments
Allows installing Python itself (and other binary packages)
Now that pip has a proper resolver and—finally added in 22.2—dry run support, IMO displaying what will be installed into where and prompting for confirmation by default (configurable) would be a major boon to avoiding mistakes and making it clearer what pip is going to do before it does it. Conda has done this since forever when run in an interactive terminal and I personally find it utterly invaluable.
For the first time in forever, I’m actually somewhat cautiously optimistic that we might be able to have something like this in the future—at SciPy there was a fair amount of high-level talk on “both sides of the aisle” so to speak of potentially working toward new standards that would achieve a much greater measure of cooperation and interoperability between PyPA and Conda metadata and distribution formats. Not sure exactly what will come out of it, but given all that’s changed over the years it seems like something at least worth trying again, provided people’s expectations don’t get ahead of themselves…but we’ll see.
Correct me if I’m wrong, but It doesn’t decouple Python version or other environment markers right? Pip still would have a hard dependency on the Python environment in which it is executed (due to the nature of python packaging not being statically resolvable or universally cross-compilable)?
Got it, thanks. That makes it easier to understand.
Was just imagining one reason why I might like a stand-alone zipapp would be so that I could resolve and/or install dependencies for multiple different python versions by just providing arguments to a pip.pyz
Now you’ve clarified how it works, it’s clear to me how that can’t work without running pip.pyz with the desired python interpreter/environment.
Pip has --python-version, --platform, --implementation and --abi flags for install that should let you specify the characteristics of a target system. Add that to the new --dry-run and --report options (along with --ignore-installed) and you could probably get what you want, without needing a zipapp version.
This piqued my curiosity so I made a proof of concept that adds a --find-script option, calling shutil.which from C code to get the absolute path. Demo (with some scripts that need their site-packages, I haven’t made zipapps yet, but at least shows two different matches):
$ ./python --find-script pip list
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in <module>
from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
$ ./python --find-script black
Traceback (most recent call last):
File "/home/<user>/.local/bin/black", line 5, in <module>
from black import patched_main
ModuleNotFoundError: No module named 'black'
I have to handle invalid cases like python --find-script or python --find-script does-not-exist, but if useful I could publish my branch. Cheers