The most popular advice on the internet for `error: externally-managed-environment` is to force packages to be system installed

Packaging
1

This is more of an informational than a call to action, as I don’t know what that call to action would be on the Python packaging side.

But someone recently reported to pip issue they were getting error: externally-managed-environment: *Error: externally managed environment* · Issue #12809 · pypa/pip · GitHub

To me the error looked very clear, but obviously the user did not know how to handle it, hence them reporting the issue. So I took a look at what I got if I googled it and this SO thread was at the top for me: python - How do I solve "error: externally-managed-environment" every time I use pip 3? - Stack Overflow

  • Most upvoted answer: A link to venv but then instructions on how to use --break-system-packages and how to permanently enable it
  • Second most upvoted answer: Instructions on how to rename EXTERNALLY-MANAGED so it doesn’t trigger
  • Third most upvoted answer: Another explanation of --break-system-packages

Clearly there is a lack of understanding or agreement by many users around this feature. There seem to be three main sticking points as best as I can tell (I’m not going to go into details on why I think each of these have a multitude of reasons):

  • Users do not want to use their OS package manager to install Python packages
  • Users find the barrier of using a venv way too high compared to forcing their way past the warning
  • Users do not understand what pipx is and if it might solve their problem
6 Likes
2

I’m going to guess that a large part of the problem is that a lot of packages is literally telling people to pip install stuff, and people are simply following instructions and then finding out these instructions don’t work.

3 Likes
3

The other half of the problem is when you tell people to create a venv first, they get mad that you’re telling them what to do (and complain when they ignore you and it doesn’t work…).

Not sure we can do much better than the fantastic naming of the option here. There’s always going to be some number of people who don’t like the advice they’re given.

3 Likes
4

What about not providing pip, except in a virtual environment?

Python could come with pipx by default instead of pip.

Of course, I know it is far from being that simple to put in place (and it will likely make a lot of people upset once again :D), but I know that you know what I mean. I mostly use Debian/Ubuntu and on those systems it is very easy to have Python without pip, it is great!

5

So why are these still two separate steps? Why doesn’t pip actually create the virtual environment first? It already has --require-virtualenv [1] but it seems like beginner-friendly UX to go the next step and create the venv in the current working directory if it doesn’t exist[2].

I know, I know, lots of fiddly details would have to be worked out, like, what directory name to use (e.g. .venv), which Python to use (e.g. first python3 on the path), still have to teach people to activate the venv, etc. etc.

Related:

  1. and related environment variables ↩︎

  2. obviously not by default any time soon ↩︎

3 Likes
6

I don’t think we can do much about this:

  • Packages telling users to pip install are correct for the majority of cases. For the rest, it needs more context which would just mean that people start complaining that “installing Python packages is too complicated”.
  • The actual message isn’t even provided by the packaging tool, it comes from the distribution maintainers. I doubt they know any better how to advise someone who doesn’t even know what the “system package manager” is…
  • We can’t remove the “do it anyway” option, as people have presented important use cases where it’s needed. But nor can we prevent people using it inappropriately.
  • If anyone knows how to solve the problem of bad advice on the internet, Python packaging probably isn’t the first thing they should work on :slightly_smiling_face:

I agree with the assessment that tools like virtual environments and pipx are too much of a barrier. But venv is a stdlib tool - if people don’t know how to use python -m venv, or that they should, that’s on basic “learning Python” level training. Which isn’t something that it’s pip’s responsibility to provide.

I’d like to see something like pipx run built into the core Python interpreter, but I don’t think the core developers have the appetite for that, and the whole “packaging should be separate” idea works against it ever happening :slightly_frowning_face: And the people finding this type of issue hard aren’t going to be able to install pipx for themselves either.

The reality is that a huge number of newer Python programmers simply don’t formally learn Python, or even good development practices. They pick stuff up by grabbing snippets of the web (or nowadays, asking ChatGPT) and they don’t have any mechanism for checking that the advice they are getting is good or bad. We can’t expect development tool maintainers to take on the role of teaching people the basics of development.

14 Likes
7

FYI uv looked to do this and there was a lot of push back and I guess they’ve decided not to go with it:
Create virtualenv by default in `pip install` and `pip sync` by charliermarsh · Pull Request #2665 · astral-sh/uv · GitHub.

There were a lot of different points, it would be hard to distill, and I think if anyone wants to push pip to do it would deserve its own thread.

1 Like
8

There might be another angle to this as well. Maybe I am mistaken, but it feels like a lot of those cases (at least the one presented in the original post), are about installing an application. We know that the story for getting from writing a Python application to distributing installers for this application (or single binaries) is not great (no one’s fault). I wonder if maybe this might be a leading reason for many cases of “just run pip install --break-system-packages spotdl”.

9

Nothing is stopping distributors which are already creating the EXTERNALLY-MANAGED from not bundling pip. AFAIK, venv should still be able to work as long as ensurepip is still provided. IDK why they don’t do this.

1 Like
10

FYI, this is why the error message to the user said:

If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed.

pipx (and perhaps soon a uv subcommand) attempts to solve this application problem. But there is a clear gap of understanding around it for a lot of users.

11

Nothing is stopping distributors which are already creating the
EXTERNALLY-MANAGED from not bundling pip. AFAIK, venv should
still be able to work as long as ensurepip is still provided.
IDK why they don’t do this.

But also, even if they did, the bad advice would go from…

  1. Break your system

…to…

  1. Install pip
  2. Break your system
3 Likes
12

Thanks for the reference. I suspected uv might have looked into it.

13

Snark: Deploying Python applications - Python Packaging User Guide

Yes, I suggested above that pipx could come by default with Python instead of pip currently.

Maybe we could write some kind of canonical page explaining the better ways of dealing with this kind of thing. A page that we could send people to. For example we could go to spotdl’s repo and ask them to remove pip install spotdl from their README and link to that page instead. I guess the first version of this page could be an edited version of this (this is a great post).

Rather:

  1. apt install pipx
  2. pipx install spotdl

… hopefully, if we do the documentation and “evangelization” effort. It is not totally fair to blame popular advice if we do not have a competitive alternative to offer.

4 Likes
14

It’s better than people give it credit for. Pipx exists and you can get it without even having Pip already. Zipapps exist, and you can get Pipx as a zipapp. (In fact: maybe you didn’t hear this from me, but on Linux you don’t need to use the system package manager to install Pipx, either. What I did was: download the zipapp, chmod +x it and sudo mv it into /usr/local/bin. Works like a charm.)

I kinda hope you’re right that these cases are mainly about installing applications - because pipx is not very helpful with libraries. (Unless perhaps you’ve already installed your own application, and pipx inject your dependencies. But that seems actually harder than the normal pip and venv route.)

(Edit: By the way, that reminds me - did anything ever happen with the plan to distribute Pip as a zipapp?)

That said, I’ve always been firmly in the “people who want to be programmers should be expected to understand how to use their computer first” camp. And I do mean always - when I first learned programming as a wee lad, it was on Commodore 64s that booted directly into ROM BASIC, so those two things were the same anyway.

I did my best to help by writing my own explanation, away from the chaos of Stack Overflow:

In all honestly, Stack Overflow is kinda bad at dealing with fundamental Python questions, especially the ones related to installing things, importing things, or otherwise setting up an environment. There are all kinds of answers pushing terrible uses of sudo pip, cargo-culting around the necessity or purpose of __init__.py, a huge array of terrible and grossly unnecessary sys.path hacks, etc. etc. And a lot of older questions about basic programming techniques have dozens of answers, with huge amounts of overlap and redundancy that the mods won’t clean up (it’s not part of their job to assess technical correctness, and they don’t want to take away from people who got reputation in the past for saying the same thing as someone else - especially if both of them said it years ago, within an hour of the question being posted).

They do. In fact, in Debian’s case (and thus also Ubuntu and variants, etc.), they were already not bundling pip, and modifying ensurepip so that it would only work within virtual environments and give a custom error message otherwise, so that you’re forced to use the system package manager if you want the optional pip.

But think about it for a second. You’re talking about people who would happily type --break-system-packages on a command line, and put up with the fact that the built-in Python might not include, say, Tkinter - rather than learn the tiniest bit of theory about what a virtual environment is or why they’re useful. Do you really think the same users are going to slow down when prompted to sudo apt install python3-pip?

In short, what @fungi said. Except that’s implicitly already happened.

Ironically, for once it’s Windows users who naturally avoid running into problems with basic computer use or reading and understanding the warning messages. :wink:

I guess you meant to draw attention to the fact that it’s out of date and full of FIXME annotations?

2 Likes
15

… right, so back to the other angle I mentioned earlier…

I could have been more explicit. There I meant that we should not particularly expect that Python is installed. We should rather assume that Python might not be installed, and even that the user does not need to know that Python is involved.

So no pipx, no pex, no zipapp. I am familiar with all of those, but it is always good to mention them for whoever happens to read this thread, because surprisingly these technical solutions are not as well-known as I would expect. We can also mention shiv, pyempaq, pyapp, and probably more that I forgot.

Same. I am always quite surprised at the extent we work to “dumb things down” for people who are here because they want to write Python code. I doubt that understanding and writing Python code is any easier than understanding and using a virtual environment. I believe there is a point where lowering the bar does more harm than good. Especially no money is being exchanged here, no one owes anything to anyone.

But anyway, it does not matter, in this case what I have in mind does not require knowledge of the Python ecosystem (no pip, no venv). I am thinking rather that there could be a distributable like an installer or a single file binary. If I am not mistake pyapp does something like this, among other technical solutions. So maybe we should work to evangelize this kind of solutions. No more pip install spotdl, but instead save the downloadable file on your hard drive and run it.

I also wonder if things like flatpak could be a solution. Maybe there is an opportunity in building up the ecosystem so that getting from a pyproject.toml in a git repository to a published app on flathub is easy and straightforward.

Obviously, I am very far from having a clear picture with all details clarified, but I believe that we should look beyond just fixing things like --break-system-packages and pip outside of a venv. In particular when it is about applications, I believe we should look beyond the Python packaging ecosystem.

Yes : )

1 Like
16

This is something distributions need to fix on their end. System python being breakable and something their users expect to be able to use is a mismatch in expectations. RHEL did the right thing here with making system python an implementation detail and not easily exposed to end users while also allowing other pythons to be installed meant for users.

17

Pip’s been available as a zipapp for a number of releases now. We don’t promote it as the canonical way to install/use pip because there was too much pushback (too many people expect subprocess.run([sys.executable, "-m", "pip", ...]) to work).

1 Like
18

You mean, like the output of cx_Freeze, or PyInstaller, or py2exe, or py2app, or PyOxidizer, or pynsist, or Briefcase, or auto-py-to-exe, or (at a bit more of a stretch) Nuitka or Cython?

The impression I get from the community is that Flatpak isn’t there yet for any programming language/environment. And users are not very impressed with it, either - in theory it’s supposed to be able to reuse dependencies, but in practice you end up losing gigabytes of disk space to things like redundant complete desktop environments with slightly different version numbers to run GUI programs that are scarcely toys.

I’m not sure what you’re hoping to see when you look beyond Python, but in principle I support you.

Agreed generally, and interesting to know about RHEL. The default Debian distribution of Python is quite a bit different from a standard compiled-from-source version. (And the Mint community will direct you to a PPA rather than building from source, even though they’re generally willing to help people build things from source and Python Linux builds are generally pretty straightforward.)

19

Yes. I have not had the need for any of those yet, so I keep forgetting their names. And also I wonder why none of those solutions has become more popular. Are they bad? Is it hard to get those to build in CI/CD? Are they complex to handle? How can we help them become more mainstream (so that we offer a better alternative than "just run pip install --break-system-packages spotdl”)? Is some standardization work needed? Better documentation, tutorials, guides?

One thing that worries me, is that it might mean that each application contains its own copy of the Python interpreter (with standard library, and so on), but maybe that is not as bad as electron was. Which also reminds me, how come electron got so popular some years ago but nothing in Python realm? Or did I miss it?

1 Like
20

Personally I don’t use them because they try to “optimise” what gets included by walking the import tree. That’s fine if it works, but fiddly and annoying when it doesn’t. What I want is something that just combines a zipapp with a Python interpreter into a single file executable.