Using `--user` installs

Packaging
1

It’s as easy as pip install --user requests.

You don’t have to use venvs.

Simple https client in standard library
2 
$ pip install --user requests
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
3 Likes
3

I’m pretty sure that’s a bug in Debian. The --user location should be for the user, and hence not “externally managed”. I agree, though, that it doesn’t really matter whose bug it is, ultimately it’s a bad experience for users. But at the same time, we can’t do much about bad[1] decisions we don’t control :slightly_frowning_face:

I can tell you how I’d like things to be fixed, but that’s just wishful thinking as I have no influence with a lot of the people who’d have to change to implement my ideas. So we have to live with what we have. And that’s basically “don’t use the OS-managed Python on Linux - either use a venv, or use a user-space Python like uv python”.

  1. And before we get into a meta-debate on design principles, I know Debian’s approach is the way it is because it needs to work with the distro’s design rather than Python’s, but that doesn’t make it any better for Python users… ↩︎

4

Which kinda sucks, because it means “Linux no longer comes with Python installed, it just has a vendored thing that happens to be a Python interpreter but is only for system tools”.

1 Like
5

I don’t disagree. But that’s a choice Linux distros make. They could ship a user-focused Python interpreter, in addition to the system one. But they don’t, leading to mixed expectations and confused messages. To be fair, this is largely a problem caused by backward compatibility concerns, so I don’t want to imply that the Linux distros are the bad guys here.

On the other hand, Windows users have had the same situation for years[1] and it’s not that bad.

  1. If you ignore the Windows Store python, which until 3.14 had its own quirks and limitations ↩︎

6

I think this varies quite a bit. I tend to use the system provided python on Fedora, but don’t bother with the one included with Ubuntu.

I created ducktools-pytui to handle my own desire for any global Python environment. Though I’ll admit that for almost any actual scripts, I just end up using script metadata and using script runners (either uv or my own).

1 Like
7

There are plenty of other reasons to avoid --user without the help from Linux distros. You can end up with two different versions of the same package. It’s another Scripts/bin directory to not be in PATH. It can cross-talk to other Python installations that happen to have the same minor version – possibly from applications you didn’t even realise had an embedded Python inside them[1].

  1. there are compile time options to prevent this but they have a lot of known issues ↩︎

8

I’m pretty sure that’s a bug in Debian. The --user location should be for the user, and hence not “externally managed”.

Not really a bug, no, it’s an accurate warning against what happens if you install a distro package of an application that happens to be in Python, then pip install --break-system-packages --user ... an incompatible version of one of its dependencies and try to run the application as that user. The distro-installed application will find the dependency in your homedir via default module search and then the application will break confusingly leading to bugs getting filed against the distro or the wrong upstream project.

2 Likes
9

Oh, I agree 100%. But I’m in the camp that says “just use a venv, it’s not that hard”. I only consider --user to be a possible option for someone who adamantly refuses to use venvs, and thinks that putting everything into their system Python is a good approach. And who isn’t satisfied to use the system packager to install packages for the system Python.

We can improve the ergonomics of venvs. 3rd party tools like uv, PDM, hatch, and Poetry all have their own workflows built on venvs. There’s a lot of activity and experimentation going on, but nothing’s yet emerged as a clear winner, so standardisation on a single answer would be premature. People can try those tools if they don’t like the “raw venv” experience. And uv in particular is becoming a very popular UX that uses venvs “under the hood” but hides the complexity from the user.

2 Likes
10

I don’t disagree. But that’s a choice Linux distros make. They could ship a user-focused Python interpreter, in addition to the system one. But they don’t, leading to mixed expectations and confused messages.

Once upon a time, Red Hat used to do some really wonky patching of the Python interpreter they packaged, including at one point backporting Py3000 features to 2(.4?). Back then, the official guidance from RH was that the Python interpreter shipped in their distro was only for use by Python-based software included in the distribution and if you had any Python scripts of your own you should install a separate Python interpreter somewhere convenient and use that instead of the one in the default $PATH.

What I saw was that users were typically ignorant of that guidance or simply ignored it and got confused, which I expect is why RH eventually changed their approach.

1 Like
11

We are WAY off topic and this should be split out, but --user installs are well covered by PEP 668 as breaking system tools, and the intent is they should fail when EXTERNALLY-MANAGED is present. From the PEP:

This applies both to system-wide installs (sudo pip install) as well as
user home directory installs (pip install --user), since packages in
either location show up on the sys.path of /usr/bin/python3. breaking system tools

And:

…there are two related problems that risk breaking an externally-managed
Python: you can install an incompatible new version of a package system-wide
(e.g., with sudo pip install), and you can install one in your user
account alone, but in a location that is on the standard Python command’s
sys.path (e.g., with pip install --user).

I can attest to this from my own experience of managing a Python distribution pre-PEP 668 at a large enterprise, I disabled user installs by default in pip because they were one of my most common support issues, and I pushed users to virtual or conda environments.

This has come up several times on the pip issue tracker, e.g. GitHub · Where software is built

1 Like
12

Thanks for the correction. I’d misremembered. And indeed, we’re way off topic so I’ll shut up now.

13

In which case I would install the debian package of requests.

14

A possible way out is to share your thoughts with distros that are interseted, and switch “Linux” for “Debian/Ubuntu” in the advice :‍)

I don’t know how --user is supposed to work correctly. It’s not tied to a specific interpreter version (so an update to “the user’s Python” will eventually break stuff, even if an older Python remains installed); and the global namespace calls for coordination that can’t really happen with PyPI packages. If you have a solution, I’d be happy to hear it.

So, pipx/uv approach of tool-specific venvs is great. Especially since uv can use the system Python to back those venvs :‍)

15

My “wishful thinking” is that Linux distros create two Python packages. One is fully isolated, for system use only, unavailable to users, and is entirely managed by the system package manager. The other is exposed to the user, and is by design managed by the end user. So in that package, --user is intended for the user to install stuff into, and using vanilla pip/uv to install packages is supported. Maybe they could even do something like the new Python install manager for Windows, so that even installing into the system site-packages would be safe (at least as safe as installing PyPI packages into the system site-packages ever is…)

That’s a lot of work for distros, so I don’t think it’s actually feasible to push for it (especially as the idea is coming from someone who doesn’t even use Linux). But if anyone wants to run with it, don’t let me stop you.

+1 on this.

2 Likes
16

My suggestion to use --user was clearly naïve, but I’m still puzzled as to why it breaks. Don’t the system’s use of the Python interpreter for system-managed purposes use the -s option? Couldn’t they just do that?

17

I argued for using -s on Fedora Devel and they shows it was not sufficient.
I forget for the details after all this time.

18

Also, “distro maintainers work is worthless, and it is better to live with the utter garbage you get when combine all packages from PyPI on your system.”

1 Like
19

I don’t know how --user is supposed to work correctly.

As a distro maintainer I do believe that sudo python3 -mpip install is 99% a regrettable idea, but at least in The Sensible Distros™ (I don’t know enough about Debian/Ubuntu any more, whether it fulfils this requirement, and I don’t want to make any statement about my colleagues there) it does install OUTSIDE of the main /usr/lib*/python3*/site-packages trees (most likely into
/usr/local/lib*/python3*/site-packages).

It’s not tied to a specific interpreter version (so an update to “the user’s Python” will eventually break stuff, even if an older Python remains installed); and the global namespace calls for coordination that can’t really happen with PyPI packages. If you have a solution, I’d be happy to hear it.

I prefer to keep even user packages in the versioned distros, when user updates major version of the Python interpreter (i.e., the minor one, because the major one is forever(?) 3), it shouldn’t
hopefully happen that often, and they can install those few packages they don’t have from the system packages again.

20

At my previous job I’ve spent some time trying to make something like this work, and, gave up.
On one end, sometimes users want system integration, so you’d need to poke holes in the isolation; on the other end, a global environment for --user has exactly the same issues as a global “system” environment: the various transitive dependencies will start conflicting if you use it often enough. And of course, installing things twice won’t get you points from the minimal container crowd.
(These issues popped up in my head; I might be misremembering a bit.)

Back to separate venvs for each tool/project.
(One thing that distros can help with here is ensuring that my venvs won’t break on the 3.14→3.15 update.)

1 Like