PEP 582 - Python local packages directory

[pf_moore (Paul Moore)]

@BrenBarn

What I mean by “harder” here is more like “it should be hard to get to a situation where you are able to run Python without some awareness of what environment you’re in”. If that environment is “the environment in which critical systemwide OS components run”, that’s fine, it’s just that the person using it should realize that is what it is.

This is a good point. Personally, I’m at the point where I am typically aware of what environment I’m in, so it’s easy for me to forget that this isn’t as obvious as it might be. However:

1. Being in the system environment by default is a problem, because it means we’re defaulting to something that we recommend against.
2. Activation sucks as a method of switching environments. It’s too easy to forget, you need to manage (mentally, if nothing else) your environments independent of your idea of “what I’m doing right now”, and doesn’t even deliver on the promise that it’s something you do once and then can forget about (open a new terminal window from the GUI and you’re back in the default environment).

The one thing that PEP 582 is right about is that having the system work out the right environment for you is a better user experience. The problem, just like with any “do what I mean” system, is getting the balance right between guessing correctly and requiring the user to be explicit. But by this point, I’m pretty convinced that we should be fixing the user experience issue by improving how virtual environments^[1] work, not by abandoning them or declaring them “expert only”^[2].

---

1. I get that there are other environment management tools like conda, but core Python can’t influence them, so we have to focus on the core-provided feature. ↩︎

2. The UX issues still frustrate me, and I’d consider myself an expert. ↩︎

[fungi]

For me, it’s far easier to be sure I’m using the environment I want
by specifying the full path to the interpreter in that environment
on invocation (~me/lib/foo/bin/python3 myscript.py or whatever)
rather than using the activate stub; I basically never “activate” a
venv.

But that probably qualifies as an expert workflow too… recent
posts under the help topic have shown basic understanding of
directories and paths isn’t even something new users necessarily
possess.

[PythonCHB (Christopher H.Barker, PhD)]

@BrenBarn

I mean, I don’t use us VS Code myself, but the way I think of it is even more extreme. It’s more like, you cannot run code at all without an environment, because without an environment, you don’t have Python.

I really like this way of thinking about it. IN fact, one problem I DO often see in my beginning class is that folks find themselves with multiple python installs: maybe 3.8 and 3.10, maybe also Anaconda – maybe even Anaconda and miniconda! EVen without any “virtual environment”, they hav multiple different pythons to choose from, and this leads to a LOT of confusion. Even more so with IDEs, where the configuration can be inp,icte and/or hidden (and as an instructor, I may know nothing about the IDE a student is using)

(part of why python -m pip install is generally recommended over pain pip install…)

Anyway, if the community could rally around the idea that it should be clear an obvious what “instance” of Python one is using – that would be great.

But it does get very tricky when, as Paul mentioned, you want a “it just works” user experience – maybe Brett’s work with VS Code will help us get there.

[BrenBarn (Brendan Barnwell)]

@pf_moore

Being in the system environment by default is a problem, because it means we’re defaulting to something that we recommend against.

No argument there!

@pf_moore

Activation sucks as a method of switching environments. It’s too easy to forget, you need to manage (mentally, if nothing else) your environments independent of your idea of “what I’m doing right now”, and doesn’t even deliver on the promise that it’s something you do once and then can forget about (open a new terminal window from the GUI and you’re back in the default environment).

I agree with this to an extent, but I think there’s a lot of mitigation that can be done here. One obvious one is the prompt manipulation so you’re always seeing (myenv) ~/myproject#, which I think is already pretty common and helpfully forces the idea of the environment right in front of the user’s face at all times.

But your comment about “open a terminal from the GUI” again makes me think the solution to that is twofold:

1. Make the default environment not be the system environment. The biggest problems come not from the fact that people didn’t manipulate the intended environment (e.g., by installing a package), but from the fact that they did manipulate an unintended environment, and those problems are exacerbated if the one they did manipulate is really important (e.g., needed to run core OS functions). If opening a terminal puts you into some kind of “scratchpad” environment and you accidentally install stuff, that’s not such a catastrophe as accidentally mucking up your system. You can just do activate env_i_meant_to_use and redo whatever you did.
2. Make it so you cannot use Python without an environment. In other words, if you open up a terminal and your prompt does not say (someenv) and you type python (or even python3) you should get “command not found”. If you really want to use the system python you should be typing activate systempython first, or using something called syspython, or using a conda run type thing that lets you run a command in a specified environment without a separate activation step, or whatever.

Of course, the problem with #2 is that it’s really a matter of how the system installs Python than anything directly under Python’s control. But still I think that’s the direction we should shoot for. I was heartened to see (on some github issue I can’t find now) that some progress was actually made getting “python-full” as a debian package, and there was even talk about going further so that python gives you, well, Python, and some other package like system-python gives you the locked-down version that has no pip or virtualenv. More pressure should be applied in that direction.

OS-level tools should not be using a Python that is exposed as python on the global system path. No one should be using anything that is exposed as python on the global system path. Every usage of Python should be gated through some kind of environment. ^[1] And the “default” environment (if one is created on install) should be a basically empty one. People can of course still get themselves in a tangle by installing everything willy-nilly into that default, but least they’ll be able to create a new environment and instantly have a clean slate, without having mucked up any system-critical stuff in the process.

On Windows, where there is no system Python, I think this is much easier: just make the official Python.org installers not install Python to the path, but instead install a full-featured environment manager to the path, so that people literally cannot start to use Python without understanding that they must first decide what environment to use it in.

@pf_moore

The one thing that PEP 582 is right about is that having the system work out the right environment for you is a better user experience. The problem, just like with any “do what I mean” system, is getting the balance right between guessing correctly and requiring the user to be explicit.

I agree to some extent, but I’m a bit skeptical that the DWIM can be made good enough to cover more than a small proportion of cases, at least with the current ways of guessing. Are there any existing ways of auto-determining the environment other than checking the current working directory or using a third-party tool that internally stores environment info (e.g., an IDE where you can save a “project” that knows which environment it’s supposed to run in)? The former is pretty coarse, and the latter isn’t generalizable because it’s inherently not guessing an environment but just “saving my choice of what environment to use for this thing”.

@pf_moore

I get that there are other environment management tools like conda, but core Python can’t influence them, so we have to focus on the core-provided feature.

We do, but I think it’s important to consider whether we can improve the core-provided feature so it offers more of the benefits of other environment managers.

---

1. I’m talking here about systems where users actually run programs directly. It might be different in a context like Python embedded in some device, but my impression is that people setting up that kind of thing already wind up having to diverge from the standard Python setup as provided by Python.org, Linux distros, etc. ↩︎

[PythonCHB (Christopher H.Barker, PhD)]

@BrenBarn

there was even talk about going further so that python gives you, well, Python, and some other package like system-python gives you the locked-down version that has no pip or virtualenv. More pressure should be applied in that direction.

I haven’t used Linux as a development workstation for a long time, but WAY back when, RedHat shipped Python 1.5, and had a bunch of system scripts that used: #!\usr\bin env python (I think, something like that). Anyway, if you installed a Python2, all sorts of stuff broke. I never understood why they didn’t at least put a python1.5 in their #! lines, or, better yet, install a separate syspython as you’re suggesting.

That was over 20 years ago! sigh.

Now that I think about it – we’ve recommended that folks don’t use Apple’s provided Python since OS-X 10.2 or so … but folks were resistant.

Anyway, a Python used for truly system functions aside, I really like your idea that the default Python not be the core install, but rather a user’s environment that they can easily rebuild and replace if they make a mess.

@BrenBarn

instead install a full-featured environment manager to the path, so that people literally cannot start to use Python without understanding that they must first decide what environment to use it in.

I’m not sure I agree – I think it’s fine for there to be a default environment that’s not the Main one. This is how I set up my systems – I create a conda environment to use as my default – and it gets activated in my shell startup. I really like (and newbies sure will) to have a Python available for quick and dirty work at the command line. Or to run simple scripts.

[ntessore (Nicolas Tessore)]

I’m just cross-posting my reply to another thread, apologies in advance for not reading through everything already said here:

@ntessore

It’s a bit of a tangent, but I often wish that pip install had a --local switch akin to --user that installed packages into __pypackages__, say, in the local folder. This local installation would be lightweight and completely transparent, without explicit activation, and would work on top of venv, conda, etc. As such, it would be part of the import system in Python rather than any specific package manager.

In the spirit of this PEP, if global installation is impossible, e.g. because of EXTERNALLY-MANAGED, then pip could say: “You can install packages locally into the current folder using pip install --local.” This would not elevate venv or any other tool above any others.

Ideally, the local install would keep a list of installed packages that can be committed, as well as the actual binaries separated by Python ABI, so that you can just reinstall the local packages with some command should you switch machines or Python version.

To give a little context to that, I often write scripts for one specific task, e.g. processing some data set. For that, I might need a specific set of packages outside the “usual” ones. At the moment, I usually create a separate conda environment for each, install the “usual” packages, install the specific packages for this one task, note them down in a requirements.txt file, do the work, delete everything. It would improve the workflow somewhat to just be able to pip install --local whatever I don’t usually have in my environment. That could still just be pip install --local -r requirements.txt to keep things simple.

[sinoroc]

@ntessore Maybe something like the following can help for your use case:

https://virtualenvwrapper.readthedocs.io/en/latest/tips.html#automatically-run-workon-when-entering-a-directory

[frostming]

According to the latest version of the PEP, the scheme for Windows is still not clarified, while leaving this in the proposal:

The example is for Unix-like systems - on Windows the subdirectories will be different.

How are they different or are they going to be the same as Unix-like systems?

Is there any other on-going discussion on this I can follow?

[thomas (Thomas Wouters)]

(For the whole Steering Council.)

After careful consideration, the SC has decided to reject PEP 582 (Python local packages directory), at least in its current form.

While the PEP’s implementation is seemingly simple (add another directory to sys.path, similar to the program’s directory), the consequences in semantics are not immediately apparent, in particular when combining with other ways to affect the module search paths (virtual environments, user-local installs, PYTHONPATH, .pth files, sitecustomize, etc).

The expected benefit of the change, a smaller stepping stone for new users first learning about using third-party dependencies in their programs, is fairly limited. As soon as a __pypackages__ install of dependencies is no longer enough and the user needs to “upgrade” to virtual environments or some other form of dependency management, the __pypackages__ installation just becomes an added complexity. The end result is something more complex than what we currently have.

Overall, there do not seem to be compelling arguments that this would, indeed, be a net benefit. There is disagreement among the packaging community, and no clear-cut beneficial use-case for the new feature. Moreover, experimentation with __pypackages__ or a similar solution is already possible via one of the many existing customization mechanisms for sys.path, like .pth files or a sitecustomize module.

The SC is open to reconsidering if there is more clear consensus among the community, or a stronger argument showing the benefits of the proposal.

[astrojuanlu (Juan Luis Cano Rodríguez)]

Sad to read the PEP was rejected but on the bright side, the SC gives lots of useful and actionable advice on how could be it improved. Namely:

• More technical discussion on how this interacts with other sys.path mangling mechanisms.
• More discussion on “progressive enhancement” or a transition path to other environment management methods (if needed?!).
• More “social proof” that “the community” (and not only the people that happen to be engaged in this Discourse?) (although maybe not all the community, but a piece of it, namely beginners) would find this valuable, useful, and a net improvement over existing methods.

[boxed (Anders Hovmöller)]

This is pretty sad.

If a proposal is too big it’s rejected because it’s too radical.

If a proposal is too small it’s rejected because it’s a too small improvement.

Many small improvements over time is how we got humans from single celled organisms. We must do much much more of them. This is how we make radical progress over time. Not by fancy big projects, but by boring small improvements at scale.

[Saphyel (Carlos)]

This is a shame. I hope this PEP can come back stronger and be accepted.

[brettcannon]

@boxed

If a proposal is too big it’s rejected because it’s too radical.

If a proposal is too small it’s rejected because it’s a too small improvement.

As the person who has to make these decisions I would like to say I don’t find that generalization accurate or helpful. Every change is viewed from a perspective of its impact on the (very large) Python community. So while the change to sys.path from this PEP is “small”, that doesn’t mean its impact is based on expectations of use of that change, how it could be misused, etc.

[dstufft (Donald Stufft)]

I think the real criteria is roughly “how much benefit do we get, for how much cost”.

Cost can come from many places. It can come from complexity of the proposal, it can come from backwards compatibility concerns, it can come from the long term overhead of maintaining and explaining a feature, or many other avenues.

My read of that post is that, the opinion of the SC, is that although the cost is small in implementation complexity, it’s much higher in long term overhead of adding yet another way for dependencies to get located and environments to get managed. They weighed it against the benefits, and found the cost too high compared to the relevant benefit, as the PEP currently stands.

This is generally a reasonably sound structure for making decisions in Software Engineering (and in general tbh). There’s very little in life that comes with all benefit and no cost, so every choice should be weighed with that in mind.

[pf_moore (Paul Moore)]

I agree. One of the most common misconceptions I see is “it’s easy, why not just do it?” This (IMO) is the problem here - there is a lot of social and transition cost that’s pretty fundamental to the idea of “let’s add a new isolation mechanism” and the PEP didn’t offer enough benefits to offset these, nor did it offer any real mitigation of those costs.

[PedanticHacker (Boštjan Mejak)]

Why are we still discussing this? Hasn’t PEP 582 already been rejected by the steering council member and Python 3.12 release manager Thomas Wouters?

I guess this is too big of a paradigm shift for the current Python 3.x version. It would need Python 4.0 for this to be implemented, if at all. Don’t get me wrong, it’s a great idea, by all means, but all the intricate details haven’t been worked out, so it fell.

I personally don’t see what is so wrong with the current site-packages to introduce such a (funda)mental change? There was a hint in one of this discussion’s thread expressing __pypackages__ == site-packages. I think it says it all. It seems to me that apart from renaming site-packages to __pypackages__, the latter naming might indicate to a Python developer that __pypackages__ could be redefined in a class (for some strange reason?) as

import pkg_resources

class MyClass:
    def __pypackages__(self) -> list[str]:
        return [
            distribution.project_name for distribution in pkg_resources.working_set
        ] + ["imposter"]

Was this the intention for the dunder naming to communicate to Python developers that they can manipulate __pypackages__ in a certain way, like in the example above?

The naming is bad, especially if the intention is not for __pypackages__ to be manipulated as in the above example or in a similar fashion. Also, having to type double underscores (front and back, so twice!) is what a lazy Python developer really wants to avoid. Remember when the isinstance() and issubclass() built-in functions were introduced? Well, their Pythonic namings would certainly be is_instance() and is_subclass(). But to type one underscore, now, that is too much. But here, we wanted to force typing 4 of them!

On a serious note, naming things properly is very important. Naming them consistently is even more important. But I guess in Python, everything goes.

Maybe we should just rename site-packages to packages and make it behave as a virtual environment. The word site, for me personally, is (still) very confusing and unnecessary. Or maybe we should leave the name site-packages as is, not to break anything, but only make it as a virtual environment.

[thomas (Thomas Wouters)]

@PedanticHacker

Why are we still discussing this? Hasn’t PEP 582 already been rejected by the steering council member and Python 3.12 release manager Thomas Wouters?

It’s not been rejected by me personally, as SC member or Release Manager. It’s been rejected by the Steering Council as a whole. (Individually I have no authority to reject a PEP like this.)

However, just because a PEP has been rejected does not mean all further discussion is over. There can be plenty of reasons to keep discussing a PEP, for example to address the reasons it’s been rejected, or to understand the reasons and use that understanding in other proposals.

[PedanticHacker (Boštjan Mejak)]

Ah, yes, such an important PEP would need the whole Steering Council for it to be rejected. I’ve read an article on Real Python that mentions you as the one rejecting this PEP. I guess they were only partially correct.

And you are totally right: it doesn’t mean a PEP discussion has to end abruptly just because the PEP in question was rejected. Maybe someone comes along afterwards with an idea that clears all the doubt and shows us all the right path to go. Maybe someone will be a lightbringer for PEP 582.

Speaking about the PEP being rejected. I am surprised that on the PEP’s site it says Status: Draft and not Status: Rejected. Was it conclusively rejected or is there still some hope left for it?

Also, why is the naming such, i.e., __pypackages__? To my brain this immediately triggers a dunder method. Why not pypackages, or just packages by the example of the naming site-packages but without the site- part? Not sure why site- is there in the first place, but okay.

[jeanas (Jean Abou Samra)]

@PedanticHacker

Ah, yes, such an important PEP would need the whole Steering Council for it to be rejected

I don’t think SC members ever accept or reject PEPs individually except if they were nominated as “PEP delegates” by the SC at the start of the discussion. Otherwise the whole SC debates.

At least, that’s my understanding.

[pf_moore (Paul Moore)]

@PedanticHacker

I’ve read an article on Real Python that mentions you as the one rejecting this PEP.

That article’s just wrong - that’s not how PEP approvals work. Or maybe it said that @thomas announced the rejection, and you misread it?

@PedanticHacker

Speaking about the PEP being rejected. I am surprised that on the PEP’s site it says Status: Draft and not Status: Rejected.

Someone simply needs to get the PEP updated. That’s an admin task, and I’ve created Reject PEP 582 by pfmoore · Pull Request #3089 · python/peps · GitHub to sort it out, to save @kushaldas the depressing task of having to mark his own PEP as rejected.