PEP 582 - Python local packages directory

Packaging
425

No argument there! :slight_smile:

I agree with this to an extent, but I think there’s a lot of mitigation that can be done here. One obvious one is the prompt manipulation so you’re always seeing (myenv) ~/myproject#, which I think is already pretty common and helpfully forces the idea of the environment right in front of the user’s face at all times.

But your comment about “open a terminal from the GUI” again makes me think the solution to that is twofold:

  1. Make the default environment not be the system environment. The biggest problems come not from the fact that people didn’t manipulate the intended environment (e.g., by installing a package), but from the fact that they did manipulate an unintended environment, and those problems are exacerbated if the one they did manipulate is really important (e.g., needed to run core OS functions). If opening a terminal puts you into some kind of “scratchpad” environment and you accidentally install stuff, that’s not such a catastrophe as accidentally mucking up your system. You can just do activate env_i_meant_to_use and redo whatever you did.
  2. Make it so you cannot use Python without an environment. In other words, if you open up a terminal and your prompt does not say (someenv) and you type python (or even python3) you should get “command not found”. If you really want to use the system python you should be typing activate systempython first, or using something called syspython, or using a conda run type thing that lets you run a command in a specified environment without a separate activation step, or whatever.

Of course, the problem with #2 is that it’s really a matter of how the system installs Python than anything directly under Python’s control. But still I think that’s the direction we should shoot for. I was heartened to see (on some github issue I can’t find now) that some progress was actually made getting “python-full” as a debian package, and there was even talk about going further so that python gives you, well, Python, and some other package like system-python gives you the locked-down version that has no pip or virtualenv. More pressure should be applied in that direction. :slight_smile:

OS-level tools should not be using a Python that is exposed as python on the global system path. No one should be using anything that is exposed as python on the global system path. Every usage of Python should be gated through some kind of environment. [1] And the “default” environment (if one is created on install) should be a basically empty one. People can of course still get themselves in a tangle by installing everything willy-nilly into that default, but least they’ll be able to create a new environment and instantly have a clean slate, without having mucked up any system-critical stuff in the process.

On Windows, where there is no system Python, I think this is much easier: just make the official Python.org installers not install Python to the path, but instead install a full-featured environment manager to the path, so that people literally cannot start to use Python without understanding that they must first decide what environment to use it in.

I agree to some extent, but I’m a bit skeptical that the DWIM can be made good enough to cover more than a small proportion of cases, at least with the current ways of guessing. Are there any existing ways of auto-determining the environment other than checking the current working directory or using a third-party tool that internally stores environment info (e.g., an IDE where you can save a “project” that knows which environment it’s supposed to run in)? The former is pretty coarse, and the latter isn’t generalizable because it’s inherently not guessing an environment but just “saving my choice of what environment to use for this thing”.

We do, but I think it’s important to consider whether we can improve the core-provided feature so it offers more of the benefits of other environment managers.

  1. I’m talking here about systems where users actually run programs directly. It might be different in a context like Python embedded in some device, but my impression is that people setting up that kind of thing already wind up having to diverge from the standard Python setup as provided by Python.org, Linux distros, etc. ↩︎

1 Like
426

I haven’t used Linux as a development workstation for a long time, but WAY back when, RedHat shipped Python 1.5, and had a bunch of system scripts that used: #!\usr\bin env python (I think, something like that). Anyway, if you installed a Python2, all sorts of stuff broke. I never understood why they didn’t at least put a python1.5 in their #! lines, or, better yet, install a separate syspython as you’re suggesting.

That was over 20 years ago! sigh.

Now that I think about it – we’ve recommended that folks don’t use Apple’s provided Python since OS-X 10.2 or so … but folks were resistant.

Anyway, a Python used for truly system functions aside, I really like your idea that the default Python not be the core install, but rather a user’s environment that they can easily rebuild and replace if they make a mess.

I’m not sure I agree – I think it’s fine for there to be a default environment that’s not the Main one. This is how I set up my systems – I create a conda environment to use as my default – and it gets activated in my shell startup. I really like (and newbies sure will) to have a Python available for quick and dirty work at the command line. Or to run simple scripts.

427

I’m just cross-posting my reply to another thread, apologies in advance for not reading through everything already said here:

To give a little context to that, I often write scripts for one specific task, e.g. processing some data set. For that, I might need a specific set of packages outside the “usual” ones. At the moment, I usually create a separate conda environment for each, install the “usual” packages, install the specific packages for this one task, note them down in a requirements.txt file, do the work, delete everything. It would improve the workflow somewhat to just be able to pip install --local whatever I don’t usually have in my environment. That could still just be pip install --local -r requirements.txt to keep things simple.

1 Like
428

@ntessore Maybe something like the following can help for your use case:

https://virtualenvwrapper.readthedocs.io/en/latest/tips.html#automatically-run-workon-when-entering-a-directory

429

According to the latest version of the PEP, the scheme for Windows is still not clarified, while leaving this in the proposal:

The example is for Unix-like systems - on Windows the subdirectories will be different.

How are they different or are they going to be the same as Unix-like systems?

Is there any other on-going discussion on this I can follow?

430

(For the whole Steering Council.)

After careful consideration, the SC has decided to reject PEP 582 (Python local packages directory), at least in its current form.

While the PEP’s implementation is seemingly simple (add another directory to sys.path, similar to the program’s directory), the consequences in semantics are not immediately apparent, in particular when combining with other ways to affect the module search paths (virtual environments, user-local installs, PYTHONPATH, .pth files, sitecustomize, etc).

The expected benefit of the change, a smaller stepping stone for new users first learning about using third-party dependencies in their programs, is fairly limited. As soon as a __pypackages__ install of dependencies is no longer enough and the user needs to “upgrade” to virtual environments or some other form of dependency management, the __pypackages__ installation just becomes an added complexity. The end result is something more complex than what we currently have.

Overall, there do not seem to be compelling arguments that this would, indeed, be a net benefit. There is disagreement among the packaging community, and no clear-cut beneficial use-case for the new feature. Moreover, experimentation with __pypackages__ or a similar solution is already possible via one of the many existing customization mechanisms for sys.path, like .pth files or a sitecustomize module.

The SC is open to reconsidering if there is more clear consensus among the community, or a stronger argument showing the benefits of the proposal.

18 Likes
431

Sad to read the PEP was rejected but on the bright side, the SC gives lots of useful and actionable advice on how could be it improved. Namely:

  • More technical discussion on how this interacts with other sys.path mangling mechanisms.
  • More discussion on “progressive enhancement” or a transition path to other environment management methods (if needed?!).
  • More “social proof” that “the community” (and not only the people that happen to be engaged in this Discourse?) (although maybe not all the community, but a piece of it, namely beginners) would find this valuable, useful, and a net improvement over existing methods.
2 Likes
432

This is pretty sad.

If a proposal is too big it’s rejected because it’s too radical.

If a proposal is too small it’s rejected because it’s a too small improvement.

Many small improvements over time is how we got humans from single celled organisms. We must do much much more of them. This is how we make radical progress over time. Not by fancy big projects, but by boring small improvements at scale.

3 Likes
433

This is a shame. I hope this PEP can come back stronger and be accepted.

3 Likes
434

As the person who has to make these decisions I would like to say I don’t find that generalization accurate or helpful. Every change is viewed from a perspective of its impact on the (very large) Python community. So while the change to sys.path from this PEP is “small”, that doesn’t mean its impact is based on expectations of use of that change, how it could be misused, etc.

15 Likes
435

I think the real criteria is roughly “how much benefit do we get, for how much cost”.

Cost can come from many places. It can come from complexity of the proposal, it can come from backwards compatibility concerns, it can come from the long term overhead of maintaining and explaining a feature, or many other avenues.

My read of that post is that, the opinion of the SC, is that although the cost is small in implementation complexity, it’s much higher in long term overhead of adding yet another way for dependencies to get located and environments to get managed. They weighed it against the benefits, and found the cost too high compared to the relevant benefit, as the PEP currently stands.

This is generally a reasonably sound structure for making decisions in Software Engineering (and in general tbh). There’s very little in life that comes with all benefit and no cost, so every choice should be weighed with that in mind.

17 Likes
436

I agree. One of the most common misconceptions I see is “it’s easy, why not just do it?” This (IMO) is the problem here - there is a lot of social and transition cost that’s pretty fundamental to the idea of “let’s add a new isolation mechanism” and the PEP didn’t offer enough benefits to offset these, nor did it offer any real mitigation of those costs.

9 Likes
437

Why are we still discussing this? Hasn’t PEP 582 already been rejected by the steering council member and Python 3.12 release manager Thomas Wouters?

I guess this is too big of a paradigm shift for the current Python 3.x version. It would need Python 4.0 for this to be implemented, if at all. Don’t get me wrong, it’s a great idea, by all means, but all the intricate details haven’t been worked out, so it fell.

I personally don’t see what is so wrong with the current site-packages to introduce such a (funda)mental change? There was a hint in one of this discussion’s thread expressing __pypackages__ == site-packages. I think it says it all. It seems to me that apart from renaming site-packages to __pypackages__, the latter naming might indicate to a Python developer that __pypackages__ could be redefined in a class (for some strange reason?) as

import pkg_resources

class MyClass:
    def __pypackages__(self) -> list[str]:
        return [
            distribution.project_name for distribution in pkg_resources.working_set
        ] + ["imposter"]

Was this the intention for the dunder naming to communicate to Python developers that they can manipulate __pypackages__ in a certain way, like in the example above?

The naming is bad, especially if the intention is not for __pypackages__ to be manipulated as in the above example or in a similar fashion. Also, having to type double underscores (front and back, so twice!) is what a lazy Python developer really wants to avoid. Remember when the isinstance() and issubclass() built-in functions were introduced? Well, their Pythonic namings would certainly be is_instance() and is_subclass(). But to type one underscore, now, that is too much. But here, we wanted to force typing 4 of them!

On a serious note, naming things properly is very important. Naming them consistently is even more important. But I guess in Python, everything goes.

Maybe we should just rename site-packages to packages and make it behave as a virtual environment. The word site, for me personally, is (still) very confusing and unnecessary. Or maybe we should leave the name site-packages as is, not to break anything, but only make it as a virtual environment.

438

It’s not been rejected by me personally, as SC member or Release Manager. It’s been rejected by the Steering Council as a whole. (Individually I have no authority to reject a PEP like this.)

However, just because a PEP has been rejected does not mean all further discussion is over. There can be plenty of reasons to keep discussing a PEP, for example to address the reasons it’s been rejected, or to understand the reasons and use that understanding in other proposals.

3 Likes
439

Ah, yes, such an important PEP would need the whole Steering Council for it to be rejected. I’ve read an article on Real Python that mentions you as the one rejecting this PEP. I guess they were only partially correct.

And you are totally right: it doesn’t mean a PEP discussion has to end abruptly just because the PEP in question was rejected. Maybe someone comes along afterwards with an idea that clears all the doubt and shows us all the right path to go. Maybe someone will be a lightbringer for PEP 582.

Speaking about the PEP being rejected. I am surprised that on the PEP’s site it says Status: Draft and not Status: Rejected. Was it conclusively rejected or is there still some hope left for it?

Also, why is the naming such, i.e., __pypackages__? To my brain this immediately triggers a dunder method. Why not pypackages, or just packages by the example of the naming site-packages but without the site- part? Not sure why site- is there in the first place, but okay.

440

I don’t think SC members ever accept or reject PEPs individually except if they were nominated as “PEP delegates” by the SC at the start of the discussion. Otherwise the whole SC debates.

At least, that’s my understanding.

441

That article’s just wrong - that’s not how PEP approvals work. Or maybe it said that @thomas announced the rejection, and you misread it?

Someone simply needs to get the PEP updated. That’s an admin task, and I’ve created Reject PEP 582 by pfmoore · Pull Request #3089 · python/peps · GitHub to sort it out, to save @kushaldas the depressing task of having to mark his own PEP as rejected.

3 Likes
442

The PEP now shows the appropriate status.

2 Likes
443

The language in the article is…

After a long debate on the Python.org forum, steering council member and Python 3.12 release manager Thomas Wouters communicated the decision

4 Likes
444

I understand what you are saying, but I disagree with the statement

adding yet another way for dependencies to get located

Yes, it is another way, and it does have serious risk of being yet another standard.

But the packaging system is a mess, and the way PEP 582 handles this is just better. Virtual environments are a mess. Yes, it will take forever for them to go away. But that shouldn’t prevent us from doing the hard work of slowly removing them.

In production, everyone uses Docker images to avoid this whole mess. I feel like part of the reason Python has had such success in the last decade is because of containers. Containers allow people to write Python code and avoid the mess that is virtual environments. How about we let folks work with Python code and avoid virtual environments who aren’t throwing everything into containers?

I feel like this is painful work, but too necessary to ignore.

3 Likes