PEP 582 - Python local packages directory

Packaging
427

I’m just cross-posting my reply to another thread, apologies in advance for not reading through everything already said here:

To give a little context to that, I often write scripts for one specific task, e.g. processing some data set. For that, I might need a specific set of packages outside the “usual” ones. At the moment, I usually create a separate conda environment for each, install the “usual” packages, install the specific packages for this one task, note them down in a requirements.txt file, do the work, delete everything. It would improve the workflow somewhat to just be able to pip install --local whatever I don’t usually have in my environment. That could still just be pip install --local -r requirements.txt to keep things simple.

1 Like
428

@ntessore Maybe something like the following can help for your use case:

https://virtualenvwrapper.readthedocs.io/en/latest/tips.html#automatically-run-workon-when-entering-a-directory

429

According to the latest version of the PEP, the scheme for Windows is still not clarified, while leaving this in the proposal:

The example is for Unix-like systems - on Windows the subdirectories will be different.

How are they different or are they going to be the same as Unix-like systems?

Is there any other on-going discussion on this I can follow?

430

(For the whole Steering Council.)

After careful consideration, the SC has decided to reject PEP 582 (Python local packages directory), at least in its current form.

While the PEP’s implementation is seemingly simple (add another directory to sys.path, similar to the program’s directory), the consequences in semantics are not immediately apparent, in particular when combining with other ways to affect the module search paths (virtual environments, user-local installs, PYTHONPATH, .pth files, sitecustomize, etc).

The expected benefit of the change, a smaller stepping stone for new users first learning about using third-party dependencies in their programs, is fairly limited. As soon as a __pypackages__ install of dependencies is no longer enough and the user needs to “upgrade” to virtual environments or some other form of dependency management, the __pypackages__ installation just becomes an added complexity. The end result is something more complex than what we currently have.

Overall, there do not seem to be compelling arguments that this would, indeed, be a net benefit. There is disagreement among the packaging community, and no clear-cut beneficial use-case for the new feature. Moreover, experimentation with __pypackages__ or a similar solution is already possible via one of the many existing customization mechanisms for sys.path, like .pth files or a sitecustomize module.

The SC is open to reconsidering if there is more clear consensus among the community, or a stronger argument showing the benefits of the proposal.

18 Likes
431

Sad to read the PEP was rejected but on the bright side, the SC gives lots of useful and actionable advice on how could be it improved. Namely:

  • More technical discussion on how this interacts with other sys.path mangling mechanisms.
  • More discussion on “progressive enhancement” or a transition path to other environment management methods (if needed?!).
  • More “social proof” that “the community” (and not only the people that happen to be engaged in this Discourse?) (although maybe not all the community, but a piece of it, namely beginners) would find this valuable, useful, and a net improvement over existing methods.
2 Likes
432

This is pretty sad.

If a proposal is too big it’s rejected because it’s too radical.

If a proposal is too small it’s rejected because it’s a too small improvement.

Many small improvements over time is how we got humans from single celled organisms. We must do much much more of them. This is how we make radical progress over time. Not by fancy big projects, but by boring small improvements at scale.

3 Likes
433

This is a shame. I hope this PEP can come back stronger and be accepted.

3 Likes
434

As the person who has to make these decisions I would like to say I don’t find that generalization accurate or helpful. Every change is viewed from a perspective of its impact on the (very large) Python community. So while the change to sys.path from this PEP is “small”, that doesn’t mean its impact is based on expectations of use of that change, how it could be misused, etc.

15 Likes
435

I think the real criteria is roughly “how much benefit do we get, for how much cost”.

Cost can come from many places. It can come from complexity of the proposal, it can come from backwards compatibility concerns, it can come from the long term overhead of maintaining and explaining a feature, or many other avenues.

My read of that post is that, the opinion of the SC, is that although the cost is small in implementation complexity, it’s much higher in long term overhead of adding yet another way for dependencies to get located and environments to get managed. They weighed it against the benefits, and found the cost too high compared to the relevant benefit, as the PEP currently stands.

This is generally a reasonably sound structure for making decisions in Software Engineering (and in general tbh). There’s very little in life that comes with all benefit and no cost, so every choice should be weighed with that in mind.

17 Likes
436

I agree. One of the most common misconceptions I see is “it’s easy, why not just do it?” This (IMO) is the problem here - there is a lot of social and transition cost that’s pretty fundamental to the idea of “let’s add a new isolation mechanism” and the PEP didn’t offer enough benefits to offset these, nor did it offer any real mitigation of those costs.

9 Likes
437

Why are we still discussing this? Hasn’t PEP 582 already been rejected by the steering council member and Python 3.12 release manager Thomas Wouters?

I guess this is too big of a paradigm shift for the current Python 3.x version. It would need Python 4.0 for this to be implemented, if at all. Don’t get me wrong, it’s a great idea, by all means, but all the intricate details haven’t been worked out, so it fell.

I personally don’t see what is so wrong with the current site-packages to introduce such a (funda)mental change? There was a hint in one of this discussion’s thread expressing __pypackages__ == site-packages. I think it says it all. It seems to me that apart from renaming site-packages to __pypackages__, the latter naming might indicate to a Python developer that __pypackages__ could be redefined in a class (for some strange reason?) as

import pkg_resources

class MyClass:
    def __pypackages__(self) -> list[str]:
        return [
            distribution.project_name for distribution in pkg_resources.working_set
        ] + ["imposter"]

Was this the intention for the dunder naming to communicate to Python developers that they can manipulate __pypackages__ in a certain way, like in the example above?

The naming is bad, especially if the intention is not for __pypackages__ to be manipulated as in the above example or in a similar fashion. Also, having to type double underscores (front and back, so twice!) is what a lazy Python developer really wants to avoid. Remember when the isinstance() and issubclass() built-in functions were introduced? Well, their Pythonic namings would certainly be is_instance() and is_subclass(). But to type one underscore, now, that is too much. But here, we wanted to force typing 4 of them!

On a serious note, naming things properly is very important. Naming them consistently is even more important. But I guess in Python, everything goes.

Maybe we should just rename site-packages to packages and make it behave as a virtual environment. The word site, for me personally, is (still) very confusing and unnecessary. Or maybe we should leave the name site-packages as is, not to break anything, but only make it as a virtual environment.

438

It’s not been rejected by me personally, as SC member or Release Manager. It’s been rejected by the Steering Council as a whole. (Individually I have no authority to reject a PEP like this.)

However, just because a PEP has been rejected does not mean all further discussion is over. There can be plenty of reasons to keep discussing a PEP, for example to address the reasons it’s been rejected, or to understand the reasons and use that understanding in other proposals.

3 Likes
439

Ah, yes, such an important PEP would need the whole Steering Council for it to be rejected. I’ve read an article on Real Python that mentions you as the one rejecting this PEP. I guess they were only partially correct.

And you are totally right: it doesn’t mean a PEP discussion has to end abruptly just because the PEP in question was rejected. Maybe someone comes along afterwards with an idea that clears all the doubt and shows us all the right path to go. Maybe someone will be a lightbringer for PEP 582.

Speaking about the PEP being rejected. I am surprised that on the PEP’s site it says Status: Draft and not Status: Rejected. Was it conclusively rejected or is there still some hope left for it?

Also, why is the naming such, i.e., __pypackages__? To my brain this immediately triggers a dunder method. Why not pypackages, or just packages by the example of the naming site-packages but without the site- part? Not sure why site- is there in the first place, but okay.

440

I don’t think SC members ever accept or reject PEPs individually except if they were nominated as “PEP delegates” by the SC at the start of the discussion. Otherwise the whole SC debates.

At least, that’s my understanding.

441

That article’s just wrong - that’s not how PEP approvals work. Or maybe it said that @thomas announced the rejection, and you misread it?

Someone simply needs to get the PEP updated. That’s an admin task, and I’ve created Reject PEP 582 by pfmoore · Pull Request #3089 · python/peps · GitHub to sort it out, to save @kushaldas the depressing task of having to mark his own PEP as rejected.

3 Likes
442

The PEP now shows the appropriate status.

2 Likes
443

The language in the article is…

After a long debate on the Python.org forum, steering council member and Python 3.12 release manager Thomas Wouters communicated the decision

4 Likes
444

I understand what you are saying, but I disagree with the statement

adding yet another way for dependencies to get located

Yes, it is another way, and it does have serious risk of being yet another standard.

But the packaging system is a mess, and the way PEP 582 handles this is just better. Virtual environments are a mess. Yes, it will take forever for them to go away. But that shouldn’t prevent us from doing the hard work of slowly removing them.

In production, everyone uses Docker images to avoid this whole mess. I feel like part of the reason Python has had such success in the last decade is because of containers. Containers allow people to write Python code and avoid the mess that is virtual environments. How about we let folks work with Python code and avoid virtual environments who aren’t throwing everything into containers?

I feel like this is painful work, but too necessary to ignore.

3 Likes
445

I disagree with the assertion that “In production, everyone uses
Docker images to avoid this whole mess.”

In production, I have servers with Python applications installed
from distribution packages, I have Python applications installed in
venvs with entrypoints symlinked in /usr/local/bin, I even have
Python applications installed in venvs inside containers in some
cases.

I work in a community of hundreds of projects with collectively
thousands of transitive dependencies, and the developers manage to
maintain interoperability and global coinstallability through
consistent use of a cross-project continuous integration system,
with the goal of making packaging into distributions like Debian and
RHEL easier.

I don’t think venvs are the terribad hack lots of people want to tar
them as, and you can absolutely do Python applications in production
without containers, but those are also not the only two options.

1 Like
446

The PEP has been rejected, drive-by comments restating things from earlier in the thread aren’t going to change that. Closing as this topic seems unlikely to go in a constructive direction at this point.