PEP 582 - Python local packages directory

Users don’t know about virtual environments. They also don’t know where pip installs to. Combine the two and we get a lot of “why am I getting an ImportError when I installed using pip?” This is also ignoring the fact that people are installing globally without realizing it. We can obviously help guide users towards creating environments (and we probably will, especially if this PEP doesn’t go anywhere), but not having to do that step or extra layer of complexity would be nice.

Probably by defining our own API for other extensions to do the right thing to then plug into our UX (e.g. a Hatch extension that would get called when we hit a scenario that a user should be doing in a virtual environment). Then we make the difficulty of creating such an extension as dirt-cheap as possible (see our repo template for creating extensions for linters and formatters). Then we either work with the tool maintainers or seed the community with extensions supporting the most-used tools (e.g. our Pylint and Black extensions).

And we would directly support venv ourselves since it’s (practically) in every Python install so the baseline is always there (see the WWBD extension for an initial stab at a Create Environment command).

:smile: I hear you, and I’m not opposed to going that route, but I’m also not interested in differentiating VS Code that way specifically. If we can help e.g. vim users, I’m happy to work towards that first and foremost. Otherwise we can come up with ideas and I can share them here to garner feedback, but that does put it in a “box” within VS Code instead of wider sharing/use.

3 Likes

I understand that, but I’m still not clear how it’s a VS Code problem, or how VS Code will fix it (unless you write your own installer or wrap pip, which is what I thought you were suggesting). Because as long as users have to use pip to install, it’s the pip experience (for better or worse) that impacts them. Which is why I’d rather a solution that works with pip, than one that needs pip to change to accommodate it. There’s just too many users of pip to lightly consider changing its default behaviour IMO.

Because we get the “bug” reports that somehow auto-complete is broken or our terminal isn’t doing the right thing. :sweat_smile: It’s also a VS Code problem because I’m being asked to make the getting started experience for Python developers as good as I can.

There’s a couple of options that effectively revolve around forcing people into virtual environments. One is to try to get people into an environment early so that they get an activated terminal and thus make sure pip points at the one in the virtual environment (which is a start-up performance issue since creating environments on Windows is slow). The other is to provide a GUI around installation and package management (which is problematic as everyone has an opinion/command on how to do that).

So when looking at this PEP, the wins from a VS Code perspective are:

  1. No need to create an environment (which is slow on Windows, especially for people on old/low-end hardware)
  2. People wouldn’t necessarily use pip to install into the wrong place
  3. We know exactly where to look for the Python “environments” for the workspace (compared to now where we have to know what tool they used and then we have to ask the tool to tell us)
3 Likes

Not at all anymore with virtualenv rather than stdlib venv.

Also not when pip no longer has to be installed into every venv. That’s the slow part - creating the environment is nearly instantaneous.

2 Likes

This part can be improved if finding __pypackages__ is built into CPython, if it is possible.

1 Like

As of the next version, pip will have a --python option that lets you manage a different environment. So you can have a private copy of pip and do pip --python .venv install xxx.

Also, pip will hopefully be available as a zipapp, so your private copy of pip doesn’t even need to be installed - .venv/Scripts/python /path/to/pip.pyz install xxx will work too.

1 Like

Without looking it up, --python seems like a weird option name to choose a venv. I would think it would be used to choose a Python executable or version. Or do you mean that --python could be used to point to the python inside a .venv?

It points to an executable. As a shortcut, if you give it the name of a venv, it will use the Python executable inside the venv. Sorry, I described it confusingly, in an attempt to show how it would typically be used in this situation.

1 Like

From reading recent comments, it seems to me that the main problem we have is there are two distinct use cases that both want to use the shortest possible command invocation form:

  1. People running python directly want it to automatically pick up the environment since difficult to teach newcomers otherwise (any additional flag creates friction).
  2. People using python to run a script (say in a shebang) want it to not pick up an environment, since doing so creates security implications to all existing scripts regarding cwd (similar to DLL injection).

And I would venture to say this is probably a sign that we really should have two commands (both shipped in the base Python distribution), instead of trying to argue which one python should default to, or even intelligently guess based on context (which I suspect is not possible to do perfectly and would create more secutiry concerns). Those are just very different scenarios and should have different entry points, with people being instructed to use one or the other depending on the use case.

Just a thought. I wonder if py would be a possible choice? Given where the community is at, it’s probably better to keep python as-is, and add auto environment discovery to py.

4 Likes

py is the standard way to invoke Python on Windows. In fact, python often isn’t on sys.path, so py is the only option. So I don’t think changing py is any more practical than changing python. But adding a new command could be an option. Or maybe just a command line flag? Which, if linked to an environment variable, could be set by the user as a preference[1].

However, the sys.path side of this feels almost exactly like the debate over whether the CWD (or script directory) should be part of sys.path by default. New users like the idea, experienced users find that it’s a source of errors. There’s endless debate but no real consensus. Python 3.11 added the -P option to disable adding the CWD to sys.path. It seems to me that we’re heading for a situation where __pypackages__ causes similar controversy and confusion.


  1. Although then it would be something else we’d need to check when debugging user issues… ↩︎

2 Likes

The Launcher for Unix already automatically picks up .venv. As for supporting more possibilities, there’s a proposed solution at Support a way for other tools to assist in environment/interpreter discovery · brettcannon/python-launcher · Discussion #168 · GitHub .

I don’t know if that quite holds as much as it used to thanks to the Microsoft Store install of Python.

I think it depends on how it’s changed, but it would probably be easier than you might think. Since the Launcher is Windows-only on one side and an entirely 3rd-party install on Unix on the other, it isn’t nearly as constrained as you might expect (based on how both Steve and I have played with that CLI for that command :grin:).

I would rather not go that route; there’s already enough command cruft that we install (e.g. pydoc). If people really want to go the route of more flexible commands, your best is probably making py be a separate install and then expanding its functionality to be flexible enough to do what people are suggesting.

3 Likes

I’m completely happy with extra options/flexibility (in either py or python). The debate isn’t (as far as I can tell) about that. It’s about wanting to change the default behaviour - and that’s a backward incompatibility, which I feel the proposal should handle as such.

I’ll also say it isn’t just about picking up an existing environment, it’s also about creating the environment to begin with.

I like this direction. I prefer to keep python (or python3) cleaner with py as a nice launcher that can have some more user friendly defaults and configuration.

I opened Provide pluggable/extensible subcommand support · brettcannon/python-launcher · Discussion #222 · GitHub to talk about supporting sub-commands.

Do you expect the Windows launcher and this one to merge at any point? If not, aren’t we risking yet more confusion for newcomers?

Don’t get me wrong - I actually really like the idea (and I’d love to see it in the Windows launcher), but it seems a bit incongruous in the context of a discussion that’s aimed at finding a way to reduce the confusion for beginners…

Steve and I discussed that this week. It’s a possibility.

Depends on how py on Windows gets distributed long-term (i.e. does it stay in the installer or become something you get from the Microsoft Store or as a separate download?). It also depends on how hard we would push such a workflow on to people, as well as whether the wider community would even buy into such a workflow.

1 Like

Any solution to a technical problem using the words “just” or “simply” is inherently likely to be far more complex than the author of the solution anticipates.

Touché! In fact, that’s exactly why I wrote the third part of my blog posts on packaging.

Alas, I also believe that the Python packaging is too complex, and too complex especially when compared to other languages’ setups. So please pardon that the simplicity of the __pypackages__ proposal combined with past positive experiences with Node.js’ similar setup caused injudicious exuberance.

In short, I respect the skepticism, but if not __pypackages__, then what else will take its place?