I’m confused again, but I’m going to put it down to “this is a tooling question, and I’ll wait till someone proposes concrete changes for the tooling side before worrying about it” 
You seemed to have one piece of feedback from reading the PEP, which roughly summarised as “what if someone injected a hidden __pypackages__ directory and hijacked python -m pip”.
My response was that scenario is exactly the same as “what if someone injected a hidden pip.py …”, which already exists today.
Both cases suck, but it’s not something that we can fix in this proposal. It is something that better application deployment would help with, though, which is the other linked thread.
Ouch. Yes, I’d not realised that, and you’re right, it’s the same security vulnerability so __pypackages__ doesn’t expose anything new. I don’t know if PEP 582 should mention it explicitly, just to make that fact clear.
1 Like
I’ve already pushed back on describing existing issues that this PEP doesn’t make worse, but perhaps we need to since they keep coming up in discussions (but then that raises the question of “why mention it here if it’s not worse?”, so it really has to be explained carefully).
2 Likes
But then we get into the desire that @njs brought up of wanting to version his tooling with his project, at which point __pypackages__/-m makes sense.
I think as developers we will forever burden ourselves with wanting to version and pull in tooling that’s dependent on the code we are working on. Add on to that the difficulties of getting co-workers/contributors to have the right tooling to begin with and I really don’t think we are going to escape from this.
And as @pf_moore pointed out, how to distribute apps is an entirely separate discussion from this PEP and so I don’t think it should be directly dragged in here (if people want a more holistic discussion of packaging and distribution all-up then that should probably happen at a higher level than this specific discussion about tightening up PEP 582).
Agreed, but if we expect __pypackages__ to be a solution for this then we need to either solve the problem that -m will work differently depending on whether you’re in the project root or not, or we need an alternative solution that lets the user put some project-specific directory on PATH (maybe __pypackages__/Scripts, depending on what pip does with console scripts), but that brings us back around to “activating” a project.
(Oh, and just as a counterexample, I’ve never particularly felt the need to tie the version of my development tools to my project. I’m not sure why anyone would, to be honest - I don’t insist that I use a specific version of my editor, why would I want a particular version of my linter, or code formatter? But I accept that this is a matter of opinion).
With my “dev manager of the Python extension for VS Code” hat on, I anecdotally can say that some very common issues users of our extension run into are:
- No idea of what virtual environments are
- Using the
pip/pip3command and then not realizing which interpreter the package(s) was installed into (which ties into not knowing about virtual environments compared to global/system interpreters)
I’m not an instructor, so I don’t know how difficult it is to teach the concept of virtual environments, but it isn’t being taught universally.
This seems more like an issue of people installing into their global interpreter which we know is a problem. Maybe this PEP will force us to address that by changing pip’s default?
I’m not sure how.
Consider a directory myproject containing two subdirectories, __pypackages__ and src. The user has black installed in __pypackages__, but not in the system Python.
If the user is in myproject, python -m black will run black. If the user is in myproject/src, python -m black will give an error. This is what I mean when I say the same command will work differently depending on where you are.
Conversely, running python .../myproject/__main__.py will work from wherever it’s run (because the __pypackages__ in the script’s directory is put on sys.path. This is why @steve.dower’s idea of having a black.py in myproject will always run black. The problem with it is that I don’t think we want to encourage projects to scatter tool driver scripts in the project top-level directory.
If Python were to scan up the directory hierarchy looking for __pypackages__, then things would work more smoothly in these examples - but I think there are likely to be enough issues with doing so, that the PEP is right not to suggest that.
Ah, see that doesn’t bother me. I use -m from the directory I’m sitting in while developing so much that it doesn’t bother me that it wouldn’t work if I changed directories. Because of . being on sys.path Python is pretty much always contextual on your directory location.
Which is kind of essentially how shiv works. It just takes that directory, adds some bootstrap code, and zips it all up.
I’m starting to think maybe it’s not the best idea to build this into python. Rust seems to do it right1: rustc does not look for non-builtin libraries, but cargo passes the necessary flags to rustc when you build with it. So maybe python -m should work as-is, and we should ship and additional command with pip to set up the correct sys.path, so (say) pip run -m black looks for __pypackages__ in parent directories, but python -m black only looks in the global installation. Some additional conditions can be added to further restrict lookup, e.g. stop looking for __pypackages__ if a pyproject.toml is found.
1 The context is different, of course, Rust being a compiled language. But I feel the anology makes sense.
This sounds like a nice idea, but -1 on it being part of pip. There’s no need for users to need to install pip just to get the ability to run local commands.
From @cs01’s original post, he’s already got support for this in pipx, so maybe that’s the thing we should be recommending (or maybe just splitting out precisely this functionality into a lightweight launcher would be better).
Agreed. I used pip only to easily show how the command would look like. Ideally this should be part of the Python distro, and as a “real” binary to hook directly into C (instead of hacking PYTHONPATH).
These are a bit contradictory. So do you mean to have some new command look in __pypackages__ but not -m, or are you talking about modifying what -m does today?
This will complicate the PEP as shipping yet another command that’s tied to a specific Python version is something we have been moving away from (e.g. the removal of pyvenv).
IMHO, adding an option the interpreter or runpy wouldn’t be horrible.
python works as it currently works right now, without the PEP.
A new command (not named python) doing the job designated to python in the PEP.
Another possibility would be to add a flag to python, say python -p to enable __pypackages__ lookup.
pythonloc currently exists for testing this. Also piploc to provide the pip side.
I’d be interested to see whether it’s easier to teach everyone to use a new command and overcome all the existing material online/offline, or to figure out the problems and just make the existing documented command work.
2 Likes
I am open to adding collaborator access to anyone interested. Also open to renaming the tools or moving them to a GitHub org rather than my personal account.
1 Like
I’d argue that the problem exists regardless, unless we can somehow make pip install magically install into __pypackages__ by default 
I have also realised the potential new command does not necessarily preclude accepting PEP 582 as-is. It is perfectly reasonable for me to have python -m only look for cwd, and require additional tooling when you want the src scenario to work.
1 Like