Teach python how to locate a well known named virtual environment and recommend tools install into it by default?
Like you could change PEP 582 to just turn __pypackages__ into a virtual environment, and teach python how to activate that instead. It’s roughly the same thing from a “I am starting from zero” UX, but now if they graduate to say, needing to access something that was installed into bin/ (I don’t think PEP 582 discusses how to handle bin/ ~at all), that provides a natural stepping stone to be like, oh hey, actually everything got installed into this directory, and you can just open it up and run commands directly from there too! Which then provides a stepping stone into “well that well known directory can just be located anywhere you want”.
To turn this around, what’s the benefit to creating somethign that is like a venv, but isn’t quite the same, when venv is right there? AFAICT none of the benefits of this PEP come from the not-quite-venv implementation of it, and all of it comes from the fact there’s some default location that python will load by default.
Let me summarize some problems raised recently in this thread that PEP 582 needs to clarify or improve:
1. OS-isolated library paths
The folder structure of the current proposal is not enough to isolate packages between different platforms, a possible solution is to name the directory with platform tags like cp310-win-amd64.
2. System site-packages ignorance
Partially agree, but it may cause problems for some “interpreter wrappers”. Let me explain, say we make a “beautiful python”, or bpython, with changes to the output and exception hook to make it output beautiful ANSI colors to the terminal, and distribute it as a python library uploaded to PyPI. A user installs it into the global site-packages with the system python. When he executes bpython /path/to/myscript.py, should system site-packages be ignored? If we are to ignore it, bpython will be broken. But if we don’t, and put __pypackages__ to the sys.path in front of system site-packages, chances are incompatible versions of dependencies in __pypackages__ being prioritized and, again, break bpython(refer to pdm#849). It seems this problem is non trivial to solve without tweaks to the interpreter on how and when site-packages are loaded.
3. Problem of finding __pypackages__
I agree that this proposal must be extended with how an interpreter looks for the __pypackages__ to load. It shouldn’t be restricted to the same directory in which the script resides. The practice of PDM is to find the current directory and its recursive ancestors at a configurable max depth.
4. Project-level __pypackages__
In addition to the last point, a project-level __pypackages__ must be seen by all scripts and modules inside it. A project can be defined by a pyproject.toml in the root. This is also what PDM is doing at present.
5. bin handling
I highly suggest PEP 582 to consider this, making __pypackages__ a full install scheme in sysconfig, with bin, includes, lib, and others. It seems the current PEP only focuses on running a standalone script, and ignores usages inside a project, while in the latter case, it is common to install binaries as well as libraries into the project-level __pypackages__, and users may expect them to run with pip run <executable>
P.S. For all the problems mentioned above, PDM has solved 3,4,5, partially solved 1, and hasn’t solved 2.
Yeah, because I burnt out on this discussion and never updated the PEP with any of the feedback from the first ~100 replies But if I did, it would say “if you need bin/ and -m won’t do, use a venv”
no env variables
no symlinks/launchers
no absolute paths embedded in configs/executables
no PATH manipulations
no additional tools needed to be known or acquired
<Deleted: one extended rant which basically shows me that I’m still not ready to come back to this topic… whoever picks this one up, good luck. I’m muting this again.>
This already happens with -S. It would break if I try to use that option with bpython, so you will either need to figure out a way to make that work, which would also fix the isolated PEP 582 directory issue, not present it as a python executable replacement, or distribute it some other way.
Overall, I think this is a small price to pay to make sure PEP 582 installs don’t break when something on the system updates.
Totally off-topic, but if anyone is aware of any standard or anything around environment variable definition files like .env, please let me know! It’s a constant headache for me at work that there isn’t one, especially when everyone seems to assume they are formatted for the OS you’re running which is not necessarily the case (i.e. Steve’s comment when someone wrote a .env file for bash; not every shell uses : as a path separator).
I made this recommendation over in the PEP 704 thread, so I’m +1 at least on the standardized naming/location scheme for where to look for virtual environments when one isn’t explicitly detected via $VIRTUALENV. As for python (but not python3 or python3.11) garnering the smarts to automatically use an environment, I’m probably +0.
I implemented pipx run script.py which picks up dependencies from a comment block in the script. The PR is awaiting merge and a new release of pipx, which is down to the pipx maintainers’ availability.
This is getting way off-topic, but whether or not building universal2 wheels is hard primarily depends on whether or not you have external dependencies. For simple projects that only use system libraries building universal2 wheels is automatic when using a universal2 python installation (such as the ones we ship on the python website).
For projects with external library dependencies and/or other languages than C (e.g. Fortran) it can be harder to get those dependencies in a “universal2 form”. Still doable, but takes more effort and that’s effort that can be spent on other work.
This. Or rather, a solution that makes doing something like this easy, without needing to manually manage the environments. Not as a competing proposal, but as a complementary one, for use cases where it makes sense. Most solutions I’ve seen require the user to manually remember which script uses which (named) environment.
The pipx solution works somewhat like this, but it doesn’t share environments except in a very basic way, and its handling of cleanup (treat the environment as a cache, so it gets deleted when it’s not used for a while) isn’t that good.
And implicitly, I’m asking that we don’t exclude workflows like this by accepting __pypackages__ or .venv, and then baking it into tools and tutorials and documentation to the point where people with different backgrounds/preferences have to “fight the system” to work the way they want to.
I never set environment variables, and I limit what I put in config files. But yeah, fighting to eliminate configuration details (for example when trying to track down a bug) is often annoyingly hard.
And if you saw my working environment you’d appreciate why I hate triggering a behaviour change on the presence of a magic directory (or file). I routinely work in a “scratch” directory full of small Python scripts, C programs, HTML files, Powershell scripts, etc. If something dumped a __pypackages__ directory in there, I’d quite likely not notice for ages (until it caused a problem which meant I was trying to track down a bug, most likely).
This is a good point. For example, if I have an activated virtual environment, and I run python in a directory with a __pypackages__ subdirectory, which takes precedence? The virtual environment or the __pypackages__? How do we explain that to users (newcomers and/or experts)?
@pf_moore you are actively against this PEP because of your despite of how works very well and understood in other languages and you have some sort of preferences with scripts having the dependencies on top very similar to the approach in Deno (then what’s the point of having pyproject.toml at all??)
@frostming and @brettcannon already provided solutions (or partial solutions) to all the problems mentioned, not sure why this PEP should be rejected or delayed.
I saw using PDM a lot of people and they didn’t have any issue understating the workflow with this PEP.
No need to be pushy here — the PEP authors will update the PEP to address the criticism, and discussion is going on. Ignoring or dismissing people’s concerns is not the way to advance this.
Prefacing this with the fact that I haven’t read all of the almost 300 comments in this thread but stumbled upon it by finding the PEP submission.
Has anybody considered the impact this would have on common linters and tooling? As a pylint maintainer I foresee many issues with adding another layer of complexity to an import system that is already quite hard to mimic while not actually executing any import statements directly (we don’t want this because of code execution concerns).
Adding stuff to sys.path depending on certain conditions would certainly be something we won’t be able to add support for in a simple fix.
I haven’t seen any comments about the effect of this on other tooling but I wonder if they share similar concern. Obviously every tool is different so I wonder if this is purely static linter specific, or if it is a broader concern.
I understand that a simple counter argument would be that if you want to use such tooling you should consider a virtualenv anyway, but this would then increase the barrier of adoption for tooling for new Python users.
What’s the specific complexity you’re foreseeing? From a Python perspective this would just be another directory on sys.path. Granted there would be some calculation as to what directory to use, but it’s no worse than where the stdlib is kept or where site installs are kept.
Pylint patches sys.path based on the directory of the file that is being linted and the arguments that were given when invoking the tool. This is already causing us headaches as this is near from perfect but for now the only way we found to avoid code execution. This also means that the sys.path pylint uses to evaluate imports is not the same as the one you would get by just invoking the interpreter. (We’re open to better solutions… )
Similarly I wonder whether users would (for example) expect pylint to add the new directory to sys.path if it was found at the level of the file being linted even though it is not available to the interpreter that is being used to run pylint itself.
For example, what if I have multiple hobby projects with different package directories that all live under a larger “Python projects directory”. If I understand the proposal correctly you wouldn’t be able to lint these files from the top level directory but would need to enter each individual directory itself to get the correct sys.path. As a new user I would definitely find that confusing I think: why does this tool behave differently based on the directory I’m calling it from? (I’m specifically thinking of people just using the direct pylint command without a python -m)
We get some reports every few months about pylint not finding imports that “work perfectly fine when I execute the file”. From an initial reading of the PEP this seems like it could cause further confusion as it adds another variable to deal with.
But perhaps we should also just see it as a test for our import system! I mostly just wanted to see if others had similar (initial) reservations/thoughts.
I think so. This is the point of defining the directory in a PEP, so that tools know where it will be and when it should apply (while still getting to/having to make their own decision about how “realistic” their case is).
So if someone did python dir/somefile.py then dir/__pypackages__ would be included, and so doing pylint dir/somefile.py should assume that dir/__pypackages__ would be part of the search, regardless of what pylint itself is using.
Though python -m dir.somefile would be different, because the “root” of the project is now . and not dir. Pytest has options to distinguish these two invocations, I expect Pylint has something similar, though it’s been such a long time since I wrote a new Pylint invocation.
Users already think that about Python, I get questions exactly like this on a regular basis from the engineers that I work with. It’s definitely possible to get into a similar situation with Pylint, as you can’t really be more consistent than Python itself.
Maybe this offers one more variable, but it’s a variable based on concrete file system information, whereas most of the other variables are transient environment settings or hidden somewhere in CPython’s path calculation algorithm (which nobody fully understands, including me, and I recently spent months working on it).[1]
Also, I haven’t touched the PEP text in years, and not at all in response to this thread, so from my POV please treat it as a vague hint of an idea and not a reliable specification. ↩︎
(Sorry for posting a late response to this, couldn’t keep up with all the threads)
That’s just horribly, horribly wasteful for bandwith and storage, none of which are exactly free (and even more so outside of the so-called 1st world countries) – completely aside from resource waste in general, see climate change… Unless we’re talking about a <10% footprint increase (which, granted, some packages light on native code might achieve), we should really be able to do better.
There’s more problems with this (exacerbates vendoring issues, creates incentives against supporting new platforms, how to deal with GPU & SIMD variants, musl, etc.), so I really hope we don’t go this way. More tightly scoped things like universal2 (just on osx) might still be beneficial, but all-arches-in-one is a non-starter IMO.
Pushing more changes to the PEP based on feedback. I hope it will be easier for people to understand that this PEP is not trying to replace virtual environments. This is to help that big chunk of newbies and trainers who needs to install and use some Python modules and packages without spending hours to learn/teach virtual environments in their operating systems.
For anything advanced (which most of the folks are asking here) use case, use a virtual environment.
The PEP should explicitly note that when it says “installers”, this is not just pip - conda will need to change as well, for example. Have the conda maintainers confirmed how this will impact them? They aren’t particularly active here, so they may not be aware of the implications.
The spec needs to explain that sysconfig needs to change to include the pypackages scheme. Without that, pip (and presumably other installers like conda) would not be able to install there.
As written, the PEP says
If any package management tool finds the same __pypackages__ directory in the current working directory, it will install any packages there and also create it if required based on Python version.
(My emphasis) This makes it impossible to install in a virtualenv (or in the system site-packages or a conda environment). Or am I misreading “and also create it” somehow? What are you asking the installer to create if not the __pypackages__ directory? Assuming I am not misreading, this is a major compatibility break for installers - the PEP should explain how the transition will work.
But if I did, it would say “if you need 
)