Paul captured part of what I had in mind, except that I think PEP 405 sufficiently standardised the existing venv layout that PEP 582 could just re-use that definition rather than needing to define it anew.
The other aspect is that we have the first two tutorials on packaging.python.org where I would like to see PEP 582 discuss how we expect the initial user experience to be improved if the PEP were to be accepted:
There are enough grey āNoteā and red āWarningā boxes on those guides (especially the first one) that I think itās entirely plausible that the PEP could improve them, but at the same time Iām conscious of the UX discussion in Consider initial bootstrapping concerns for pipenv-based tutorial Ā· Issue #394 Ā· pypa/packaging.python.org Ā· GitHub that convinced us that we need to separate the initial āinstall the packages I want to use for my Python projectsā guide from the āinstall the packages I want to use for this particular Python project in this particular directoryā dependency management guide.
Iām sorry you feel that way. I like the idea in principle (after all, itās very similar to the ideas behind zipapps, which Iām a big fan of).
The stumbling block seems to be simply āwhat is expected of pip (and similar tools)ā. Iād be happy enough to defer that for now if it takes some of the pressure off. But I do want to avoid it getting ignored (because I think doing so would harm the proposal, in the same way that zipapp suffers from pip install --target being a clumsy option).
BTW, I think āentrenchedā is a bit hard. Pip has a huge user base, supporting an immense range of workflows, and very little developer resource. Iād describe our situation as āsaddled with huge backward compatibility issuesā rather than āentrenchedā. (Although Iād accept ātoo concerned with backward compatibility to easily accept changeā as a fair, if depressing, characterisation )
zipapp is actually one of the big reasons I like this proposal (if amended to use a PEP 405 layout internally), as I can see the benefit clearly there - putting the dependencies in a nested venv would be quite a bit closer to the way shiv already works.
tox version is tied to tox config; new versions of flake8 add or change rules and unpinned version suddenly breaks CI, requiring to change the code or config; pytest versions need to be compatible with pytest plugins and my own fixtures; pip-tools may change details of the output format or support more options, etc. The only tools that I can install globally and without caring about the version are httpie (local dev tool, not called from tests), pdbpp, safety, for example.
(Reply to old question, was clearing messages and wanted to give an answer to this specific point!)
Iām unclear on how āI think this general idea can work, with these particular changes to the internal technical plumbing that will make the implementation easier but should otherwise be invisible to most end usersā ends up being interpreted as ātoo entrenched to shiftā.
It was intended as āthe core idea seems good, but it has some practical issues with potential barriers to adoption, which I believe we can solve with these particular changes to the technical detailsā.
Iāve been down the path of proposing major packaging workflow redesigns myself with PEP 426, and learned that unless I can somehow bring copious amounts of contributor effort to all the affected tools, as well as a major educational push to get existing developers to change what they teach newcomers, then incremental improvements that are straightforward for developers of existing tools to implement and publishers of existing projects to adopt are going to be more effective in practice than seeking step changes (hence my own overly ambitious metadata 2.0 spec in PEP 426 ultimately being withdrawn, while Dustin Ingramās more pragmatic metadata 2.1 spec in PEP 566 was accepted)
Thatās a question that is easily answered with some level of experience in software development: your systems need to crash in order for the answer to become apparent.
For example, if pip installs packages globally by default, newcomers may globally install things needed for their projects with simple pip install foo commands (I have peers at my workplace that do exactly this), then eventually they will have enough projects on their system that something only works with one version of a dependency while something else works with an incompatible version of the same dependency, and then one project will work while the other doesnāt. This causes waste of time having to switch dependencies or to learn non-standard non-official venv workflows that can vary substantially from project to project.
Locally-installed dependencies are the solution to these pains by simplifying the workflow for people making python projects. Itāll be like npm for Node or conan for C++, where any given project on any machine can depend on its dependencies in a much more reliable fashion.
I think I have a small amount of experience with software development
I was not suggesting that a tool like flake8 and all its dependencies should be installed in my system Python. That is a bad idea for all the reasons you mention. But that doesnāt mean I need to install it repeatedly, in every single project I maintain. Tools like pipx exist to manage centrally installed but isolated copies of Python packages. And you can build your own isolated environment with virtualenv. Python development tools tend to be distributed as libraries (with console entry points) rather than as standalone apps, but that doesnāt mean you have to use them like that.
As I said originally, no-one installs a separate copy of their editor, or their C compiler, for each project they work on. Why do it for their code formatter or linter?
Agree for the formatter. I think the linter is a bit more unique, as linter generally keep adding new features/checks all the time, you can easily get to the point that your projects lint run fails today, though was still ok yesterday.
Anyways. for whoever is reading this topic: the python community did not give up on this idea, though still is looking for a champion willing to take up the mantle and iron out the details of how we make this better than what we have.
The options are there so we can choose (global installs or local installs, though I wish local installs were the default with global being opt-in). Personally Iāve had too many issues with linters and formatters due to version incompatibilities (causing more waste of my time than local installs) that I simply prefer to enter a project and install its requirements locally and avoid those problems. It doesnāt take much extra time to do that.
Itās one year passed and there is no new updates in this thread. But I read this and decided to write my reply:
Anyways. for whoever is reading this topic: the python community did not give up on this idea, though still is looking for a champion willing to take up the mantle and iron out the details of how we make this better than what we have.
Currently, I am maintaining a package manager PDM that is built on top of this PEP, but with some tweaks:
Besides lib, there are also bin(Scripts on Windows) includes directories in __pypackages__.
The version namespace can have a -32 suffix for 32-bit Python.
The __pypackages__ in the current work path will be loaded if there is no __pypackages__ inside the directory where the script called exists. So that there would be only one __pypackages__ for nested scripts folders.
If there is no __pypackages__ found, will look for a given depth of parents for one to exist.
In PDM, I was able to make python(not wrappers) be aware of local packages by hacking the startup of the interpreter, more specifically, users just need to add a sitecustomize.py to the PYTHONPATH to make it happen.
The biggest benefit of PEP 582 is you can have only ONE copy of each utility tool. For example, I can install ipython via pipx and work for arbitrary projects. When you enter the project directory, the local packages will be loaded, so is useful for many debugging and profiling tools.
Thanks to the 3. and 4. tweaks, it is also able to call a console script with the absolute path of the executable directly, so that one can achieve the similar to what pipx does.
But beyond that, PEP 582 has shortages which virtualenv does better: multi environment testing ā you canāt make multiple local packages directories that have different set of dependencies. This is why I didnāt force users to use PEP 582 in PDM.
I post to make core developers know there are tools that rely on this PEP and it turns out happy using for the most of time. This comment also acts as a ping to wonder if there is any possibility to make this PEP go further.
Iām planning to send you and David OāConnor of Pyflow an intro email in about two weeks to connect you with @kushaldas who is the co-author of the PEP who still wants to actively push this (heās out sick ATM, hence the wait time).
Found this PEP recently while struggling with virtual environments and dependency management as fresh python developer.
It seemed to solve all issues at once, but along the way I became more sceptic.
At first sight it avoids the need for virtual environments by having a local copy with dependencies and dev dependencies. In fact we need multiple arbitrary dependency configurations, since a software project deals with 4 types of dependencies that should be separated:
compile time dependencies (absent in a pure python project)
runtime dependencies
test dependencies (typically a superset of runtime dependencies)
Gradle (in Java land) addressed this from the start by allowing you to define arbitrary dependency configurations.
They even made the build tool itself a sort of versioned dependency: you specify the gradle version in your project and add a bootstrap script to your project that downloads the desired version as required.
When maintaining multiple projects, this becomes indispensable as different projects require different versions of tools.
IMO the approach taken in this PEP has the same issues as NPM: eventually all dependencies get mixed together leading to conflicts in the long run.
python-wraptor is yet another approach to abstract away virtual environment management and to manage both dependencies and tools.
I personally think that saying itās a āneedā is a bit strong. I canāt remember the last time I had a conflict among the four groupings you listed in a project.
But how is this any different than today with virtual environments? And how often have you run into version conflicts between the tools you install and your own project requirements?
Are you specifically advocating for a new approach to installations in Python? What Iām trying to find out is what your proposed solution is as I feel like youāre saying you donāt like how Python handles installation of packages already, and so you want to change things more drastically. That in and of itself requires its own PEP and a consideration of backwards-compatibility as all tooling out there expects there to be a single version of a package to be used at any one time in a running interpreter.
You can create an arbitrary number of virtual environments, one for each need. This is basically how tools like tox, nox, and pre-commit do it, and personally I think itās extremely useful and should be handled first-class if we are to standardise environment (virtual or PEP 582 or whatever) organisation in a project.
With that said, someone just told me today this is a wrong solution so maybe Iām just doing things terribly wrong and donāt know what Iām talking about, and we should definitely mix all those dependencies in one environment
The way I imagine this is a directory similar to pyproject-lock.d (say .pyproject-env.d). Each directory in it is a directory containing an environment, named something like
# Generate pyproject-lock.d/default.toml.
pyproject lock --group=default
# Syncs default.toml into .pyproject-env.d/default-cp310-win_amd64/.
pyproject sync --group=default --python=cp310 --arch=win-amd64
# Execute a command against a populated environment in default-cp310-win_amd64.
pyproject run --group=default --python=cp310 --arch=win-amd64 -- my-console-script
There are some other UX details to figure out, e.g. can the various options be inferred or set with some project-wise configs so we donāt need to type them in every time, can we just say --python=3.10 and --arch=64, and so on. But those can be explored by implementations, once the basic structure is layed down.
There is a significant thing that needs clarification from the PEP ā What is the path scheme other than lib
pythonloc and pipx seems to pick __pypackages__/X.Y/lib/bin, but pdm picks __pypackages__/X.Y/Scripts for windows and __pypackages__/X.Y/bin for other platforms. This would be a main blocker for IDEs to add support for PEP 582.
IMO, any proposal to add a new place where āthings can be installedā should provide a sysconfig-compatible layout, by which I mean provide a clear explanation of how to calculate a path corresponding to every location returned by sysconfig.get_path_names() - ('stdlib', 'platstdlib', 'purelib', 'platlib', 'include', 'scripts', 'data') in Python 3.9.
Itās a shame that sysconfig doesnāt provide an extensibility mechanism to allow 3rd parties to register additional schemes - then we could just say that the proposal needs to define a sysconfig scheme and be done with it.
Having said that, this proposal, as itās a PEP to change Python, could (and probably should) define a new sysconfig scheme (Iād suggest the name ālocalā, but feel free to bikeshed ) and then tools can simply do sysconfig.get_paths("local") and know they will be able to treat the result just like any other install target.
)
Ah, the internet.
) and then tools can simply do