I would argue it somewhat changes things because it makes switching tools easier. Look at what a project had to go through to change build tools before and after pyproject.toml. If we can agree on the feature matrix we want to hit and get standards around them, then the default/baseline UX becomes more common and various tools are really driving experimentation and unique workflow needs instead of people having to use them for the same workflow that just happens to be implemented different due to the lack of a standard.
Iâm definitely not arguing against evolving pip to handle these virtual environment cases. I just wanted to make sure people were on the same page in terms of what âhandling environmentsâ means.
I donât think I understand this statement. Youâre saying youâre currently willing to contribute to pip, but if we decide to evolve pip to unify those things, youâre now unwilling to contribute to it? Or are you saying youâre willing to contribute to a unified tool, but only if itâs a greenfield project?
In any case, I think adding features to pip (something that has occurred regularly for about 15 years) is more feasible then us all deciding to bless a single tool besides pip as the recommended thing (something that has never happened, other than for pip itself).
What I mean I donât think implementing those features is going to change the difficulty in making a decision. Like letâs wave our wand and pretend we have lockfile support standardized and everyone now supports it.
Great, how does that help us decide which tool to bless? Are we concerned that whatever tool we decide to bless wonât implement a hypothetical lockfile standard in the future, such that we need to have it done prior? Do we think that implementations are going to differ so wildly that what tool we choose is going to hinge on exactly the semantics of how they implemented lock file support? Are we afraid that if we pick a tool and bless it, that they might come up with their own non standard lockfile format?
In all of the above cases, I think the pip developers have shown that theyâre going to implement the standards that get defined, that theyâre going to do it with an eye towards compatibility, and theyâre going to avoid introducing new, pip specific features, that should be standardized prior to such standardization existing.
In other words, I donât think those extra features provide any extra information to inform our choices here, it just serves to delay making the choice. In some cases in the past, we avoided making a choice because ultimately we didnât want to, and instead we implemented things like PEP 517 which allowed choices to be pushed onto the users. In those cases delaying making a choice was a good thing.
However, the status quo both before and after this proposed idea is that choices are still all wholly possible for end users, they can choose pip, or hatch, or poetry, or something completely different. The problem is that end users do not feel adequately served by the status quo, specifically because it requires them to make choices. Thereâs no way to avoid making a choice here, other than by pushing that onto a user base who have clearly communicated that they do not want that.
With that in mind, wait until X, in my opinion, only makes sense if X is going to alter the outcome what choice we make, which I donât think any of the proposed or hypothetical features are going to, because those are all standards that we would expect any choice we make to commit to implementing.
Yes that is correct, for the same reason I wouldnât want to contribute to Chrome if it decided to add features for task management like Jira. That would simply not be worth my time to rearchitect an existing large code base that has generally a singular purpose into a general-purpose jack of all trades.
Ok. That doesnât make much sense to me as a stance, but I understand what your stance is here.
Iâm personally not too concerned about it. I donât think pip doing these things is some crazy out there scheme. Theyâve been a common suggestion by various people for something like 10+ years, I donât think thereâs going to be a large number of would be contributors swayed too much one way or the other about it, and that same argument could be made for practically any feature added to any tool, because generally speaking nobody ever fully agrees on where exactly the lines are drawn between what âpurposeâ a specific code base serves.
I understand your view but that also doesnât make much sense to me Allow me to express my point in a different way:
Do we think the backends of Flit, Hatch, Poetry, PDM, etc. were created just for fun or because PEP 517 told us we could? No, it was because setuptools was too difficult to contribute to. And consider in that case that is merely improving upon its central and only purpose of building packages. In the case weâre talking about here weâre in a code base of equivalent size with even more complexity and weâre talking about not just adding new features but fundamentally changing what it does/is.
I donât think there is agreement that this is a desirable goal.
My understanding from PEP 517 to today is that each build tool (that compiles extension modules) can be configured in any way it needs or wants, and people will follow tool-specific docs to know how to configure it. As long as it integrates as a build backend then pip will be able to build the project, and that was the goal.
Your recent message about standardizing metadata (source files and compiler commands) to abstract these extension module build tools is the first time I recall the idea coming up, and I havenât seen enthusiasm for it in replies.
I will say I am quite confused why there is not much enthusiasm or understanding of the rationale behind that proposal.
Perhaps it is because I maintain a build backend that my understanding of the internals is blinding me to othersâ views. In my mind and how the process works in reality is you have a component that selects what files are to be included in an archive and some other component that interacts with a compiler to generate some files. There is no reason at all that they should be the same component.
I think that they were created because the vision of what they wanted wasnât possible within the constraints of existing solutions. Like there is no world where setuptools ends up shaped like flit, because the underlying goals and desires of those two projects are different.
I think that solving whatever problems people have to contributing to pip is a more tractable problem then getting agreement on something that isnât pip. These discussions tend to go nowhere, because they devolve into politics about which tool gets blessed as the default.
Pip is already the âdefaultâ, so this side steps that. In fact, pip can start adding those features today, without anyoneâs permission, and I suspect if they did so the âplease provide a unified toolâ talking point would just go away, because pip is already the default tool, itâs just implementing the features that people keep asking for.
Further, I donât really think itâs "fundamentally changing what it does/is any more than adding the wheel sub command changed what it does/is. Itâs not like itâs suddenly changing into a blog authoring tool or something.
I havenât seen these things discussed before but that doesnât mean that they arenât good ideas. Iâd be very happy for things to get a bit more standardised on the backend side of PEP 517 but for now just getting things working post-distutils is obviously the priority. I expect that later it will become clearer which things can be abstracted.
Aside from different historical alignments, I think itâs hard to ignore that one factor that also played a role was limited reviewing capacity / maintainer availability[1]. Similarly, there are limits to the capacity that pip maintainers currently have. Saying âletâs become that unified toolâ (and I agree that pip certainly is the most central today and would offer likely the easiest transition), is like sticking a sign âStampedes Welcomeâ on the door, but that doesnât make the door larger.
So while I think itâs an intriguing idea to flesh out pip in this way, it would IMO have to come with a corresponding update of governance / expansion of maintainership to allow it to realistically grow all those features in something less than âyearsâ.
Perhaps thereâs another variant involving pip along the lines of what @rgommers described further up in his blogpost (using a hypothetical pyp) â rather than re-implementing everything in pip, it could delegate a lot of these new tasks to existing solutions.
Thatâs assuming that we can find the right interfaces, but that way weâd have a unified front-end, and the backends could be switched based on user preference. This wouldnât absolve us of a choice of one tool per new task as a default, but under the hood (i.e. it would come through pip), and those defaults could change over time based on merit / popularity / necessity.
This is something the conda folks have wanted to do for years and years, to the point of having given up asking (AFAICT).
I agree, without some fundamental changes to how pip is maintained, this would be a very long process. (And Iâm not sure what precisely you cover with the term âgovernanceâ, but one constraint weâd have to look very hard at is âwillingness to break existing users for the sake of future growthâ)
Speaking as another pip maintainer, I have a slightly different view than @dstufft. I think that heâs absolutely right that growing pip into the âunified toolâ role is a great way to sidestep all of the questions about what tool to bless. But the difficult question for me is how do we get there from here.
Users want a unified tool. OK, but when do they want it? In 3-5 years? Maybe we could get pip to the point of being that tool in that sort of time period. In 6 months? Not a chance. Maintainer bandwidth is definitely one problem (and I can attest to the fact that working on pip, in particular the âmaintenanceâ side, is very draining[1]). Other issues are the various legacy cases we have to handle - removing the âdirect invocation of setup.pyâ code path has been a multi-year project blocking many new developments, similarly for the new resolver, and the idea of a âpluginâ architecture and a stable API, which if implemented would help immensely with adding new commands/functionality. And thereâs the whole scope question - while we might agree in principle with âbecoming the unified toolâ, each new command would need discussion and agreement. Thereâs been a long (again, multi-year) discussion on a pip run command that runs a script with its dependencies available in a temporary environment, for example.
In addition, this could involve significant bootstrapping issues for pip. If we offer new functionality, do we write it ourselves, or do we vendor an ever-increasing number of 3rd party modules? Do we need to re-architect pip to have a small âbootstrapâ version that can self-extend to add functionality without vendoring? How else do we isolate our dependencies from the userâs? How will vendoring further libraries impact Linux distributions, who already need to decide how or if they devendor pip? Having a single centrally-installed pip might help here (itâs what other tools do) but when I introduced the --python option, I asked about that and the idea was not well-received (some objections were simply inertia, but there were enough genuine problems that the idea was dropped - specifically the fact that subprocess.run([sys.executable, "-m", "pip"]) is our supported method of running pip from within your code).
To repeat, I agree with @dstufft that growing pip to fill the âunified toolâ role is a great option. But I donât think we should underestimate the challenges.
Equally, though, I think people offering other projects as the âunified solutionâ may have underestimated the impact on them if they suddenly became the recommended tool and gained a userbase the size of pipâs (the one exception here may be Poetry, which already has a sizeable community, albeit a lot smaller than pipâs).
One final point - weâve been gradually expanding pipâs capabilities for years now. And pip is already the standard tool shipped with Python. So in some senses âmake pip the recommended toolâ is the âdo nothingâ option - and could well be perceived by users as exactly that. If we took this route, how would we explain to users that weâd listened to their concerns and were giving them what theyâd asked for?
And Iâd be against adding maintainers who only wanted to work on shiny new stuff and ignore the maintenance side, for the record. âŠď¸
As I see it, certain areas are currently out of scope for pip, including creating sdists (use build) publishing (twine), setting up environments (venv etc.) or expanding a list of desired requirements into something like a lock file (pip-tools). Perhaps thereâs more discussion internally about these, but not being a pip maintainer, my impression is that pip maintainers have largely said a firm no to such things.
I think that has made a lot of sense in the context we have (and I like working on tools with limited scope), but if you announced that pipâs scope was opening up to cover a set of these things that we currently point to separate single-purpose tools for, even if it was going to take a couple of years to flesh that out, I donât think it would be seen as âdoing nothingâ.
Of course, forging a consensus on what set of things belonged in the new, larger scope would be hard, but it always is, and I donât think we can address these concerns without some kind of consensus.
As someone who wears that hat, and has spent a not-insignificant amount of time in pipâs issue tracker, I am ~sure that various maintainers at various stages of âhistoryâ of pip have said that they are open to the idea but that we need a broader discussion about it.
In line with that, basically all maintainers whoâve spoken here are open t/supportive of the idea of expanding scope for pip. FWIW, I have a draft blog with quotes from discussions here and on the issue tracker to that end as well.[1]
Perfectionist tendencies have kicked in, and itâs got a larger scope than what it started with (i.e. response on here). âŠď¸
I suspect if you asked them, theyâd all say âright nowâ but I also think that most of them are reasonable, and if they see movement towards it, theyâd be happy and see that weâre making progress. I donât think there is a world where they get it in 6mos no matter what we do if theyâre not already happy with the status quo options.
Yea I may have brushed over them to some degree, but thatâs largely because I view the challenges of evolving pip to that point to be largely technical challenges, and in my opinion, technical challenges are more tractable than political ones.
I think the beauty of the idea, is that it sort of is the âdo nothingâ option, like itâs not really doing nothing because weâre expanding pipâs scope and adding the features people say they want in the tool⌠but thatâs something weâve been doing all along. Itâs really just becoming a bit more aggressive about expanding those features to arrive at the destination that people want.
How to communicate that out is always a struggle for us. We could write a PEP on it if we wanted, or put up a page on packaging.p.o, or even just put it into the pip release notes, or just start doing it and let people notice that pip is gaining these features over time.
Maybe (and this is a serious comment) the key takeaway from this discussion is that we need some sort of formal publicity / user liaison team, who are explicitly tasked with this?
Yes, getting the message out is key. I think one low-cost way is pinning a GitHub issue(s) of the intent and high-level roadmap and/or placeholder issue(s) for planned milestones.
I will also add, that when pip embarked on another very difficult piece of work (new resolver), there was visibility and comms that made it out there. Sure, perhaps it didnât reach everyone, but it certainly was heard. Perhaps a similar approach or something can be learned from that experience for this new set of challenges.
One thing thatâs unclear to me in this discussion is, what exactly do we want this hypothetical new system to do? I really liked @johnthagenâs example in another thread laying out some concrete examples of things that people might do, but it seems this thread has moved in a more abstract direction.
I think it is not so important to users whether the hypothetical unification is technically âone toolâ or several, as long as they are designed coherently as an integrated set. As far as messaging, personally I think one thing that would go a long way toward making users feel like their concerns have been heard and that pip (or whatever set of tools) is responsive to them is a definitive section within the python.org docs that clearly lays out how to accomplish concrete tasks, clearly states that the way it lays out is the official way, and backs that up by clearly demonstrating how to accomplish all the tasks that people want to accomplish but that currently require navigating a labyrinth of conflicting toolsets. (In effect this would be a flowchart that includes choices like âdoes your project include code written in a language other than Python? if so, then do blah blahâ, although it wouldnât have to be structured like a graphical flowchart.)
To be honest, I do not believe the âexpand pipâ option. At least, not today. If the pip team gets together, makes an appeal for more help to push in this direction, and comes back in 6-12 months and shows that they make steps and it can get there, then perhaps I would. But the weight of history, the complex and legacy code, the backlog of issues and difficulty of working on pip, and the important lower-level role as a pure installer it already fulfills are already stacked against this idea imho. So by all means try if you want, but itâs a poor conclusion to now decide that this is the way.
Poetry/PDM/Hatch are much more along the lines of what users seem to want. Each has its own problems and isnât complete enough, however if youâd take the best features of each youâd have about the right thing. Now that has the same problem as for the pip direction above: we cannot agree to pick one here, nor can we wave a magic wand and merge them. But again, that is not needed. Itâs encouraging to see that the authors of each of these tools have shown up here (much appreciated @sdispater, @neersighted, @ofek and @frostming!). If you as authors of these projects would get together and work out a strategy, Iâd believe in that working out well in a reasonable time frame. And make both Python users and the average package maintainer enthusiastic about this.
Iâm not intimately familiar with each of Poetry, PDM and Hatch, but here is what I understand some of the key differences and issues to be:
Team: Poetry seems to have a good size team (multiple maintainers; 6 folks with >100 commits who were active in the last year). PDM and Hatch are both single-author projects.
User base: Poetry seems to have most users by some distance, Hatch seems to gain quite a few new users, PDM seem to be struggling a bit.
Build backend support: PDM supports build backends (PEP 517) the expected way, it seems that Poetry can be made to work by plugging an install command like pip install . into build.py (so Poetryâs approach is even more general than build backends, but a bit clumsier by default), and Hatch has no support at all for anything but its own pure Python build backend (and new plans do not look good).
Conda/mamba support: Hatch and PDM seem to have support (not sure how feature complete), Poetry does not.
Other package manager support isnât present anywhere, but if conda support can be added, that leaves the door open for future other package managers (e.g. if enough people care to maintain Spack or Nix support, then why not?)
Virtual environment handling seems to me to be a subset of this, or another angle on it. Thereâs no good default, as the discussion on PEP 704 shows. PEP 582 (__pypackages__) is an entirely reasonable approach. PEP 704 doesnât add much, but as long as thereâs different ways of doing things it probably doesnât hurt either. The important thing here seems to be to hide complexity by default from especially the beginning user, while allowing multiple approaches.
Plugin system: all three projects seem to have a plugin system and a good amount of plugins.
Lock file support & workflows: Poetry and PDM have lock file support, Hatch does not. Thereâs an issue around standardization here, TBD how that will work out. Poetry and PDM seem to conversely have issues with the âpackage authorâ workflow, which should not lock and not put upper bounds on dependencies by default. Looks like that needs more thought in projects and something like two workflows/modes: application development and package development.
Completeness of commands offered: it seems pretty complete for all projects, although a standardized test command at least would be a nice addition for all projects.
Thereâs probably more to compare here, but I think itâs safe to say that if these projects would join forces, weâd have something very promising and worth recommending as the workflow & Python project management tool. And it doesnât require a 200+ message thread with everyone involved in packaging agreeing - if a handful of authors would agree to do this and make it happen, weâd be good here and could âblessâ it after the fact.
I think youâre alluding to them automatically generating a lock file and then continuing to use that file going forward while it exists regardless of whether itâs been checking into VCS?
Trying to standardize that has been brought up before and typically has been shot down (e.g. Providing a way to specify how to run tests (and docs?) ). I think the issue typically comes down to choosing a scope (API or just shell command), and then what to do for OS variances. But Iâm a supporter for something like this, so Iâm the wrong person to ask why we shouldnât do it.
Allow me to express my point in a different way:
