To be honest, I do not believe the “expand pip” option. At least, not today. If the pip team gets together, makes an appeal for more help to push in this direction, and comes back in 6-12 months and shows that they make steps and it can get there, then perhaps I would. But the weight of history, the complex and legacy code, the backlog of issues and difficulty of working on pip, and the important lower-level role as a pure installer it already fulfills are already stacked against this idea imho. So by all means try if you want, but it’s a poor conclusion to now decide that this is the way.
Poetry/PDM/Hatch are much more along the lines of what users seem to want. Each has its own problems and isn’t complete enough, however if you’d take the best features of each you’d have about the right thing. Now that has the same problem as for the pip direction above: we cannot agree to pick one here, nor can we wave a magic wand and merge them. But again, that is not needed. It’s encouraging to see that the authors of each of these tools have shown up here (much appreciated @sdispater, @neersighted, @ofek and @frostming!). If you as authors of these projects would get together and work out a strategy, I’d believe in that working out well in a reasonable time frame. And make both Python users and the average package maintainer enthusiastic about this.
I’m not intimately familiar with each of Poetry, PDM and Hatch, but here is what I understand some of the key differences and issues to be:
- Team: Poetry seems to have a good size team (multiple maintainers; 6 folks with >100 commits who were active in the last year). PDM and Hatch are both single-author projects.
- User base: Poetry seems to have most users by some distance, Hatch seems to gain quite a few new users, PDM seem to be struggling a bit.
- Build backend support: PDM supports build backends (PEP 517) the expected way, it seems that Poetry can be made to work by plugging an install command like
pip install .intobuild.py(so Poetry’s approach is even more general than build backends, but a bit clumsier by default), and Hatch has no support at all for anything but its own pure Python build backend (and new plans do not look good). - Conda/mamba support: Hatch and PDM seem to have support (not sure how feature complete), Poetry does not.
- Other package manager support isn’t present anywhere, but if conda support can be added, that leaves the door open for future other package managers (e.g. if enough people care to maintain Spack or Nix support, then why not?)
- Virtual environment handling seems to me to be a subset of this, or another angle on it. There’s no good default, as the discussion on PEP 704 shows. PEP 582 (
__pypackages__) is an entirely reasonable approach. PEP 704 doesn’t add much, but as long as there’s different ways of doing things it probably doesn’t hurt either. The important thing here seems to be to hide complexity by default from especially the beginning user, while allowing multiple approaches.
- Plugin system: all three projects seem to have a plugin system and a good amount of plugins.
- Lock file support & workflows: Poetry and PDM have lock file support, Hatch does not. There’s an issue around standardization here, TBD how that will work out. Poetry and PDM seem to conversely have issues with the “package author” workflow, which should not lock and not put upper bounds on dependencies by default. Looks like that needs more thought in projects and something like two workflows/modes: application development and package development.
- Completeness of commands offered: it seems pretty complete for all projects, although a standardized
testcommand at least would be a nice addition for all projects.
There’s probably more to compare here, but I think it’s safe to say that if these projects would join forces, we’d have something very promising and worth recommending as the workflow & Python project management tool. And it doesn’t require a 200+ message thread with everyone involved in packaging agreeing - if a handful of authors would agree to do this and make it happen, we’d be good here and could “bless” it after the fact.