Python Packaging Strategy Discussion - Part 1

[brettcannon]

(I’m a maintainer of packaging and VS Code.)

@pf_moore

the big problem here will be that many of the respondents wanting a unified tool will have been making an implicit assumption that their preferred workflow will be supported…

I see this from the Python experience in VS Code perspective all the time. People often don’t have exposure to other workflows, so they innately think their workflow is “normal”, so if something doesn’t work for them then it obviously is broken for everyone, right? And when you point out that they are not necessarily normal, not everyone is up for changing to match a more common workflow.

@pf_moore

does Javascript have the concept of compiled extension libraries?

Yes, but they are not hosted on npm as built libraries; think of them like sdists.

@pf_moore

One alternative that I think we should consider is continuing to work on the goal of splitting out the various “components” of packaging into reusable libraries[2]. Projects like installer, build, packaging and resolvelib are good examples of this. Using those libraries, and more like them, it will be a lot easier to build new workflow tools, any one of which has the potential to become the unified solution people want.

This is the approach I also support. If we can get everything backed by standards and then make using those standards easy by making sure they have backing libraries, then we can make it easier for people to experiment a bit as to what the proper UX should be for any front-end packaging tool. But it also gives people an escape hatch (no pun intended) to use a different workflow if the preferred/default tool doesn’t meet their needs.

@pf_moore

It’s definitely going to result in more complexity on the way to something simpler, and I’m pretty sure it’s not what the survey respondents were imagining, but I feel that it might be a better trade off for the long term.

More complexity for us or users? I can see for us as we have to put in the effort to write the code for everyone to use (see the ongoing journey of packaging.metadata as an example ), but I personally can’t think of how users are impacted by these things.

@smm

Can we do anything else to reduce the complexity?

Probably a page on packaging.python.org that lists the tools and why one might choose them. This came up on Mastodon at Antoine Beyeler: "@daneah@fosstodon.org I'm currently reading "Publ…" - Mastodon just today. I think part of the issue about the perceived complexity is people simply don’t know where to turn for current knowledge and lack guidance without finding the right blog post or article explaining how to do something.

@smm

If we do reduce the complexity, are there any obvious or not so obvious disadvantages in going down that route?

Only if it becomes too restrictive (which I don’t see us allowing to happen). While people bemoan Python’s packaging story as being too complex, it’s flexibility is what helped it become the glue language of the programming world. While I would always prefer people wrote more Python code than wrapping existing code, we also have to accept, and to an extent embrace, that folks simply won’t do that.

@ofek

I’m fine with Hatch providing this unified UX since it really is almost there already.

I’ve previously said I’m fine with that viewpoint if we wanted to push a tool we have today.

@ofek

The only thing that is out of my control is the lock file standardization which Brett discusses here.

I haven’t forgotten. GitHub - brettcannon/mousebender: Create reproducible installations for a virtual environment from a lock file is the high-level steps left to work towards an MVP lock file format.