I wanted to share some thoughts about how I see the overall ecosystem (?) working going forward - in light of the recent discussion in Proposal to create a Python Packaging Council, and also prompted by @brettcannon’s response to me in PEP 711: PyBI: a standard format for distributing Python Binaries. Apologies in advance for any lack of organization.
Taking for granted that we succeed in establishing a PyPA-with-actual-A (I’ll call it PyPC given the original thread title), what would everyone like to see as the user experience of a) installing and b) making/distributing packages going forward?
Personally, I like the idea that packaging tools stay outside of the standard library. The upside is as described - they aren’t bound to Python’s release cadence - and the downsides are pretty minor: a lot of people are just not going to need these tools, either because they just don’t have any interest in distributing packages, or because they’ve already glommed onto some alternative and continue to be satisfied with that.
To start, let me try to enumerate what a toolchain needs to include. I think these are the general minimums:
-
something to set up a test environment (
venv) -
something to install dependencies and resolve conflicts (
pip, mostly - though it needs a proper lockfile I guess;pip freeze, aside from not really solving the problem, seems more like something the venv should supply itself) -
something to help with templating new projects and keeping
pyproject.tomlup to date -
something to build the project and produce the necessary artifacts (
build) -
something to distribute those artifacts (
twinecurrently)
Larger projects also seem to express interest in:
-
code formatting tools (like
black; the standard library amazingly enough still includestabnanny, but I haven’t heard of anyone using it or suggesting its use in approximately forever) -
other code auditing tools (linters, dependency vulnerability checkers, static type-checking …)
-
documentation generation (
sphinx) -
something in the “makefile” space that can be set up to automate test runs and builds, update version numbers etc. (
invoke, I guess)
It’s hard for me to see a point in putting anything else in this category. Full-scale CI/CD seems like something to do in a language-agnostic way. Test runners seem superfluous if you already have a makefile system; just write a rule that sets up venvs and runs tests. I guess it would be nice to have easier ways to set up venvs for new versions, but I think that’s the responsibility of the venv tool (assuming PEP 711 is accepted, it could then just add functionality to grab a PyBI for a specified version from a specified source, unzip it and add the activation etc. scripts).
Personally I don’t want to use any code auditing or formatting tools and don’t see a good reason to make anything default/official/standard there. It’d be neat to be able to include invoke (are there competitors there?) even if smaller projects would mostly ignore it.
The main thing that I feel is missing is easy setup/config-file-maintenance. There are a bunch of takes on that - I even attempted one myself many years ago, and it’s a big part of why I installed Poetry in the first place. I also feel like the task of generating and publishing documentation is a bit overwhelming, but I’m not really sure what to do about that.
On the other hand, it feels a bit strange to me that I actually need a separate program to upload to PyPI. I guess that’s mainly about authentication; but why couldn’t I also just have, say, an upload interface on the site itself?
Aside from enumerating the tools, I think the main topic I have in mind is, which parts of this toolchain are PyPC’s responsibility, and how can we simplify and streamline the process of educating people about packaging (introducing the idea in the first place; making it easy to grab the default tools all at once; giving a quick overview for experienced programmers-and-command-line-users that isn’t intimidatingly long; …)?
Thoughts? Did I overlook anything big?