I wanted to chime in that I’ve been very encouraged with the activity on this thread!
I had typed up a lot of content that I ultimately stripped from my blog post on what I thought should be done and folks here seem to be gravitating towards a lot of what I was going to say.
In case you missed it, there was some additional discussion on this blog post on Twitter/X, Lobste.rs, and HN. The largest themes I got were:
- Me too. I’m clearly not alone in this boat. Some people even said they gave up porting off
python setup.py because they couldn’t figure out how to do it.
- There was a lot of sentiment that authoritative guidance on what to do in Python packaging land is severely lacking. That good, trustworthy, modern documentation is hard to find.
- Lots of people incorrectly believe that all of setuptools is deprecated and they need to delete setup.py.
- People seemed to be largely dissatisfied with the extra complexity from the introduction of
pyproject.toml. However, I think the reasons are highly varied. (Some people just don’t like any change. Others are complaining about the lack of porting docs. Etc.)
In addition to the topics discussed so far, I want to raise a few more from my post.
Lack of a Build Frontend in the Default Distribution
I don’t fully understand why a build frontend isn’t present in Python distributions by default.
I think that shipping a build frontend in Python distributions could make things vastly simpler for end-users by eliminating a lot of cognitive overhead with having to think about which build frontend to use and how to install it.
Many languages do things this way (Ruby’s gem, Rust’s cargo, Go’s go). End-users seem to love the unified toolchain approach. And the presence of a default tool doesn’t undermine innovation in the larger ecosystem.
Before Python 3.12 (or earlier 3.x releases if we want to be pedantic about setuptools availability), we had the ability to produce sdists and wheels using just the standard library’s distutils + [ensure]pip. But 3.12 fully removed this capability. If we want to be customer focused and ease the transition for existing package maintainers, shipping a [simple-to-use] build frontend in the distribution seems like an effective way to do that.
Securely Installing Packages in
Are my blog’s assertions about
pyproject.toml build system package installation being intrinsically insecure accurate?
This question can be answering by stating how to deterministically bootstrap a Python build system frontend and backend and all transitive dependencies in a way that is robust against new package versions being published and is resistant to content tampering.
Is there any documentation on how to do this bootstrapping securely? Are there any discussions on it folks can link me to?
FWIW I have a half baked idea for package registries to store content digest indexed manifests - think requirements.txt files or poetry’s equivalent - and then for package installer frontends like pip to be able to do something like
pip install flask@sha256:deadbeef42... to download a content indexed manifest stating all transitive dependencies to install. This way, deterministic install descriptors can be generated and used for reproducible, tamper-resistant installs. All an end-user has to do is refer to a short, immutable content digest instead of having to maintain the manifest themselves. This is conceptually similar to how OCI (read: Docker) image registries work - image manifest & image index.