Since this thread seems to be my fault, let me use this opportunity to give my perspective on the state of Python packaging.
Python-only packages
I think we’re in a mostly adequate place here. Anybody who who wants to publish a Python-only package can realistically get it done, either with my blog post, flit, poetry, or by going setup.cfg
-only. There is a bunch of paper cuts left though.
I personally feel (and I do understand that it’s only my opinion) that flit and poetry are nice approaches but they only get me 90% there. If people are cool with that, more power to them but I’m not seeing myself switching to static metadata because I don’t want data duplication between my code and package metadata. We have well-known __
variables for almost all fields, why is flit the only one looking at __version__
(and ignoring the rest)? I’ve been told on the poetry tracker to ship my pyproject.toml
with my package (!) and parse it using ConfigParser(!) in my __init__.py
(!), on import (!).
I am aware of importlib.metadata
but unless I want to add a runtime dependency, backward compatibility makes it a non-starter for existing projects. I know about setuptools_scm
, I respect it, but I don’t like to treat SCM metadata as canonical data YMMV.
I also like to be able to concat the latest changes (and only the latest, if you concatenate everything on a long-running project, it gets unwieldy) to my long_description
as a service to my users – I guess I could write a setuptools plugin to do all my shenanigans? But that brings me back to why I don’t use flit and poetry, things I’d like to do dynamically keep coming up.
I also don’t see much value in them over a setup.cfg
-only setuptools package, except I wish we could finally stop using setup.cfg and put that data into pyproject.toml
. This half-transition that’s dragging for years now is not great.
Another major problem that won’t allow me to drop setup.py
anytime soon are editable installs in PEP 517. I love PEP 517 but I really need proper editable installs. I’m still a bit surprised that this went past PEP 517. You need them for src
layouts (I know, that’s my pet peeve but it’s increasingly popular) and when you have setuptools entrypoint for CLI tools.
OK these were the good parts.
Binary packages
The moment you add the least amount of compiled code to your package, flit and poetry are right out and paper cuts turn into proper guttings.
First, check this beautiful setup.py: https://github.com/hynek/argon2-cffi/blob/master/setup.py
And now look at this arcane heap of terribleness that I have to use to build wheels for argon2-cffi
: https://github.com/hynek/argon2-cffi/blob/master/.azure-pipelines/wheel-builder.yml
I’m sorry for the strong words, but this is bullshit. The average maintainer cannot be expected to deal with this.
The only reason I got it working after hours of fiddling at all, is that @Alex_Gaynor and Paul Kehrer did the most the work for me and I “just” had to adapt a few details.
The ~best~ nugget is certainly that on Linux and macOS we have to pin pip to a version before PEP 517 support (10.0.1).
If I understand it correctly, the logic goes like this:
- We need to tell wheel to build an
abi3
wheel – but only on Linux and macOS, because there aren’tabi3
wheels for Windows yet. - You cannot pass arguments to
wheel
throughpip wheel
in PEP 517 mode (and neither throughpep517.build
AFAIK – I’m personally very disinterested in the discussion whether pip does everything or not as long as it works). - Disabling PEP 517 with
--no-use-pep517
while having PEP 517 configuration inpyproject.toml
makes the build abort. - Profit?
If we could set the limited ABI unconditionally, this problem might go away? For that we’d need abi3
for Windows and wheel tolerating/ignoring that options on Python 2. However, abi3
for Windows won’t land before 3.9 so we’re stuck with the mess above for years to come.
Honestly, I have never been closer to abandoning argon2-cffi
than on that frustrating Sunday afternoon.
TL;DR
- Add
pyproject.toml
support to setuptools so we can dropsetup.cfg
andsetup.py
. - Consider support for metadata extraction for well-known fields.
- Get proper
pip install -e .
with PEP 517 support. - Get
abi3
for Windows. - Figure out
abi3
tooling in the meantime. Allow passing arguments to wheel throughpep517.build
? - I love you all, thank you unironically for your hard work. I remember the times of
easy_install
and eggs and at least for the users, the situation has much improved.