I have toi create a new Python package so I thought, for once, I’d try to be doing the right thing and use the new pyproject.toml as well introduced by Brett in https://snarky.ca/what-the-heck-is-pyproject-toml/
But I goty confused quickly unfortunately. It seems no official documentation exists yet around pyproject.toml and the PEPs aren’t user docs. It appears Pypa and the Python docs both continue talking about using setup.py.
I was wondering if there was a place I wasn’t aware of perhaps? Or maybe some help to document it if I can find the time, but then where to start?
So as I understand you, pyproject.toml is only meant to declare the build configuration and we still need setup.py/cfg for defining what’s inside the built package, right?
Correct. To be pedantic, setup.py and/or setup.cfg is needed for defining what’s inside the built package if you choose to use setuptools. But most people use setuptools anyway.
Forgive me if what I’m going to write feels like ranting; I’ve had this discussion with many people and have many thoughts on this topic.
The number of files does not correlate to “goodness”. Each configuration file is a signal to people reading your repository (including your future self) what tools the project uses. You see webpack.config.js and you know a project uses Webpack, not (say) Gulp; pytest.ini for pytest as the designated test runner, noxfile.py for Nox, etc. Each of the three files in this set-up signals different aspects of your project’s tooling. pyproject.toml means this is a Python project. setup.cfg means it is using setuptools for packaging. setup.py means it also uses setuptools during development (note that you don’t need this file if you don’t call python setup.py directly; pip recognise setup.cfg and can run it independently). Can we use only one file? Absolutely. But I’d prefer not doing that, personally.
I do regnoise that a lot of people don’t agree with my preference. And that’s OK, since it’s just a subjective preference. But on the other hand, it is also important to recognise the reasoning behind the preference to multiple configuration files. What a better way means depends on the people who use the tools (and remember, project maintainers are usually the most involved users of the project itself!), and it doesn’t help to complain when you and the tool designers disagree.
If you absolutely can’t stand this “clutter” (as some of my friends put it), there are other tools that may fit your preference. Poetry (mentioned above) and Flit are both alternatives to setuptools, and both of them only use pyproject.toml as the One Configuration File. (There is also discussion to add pyproject.toml support to setuptools as well, but don’t hold your breath waiting for it.) But regardless of the tool you choose, it’s only natural to sometimes disagree with the design. We all learn to live with the differences.
Personally, it doesn’t really pain me we have many files but it does that the explanation for them is either buried deep into the docs somewhere or not transparent at all.
It was more a documentation issue for my side than an infrastructure issue.
To be fair, on that page, it says you need the pyproject.toml and a setup.cfg file. Maybe that’s just for an example but that’s the confusion some of us were raising.
I’m guessing, projects with just the pyproject.toml end up adding every metadata, found in setup.cfg, in the pyproject.toml file, right?
There are a lot of people showing interest in the feature, but no-one committing resources to implement it yet. Not exactly what I’d call “a lot of promise”.
Since I’m the one who created this thread, I want to say publicly though, that once I had spent a bit more time reading through the rich documentation, things looked to make more sense indeed.
All in all, my project moved a the new scheme. It wasn’t that hard and I am happy I could get away from setup.py almost entirely (it’s almost empty now).
Disclosure: I am an average Python coder I am occasionally helping folks in SciPy as a core maintainer and I have my own pure Python package harold for which I decided to follow the latest greatest standards.
I can’t say I don’t regret it. I’ll try to explain what the whole user journey currently is;
There is a very obvious expert blindness seeping into the docs. I just wanted to write a toml or yaml or cfg with ?ml file syntax but ended up reading PEP 517 because everywhere on the docs mentions PEP 517 or 660 or 518 or any other random number. If people think that average users will understand that text without being a somewhat expert on packaging we have much bigger problems.
There is no clean cut description what these files say. Also the documentation language is, much to my regret, full of jargon. Take setuptools Quickstart - setuptools 62.1.0.post20220410 documentation it says basic use. It uses setup.py and pyproject.toml. One of them is discouraged and the other one is experimental. What am I going to use only setup.cfg? Are these three options or I need all? Then I go up the page because I need more info apparently and it says read stuff at the bottom of the page. Very well. There is something called setuptools Quickstart - setuptools 62.1.0.post20220410 documentation
That seems relevant because that’s what I am doing. But below that it points to Packaging Python Projects — Python Packaging User Guide which asks me to change my project structure to so-called src structure which I didn’t have before. Is this what the new way is? No answer there. Can I keep my project structure? I can’t tell.
Then I start googling and I understand that setup.py is the legacy way. So then I need pyproject file which I learn that from setuptools it is an option. But the actual setuptools file is called setup.cfg. I mean why? Why not setuptools.cfg? OK nevermind, not my problem. So I read Brett’s blog post What the heck is pyproject.toml? Ok probably more advanced people can understand and appreciate that. Not me because I still don’t get which file I should use let alone their content.
So I start randomly doing things on my own adding lines removing lines duplicating them etc. and I start hitting [BUG] · Issue #3197 · pypa/setuptools · GitHub Thanks to @abravalheri 's kind help I manage to straighten some kinks both mentally and code-wise but still this is seriously complicated compared to Python’s typical straightforward and expressive style.
Note that, I am not this baffled while dealing with SciPy’s Fortran/C/C++ build system with n different compilers. But with this new development, I can’t even get the 10000ft high-level strategy around these tools let alone actually following tutorials.
Anyways, long story short, where can I attempt to contribute to the docs for a unapologetically beginner level tutorial for this? Because clearly noone else would do it. I don’t know what I am doing and that makes me a perfect candidate for it.
And as a side note, please don’t use non-deterministic word randomly in the docs. If you know what I mean…
Not much of an answer, but maybe I would recommend not using setuptools, and use any other modern build tool instead, flit might be a good choice, pdm is another one.
Otherwise everything you mentioned rings true to me. Documentation is not as good as it should be, but it is improving by the day, so is Python packaging in general.
Another user posted a somewhat similar post with the idea of documenting their own journey into Python packaging with fresh eyes, you might find something useful in there for you. Here it is:
By Ilhan Polat via Discussions on Python.org at 13Apr2022 21:23:
Ah thanks a lot indeed, that thread is more eloquent than what I have
written down. I’ll follow that one a bit carefully.
BTW, I’d welcome feedback on the overview docs from you as a user of the
doc (rather than the PyPA guys, who will be reading it as people who
already know how all this stuff works). Could you have a read of this:
