Where to get started with pyproject.toml?

Forgive me if what I’m going to write feels like ranting; I’ve had this discussion with many people and have many thoughts on this topic.

The number of files does not correlate to “goodness”. Each configuration file is a signal to people reading your repository (including your future self) what tools the project uses. You see webpack.config.js and you know a project uses Webpack, not (say) Gulp; pytest.ini for pytest as the designated test runner, noxfile.py for Nox, etc. Each of the three files in this set-up signals different aspects of your project’s tooling. pyproject.toml means this is a Python project. setup.cfg means it is using setuptools for packaging. setup.py means it also uses setuptools during development (note that you don’t need this file if you don’t call python setup.py directly; pip recognise setup.cfg and can run it independently). Can we use only one file? Absolutely. But I’d prefer not doing that, personally.

I do regnoise that a lot of people don’t agree with my preference. And that’s OK, since it’s just a subjective preference. But on the other hand, it is also important to recognise the reasoning behind the preference to multiple configuration files. What a better way means depends on the people who use the tools (and remember, project maintainers are usually the most involved users of the project itself!), and it doesn’t help to complain when you and the tool designers disagree.

If you absolutely can’t stand this “clutter” (as some of my friends put it), there are other tools that may fit your preference. Poetry (mentioned above) and Flit are both alternatives to setuptools, and both of them only use pyproject.toml as the One Configuration File. (There is also discussion to add pyproject.toml support to setuptools as well, but don’t hold your breath waiting for it.) But regardless of the tool you choose, it’s only natural to sometimes disagree with the design. We all learn to live with the differences.

5 Likes

Personally, it doesn’t really pain me we have many files but it does that the explanation for them is either buried deep into the docs somewhere or not transparent at all.

It was more a documentation issue for my side than an infrastructure issue.

3 Likes

We don’t actually - setuptools is fully PEP 517 compliant, so you just need pyproject.toml.

1 Like

Do know of some sample source code, ideally an open source project, that shows how this is done with only pyproject.toml?

It’s right there in the setuptools documentation: Build System Support — setuptools 53.1.0 documentation

1 Like

To be fair, on that page, it says you need the pyproject.toml and a setup.cfg file. Maybe that’s just for an example but that’s the confusion some of us were raising.

I’m guessing, projects with just the pyproject.toml end up adding every metadata, found in setup.cfg, in the pyproject.toml file, right?

Oh you’re right, that’s a good point, I missed that – especially as the syntaxes for TOML and .cfg are so similar… :roll_eyes:

So yes, for now you need both, but there is currently an ongoing discussion which shows a lot of promise…

There are a lot of people showing interest in the feature, but no-one committing resources to implement it yet. Not exactly what I’d call “a lot of promise”.

2 Likes

Since I’m the one who created this thread, I want to say publicly though, that once I had spent a bit more time reading through the rich documentation, things looked to make more sense indeed.

All in all, my project moved a the new scheme. It wasn’t that hard and I am happy I could get away from setup.py almost entirely (it’s almost empty now).

3 Likes

Disclosure: I am an average Python coder I am occasionally helping folks in SciPy as a core maintainer and I have my own pure Python package harold for which I decided to follow the latest greatest standards.

I can’t say I don’t regret it. I’ll try to explain what the whole user journey currently is;

  • There is a very obvious expert blindness seeping into the docs. I just wanted to write a toml or yaml or cfg with ?ml file syntax but ended up reading PEP 517 because everywhere on the docs mentions PEP 517 or 660 or 518 or any other random number. If people think that average users will understand that text without being a somewhat expert on packaging we have much bigger problems.

  • There is no clean cut description what these files say. Also the documentation language is, much to my regret, full of jargon. Take setuptools Quickstart - setuptools 62.1.0.post20220410 documentation it says basic use. It uses setup.py and pyproject.toml. One of them is discouraged and the other one is experimental. What am I going to use only setup.cfg? Are these three options or I need all? Then I go up the page because I need more info apparently and it says read stuff at the bottom of the page. Very well. There is something called setuptools Quickstart - setuptools 62.1.0.post20220410 documentation
    That seems relevant because that’s what I am doing. But below that it points to Packaging Python Projects — Python Packaging User Guide which asks me to change my project structure to so-called src structure which I didn’t have before. Is this what the new way is? No answer there. Can I keep my project structure? I can’t tell.

  • Then I start googling and I understand that setup.py is the legacy way. So then I need pyproject file which I learn that from setuptools it is an option. But the actual setuptools file is called setup.cfg. I mean why? Why not setuptools.cfg? OK nevermind, not my problem. So I read Brett’s blog post What the heck is pyproject.toml? Ok probably more advanced people can understand and appreciate that. Not me because I still don’t get which file I should use let alone their content.

  • So I start randomly doing things on my own adding lines removing lines duplicating them etc. and I start hitting [BUG] · Issue #3197 · pypa/setuptools · GitHub Thanks to @abravalheri 's kind help I manage to straighten some kinks both mentally and code-wise but still this is seriously complicated compared to Python’s typical straightforward and expressive style.

Note that, I am not this baffled while dealing with SciPy’s Fortran/C/C++ build system with n different compilers. But with this new development, I can’t even get the 10000ft high-level strategy around these tools let alone actually following tutorials.

Anyways, long story short, where can I attempt to contribute to the docs for a unapologetically beginner level tutorial for this? Because clearly noone else would do it. I don’t know what I am doing and that makes me a perfect candidate for it.

And as a side note, please don’t use non-deterministic word randomly in the docs. If you know what I mean…

6 Likes

Not much of an answer, but maybe I would recommend not using setuptools, and use any other modern build tool instead, flit might be a good choice, pdm is another one.

Otherwise everything you mentioned rings true to me. Documentation is not as good as it should be, but it is improving by the day, so is Python packaging in general.

Another user posted a somewhat similar post with the idea of documenting their own journey into Python packaging with fresh eyes, you might find something useful in there for you. Here it is:

And the output:

2 Likes

Ah thanks a lot indeed, that thread is more eloquent than what I have written down. I’ll follow that one a bit carefully.

If you have to deal with C extensions, then you might want to follow and contribute to this topic as well: "Modern" way to build packages with C (Cython) extensions

1 Like

By Ilhan Polat via Discussions on Python.org at 13Apr2022 21:23:

Ah thanks a lot indeed, that thread is more eloquent than what I have
written down. I’ll follow that one a bit carefully.

BTW, I’d welcome feedback on the overview docs from you as a user of the
doc (rather than the PyPA guys, who will be reading it as people who
already know how all this stuff works). Could you have a read of this:

https://github.com/cameron-simpson/css/blob/pypi/doc/pypa-the-missing-outline.md

and report your impressions or suggestions?

I’m going to try to get it included in the PyPA docs up the front where
the tutorial is mentioned.

Thanks,
Cameron Simpson cs@cskk.id.au

3 Likes

This related thread might be of interest: What new Reference or Explanation would usefully reduce confusion? It suggests a few specific document topics which might address the confusion which this topic reports.

1 Like

Cameron, I just took a look at PyPA The Missing Outline. Some suggestions:

  • Suggest you reformat your 13 April message to make the link clickable. (Probably indent with quoting (’>’) instead of with unformatted code (three backticks ‘```’.)
  • I like the Diàtaxis framework for technical documentation. It seems you are writing what they call a “How-To Guide”. Consider using their framework to sharpen your focus. Be clear on the audience and the goal. Avoid sliding into Tutorial or Reference or Explanation.
  • There is another way to “publish a package”, which is by making it a self-contained command-line executable, which need not be installed to the user’s Python environment, and treats the user’s Python interpreter as a runtime platform. You don’t intend to cover this, but it would be helpful to say in the first section that you don’t intend to. Deflect the readers who actually want a Guide to Packaging a Command-Line Executable.
  • In “The Build System”, it might be helpful to state that “build” in this context is limited to the Source Distribution and Wheel artifacts, and does not apply to other artifacts which one might “build”: command-line executables, documentation, etc.

I hope this is helpful for you. Thank you for the writing!

Two more comments on your draft, having thought some more.

  • Consider a more specific title than PyPA The Missing Outline. You are giving instructions for ending up with sdist/wheel artifacts, and uploading to PyPI. So maybe include these terms in your title. Also, what happens when PyPA fills in their missing outline? You will still have a useful guide, but your title will not make so much more sense.
  • I think I want to attempt a Guide to Packaging a Python Application File, based on using shiv, pez, and zipapp. I think it will be parallel in audience and intent and structure to yours. When I have a draft, we can compare them.

I feel exactly the same. I’m lost and am going to stick with setup.py until a benevolent dictator wakes up and cleans the packaging mess up.

3 Likes

Just FYI, since somehow I never got back to John:

  • Diàtaxis: this doc is an Explaination, not a Tutorial. It aims to provide the mental framework on which any publish-to-PyPI How To would hang in an author’s mind.
  • The PyPA people were very welcoming, and after some review my doc is up on the site at: The Packaging Flow — Python Packaging User Guide
  • Your intended guide would be a tool specific How To.

2 posts were split to a new topic: Help with my pyproject.toml