User experience with porting off setup.py

[kknechtel (Karl Knechtel)]

@pf_moore

I think that in the light of the current thread, it’s going to be crucial to provide a clean and well-documented transition for people who currently have complex setuptools build processes

@indygreg

Me too. I’m clearly not alone in this boat. Some people even said they gave up porting off python setup.py because they couldn’t figure out how to do it.

Okay, so what I’m wondering at this point is: What are the key kinds of complexity found in these build processes? Specifically what do people so with setup.py, that’s difficult to (do in pyproject.toml / migrate to a pyproject.toml-based approach), that’s really valuable to the people doing it (rather than just being e.g. some old legacy workaround that isn’t really needed any more)? Reading through the rest of the thread, I don’t get a clear picture of this. I do get a clear picture that many people are finding it difficult to do certain specific things, but not what those things are. For me, it’s only really ever involved turning some keyword arguments for the setup call, into corresponding entries in pyproject.toml. (And I was someone already accustomed to writing JSON by hand, so TOML seems trivial.) I do use Poetry, but I’m honestly considering dropping it and just using the built-in tools.

@indygreg

Lots of people incorrectly believe that all of setuptools is deprecated and they need to delete setup.py.

I honestly feel like we should start talking about getting people to “delete setup.py”. Of course, Setuptools is still a default backend, thus it is not by any means deprecated. However, I can imagine a future in which it either eventually drops support for setup.py (and a bunch of legacy cruft is cleaned up), or it otherwise morphs into or is replaced by something that’s explicitly only a backend. Once it exists, pyproject.toml is simpler, if only because it isn’t executable code used for configuration.

@indygreg

I don’t fully understand why a build frontend isn’t present in Python distributions by default.

Did I misunderstand something? I know that distutils is being removed in 3.12 and that Setuptools is technically third-party, but I thought that pip wheel still exists, still sets up isolated build environments, and still provisions whatever version of Setuptools is necessary to function as a backend within that isolated build environment. Does that not, then, qualify as “a build frontend present in Python distributions by default”?

(Actually, I’m honestly not clear on why build exists in the first place.)

---

That said, I have some thoughts about organizing guide/tutorial content about pyproject.toml, which I think apply independently of whatever those issues might be.

@oscarbenjamin

Both pip and setuptools support all of the relevant standards and the only real change needed is probably to add pyproject.toml listing setuptools as the backend and any other build depenmdencies and then… It is not necessary to go and learn about all of the other backends or frontends or whatever. What is needed is a clear guide that shows how a setuptools based project (the vast majority of existing projects that exist on PyPI) should be modernised to work with current standards…

What are the minimal changes I should make to ensure that I am not using deprecated features of packaging any more?

Agreed 100%, but with the caveat that there is clearly more to talk about than just that use case.

@BrenBarn

Note this means it won’t solve the specific problem discussed in this thread, where there’s a preexisting desire to use setuptools

The desire to “use setuptools” (i.e. specify it as a build backend) isn’t a problem AFAICT, and won’t be for some years yet. The problem is the desire to use setup.py. It’s exactly as Oscar says IMO.

@BrenBarn

I just axed the entire part of the tutorial that involves choosing a backend and made it use Hatchling. I’ve never even used Hatchling and I don’t care what the backend is, but (as mentioned on other threads) I don’t think a first-look “here is how to do this” tutorial should say anything about backend choice other than “look elsewhere if you want to do this any other way than exactly how this tutorial does it”. The user shouldn’t need to make any choices like that to follow the tutorial.

I’m not comfortable with the idea that an officially blessed tutorial makes an arbitrary choice of tooling from among competing third-party options. The tutorial should IMO explain that a backend is being chosen (and briefly point out some common options), but then show how to choose the built-in, default option (i.e. setuptools, but still having it use pyproject.toml) and make it work (i.e., pip wheel, unless again I misunderstood something).

@BrenBarn

I don’t believe the tutorial achieves that aim, because I think there are only two ways to do that:

1. Pick a backend and tell them to use it.
2. Include a complete discussion of the pros and cons of various backends so they can choose one.

Right now the tutorial doesn’t do either of those, so the net result is an increase in confusion.

I think this observation is fair. It’s just, why not pick Setuptools to get started, and shuffle the discussion off to the side?

@oscarbenjamin

This is why I have suggested two guides that should be aimed at people who are (as in the title of this thread) porting off of setup.py i.e. maintainers of existing setuptools-based projects:

1. A guide for updating to recommended usage while still using setuptools.
2. A guide for considering the possibility of migrating from setuptools to something else.

@henryiii

I think there’s a very important third category, and that’s exactly what the current guide is aimed at:

3. Give users new to packaging Python projects a clean, modern introduction to how things should be done from scratch.

That’s what the current guide is for, and I think it does that well. It’s not meant to push anyone toward changing backends - both by not forcing a backend, and by not being targeted toward existing projects at all.

I think all these categories of users are important. I’m not convinced that they need separate guides. (The fact that the current guide can do what it does “without forcing a backend”, btw, makes me disinclined to agree that “choosing a backend is important”. Choosing Setuptools doesn’t feel like making a choice, for the most part. Although in the sketch below, I do propose describing it as such.)

Here’s how I think a unified explanation should be laid out, roughly:

For modern projects we don’t recommend using setup.py or setup.cfg any more. Instead, the metadata that used to go in setup.cfg, or as keyword parameters to the setup() call in setup.py, are now covered by pyproject.toml. If you’re starting fresh, you don’t have to worry about the history here. However, people maintaining older projects should be aware of these changes. Having code run at install time has numerous downsides [explain in detail]. [etc. etc. as necessary] If you’re in this situation, please read through in order to get a basic understanding of the new system; section X covering the detailed contents of pyproject.toml will also show how to migrate your old setup with minimal changes.

Here’s pyproject.toml. You use it to give metadata about the code that will be distributed (the [project] table), and to configure the tools that will prepare the code for distribution (the [build-system] table). Code can be distributed as a sdist that is basically a zip archive of the source code, or as a wheel - this is necessary to handle compiled C extensions. We say that a wheel needs to be “built”, but for pure Python projects this generally just means bundling the Python source code together with some metadata. The advantage is that an installer can then directly copy files into place, since they’re now organized the way that they need to be within the site-packages directory of the target Python. [Ed.: Or something along these lines, basically just explain why wheels are important even for pure-Python, open-source projects]

Building a wheel generally involves having a frontend communicate with a backend. (Explain these terms in-line, in addition to linking the glossary, along with justifying why the split exists.) In modern Python, an up-to-date version of Setuptools will work fine as a backend, and Pip serves as a frontend. Some people prefer to use third-party tools to fill these roles - many of them provide their own frontend and backend, as well as offering other functionality such as uploading wheels to PyPI. (Pip does not do this; see the guide for Twine or the guide for manual upload if you want to use only native Python tools.) Skip to section Y if you’re interested in choosing a third-party frontend and/or backend.

[This is section X.] Here’s how you use the [build-system] table to tell Pip to use Setuptools as a backend. There’s also legacy support for using Setuptools if the table is omitted, but you shouldn’t rely on this because [explain why]. Here’s how you use the [project] table to give the necessary metadata. Only the name and version are strictly necessary, but others are important or useful because [explain why]. If you already have an old setup.cfg file lying around, here’s how to migrate it. If you already have an old setup.py file lying around, you might have to do some more complex analysis, but here are some hints on filling in the metadata from there.

After that, here’s how to use pip wheel after that in order to create the wheel. [Explain where the .whl file ends up, basic anatomy etc.]

[This is section Y. Now we can mention Flit, Hatchling, Poetry, PDM etc. ad nauseam.]

The idea is that people can be directed to skip over the parts that don’t apply to them, and there’s one overall canonical source of information. More specific guides could then be extracted from this, potentially (such as what @bryevdv outlined).