The package setuptools is not part of Python itself but the current situation is very confusing - at least for people having just basic experience with packaging (as me):
Also on Linux freshly created venvs contains setuptools.
Besides that in the venv documentation ( venv — Creation of virtual environments) setuptools is described as:
core dependency
common installation tool
core venv dependency package
The documentation does not mention if the core dependencies are just a default setting or if it can be changed or what is the status of setuptools.
The main point was that whatever you have installed doesn’t matter, since both pip and python -m build do build isolation by default and make a fresh venv for the build and install the build system requirements there. Whatever you have in your current venv is not used at all.
And unlike pip, setuptools doesn’t provide any interface to invoke it, so the newcomers probably won’t even know that they have setuptools (unless you count python setup.py sdist bdist_wheel as the interface, which is deprecated and we definitely don’t want newcomers to be using that).
It depends on what you mean by “concrete”. Setuptools is only installed in a base Python as a result of ensurepip, and that will change as soon as the pip developers decide that pip can run smoothly without setuptools - which means that we’ve removed the last few legacy bits of behaviour that expect setuptools to be present. So the removal won’t happen just yet, but it could be relatively soon. At that point setuptools would also vanish from venv. I can’t speak for the virtualenv maintainers, but I’d hope it (and wheel) would be removed from virtualenvs as well.
And this is one of the reasons I feel that we should recommend a backend other than setuptools. There’s a lot of legacy baggage around for setuptools, that will likely be confusing for users. Advice to use setup.cfg rather than pyproject.toml, examples that show metadata being computed dynamically in setup.py, etc. All of that is out of date, but good luck getting it removed from the internet Newer backends don’t have this legacy, so there’s no confusing messaging out there for users.
I think some of the recent comments from beginners suggest that at least a little more explanation of backends and build isolation might be useful in the tutorial.
@saaketp and @pf_moore already answered this in detail (in the affirmative), so I won’t repeat what they said.
As others have implied, the opposite is true; since it is intended to be a pure backend intended to be downloaded and installed at build time in the isolated build env like any other PEP 517 backend and invoked by PEP 517 frontends, and there’s no supported interface for for users to invoke it directly (nor has there ever been, formally), it doesn’t matter from a user perspective whether it is installed locally.
Right, so it makes no difference to users whether it happens to be installed in the rutnime env, because the backend is always downloaded and installed in the isolated build env anyway, other than it comes with a lot of legacy baggage and bad third-party advice, while not able to be as simple and helpful as newer build systems.
The idea is to make clear that with PEP 517 and PEP 621 this is an implementation detail; all users need to worry about is specifying their backend of choice and the metadata in a standard format (and possibly package data, if their build system doesn’t handle it automatically as most do), and their frontend will automatically take care of the rest.
To note, @henryiii 's PR does address and clarify this, but perhaps not sufficiently for users who already have some incomplete knowledge of the old legacy ways of doing things prior to PEP 517. @sbaack , do you think you could take a look at the preview, particularly the creating pyproject.toml section, and see if it makes this more clear to you, and if not, what you’re still confused on and why? @vbrozik , maybe you’d want to do the same?
Note that currently this preview tutorial doesn’t work, because of the conventions around package names in newer backends; see the PR discussion for more details.
To note, my request to the users was in regards to to conceptual explanation of backends, rather than actually executing the relevant steps.
In any case, this should be mostly a matter of updating the PR to ensure that the import package name and distribution package name match, as they always should for any modern package (interestingly, the lack thereof just came up again on here). @henryiii , any update on that?
Other than that, it seems the remaining issues are just a bug in PDM, and my fault for still not finishing updating PEP 639 again yet.
To clarify the distinction between front- and back- ends, I think pip and build should be identified as ‘frontends’ in the pyproject.toml section. Also, my understanding is that some of the ‘backend’ tools can act as frontends as well, so that should be made clearer. (e.g. you’d use ‘flit build’ rather than ‘py -m build’, right?) Perhaps revert the first clause to: pyproject.toml tells frontend build tools (like pip and build) which backend tool to use…
Following on from that, the later steps in the tutorial use build and twine, etc., whereas with flit and others you’d use their commands for the build and publish steps. Could it work to use tabbed content in those sections, too? It might get too cumbersome to do this, I don’t know. But, as it stands, it is confusing to show the four tabs for different backend tools, and then have the remaining steps not match the actual process you’d use if you’d chosen to follow, say, the hatchling track.
The two sections titled " Generating distribution archives" and " Uploading the distribution archives" use the term ‘archive’ which is not the usual term. I think this should say ‘packages’ to match the usual terminology (and what is listed in the glossary).
@bernatgabor can confirm, but yes that is the plan for virtualenv since pyproject.toml-based builds don’t need any particular deps preinstalled. Paraphrasing from Discord:
I want to remove setuptools and wheel if anything […] people should use build-system.requires and not the embedded wheel functionality
I was thinking something along these lines as well.
This echoes my initial concern about using build backends that come with their own frontend. However, the discussion in this thread indicates a strong desire to a) adopt a simpler backend than setuptools and b) demonstrate the backends can operate independently of frontends. I think the assumption is that a newcomer to Python packaging won’t know about all the frontends, and will be well-served by the agnostic tools like pip and build.
I tried to normalize the terms a bit already, but held off on completely replacing “archive”, because the change already felt like scope creep.
Thanks to everyone for all the clarifying answers! This was very educational and I learned something However, I still do not fully agree with the approach taken in the updated tutorial from a beginners perspective.
While it’s great that build will automatically download the tools you need for you, I think it’s a disservice to beginners to implicitly suggest that they don’t need to install hatch/flit/setuptools/pdm at all because if I’m not wrong again, you still need those if you want editable installs? I certainly want those and I suspect a lot of people do. At least I would add a note that you can optionally install those tools if you want editable installs.
Related to this, I agree with @craigf that the tutorial should at least mention that flit and other tools have their own build and publish steps. Because from a beginners perspective, I found it rather confusing the tutorial for flit etc. tells me to install them and use their publishing tools while the ‘official’ tutorial uses something else entirely.
I totally get why build is highlighted the way it currently is, but from a complete beginners perspective, I think it would good to at tell people that they might want to install hatch (if that’s the default choice now) and also show the build and publish commands from hatch as tabs.
No, you shouldn’t need the backend installed to get an editable install either. (Once again, setuptools’ legacy behaviour might be an exception here )
I think it could be worthwhile to note that some backends have a UI, and if you want to use that UI, rather than the sundard backend-agnostic tools, then yes you will need to install the backend locally, and refer to the specific backend’s documentation for details. I don’t think we should document backend-specific commands in the guide, though, as we’ll only get into debates as to which backend we should describe
Really? I thought so not just because of setuptools, but also because other tools like flit needed a PR to explicitly support PEP 660. Why do they need this if I can make an editable install without any backend installed?
Because PEP 600 requires the backend to provide extra hooks, which are used to install the mechanism that exposes your project as editable (which may be just a .pth file, or soemthing more complex). But the mechanism that gets installed doesn’t need the backend installed to work.
I don’t think you need setuptools installed for something that you’ve installed as editable via the legacy method to continue working either, but I’m wary of making any definitive statements about setuptools, as they’ve got so much backward compatibility stuff to deal with that anything’s possible…
Sorry to dwell on this, but just to be sure I understand: Does this mean that the backend will automatically be downloaded when I call python -m pip install -e ., just as with python -m build? So when my pyproject.toml says I’m using hatch, hatch will be downloaded automatically and used to make the editable install?
As mentioned above, if you would mind explaining either what in the proposed revision of the tutorial gave you these misconceptions, or where you might expect to have them addressed, that would help us further improve the tutorial for you and everyone.
Just to clarify this explanation a bit for a beginner, since its rather dense, the backend does get installed, but this is done automatically by the frontend in an isolated environment it controls, and happens regardless of whether you may happen to have it installed in your local working environment or not.
Indeed that’s correct; as I understand it pip just runs setup.py develop in the isolated build env until Setuptools’ PEP 660 support is merged.
Thanks again for clarifying! This is actually pretty awesome.
My misconception comes from not knowing that both python -m build and python -m pip install -e . don’t require me to have any backend tool installed via pip. This misconception partly is due to the fact that I checked the various backend tools before and all of them instruct you to install them via pip and in some cases to use their build and publishing commands. It’s not obvious for a beginner like me that python -m build is a more ‘universal’, tool-agnostic solution.
I think it’s really important that this is communicated clearly with a simple explanation (something along the lines of ‘whatever tool you specify in your pyproject.toml is automatically downloaded and there is no need to install it yourself’). In addition, as others have suggested, it should point out that some publishing tools mentioned in the tutorial alternatively provide their own solutions to create and publish packages
What I was meaning was that once a project has been installed in editable form, there’s no dependency on setuptools for it to keep working like that - all that’s installed is a .pth file in sys.path. The one thing I’m not sure about is console scripts installed by legacy setup.py develop - I believe they rely on pkg_resources to run (unlike pip-managed console sripts) and so won’t work if setuptools isn’t present.
As I’ve said though, all of this is annoying and confusing complexity involved in supporting the legacy behaviours that setuptools has to deal with. The message for “modern” packaging tools (which includes modern setuptools, of course) is much simpler, and we should very much promote the simple model and guide new users to ignore all of the history and legacy behaviour for as long as they can get away with doing so
What’s not clear to me is how to promote the new message without getting a lot of pushback along the lines we’re seeing here (from people with scars from previous enounters with how packaging used to be )
Edit: Or pushback from people who are getting a different message from the backend documentation which focuses on the “backend as a build tool” user interface (thanks @sbaack for clarifying that’s where you got the confusing message from).
Newer backends don’t have this legacy, so there’s no confusing messaging out there for users.
However, I still do not fully agree with the approach taken in the updated tutorial from a beginners perspective.