Clarifying the differences between PEP 517 and setuptools

I am mostly an onlooker that is tired of the awkwardness of setuptools, but I think one reason why PEP 517 is not on-par with setuptools is that the available tools still do not cover all the use cases. In particular, it seems that using PEP 517 tooling to build python extensions is not quite there yet (at lest this is the impression I have after trying and asking for advice here). Another issue is that all the “canonical” Python packaging documentation assumes setuptools (well, actually in most places it assumes distutils). I think it would be premature for widely used complex packages to rework their build story around PEP 517, unless the maintainers are committed to work on the tooling itself and to spend some extra efforts in helping users to understand how the new system works.

1 Like

I think here you’re mistaken. PEP-517 is only the setup.py CLI part of setuptools. PEP-517 does offer feature-wise on par support with what setup.py CLI does. It’s perfectly fine to use setuptools via PEP-517 interface, instead of setup.py calls, in which case there’s no loss of functionality.

1 Like

To be absolutely precise here, there is a known limitation in that PEP 517 does not provide “editable install” (setup.py develop) support, but that’s not relevant for the topic under discussion here, as distributions wouldn’t need that functionality.

Editable support is the subject of a different (currently stalled) discussion.

1 Like

I think there is a even more pressing need for clear documentation then.
I have spent multiple hours trying to understand what the Python
packaging future is supposed to be, and I haven’t yet understood what
the large picture is.

What is PEP-517 bringing if the only practical difference is typing pip install . instead than python setup.py install while still needing to
maintain a setup.py setuptools based build systems?

In this thread is has been debated that package maintainers should
abandon setuptools to embrace PEP-517. Here you are saying that (if you
want a working build system) PEP-517 is just another way to invoke
setuptools. I don’t see what the discussion is about then.

I am honestly confused.

Cheers,
Dan

1 Like

That’s what I meant by “stalled” :slightly_smiling_face:

This subthread is really off-topic, though, so I suggest we either move it somewhere else or drop it at this point. This topic is about what distributions need to handle projects that build with PEP 517, rather than with setup.py.

2 Likes

I am even more confused. @bernatgabor is saying that to have feature parity I need to write a setuptools based setup.py and invoke it via the pep517 interface. If this is the case, how am I loosing the capability to have editable installations? I can always invoke the setup.py script to obtain that.

1 Like

IIUC this is not an exact definition, as PEP 517 defines an interface (an the most common choice is currently to use that interface to invoke a setuptools based setup.py), thus nothing is build “with PEP 517”. I think this terminology is what confused me as I thought that pep517.build is actually doing the work, however now I understand it is only an interface to the underlying build tool. Maybe some more clear language could help people that has not been following closely to avoid a similar mistake.

1 Like

Thanks @bernatgabor, I just scanned the post quickly, but it seems to be exactly the kind of resource I was looking for.

1 Like

My bad here. We have feature parity minus the develop command. And yes you can manually keep invoking setup.py develop, but other frontend tools (e.g. pip/IDE/etc) who would like to use the PEP-517 API to make this invocation can’t do it yet (and might need to fallback to invoking python setup.py develop, but this is only possible if you have that setup.py file around).

1 Like

I keep seeing people be confused by such lazily worded statements as this, so let me try and clarify the various aspects:

  • if your package builds with setuptools, you will have a setup.py file
  • if your package builds with setuptools, you can invoke setuptools directly through setup.py develop to use its editable install feature
  • that form of editable install is currently a setuptools only feature (it would be nice if other build tools also provided an equivalent)
  • you use PEP 517/518 to programmatically inform other tools that your package build uses setuptools (or something else)

That last point never seems to be clear enough: the build backend specification is nothing more than that, a specification. You are still fully reliant on the build backend for configuration and the actual mechanics of the build, and nothing about PEPs 517/518 make them interchangeable from the POV of a package developer or a package installer - only for the package builder.

To drive the point home, let me rephrase those bullet points in terms of my own build backend:

  • if your package builds with pymsbuild, you will have a _msbuild.py file
  • if your package builds with pymsbuild, you can invoke pymsbuild directly through python -m pymsbuild to use its in-place build feature
  • that form of in-place install is also available in setuptools through a different invocation
  • you use PEP 517/518 to programmatically inform other tools that your package build uses pymsbuild (or something else)

The critical piece: PEP 517/518 fulfils exactly the same role here as it did earlier. And it doesn’t impact wheel installation at all.

So distributions that need to rebuild things almost certainly want to use PEP 517/518 directly, and likely need to invoke it directly to provide platform-specific settings (which are not very well defined right now, so this seems like a good opportunity). This will generate a wheel, which can then be installed using any mechanism from an unzip tool through to pip install --no-deps <name of wheel file>.

And if you’ve just built the wheel yourself and know exactly which platform it’s for, the unzip tool is going to be just fine, because you’ve manually done the dependency resolution and build. Everything pip has to automatically install dependencies (including for build) and isolated environments is for random users, not build experts. If you’re deliberately participating in the build, rather than "pip-install-and-hope"ing, you’re going to find pip getting in your way more often than not.

4 Likes

To this I’d add:

  • python setup.py develop and pip install -e create a .pth in site-packages containing one or more paths to code (in development mode; where the program has permission to overwrite itself)

Also,

  • Does a distro-specific wheel installer need to record what nonstandard paths package resources have been installed to for all or only some packages in the env? How should an installed module find the other files of (what was) its [wheel] package?

I think this is the first point of contention? Is injecting the source tree into the sys path without any packaging transformations considered develop mode? If that’s all you after you don’t need setup.py you can just do ‘PYTHONPATH=src python -m’. In a pep 517 editable mode I hope the backend to have more control :man_shrugging:

Yeah but there are often many scripts in $VIRTUAL_ENV/bin where PYTHONPATH= would need to be manually configured (“postactivate”), src/projectname/projectname/__init__.py means you can’t just add one src/ maybe with symlinks, and symlinks didn’t work on windows; so: .pth files.

Thank you @steve.dower. I think one of the main pain points with Python packaging (at least for package authors that do not follow closely the PyPA team activity) is documentation and imprecise statements do not help understanding. It would be great if the Python packaging official documentation could be updated to current state of things and hits to the direction where development is aimed be added. AFAIK the “official” documentation still assumes setuptools (or most likely distutils).

1 Like

“Build System Support”
https://setuptools.readthedocs.io/en/latest/build_meta.html

https://github.com/pypa/setuptools/blob/master/docs/build_meta.rst

  • conf.py: include “edit on GitHub” links in the setuptools docs theme or maybe just the RTD theme?
  • sphinxcontrib-srclinks
  • RTD button in the lower right corner >

https://packaging.python.org/guides/distributing-packages-using-setuptools/
https://github.com/pypa/packaging.python.org/blob/master/source/guides/distributing-packages-using-setuptools.rst

“6. Helping with Documentation”
https://devguide.python.org/docquality/