PEP517's definition of "frontend" and "backend" is unclear to me

Packaging
1

PEP517’s definition of “frontend” and “backend” is unclear to me.

PEP 517 – A build-system independent format for source trees, section ‘‘Terminology and Goals’’, says:

A build frontend is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree’s build backend. In a command like pip wheel some-directory/, pip is acting as a build frontend.

This says that the frontend is a) what users run, and b) builds wheels. But it says that the backend is what builds wheels. This is confusing. That thing which builds wheels — is it the frontend or the backend?

PEP517’s example confuses me more.

In a command like pip wheel some-directory/, pip is acting as a build frontend.

Well, the user invokes pip from the shell, so that makes it a front-end. But doesn’t pip also make the wheel? Doesn’t that make it a back-end as well? And, am I correct in understanding that pip does not consult pyproject.toml, so it really predates the PEP517 division of build into frontend and backend? If so, that seems to make pip a poor example.

PEP517 defines a key build-backend for the pyproject.toml file. Its value is “a string naming a Python object that will be used to perform the build”… That implies to me that the object which performs the build is the build-backend, so therefore the frontend must be the object which does not perform the build.

It seems to me that what PEP517 might be trying to say is,

A build is the action of building a wheel or sdist from an arbitrary source trees or source distribution. A build frontend is a tool that users might run to invoke a build. The build backend is a library called by the build frontend, and which actually performs the build action. The build frontend consults the pyproject.toml file to identify the build backend, and invokes it. The build backend consults pyproject.toml and perhaps other configuration files in the source tree and/or parameters passed from the frontend to guide how it performs the build.

Do I understand that correctly?

And, what tools are good examples which clearly show the distinction between build frontends or build backends, in the PEP517 sense?

Background: this question arises because I used my understanding of PEP517 to pose a question in StackOverflow, How to build pex or shiv package from pyproject-compliant project? A contributor there suggested that I misunderstood what PEP517 means by frontend and backend, and that maybe I could clear up my misunderstandings on this forum.

1 Like
2

That’s reasonably accurate. (I’m not going to nitpick over details, as I assume what you’re after is a broad understanding, not technical details).

The problem you seem to be having is that build backends are intended to build wheels or source distributions (which are distributed on package indexes like PyPI and consumed by installers like pip). They are not for building other types of build artifacts, like zipapps. So you don’t “build a pex or shiv package” with a build backend, you use pex or shiv to do that.

3

No, no and no. Copying from what I wrote in Build System Interface - pip documentation v23.3.2

When dealing with installable source distributions of a package, pip does not directly handle the build process for the package. This responsibility is delegated to “build backends” – also known as “build systems”. This means that pip needs an interface, to interact with these build backends.

Basically, when given a source tree, pip will make an isolated environment and install+invoke the build backend from that environment. The build backend (eg: setuptools, flit-core, hatchling, poetry-core etc) do the actual work of generating the source distribution / wheel and providing it to pip, using the mechanism described in PEP 517.

For pyproject.toml-based builds, see pyproject.toml - pip documentation v23.3.2 which describes the way that pip builds packages.

Yes.

build · PyPI is a good example for a build frontend and sphinx-theme-builder · PyPI is a good example of a build backend.

4

Actually, the following documentation section (in pip’s documentation) discusses the two interfaces pip uses (legacy and pyproject.toml-based) to build wheels, both of which involve delegating the work of performing the build to an underlying tool.

https://pip.pypa.io/en/stable/reference/build-system/

5

Thank you for your prompt and informative reply.

Wow.

That is not the impression I got from reading PEP517, and packaging documentation like Overview of Python Packaging - Python Packaging User Guide . I came in with an understanding of “build” from workflows like make: automated generation of multiple kinds of output from a set of source materials.

Looking at PEP517 again with this frame of reference, that “it is only for wheels or source distributions”, I suppose I can find several bits of wording that imply this limitation of scope, and several bits of wording that do not imply this limitation. I don’t find wording which states this limitation directly.

My understanding is that sphinx-theme-builder generates HTML content. Is it a good example of a build backend, even if it does not generate a wheel or source distribution?

6

I don’t want to say that @pradyunsg is wrong (he’s just as much of a packaging expert as me, and it appears sphinx-theme-builder is his project) but it doesn’t seem like the sort of backend that would help clarify what a backend is for you. I’d point to setuptools, flit, or hatch as more typical build backends.

But the key point you need to understand is that the core responsibility of a build backend is to build wheels and sdists. That’s what the build_wheel and build_sdist hooks from PEP 517 are for.

The fact that there’s no hook for building anything other than wheels or sdists should make it pretty obvious, to be honest…

1 Like
7

Thank you for your reply, and for helping me understand.

OK, I have some misunderstandings to correct!

Ah, I see.

I think that part of my problem is that I approach the packaging issue having already decided that I don’t want to use pip (directly at least), I use pipenv instead, so I’m not reading the pip documentation. And when I read the tutorials and guides in the Python Packaging User Guide, the boundaries between pip and setuptools are blurry for me. They are both things I don’t know well and don’t use.

One thing that would help me is an explanation of the PEP517 build workflow above the level of specific build tools, in the Python Packaging User Guide.

The ‘‘Packaging Python Projects’’ tutorial starts off almost filling this role. Then it sends the reader off to PEP518 and PEP517 to explain the architecture — but they did not succeed at being good explanations for a naive reader, at least for me, because that was not their original purpose. Then it goes ahead and uses setuptool as the build backend, without explaining what other backends are options. And it dives into the contents setup.cfg and setup.py, without explaining that these are artifacts of setuptools and not of the PEP517 architecture.

Another thing that would help me is to have the documentation for these tools include explicit sections on how they work with the PEP517 architecture. I looked for this in the pex and shiv documentation, and could not find it. (If they are outside the PEP517 scope, then it would have helped me for them to say that, but a document cannot mention all the infinite scopes it is outside of.) The PEP user guide section, pyproject.toml is pretty good, but it does read a lot like “let me introduce all this new PEP517 and PEP518 material to you” rather than “here is the interface: get_requires_for_build_wheel and build_wheel and so on”.

8

But “explicit is better than implicit.” It would help for the Python Packaging Guide to say explicitly that the PEP517 architecture is only for tools which generate wheels and source distribution, and that other artifacts are out of scope.

1 Like
9

Maybe. I’ll leave it to the authors of that document to consider that suggestion.

But just to be clear, PEP 517 itself is a specification document, intended to be read by people implementing the spec, not user documentation for people not familiar with packaging. Maybe it’s a problem that we don’t have something more user facing, but it’s not the PEP’s role to be that document.

10

Agreed. That’s why I said, “It would help for the Python Packaging Guide to say explicitly…”.

But, to be fair, the Packaging Python Projects tutorial does cast PEP 517 in the role of user documentation:…

… See PEP 517 and PEP 518 for background and details.…

So notwithstanding the topic title of “PEP517’s definition… is unclear to me”, I am happy if the outcome of this thread is an improved Packaging Guide for packaging users like me.

If PEP 517’s scope is clear enough to its target audience, good for PEP 517. It did not make itself clear to me, so that is perhaps feedback. Where are clarifications and corrections to PEPs recorded, if and when they are accepted? As further PEPs?

11

Just to note—while the interface defined in PEP 517 can only produce sdists and wheels directly, many if (or perhaps even most, someday) downstream tools, repackagers and redistributors either consume them, and/or invoke a PEP 517 build to produce a sdist or wheel as part of their own build process, and then consume that generated artifact. So, when a tool PEX and/or Shiv are generating a package, they may invoke the project’s PEP 517 build backend to create a sdist or wheel as an intermediate artifact which they then consume—the comments on your SO question seem to suggest that is the case.

Right, but I’m not sure if you’re in the target audience for PEP 517, a highly technical specification document targeted at packaging tool developers as opposed to regular package authors like yourself.

Nominally, on the PEPs repo, though generally more than minor editorial defect fixes are discouraged for Accepted/Final PEPs, as they are generally considered historical rather than living documents. However, the official packaging specifications are canonically hosted on the PyPA specifications page of the PyPUG site, with the PEPs being nominally change proposals; a non-normative editorial clarification like that wouldn’t require another PEP, as opposed to a PR to the specifically site.

Unfortunately, we haven’t yet migrated it over there yet which I’m a bit guilty myself for; its something I’m interested and suitably equipped to do as a technical writer, PEP editor, docs team member and I just haven’t found the spare time yet.

So, I’m not sure whether we’d accept a PR on it; if such a clarification is needed, it would belong more in the user-facing side of the PyPUG. Also, to note, the PyPUG tutorial may be already be made clearer on this point per this discussion; your feedback over there as a non-expert interested in packaging would be most welcome.

1 Like
12

Thank you for your answers.

I’d be happy to give that discussion a read. I have the advantage of pretty fresh eyes on the packaging topic. Could you double-check that issue number, please? It looks like a duplicate of an earlier link.

13

Indeed, I pasted the wrong link—sorry about that, and thanks for the catch! I meant to paste this:

1 Like
14

As PEP-delegate for specs of this nature (although technically not for PEP 517 itself, which was before my time) my view is that I might accept a clarification that said that the PEP was a technical specification, not user documentation[1], but that’s all - I wouldn’t want user documentation added to the PEP. In general, PEPs are not meant to be altered at all after acceptance.

Well, (a) that’s something that can be clarified in that tutorial, as you say, and (b) they are probably saying that for no better reason than there’s nothing else to link to. For better or worse, no-one has (to my knowledge) ever written an introduction to the history and motivation for pyproject.toml and the build backend interface…

  1. I say “might”, because that’s true of all PEPs, so I consider it redundant to add it just for this one. ↩︎

15

Maybe this discussion (and its links) can help clarify some things:

16

This is maybe true in a very narrow sense, but not really in practice. And unfortunately that is hard to understand given the language used in the PEP and in other places. Let me clarify a bit.

There’s two very different kinds of wheels:

  1. A basic zipped up package (.py, .so/.dll files, etc.) plus some metadata.
  2. A wheel meant for distribution on PyPI and meeting all the extra requirements that come with that (e.g., manylinux compliance, portability to other machines).

When you want to build an artifact like a .rpm, .deb or a conda package, what you often do is:

  1. Build a wheel in the sense of (1) and then unpack/install it, e.g. with python -m pip install . --no-build-isolation.
  2. Unpack the wheel, add the required metadata, do things like make binaries relocatable, and repack into your final artifact.

When you want to build a wheel for distribution on PyPI, you basically do the same. You do step (1), then run auditwheel/delocate, vendor dependencies (native libraries, a gfortran runtime, etc.) - and then you have your final wheel for PyPI.

Of course, when you have “simple” Python packages (= the vast majority of them), there is no difference between the two kinds of wheels - what you built with a simple pip or build call is what you throw up on PyPI. This is probably why the language in the PEP and docs is confused. But it’s certainly not true that you cannot or should not use the build_wheel hook in the process of building other artifacts.

Here is roughly my mental model of how building artifacts works:

image
image866×479 29.3 KB

When you build a wheel locally or for distribution on PyPI you take a particular path through these steps. Same for building another packaging artifact - and the paths for all artifacts are likely similar, and may invoke build_wheel (and it’s unlikely that we’d want to add build_rpm, build_xxx etc. to pyproject.toml).

I plan to expand on this elsewhere soon.

3 Likes
17

I agree that there was nothing else to link to, especially at the time. It was an understandable choice. But it comes with a cost: until that tutorial gets rewritten to point to a more appropriate specification or explainer, new readers will come across PEP 517 expecting it to be user documentation. They will notice its shortcomings in that role. It should not be surprising.

Fortunately, another discussion thread, Removing setup.cfg and setup.py from the packaging tutorial - #91 by craigf, has also suggested that more suitable user documentation should replace the references to PEP 517 and PEP 518 in the tutorial.

18

This explanation, and the flow chart, is helpful explanation. Thank you. It would be great to get that out into the Packaging Guide somehow.

Let me rephrase my claim. To someone who merely wants to build a certain kind of artifact, their interest is in how to invoke the tool which builds that artifact, and the final artifact which results. Intermediate tool steps and intermediate artifacts are not so important. They might be no more than an implementation detail.

The PEP 517 architecture applies (as I understand it) only to tools which generate wheels in the sense of (1), and source distributions. For other artifacts, and the tools which generate them, PEP 517 does not guide how to invoke those tools, how to configure them, or the non-wheel-in-the-sense-of-(1) artifacts which result. Those other tools might turn around and invoke a PEP 517-compliant tool to make an intermediate artifact, but this is an implementation detail. Someone who wants that other artifact need not concern themselves with PEP 517.

In the terms of your diagram, it looks to me like PEP 517 is concerned with only the “invoke build system” step. It might provide a way to list dependencies, thus spreading upstream into part of “set up a build environment”. But PEP 517 has nothing to say about the “Obtain sources” step, or the “Post-process build artifacts” step, or the “Install or upload artifacts” step.

To someone coming from the wider world of software development, “build” can be a broad term, that includes the “obtain sources” and “post-process build artifacts” steps. I am thinking about C-language autotools-based projects, where “build” includes a configuration step, generating makefiles, compiling, linking, installing, etc. I am arguing that if the Python Packaging world wants to say that “build” means only your diagram’s middle step, it is helpful to say so explicitly.

19

IMO, such a clarification might be a better fit when/if PEPs are referenced in user documentation (if they need to be at all). That seems to be a more appropriate place to say such, seems to be where the people who’ve been confused found the PEP in the first place, and doesn’t require modifying the final PEPs.

Well, I’m not sure about that; @brettcannon 's written a number of blog posts on the history and motivation of pyproject.toml and how it fits into the ecosystem, and there are innumerable other ones by folks with varying degrees of direct connection to the packaging ecosystem. One or more of those could be linked instead, which would probably be much more useful and approachable to the average reader than the spec text.

Right, though if they are a package author, they still should and quite likely might need to concern themselves with providing the metadata, build system dependencies and tool specific config in pyproject.toml, and if they are involved with the development of downstream repackaging tools, it is the general backend agnostic way of obtaining an artifact they can repackage and redistribute. Other than that, yes, it is essentially an implementation detail for users who are not authors or repackagers.

1 Like
20

Well, actually, that’s big news to me as someone who’s trying to figure out how this all works! I’ve just been working on converting my project (from using setup.py) to use the ‘new shiny’ pyproject.toml system (with setuptools & setup.cfg).

I was building my sphinx user-docs using an alias in setup.cfg (build = build_sphinx build). The last few days I’ve been hunting everywhere for info on how to build my sphinx user-docs as part of the new py -m build command, like it did with py setup.py build, and I think this is the answer – you can’t. This is a big surprise to me, as I’ve been reading about PEP517 & 518 being the ‘new way’, and I was expecting it to handle the full build process. From what I see in this thread, I need to revert to using py setup.py build.

I think this narrow scope definitely needs to be made clearer in the documentation (as @JDLH says, it is not an unreasonable assumption that a build process will support building all of the content). It would also be helpful to link to explanations on how to do the more ‘complex’ things (although, I don’t consider adding documentation to a project distribution to be an advanced use-case; that could be a good candidate for Tutorial #2.)