Dependencies for build backends, and debundling issues

Packaging
1

I’ve just had a Linux distro maintainer reach out to me on the editables project, asking about switching to flit as a build backend. Apparently, the dependency chain for setuptools involves platformdirs, which is built using hatchling, which uses editables, which uses setuptools (!)

The problem comes when distributors want to debundle dependencies from setuptools, when having a circular set of build dependencies is an issue. This is something that setuptools doesn’t formally support, but debundling is a common request and something that I think deserves a certain amount of consideration. I.e., even if the consensus is that as a community, we won’t support debundling, I think it’s better as a community agreement rather than expecting each project in the chain to make their own decision (as the author of editables, the debundling is happening 3 levels higher up in the chain than me, for example, so why do I even need to have an opinion on it?)

It appears that flit_core is the only build backend with no dependencies (apart from tomli on older versions of Python, so not even that really!) so in order to break a chain like this, people need to use flit. I have my own personal reservations about flit as a build tool, but they aren’t relevant here. The question is whether it’s reasonable to have flit be in a “special position” as the only dependency-free build backend. (I don’t even know whether flit’s author would want to promise that property as a long-term commitment).

The ability to have “in-tree” build backends was added to PEP 517, specifically to try to avoid bootstrapping issues for build backends. It wasn’t focused precisely on the debundling issue, but it seems like it should be relevant here, and yet it’s either not being used by backends, or isn’t actually helpful.

What are people’s views on this issue? Is “sorry, debundling isn’t my problem” a sufficient response here, and are we as a community OK with the implications of this?

As far as editables is concerned, I won’t be moving to flit. I would consider other build backends (indeed, I intend at some point to move off setuptools anyway) but I won’t be considering the dependency issue and its impact on debundling except as a potential tiebreaker between two otherwise-equal options (which is not something I think is likely in practice).

1 Like
2

I think so. I don’t think we should be trying to solve this problem in multiple build-backends. We’ve largely consolidated on Flit as being the build-backend for low-level packaging tooling, precisely because it’s the only tool that’s solved the bootstrapping story for Linux distributors.

If you don’t mind, could I ask you to elaborate on the reservations you have around Flit?

That seems to be relevant here since, if you didn’t those reservations, you would’ve switched to it and this discussion wouldn’t need to happen. :slight_smile:

1 Like
3

It depends on what “layer” the tooling is on.

If it’s needed for redisibutors to get to the point of bootstrapping up to getting a functional (non-debundled) pip: No, I don’t think that’s a reasonable approach to take and is, TBH, borderline hostile to redistributors.

Otherwise, yes. It’s 100% OK to say that.

1 Like
4

Specifically, I don’t like (and don’t want to use) the flit command itself. It doesn’t offer a particularly useful or complete set of project workflow tools, and so it offers very little in comparison to setuptools. Furthermore, I am strongly uncomfortable with the fact that by default flit build and py -m build produce different sdists (because flit looks at VCS data, whereas flit_core doesn’t). Having to manually configure things so that these two commands work the same, as well as being fiddly and error-prone, is also not well-documented as far as I can find (this is a side-effect of the fact that there’s no independent documentation for flit_core as a backend).

On a more general note, I’m looking to move to a more modern build workflow for all of my projects. And I’d like to standardise on one tool, which will likely be either PDM or hatch. Having to use a different tool for editables, simply to address an issue which is not even with my project, isn’t something I’m particularly comfortable with.

OK. Is that documented anywhere, and is the maintainer of flit OK with that position? Also, is the decision to use flit, or is it simply to use the flit_core backend? If the latter, then can that backend be documented as an independent project so users don’t have to disentangle the details of flit_core from the flit documentation?

Also, what counts as a “low-level tool” here? The editables package is intended as a dependency for build backends, but if a build backend has a dependency, then it’s already made the decision to require a complex build process, so isn’t that the problem?

Looking at this another way, why isn’t it up to platformdirs to build with flit because setuptools uses it as a dependency (vendored, but the point here is that debundlers choose not to accept that mitigation)? Both platformdirs and editables are direct dependencies of build backends, so shouldn’t the same rules apply?

I’m not sure I follow your logic here. I assume you’re implying “by building everything needed from source, not using prebuilt wheels”. But that’s not the issue here, because that’s perfectly manageable (I believe) as long as you don’t insist on debundling setuptools.

Arguably, your logic means that pip should switch to flit (or just flit_core, if requiring build for bootstrapping is acceptable - I’m not sure if build can bootstrap itself without a working pip), so that we avoid a complex chain for building (non-debundled) pip from source. And then everything else can do what it chooses, with debundling being purely a discussion between projects that vendor dependencies and redistributors wanting to debundle them, with no impact on the vendored dependencies themselves. All of which (including using flit for pip), I’m fine with :slight_smile:

5

It can, this is part of the core design. See Installation - build 1.0.3 (now that I am looking at this page, it needs to be updated for Python 3.11 and tomllib).

6

No, because installer exists. If you can build a wheel for pip, you can install it with installer. And, IIUC, setuptools is more difficult to bootstrap than flit is, even if you bootstrap other pieces.

FWIW, my take has always been that redistributors who wish to debundle when projects tell them not to, are subject to not getting support from upstream, so I’m not gonna elaborate on how/what they should be doing. :slight_smile:

As one of the maintainers of Flit, I certainly am. :slight_smile:

1 Like
7

I’m not sure I’m following your point here, but I guess it’s irrelevant, as I’m not really worried whether anything changes or not. My original question has been answered, that (basically only) flit works if you need to avoid bootstrapping issues, but in general there’s no need for downstream dependencies to change to support debundling.

Thanks for the help.

Ah, I didn’t realise you were a flit maintainer, sorry.

8

To be clear, it’s really flit_core as a the build back-end, and I would be surprised if either @frostming or @ofek designed their tools to require a specific back-end.

9

I’d also mention flit_core isn’t really special here except that it exists and already satisfies the self bootstrapping case, the real constraint is that you can’t have dependency cycles a build time. Anyone can create an alternative to flit_core that also satisfies that constraint and use it instead.

2 Likes
10

setuptools_scm depends on setuptools, but only for legacy backward compatibility reasons, it is now completely uncoupled. When Ronny the maintainer strips the logic into a new reusable package then the Hatchling plug-in will depend on that but nothing can be done until then.

11

So two outcomes:

  • flit_core is the recommended backend to use somewhere in the build dependency chain to break cycles
  • flit_core needs enough documentation to be able to use it as a standalone backend, even if these docs are effectively internal and targeted towards packaging tool maintainers

Is this correct?

12

Personally, I’d add “using flit_core must not give different results depending on whether the flit command or another PEP 517 frontend is used”. That’s a showstopper for me. Although I’m not currently considering flit_core for any of my projects, so :person_shrugging:

1 Like
13

This already exists.

https://flit.pypa.io/en/stable/bootstrap.html

If no one else protests… yes.

14

For this specific issue the easiest fix is to not version using VCS/hatch-vcs but rather define the version statically, or dynamically from shipped code.

edit: linked doc

15

That isn’t (IMO) independent documentation for using flit_core in the absence of flit. Maybe that’s just because of the issue with how flit (vs flit_core) picks what to go into the sdist, though - the section on sdists doesn’t document what flit_core includes in the sdist by default, just what flit does.

But that one issue makes it hard to trust that the rest of the docs apply to standalone flit_core without qualification…

16

AFAICT, there’s only one other aspect: flit build can include an setup.py whereas flit_core can’t.

17

Why is there no low-level, no-bootstrapping-required simple build backend in the stdlib? That seems like the safest direction to move toward, even if it can’t be depended on everywhere for a few years/several Python releases.

4 Likes
18

Nobody made it :slight_smile:

19

That’s kind of what I was getting at :smile: - it shouldn’t be that hard, right?

I really feel like the time is ripe to improve the stdlib packaging libraries with minimal tools for commonality across package managers. So much good work has been done on standardization, let’s double down on that! We’re already removing cruft like distutils and we’ve got building blocks like importlib.metadata and tomllib. There’s plenty of room for innovation and alternatives on top of this commonality, which we should preserve, but all of those tools shouldn’t have to reinvent the … ahem … wheel [1].

  1. modulo all the usual stdlib-is-where-modules-go-to-die caveats ↩︎

2 Likes
20

I will say, after thinking about it more, I’m not sure that a stdlib build backend solves any of the problems we have right now?

In the original problem, the error was caused by a dependency cycle in the build dependencies, which can be used using flit_core. In a hypothetical world where we have a stdlib backend, the original dependency cycle would presumably still exist, and the outcome here is that @pf_moore could have two backends to pick from instead of one.

Adding something to the stdlib could have solved things if we didn’t have a robust mechanism for downloading third party backends-- but we do, the problem that keeps creeping up is dependency cycles that inadvertently get added because libraries aren’t aware they’re a build backend dependency and change their backend and introduce a cycle.

What would potentially help is to figure out common dependencies that build backends (and other tools requires for bootstrapping) are using and try to add versions of those to the stdlib, because the less dependencies a build backend has, the less likely they are to get into a dependency cycle.

To do a quick survey:

- setuptools
  - packaging
  - pyparsing
  - ordered-set
  - more-itertools
  - jaraco.text
    - jaraco.functools
      - more-itertools
    - jaraco.context
    - importlib-resources
    - autocommand
    - inflect
      - pydantic
    - more-itertools
  - importlib-resources
    - zipp
  - importlib-metadata
    - zipp
    - typing-extensions
  - tomli
  - platformdirs
    - typing-extensions

- flit
  - flit-core
  - requests
    - certifi
    - charset-normalizer
    - idna
    - urllib3
  - docutils
  - tomli-w

- hatchling
  - editables
  - importlib-metadata
    - zipp
    - typing-extensions
  - packaging
  - pathspec
  - pluggy
    - importlib-metadata
      - zipp
      - typing-extensions
  - tomli

All of the tomli uses can be replaced with toml-lib, obviously the importlib-metadata and importlib-resources backports could be replaced with their stdlib versions once Python compatibility allows.

Unfortunately I’m not seeing a great path forward for big reductions in chances for dependency cycles. There’s not a ton of overlap other than the packaging library, which already uses flit-core [1].

  1. Though it might be interesting to consider packaging for other reasons, a lot of the code in there shouldn’t need to change often, and maybe backports like importlib-metadata etc would be good enough… but I’m not super involved there anymore so that would be a Brett / Pradyun question. ↩︎