Dependencies for build backends, and debundling issues

The circular dependencies problem has always seemed like a hard lift to me, why does gcc get to have all the fun for example? (you have to have it to build it) or, these dependencies can be run from source checkouts?

Having a build backend in the standard library could definitely help the bootstrapping process a lot. Currently, it’s kind of a nightmare for people who can’t use pip like Linux distros.

I created GitHub - FFY00/python-bootstrap: Helper script to bootstrap a Python environment to help with that, but IMO it’s kind of an ugly solution, it’s difficult to use in a distro packaging environment, and I don’t should be needed anyway.

This would only be helpful anyway if the maintainers from the packages required to bootstrap a packaging-ready Python installation actually use it.

That’d be:

  • build
    • pyproject_hooks
  • installer

(cc-ing @pradyunsg @takluyver @pf_moore)

External build backends should be able to bootstrap themselves, but there can be situations where there are circular dependencies.

I’d be willing to champion the work of adding a simple (pure Python only, not very configurable) build backend to the standard library if people think that’s a good idea.

3 Likes

Those are all on flit_core, so… we have this resolved?

I think adding a build-backend to the standard library is a really bad idea. We are in the process of finally decoupling the packaging solutions from the standard library (distutils) and putting something in the standard library absolutely blesses it as the standard tool to use, regardless of whatever we end up communicating.

Besides, we have a competition-based model for build-backends today and a build-backend that provides the ability to self-bootstrap with a non-standard-library backend today.

1 Like

I’d like to get to a point where pypa/build can self-bootstrap and install (very limited in scope, targeting only this specific use-case) without any dependencies, then you can build pypa/installer, again, without any dependencies, and from there you can essentially bootstrap any backend you need.

Unless we have a build backend in the standard library, both pypa/build and pypa/installer would have to include a minimal build backend implementation on their repos.

The standard library backend could be undocumented and really just meant to target bootstrapping situations.

The current state of affairs is evidently an issue, it’s a huge PITA to bootstrap Python installations because of the circular dependencies. I’d be happy to hear any alternatives that could solve it.

Thanks @pradyunsg for the reminder that there are now several Flit maintainers. :slightly_smiling_face: FWIW, as a maintainer and the original author, yes, I’m OK with committing that flit_core should always be easy to bootstrap, probably requiring only stdlib dependencies (unless there’s another situation like requiring a TOML parser before it’s in the stdlib).

This is basically the only reason for anyone to still choose Flit now that other more feature-rich packaging tools have matured. And as others have pointed out, there’s a docs page specifically about bootstrapping, which I take as a kind of promise that we won’t lightly make that harder.

I also dislike this difference, but I haven’t yet had any great idea for how to get rid of it. Most likely the ‘use VCS’ option will become opt-in even for flit build, though this makes me sad because I like that as a default when I’m publishing packages. Would that assuage your concerns?

This is the crux of it. I think we need some bootstrapping story for core packaging tools - that’s why I’ve put in time and energy to create one with flit_core (to get build and installer set up). But I don’t think downstream packagers can expect that they can de-vendor every build backend without giving themselves headaches - e.g. poetry-core vendors tomlkit, which is built with poetry-core. Our ecosystem has decided vendoring is sometimes a reasonable trade-off. Downstream packagers can ask us to make changes, but some people are going to say no, and they will have to figure out ways to cope with that.

A library intended to be used by build backends is a grey area, IMO. I don’t think you’re under any obligation to avoid a build-dependency cycle, but it would be a generous gesture to downstream packagers to consider this.

This is essentially what flit_core offers. It can build itself from source, the sdist contains a minimal script to install its wheel. Then you use it to build installer, which can install itself, then build.

2 Likes

It would help. Without that change, using flit_core as a backend acts as too much of a signal that flit build is a supported way of building the project when it actually isn’t. With the change, flit build might still not be the supported approach, but at least it’s not flat-out wrong.

I’d also like to see the precise details of what gets included by default documented (given that it would then be common to both flit build and flit_core). And I’d like what gets included in the wheel to be clearly documented (I assume though that, apart from tool.flit.external-data, this is a case of “if you don’t like what we put in the wheel, you shouldn’t be using this backend”).

But to be clear, this would simply mean that I’d be happier using flit_core if I needed a “no dependencies” backend. In practice, though, I don’t tend to agree with people’s reasons for asking projects to switch to a “no dependencies” backend, so I’m not really someone who needs to be convinced.

IMO, vendoring is the bootstrapping story here, right now. I don’t like having to vendor dependencies for pip, so I wouldn’t say we do it for our convenience. And clearly distributors aren’t keen on it. But no-one has yet come up with a better answer that doesn’t take away choice from a bunch of projects which aren’t particularly “foundational” in the ecosystem.

Up to a point. If build backends have it as a dependency, then they’ve presumably solved the circularity problem themselves somehow. If they vendor it, there’s no build needed, and it’s the people insisting on unsupported devendoring who need to work around the issue. I’d be more sympathetic to build backends asking for this than 3rd party redistributors (who should be discussing the issue with the backend).

Having said that, using flit_core is a good solution. But only if I can do so while still using the workflow tool of my choice. It’s flit that’s the problem for me, not flit_core. And I think (but haven’t yet checked) that tools like hatch, poetry and pdm allow you to pick your own build backend. If they don’t, then maybe they should, otherwise we’re back to the monolithic one-tool-for-everything situation that PEP 517 was intended to fix.

Hatch doesn’t. :slight_smile:

:slightly_frowning_face:

Not even by hand-editing the pyproject.toml? That’s a PITA. Score one for PDM, then.

I am releasing Hatch this weekend and this feature will be in the release after that.

3 Likes

If your package is a build backend dependency, you are kindly asked not to introduce build cycles. If your package is used by multiple backends, that basically means that you have to use flit_core, as it is the only established dependency-free backend. This affects maybe a couple dozen packages tops. Is not using flit_core in these packages so important that we must upend the entire bootstrapping chain?

2 Likes

It seems like part of the problem is that authors of build backends
may decide to pick convenient libraries from PyPI in order to avoid
reinventing them from whole cloth, while not informing much less
consulting the maintainers of those libraries about suddenly being
in the critical path. It should be the responsibility of the build
backend authors to choose their dependencies carefully and make sure
the maintainers for the dependencies they have are consenting to
that situation.

5 Likes

Looking at Donald’s list from above, setuptools’ hotchpotch of vendored[1] dependencies aside, the majority of backend dependencies are packaging-specific or packaging-adjacent and their authors are intimately aware of what that implies for their build processes. I haven’t seen anything to suggest that there’s a wider pattern of “unconsenting” dependencies or that backend authors are somehow uncooperative or inconsiderate.


  1. No one’s saying that tomli should be devendored from flit_core, presumably because that would break the world :stuck_out_tongue: ↩︎

To be honest, I have yet to see anything to suggest that there’s an issue outside of cases where people are devendoring dependencies. If this had come up in the course of a supported use case, I suspect the conversation would have been very different in the first place.

I meant that was a problem with your suggestion. How will a library’s maintainers know that a build backend has started relying on it, unless someone tells them? Sure, if they know they’re maintaining something which is being used by a build backend then perhaps they can try to take your advice to not introduce new build cycles, but in my opinion the onus and responsibility is on the build backend maintainers to choose dependencies which won’t introduce build cycles and communicate with the maintainers of their dependencies in order to make sure they’re okay with that additional burden.

2 Likes

Hatch doesn’t… what? [1] :stuck_out_tongue_winking_eye:


  1. context lost, thanks flat threading! ↩︎

1 Like