PEP 517 workflow for distributions

Packaging
1

As Brett asked on https://github.com/pypa/packaging/issues/363, I am opening this thread to document and discuss what is missing to make PEP 517 usable for Linux (and other) distributions.

The problem

Pre PEP 517 we had two workflows I am aware of:

  • Install via setuptools (eggs)
  • Build a wheel via setuptools and install it with a wheel installer

In a PEP 517 world, we have just one:

  • Build a wheel via PEP 517 or by invoking the backend directly, if it supports that, and install it with a wheel installer.

Currently, the only usable wheel installer is pip.

The problem is that some distros can’t use pip, AFAIK there are two problems they may have:

  • pip is very hard to convince to behave like a simple and reproducible wheel installer. Distro packagers need to be able to tell pip to install to a custom location, ignore dependencies and to not try to connect to the internet. pip was simply not designed for this.
    (the biggest one)
  • pip vendors its dependencies. Some distros have rules against vendoring of dependencies, so they need to devendor pip. Usually this would be annoying, but should be fine, but it is very problematic in this case as pip would be a required tool for the workflow. This forces distros to manually bootstrap pip and all its dependencies, which I think is around 30 packages. This is simply not a feasible thing to do, especially considering it needs to be done every time Python updates.

This last issue has already been discussed at length, I don’t want to rehash the discussion as I believe that would not be productive. I think PyPA maintainers should just take it as pip not being a suitable wheel installer for bootstrapping a Python environment.
And as so, we also don’t need to have more discussions about the first issue as the second one is a blocker, it is simply stated here to document it. It is not a very big blocker but it is an issue.

So, distributions that can use a vendored pip do not have an issue, they can just bootstrap pip and use it to build the rest of the packages. But distros that cannot, currently are forced to use the old approach and install directly via setuptools (eggs) as that is the only other alternative to get a package installed.

It is worth noting that building via PEP 517 also used to be one of the blocker issues. We need a way to build the wheels. pypa/build now solves that.

The solution

Currently, distros that have the issue described above are forced to install via setuptools directly (eggs). We want them to move to PEP 517, which only provides installation via wheels.

For this, we need two components:

  • PEP 517 builder
  • Wheel installer

The first one has recently been solved by pypa/build.
The last one still needs to be resolved. There is an initial implementation in pradyunsg/installer but progress has been slow and is still a WIP.

So, the current blocker of distributions adopting a PEP 517 workflow is a suitable wheel installer not existing.

The provisional solution

Since there is no path for the distributions described here to adopt PEP 517, they are forced to use the previous workflow, installing via setuptools (eggs).

Some distros have a heavy preference from building for the original source, rather than the sdist. AFAIK this is not required, but important to note. As a last resort, I think distro packagers could compromise and install via sdist if it provided a setup.py.

The best option, naturally, would be to keep using setuptools until a wheel installer exists and distro can finally embrace PEP 517.

setuptools supports PEP 517 so I don’t think arguments like “we want to update the tooling to PEP 517” make sense.

I do understand some core packaging tools maintainers might want to move away from setuptools to other backends but I ask if you think the benefit you get there is worth breaking the workflow for this many people?
Distributions will still have to use the old workflow. So for you saving a few lines in your metadata definition or not having to keep some metadata fields in sync, distro packagers will have to craft a setup.py to use to build your package and will have to respond to possible bugs coming from there.

So, I would like to ask, please, keep using setuptools until there is a wheel installer suitable for distros to use. If you really need to move away from it, at least make sure your backend generates a setup.py to put in the sdist.

Improving the workflow

Okay, moving past the workflow problem. The current state of the packaging utilities is still hostile against distribution packagers and everyone bootstraping a Python environment from scratch. PyPA solved this issue by vendoring pip’s dependencies but this is not something everyone can do. I believe the PyPA, given its position, should make sure this workflow is as easy as possible, even if it is not directly affected.

There is no escaping from some manual package bootstraping, but this issue can be heavily improved by reducing the amount of packages required to bootstrap to the reasonable minimum. One way this can be easily improved is by making sure all packages use the same backend. When bootstraping the packages we would have to also bootstrap their backend, in order to build them, if all packages use the same backend we reduce the number of packages required, which even just as one, is a huge improvement.

The target packages would be:

  • toml
  • pep517
  • packaging
  • build
  • pradyunsg/installer + its dependencies

Currently, two backends are used, setuptools and flit. I would really like if we could agree on one backend to switch to once the wheel installer is in place. The only foreign package is toml, but I think the maintainer could be persuaded by presenting this reasoning. We still have to wait on pradyunsg/installer to see what else it introduces.

TLDR: Currently, the only thing blocking distributions from adopting PEP 517 is there not being a wheel installer (pip is not a solution here).

4 Likes
2

IMHO if distributions are hurting by this they can contribute to the creation of the wheel installer :thinking:we can’t really deprecate PEP-517 to accommodate them. Furthermore one thing I did not see mention here is how will distribution provision build dependencies? Aren’t them also part of the need to be bootstrapped packages. It’s a bit unclear to me what you’re advocating for here? To drop pep-518 and pep-517 until we have a wheel installer?

1 Like
3

Note that:

  1. In principle, installing a wheel is (by design) simple, mostly being just unpack and move into place. The devil is in the details, obviously, but I would have thought that distros building a custom wheel installer for their specific needs is certainly an option.
  2. I’m fairly sure that help in moving pradyunsg/installer forward would be welcomed.

It’s good to know that there’s fundamentally only the one blocker, but I’m not quite sure what else this issue is intended to convey.

Personally, I’m very much of the view that a dedicated wheel installer would be a useful additional tool. I fully intend to contribute to the existing project once it drops Python 2 support (as I’m personally no longer interested in working on Python 2) but I’m a volunteer, with limited free time, like pretty much everyone involved in packaging, so it’s likely not a good idea for distributions to depend on that happening.

So is the blocker (for distributions) that they can’t find resources to work on a wheel installer? OK, fair enough. If this is a call for volunteers, then I guess I’ve explained my position above.

Apologies if I’m misreading the message here - as a Windows user, I’m not completely clear how to think of “distributions” - some seem to be large corporations, and yet others seem to be groups of volunteers. My comments above are mostly directed towards the “large corporation” end of the scale.

2 Likes
4

I have already offered to help. I don’t think it makes sense to be developing two wheel installers in parallel. Currently, we are waiting on Pradyun to finish and commit his initial implementation. I am not sure what else distros can do.

I am not proposing stopping PEP 517/PEP 518 adoption. I am just asking to do so in a way that is still compatible with the older way, which means sticking to setuptools. pypa/build would be an example.

I mentioned: manually

This is why it’s important to keep the required package count low.

It is to document the current state of affairs, and to ask core package maintainers to please keep using setuptools, in order to not break workflows for distributions. In case you need to move away from setuptools, please make sure the backend you choose will include a setup.py in the sdist. I tried to make this clear, perhaps it wasn’t.

The only corporation involved in the community seems to be Fedora, which doesn’t have the pip problem, hence is not affected by this. The only other distribution I see involved, mainly via me, is Arch Linux, which is run by volunteers. You also have Nix and Gentoo, IIRC, which have been active in the discussions, but not contributing AFAIK, and both are also run by volunteers. I think you can effectively think of distributions regarding this issue as groups of volunteers.

But as I explained above to Bernát, pradyunsg/installer is currently blocked by the author, there is nothing I can do right now. I have written FFY00/python-install but it’s incomplete. I have limited time, I can either take that valuable time and work on this wheel installer by myself and then get it superseded by pradyunsg/installer later, or wait for Pradyun.

2 Likes
5

FWIW, it’s been a case of not-enough-hours-in-the-day for me to work on pradyunsg/installer. I’ve been stretched a little too thin lately, so installer has fallen by the way side.

If someone is willing to spend the time, I’m open to co-working on this over coming weekends – https://calendly.com/pradyunsg/weekend-hour. This is especially open to Linux distro maintainers, since, well, they’re the most invested in this.

4 Likes
6

No worries, I really appreciate what you are doing. This thread is about the current state of affairs and not to put pressure into this moving forward, as the resolution requires volunteer time.

I can sign up, I would just ask that you write down the topics we need to address so that we can more easily organize, keep track of progress and what is missing. Doesn’t need to be the full list, just the current working pool and maybe things for the near future.

1 Like
7

I think it does. Spec implementation is hard, and it is very easy to get tunnel-vision and miss details when everyone is looking at the same thing. Parallel development with good communication (“comparing notes”) is a good way to make sure edge cases are all covered, in my experience.

Also, although it is (relatively) clear what specifications are in scope of a wheel installer, the interface to access them is still in no way obvious (well, everyone agrees they know what the best interface is :stuck_out_tongue:). Friendly competition is a good way to drive UX innovation.

5 Likes
8

Strong +1 on this. The main reason we created PEP 517 was to allow multiple implementations. We now have a number of build backends. Having multiple frontends is also hugely important to the health of the community - as @FFY00 has already noted by pointing out that pip isn’t the right fit for distributions. Having multiple wheel installers, far from being “duplication of effort”, is actually the whole intention behind the packaging standards approach, and I’m disappointed if people have missed that message and are still hoping for “one true tool”.

-1 on this. PEP 517 has been around for 5 years now, and I don’t think we should need to treat it as a “second class citizen” by this point. I’m not saying that we should actively push people away from setuptools, but I see no reason to stop any project from using an alternative backend if it suits their needs. If Linux distributions still aren’t in a position to handle that, then I have sympathy, but I’m not inclined to hold the ecosystem back until the distros address that problem. Distros can ask individual packages, if they so wish, but I don’t think the “Python packaging community” or the PyPA should add their weight to such a message.

1 Like
9

+1 from me as well!

The one personal note about this: we should make it fairly clear what the relationship between the various projects is, if we go down this road, because I’d like to not deal with folks-who-do-not-wait-to-criticise coming in and saying demotivating things on either effort. :slight_smile:

1 Like
10

13 posts were split to a new topic: Clarifying the differences between PEP 517 and setuptools

14

Opens the space for other build backends than setuptools. The new build backends (such as poetry/flit) can offer a simplified improved UX (e.g. simple TOML configuration file rather than executed python code). New build backends are not obliged to offer feature parity with setuptools (importantly such as building c-extensions). Therefore they can simplify the packaging experience for the majority of users working on just pure python codebase. You can read about this more anecdotally within my blog post at https://www.bernat.tech/pep-517-518/

I don’t think it’s stalled. GitHub - pfmoore/editables I think is a direction we agreed would be great to go down. However it’s waiting for people willing to integrate that with setuptools/flit as POC, and then we can move to standardise it. I’m currently already working on the build tools and the tox rewrite, so until those roll out I’m sadly not available to work on this.

1 Like
22

I was gonna write a slightly angry reply to this but I will refrain from doing so. I tried summarizing my points. Please note that I am extremely disappointed with this response, I have been trying my best to fix the issue and this kind of inflexibility to compromise in any way is very demotivating.

  • PEP 517 has been around for 5, yes, however it is still provisional and the period it has been actually usable has been far smaller.
  • I don’t think it’s reasonable to expect distribution packagers to be actively monitoring PEPs that affect the packaging ecosystem or to follow this high-volume discourse. IMO it should be the PyPA’s job to at least make the minimum effort to understand how proposed changes will affect packaging users, not limited to pip. As it stands, PyPA’s packaging interest == pip, which I believe is very wrong. Perhaps a distributors mailing list could be created, but that still requires some movement from PyPA to try to get feedback, since PyPA members don’t seem to be thinking about these use-cases.

This comment in specific is hurtful to me as it is not what I am proposing at all.

How is asking to keep using setuptools, which fully supports PEP 517 (not that was even needed since PEP 517 is backward compatible), on less than a handful of core packaging packages, like pypa/packaging, holding the ecosystem back??

I have even outlined an alternative. If you really need another backend, please choose one that will include a setup.py on the sdist.

Again, please let me know in which way this is holding back the ecosystem?

And please let me know how do you suggest distributions should fix the problem? Because to me the only clear solution is to stop updating packages, which would just end up with more bugs being opened against pip and all other packages. That is something I really don’t want to happen, but maybe I just shouldn’t care as that is the same courtesy I have been given.

24

In which case, I apologise. But I’ve clearly missed what you are proposing - all I can say is that I did point out earlier that I was unclear what you were suggesting.

I thought you were suggesting that we encourage packages in general to stick to setuptools. Re-reading your original post, I see I completely missed the point of that section of what you were saying. I skipped down to the tldr section which didn’t mention it. My bad.

But the PyPA governance PEP explicitly states that setting policies on individual policies is a specific non-goal. So I think you need to ask the individual projects. Speaking for pip, we don’t have any plans to move off setuptools, for what it’s worth. And if any other pip developer proposes we move to flit, I’ll link to this discussion so the information is available.

I thought flit included a setup.py. It looks like maybe newer versions have stopped doing that. So basically you’re asking for a set of 5 packages to avoid using flit? As I say, that’s a much more limited scope than I realised.

That was my misunderstanding, so accept my apologies once more.

I don’t honestly know. Maybe discuss the use of flit with the projects that are using it? I can’t say how they would react, but that would be where I’d suggest starting. If they don’t want to change, I think you have to respect that - the PyPA is not the right group to go to for support in pressing for change.

Apart from that, working on (or helping with) a wheel installer sounds like a way forward, or developing ways of automating a focused manual unzip process for wheels for the projects you’re concerned about.

Honestly, I don’t know - I sympathise with your problem but don’t have any solutions for you.

I’m sorry if my misunderstanding, and my response based on that misunderstanding, upset and annoyed you. That wasn’t my intention.

(I’ll note that as the maintainer of build, you’re as much part of the PyPA as me, so I hope that you understand here that I’m not commenting from any position of authority, but simply as a colleague looking at the concerns you’re bringing to the table).

I feel that the PyPA does try to understand the user’s perspective. Personally, I’m focused mainly on people wanting to create packages, and people wanting to install packages. I don’t have much insight into how distributions differ from other people wanting to install packages, so to that extent your perspective as a PyPA member is very valuable. But I’d also note that there are many more package builders and installers than there are distribution maintainers, so I’d personally be cautious about giving undue weight to distro maintainers in the (rare) cases where there’s a conflict.

I definitely don’t see the PyPA’s focus as being on pip. Far from it - even as a pip maintainer, I’m extremely interested in making sure that the PyPA pushes to set up standards that allow people develop their own tools, and not be locked into pip as the “one privileged tool”. (I could take offense at the fact that you think otherwise, because for me, interoperability is all about breaking down barriers to competition - but I won’t, because I’m well aware that you’ve no reason to know about the relevant discussions I’ve been involved in).

Hopefully, I’m explaining my position better this time. Regardless, though, I’ve said enough, and I’ll leave it here - I don’t want to unduly influence the discussion, so I’ll leave the floor open for others to give their views.

25

I’ll note that toml is not a PyPA project, and we can’t be enforcing requirements on that project.

The rest seems OK, as long as the other maintainers are on board. I’ll be a little annoyed that I can’t use flit on at least 3 of my projects, but hey, more motivation to finish the installer. :slight_smile:

26

It’s specifically around the burden of using setuptools compared to flit. The maintenance cost is unfortunately non-zero in maintaining setuptools support in such a simple project. So it’s not just the more pyproject.toml part, it’s a workflow thing as well.

So it sounds like finishing installer unblocks people. OK, so if someone commits to help out @pradyunsg to aim to get installer done and in PyPA by the end of 2021, I will support reverting the flit usage in packaging to give folks the time they need to move things forward.

Does that seem fair to everyone: to unblock folks today to make sure no one is blocked in 2022?

1 Like
27

This. I would like to have a test that shows that it is still possible to install entirely from source (that is, from source in a vcs, not pre-made sdist).

In Nixpkgs we currently use pip but I’ve been wanting to switch to build. Due to its dependencies I haven’t made the change yet, but I should at some point.

When that’s done I am willing to help setting up a CI with Nix that checks whether bootstrapping from source remains possible.

Just now I gave it a try to see whether I could bootstrap build. Using our bootstrapped-pip (essentially unpacked pip, setuptools and wheel archives) it works fine, but there is less point in using build until installer is available. Right now I made the assumption every package has its package name as folder name so they can easily be added to PYTHONPATH. I know some projects like calling their project src. It would be convenient if this would not happen with the packages required for bootstrapping.

28

I’m ok with asking to keep to setuptools as build backend. This here though makes for these projects hard to test the site-packages, not the local source tree. As such I don’t think this request is reasonable. Project don’t do this because they like, but because this solves an active problem during development and testing.

5 Likes
29

None of the projects is using src currently, so apparently it doesn’t seem to be that big of a deal just yet. Or did I give the maintainers an idea? :slight_smile:

FIne, not being able to use the package name as folder directory would mean explicitly listing the packages you need along with their corresponding module names in order to make a PYTHONPATH. Unfortunate, but really not that big of a deal. I suppose most distro’s do that already anyway.

The PyPA is a community and the distro maintainers can be a part of it, or at least that is what I imagine it to be. In that sense, are you not also a part of it?

This I unfortunately agree with. Following python-dev, Packaging/ here and interacting in some of the discussions as well as on the issue trackers of some of the projects, it is also my experience that the use cases and issues of distro’s are just not considered well.

Distro’s have issues like bootstrapping, they deal with large amounts of packages so they want or need to have a structured approach for fetching package information, and building and testing these packages. We need to resolve dependencies of a large set of packages when updating our package set, and be able to override requirements when solving, because there’s always packages out there that pin too strictly.

30

I’ll request everyone engaging in this discussion to avoid making broad strokes arguments/statements about state of Linux distros + Python packaging and who’s responsible for what — please go make dedicated threads for those discussions. Saying those things here is inevitably going to derail constructive discussion here, and I really don’t want that happening.

I’d like to not have this thread become about people complaining/debating about how status quo isn’t ideal (look around, nothing about the world is ideal at the moment). Let’s keep this discussion focused on the very specific suggestion in OP to resolve one specific issue.

Wanna have broader picture discussions / talk about responsibilities / anything that not 100% about the OP suggestion? Please make a new topic.

1 Like
31

My suggestion then for all the Linux distro folks is to organize and create a mailing list where you can coordinate what needs and issues you have to surface them to the appropriate group as a collective rather than as individual “Nix”, “Arch”, “Debian/Ubuntu”, or “Fedora/RH” asks. I’m sure the ML could be hosted on mail.python.org if you asked.

1 Like