A reference API for creating wheels from code

The wheel project is documented here as being the reference implementation for wheels, but not as offering a programmatic API:

This library is the reference implementation of the Python wheel packaging standard, as defined in PEP 427.

It has two different roles:

  1. A setuptools extension for building wheels that provides the bdist_wheel setuptools command
  2. A command line tool for working with wheel files

As a general principle, I tend to believe that we should have library implementations of all of the packaging standards that we have defined, so that user code can be built on top of them without re-implementing the details of the specs. In this case, the obvious consumers of such a library would be build backends¹, but I have also found a need to create wheels in adhoc code and test harnesses.

The wheel project has historically preferred not to support an API (@agronholm @dholth is that a fair statement?) so it’s not clear to me where users should get that library. I don’t want to risk community upset by unilaterally creating a competing wheel project, in particular as the “obvious” way to kick-start such a library would be to copy code from the wheel project, but I’d like to open a discussion on

  1. Whether I’m right that we should have a reference library implementation of the wheel spec.
  2. How we should do that in a way that’s minimally disruptive to the packaging community.

What are people’s thoughts? In particular, what are the wheel project maintainers’ feelings on the matter? (I originally considered raising this as a ticket on the wheel project tracker, but it was difficult to frame the question in a way where the “obvious” answer wasn’t simply “the project doesn’t offer a programmatic API”).

For background, this question was prompted as a result of thinking about how I can test my project for generating editable wheels in preparation for a PyPI release - the code produces the files that need to be bundled into a wheel by the backend, but I do not want to include wheel-building code in the editables library itself.

¹ Other than setuptools, who are well-supported by the bdist_wheel extension included in wheel.

1 Like

Have you seen this issue? https://github.com/pypa/wheel/issues/262

This is mostly just waiting for tool authors to weigh in and for requirements to be compiled from that input.

When I looked at the wheels project last time it looked more like setuptools-wheels to me. IMHO an actual wheel project should not need to know about setuptools at all. Therefore my recommendation is to make a new wheels project, which could be taking parts of the current wheels project, but not all.

The only part of wheel that needs to know about setuptools is the bdist_wheel module which is a setuptools plugin. I see no value in creating yet another project. If the fact that wheel is mostly used as a setuptools companion bothers you, I’m perfectly happy to move the bdist_wheel command to the setuptools project itself (once a public API has been decided on). This has in fact been discussed before, both online (IIRC) and offline.

IMO it’d be a good idea to have a reference implementation of building a wheel that projects can reuse. We already have a high-level API to build wheels in PEP 517 hooks, and what we are missing not would be a series of utilities functions to help people implement the hooks without stepping on landmines like encodings, INI format compatibility, etc.

So what’s needed from pypa/wheel (or any other library that is degisnated as the reference implementation) is not necessarily a WheelFile class, but to expose those utility functions in documented, stable places, so people can import and use them easily.

No I hadn’t. Apologies for missing it.

So how can I help to move this forward? Which tool authors are you looking for input from? I can’t see setuptools needing anything, because of bdist_wheel. As a casual user, I’d be fine with the proposed approach of expsong WheelFile - and I don’t really see much reason not to start with that.

I guess if the proposal is that WheelFile gets exposed I can use that API right now (I’m writing adhoc scripts, so API stability isn’t crucial to me) and provide feedback based on how that works out. That addresses my immediate issue, and maybe helps to move the issue forward?

I started writing a new wheelfile at https://github.com/dholth/wgc/blob/master/wheelfile.py . Example of rewriting a wheel into another wheel with the existing WheelFile: https://github.com/dholth/wgc/blob/master/wgc2.py#L144

I’ve written really short ones https://github.com/dholth/sdl2_lib/blob/master/wscript

WheelFile is a subclass of ZipFile and takes care of the archiving bits only. Its job is to check or generate RECORD for you and to know the location of important files inside .dist-info/ from the wheel filename.

You need the archiver bit in an API obviously, and then you have to decide how much you need for normalizing names, dealing with versions, parsing and writing the metadata.

For editable wheels you probably have the metadata and just need to archive it together with your proxy modules. So you need ZipFile and to generate-RECORD

The setuptools plugin is only part of the wheel project. WheelFile is most likely to be useful as an API. The metadata parsers may have baggage from having to doc Unicode in both Python 3.2 and 2.6. It also includes a wheel [un]pack command line tool.

@uranusjr a setuptools editable wheel builder needs the API too, for the same reason that bdist_wheel uses WheelFile. bdist_wheel is concerned with adapting the setuptools install command into creating a wheel. If you’re not doing that (making a special editable wheel perhaps?) then bdist_wheel itself is useless. The other bits of the wheel project have more potential for re-use.

Therein lies a problem for me, as I’m not really “in the know” when it comes to setuptools alternatives. Flit comes to mind, what others are there?

You should probably involve both front-end and backend maintainers of the build ecosystem, this involves:

  • setuptools, scons, flit and poetry
  • respectively pip, tox, nox.

Which of these tools actually build wheels directly?

Probably all of them need to interact with wheels, and may need to build wheels directly. flit doesn’t at the moment, but makes sense to switch to do it itself. Otherwise fronteds may not need to build wheels directly but need to consume them. So if you want a wheel API beyond just build you want them involved.

What would be the best way to get them involved?

IMHO create a RFC and invite the maintainers of those project to participate? Give them 2 weeks to weigh in. Go ahead with the agreement from there. Having a rough implementation and clear design goals explained will likely speed up the operation, together with some example uses.

As an alternative, why not simply expose the WheelFile interface as it stands (or maybe only a limited subset of it) and wait for anyone with additional needs to file feature requests? At least to a first approximation you know that WheelFile is a viable API, because wheel itself (and bdist_wheel in particular) uses it.

Over the next couple of days, I’ll look at my use case and see how well WheelFile would handle it (and in particular, which bits of the interface I need). That’s the use case of “adhoc scripts wanting to write a wheel” rather than build backends, but honestly, I suspect that initially at least these are likely to be your key users (existing backends have probably written their own wheel handling code by now, and may not want to change in the short term).

I’ve now posted an RFC as suggested.

1 Like

Distlib builds wheels, too - @pf_moore did you look at that API and find it wanting? It doesn’t rely on setuptools or any other external libraries.

I didn’t to be honest. I’d forgotten it included wheel building. From a quick look, it’s completely file-based so it’s not ideal for my use case, though.

I tend to find the mix of standard and experimental features in distlib off-putting (mounting wheels via zipimport, for example, is not supported by the standard), and so I generally avoid using it much these days. But thanks for the reminder - it does have a lot of useful functionality.

1 Like

I’m personally fan of smaller well-scoped projects, distlib feels to me like tries to do too much. Maybe we should break it up. It also feels the project struggles with maintainance, no issues have been addressed or new commits since January. @vsajip can we pull in more people to maintain it (and dare I say move it to Github :man_shrugging: ).

distlib was originally put together after packaging was pulled late in the Python 3.3 release cycle. Some bits of it were originally part of that effort. All of its functionality is related to package distribution concerns. One doesn’t have to use every part of a library if one doesn’t need to.The current sub-modules feel to me like they belong together - it allows a coherent approach to related concerns.

I’ve just realised that there’s a bunch of issues that have been raised since February. At around that time, another contributor accidentally deleted the repository, and it had to be restored from a backup. Although no source information was lost, it appears that the restored repository hasn’t sent me email notifications when issues were raised, so I had no idea they were there. I’ll take a look at them now …

There was an effort then to put more packaging within the standard library. Recently though I’ve seen more and more of an effort to take standard library things out and maintain them separately on PyPi. The fact that distlib/packaging did not make it into Python ever since leads to the this conclusion too.

No worries, but makes me believe with more maintainers on a project someone else would have noticed this earlier. There isn’t a better channel currently available to contact maintainers than the issue tracker, so perhaps we need better tools to follow maintained libraries, than e-mail notifications that may end up not being sent or missed due to e-mail filters. I maintain virtualenv and raised a blocking issue in February, but then went with an ugly monkey patch to workaround the issue. If PyPa maintainers can’t access the project when you’re not around (for whatever reason) IMHO we’re doing something wrong.