Extraction of deprecated `wheel.macosx_libfile` into its own package

Packaging
1

Hi everyone, this is a follow up on the discussions in:

I am posting it here since one possible action is to create another pypa repository, and it should be discussed in the broader PyPA context.

Maintainers have expressed their desire to stop maintaining the macox_libfile functionality within pypa/wheel. This library is used to calculate the proper macOS platform tag to be used when creating wheels.

In the discussion, I proposed the following:

I see 3 options going forward:

  1. setuptools absorbs also the macosx_libfile module
  2. A different foundational PyPA project absorbs it.
  3. We create a separated project for it, e.g. pypa/macosx_libfile (I would still keep the API as “protected” for internal use of the PyPA projects only, e.g. via a private module macos_libfile._restricted 1)

Regarding 1, the module seems to be fairly complex and require advanced knowledge about macOS systems. I don’t feel right now confident in transferring the maintenance of such module into the same project as setuptools (which is already high maintenance).
Regarding 2, I think it is unlikely other foundational projects are interested in absorbing this module.

So what makes more sense to me is option 3 to decouple the maintenance effort (…) hopefully we can also have other volunteers involved.

Note: If a new project is created, I am happy to be added as one of the maintainers since setuptools is the main consumer of the library so far (… although I don’t use macOS systems myself and haven’t worked on analysing macOS binaries, but I can try to organise issues/PRs).

In Future of `wheel.macosx_libfile` · Issue #4935 · pypa/setuptools · GitHub, it was pointed out that other projects in the ecosystem may also by interested.

In Future of `wheel.macosx_libfile` · Issue #4935 · pypa/setuptools · GitHub, it was suggested that delocate may be a related project. (However, wearing my setuptools maintainer hat, I would prefer to have it as more lightweight package with no dependencies),.

So my questions are:

  1. Should we proceed and extract the code in a new package?
  2. Is there a better approach?
  3. Do we have the PyPA permission to proceed with that? (I believe that this aligns with the PyPA goals)

/cc @agronholm @jaraco

1 Like
2

To me, adding one more PyPA project is the least desirable option. Setuptools already ships with the macosx_libfile module, but because of its (IMHO misguided) vendoring system, it will not import it if the wheel package can be imported from elsewhere. Thus, if the canonical wheel.macosx_libfile module is missing, the vendored module is useless.

3

If we extract it into a separated package, setuptools.command.bdist_wheel would import from this separated package, then wheel.macosx_libfile could be removed. New versions of wheel could maintain for a while the implementation in https://github.com/pypa/wheel/blob/0.46.0/src/wheel/bdist_wheel.py, and eventually remove it once pypa/wheel feels comfortable in doing so.

Regardless of the discussion if the vendoring system on setuptools is doing the right thing or not with the priority of the imports, the vendored copy of wheel is only that, a vendored copy.

I cannot speak for all setuptools maintainers, but I don’t think it is the intention of the project to keep forever the same froze versions of the vendored dependencies. If the community wants for setuptools to import macosx_libfile from its vendored copy of the wheel, then wheel would still need to keep and maintain macosx_libfile in the pypa/wheel repository.

Extracting the code into a separated project would give pypa/wheel a clear path ahead on how to drop it from the code base.

4

IMO, what’s most important is for someone to take on the responsibility for maintaining this code. Organisational details are far less significant.

If neither setuptools nor wheel want to own the code going forward, then the first step (IMO) is to explicitly declare that - state that the macosx_libfile module is now in “maintenance mode” and is looking for new maintainers. If no new maintainers step forward, then setuptools and wheel will need to decide what they want to do.

Assuming that new maintainers do step forward, I suggest that we start by simply letting them take over maintenance as a standalone project. No need initially for it to be a PyPA project - that can be done later once the project is established. My logic here is that if the project is “owned” by the PyPA, this will discourage people from volunteering to own it themselves, as they will assume (incorrectly) that the PyPA is providing maintainer resources.

2 Likes
5

Thanks Paul. I do agree that we should be moving the module “into maintenance mode”.

Just to clarify, I am volunteering myself, but to the limited extend to what I can help with. Ideally it would be nice to also have other volunteers in a better position to help (e.g. experience with the platform).

My logic here is that if the project is “owned” by the PyPA, this will discourage people from volunteering to own it themselves, as they will assume (incorrectly) that the PyPA is providing maintainer resources.

I disagree slightly with this point. These are my personal feelings: I don’t feel confident enough to “own” the package for my self, but I am happy to be “one of” the people taking care of this (under the limited extent I can help). The inclusion under the PyPA umbrella makes me more confident because of this goals of PyPA:

Provide support for existing projects under the PyPA

In the event that a given project needs additional support, or no longer has active maintainers, the PyPA will ensure that the given project will continue to be supported for users to the extent necessary.

I know that in the end of the day, PyPA will not magically be able to support all the projects and that the work is fundamentally carried out by the individual package maintainers, but psychologically this wording is nice to have. I don’t think that moving into the PyPA organisation discourages other people to volunteer, I think it has the opposite effect[1].

For the sake of brainstorming, this is another possibility on how to react if pypa/wheel drop macosx_libfile:

I don’t have a further assessment on what would be the impact of implementing that change.

  1. E.g., I believe we saw an increase in terms of contributors/maintainers when pipx moved into PyPA. ↩︎

1 Like
6

I understood that, but presumed you wouldn’t want to be sole owner and maintainer of the library, otherwise you’d have been happy to incorporate it into setuptools (or you would have just created a new project and left it at that). So “how do we get enough maintainers (and someone willing to be the owner of the project)” remains the key question.

7

(I pressed “Send” too soon on the last post, and then got distracted - sorry).

To be honest, I feel that’s optimistic at best. The PyPA doesn’t have any particular way of getting extra resource, so that commitment feels to me more like a “good intention” than a practical safety net. But as I say, organisation is secondary here - I’m not against having the project under the PyPA, but let’s start by someone[1] creating the project, marking themselves as the project owner and maintainer, setting it up on github/PyPI/etc. They can then ask for PyPA membership, and I don’t think there’s any chance that request would be rejected.

Until now, I’ve avoided commenting on what wheel and setuptools are doing. We have some functionality that’s currently needed by a PyPA project (setuptools), which has issues because there’s no-one with the necessary knowledge to maintain it. We can discuss how to help address that issue. But this comment is a little different (at least based on what I know of the situation) and I want to call it out, because if you’re asking for community help here, we need to work together. And by framing the issue as “pypa/wheel drop macosx_libfile” you’re presuming a certain view of events, that I don’t agree with.

As far as I understand it, macosx_libfile is part of the bundle of code that constitutes the implementation of the bdist_wheel command. That bundle of code is being passed from the wheel project to setuptools. At the point where that transfer is complete, a weirdness in the way setuptools chooses to “vendor” its dependencies means that the wheel project is prevented from tidying up the no longer needed code. In theory, that’s be fine, and is simply a problem with setuptools’ vendoring process that they need to address. But now we have a situation where macosx_libfile becomes a sticking point, because no-one wants to support it, so the two parties are in conflict over how to move forward.

Framing that as “pypa/wheel dropping support” is a bit strong. It is just as valid to describe it as “pypa/wheel trying to remove dead code they thought they had handed over”.

Going back to my original point “let’s get the organisational issues sorted out first”, who owns macosx_libfile now? Clearly wheel thinks it was passed to setuptools as part of bdist_wheel. Does setuptools feel that bdist_wheel still depends on the wheel project via macosx_libfile, and so wheel still owns that code?

We could decouple setuptools and wheel by setuptools unconditionally using the code in their vendored copy of wheel (either by changing their vendoring mechanism, or by moving the code somewhere else). That doesn’t address the matter of long-term maintenance, but it does simplify the problem to just being “there’s a bit of code that no-one has the expertise to maintain, what do we do about it?” rather than being a weird conflict between wheel and setuptools, which is holding both of them back from progressing the move of the bdist_wheel command.

  1. Maybe that will be you - I don’t want to presume, though. ↩︎

8

Hi Paul, thank you very much for the detailed analysis.

I apologise, I chose the wrong set of words. I want to emphasize that (since the beginning) I completely support if pypa/wheel wants to remove any particular piece of code from their base, and to be honest, it does not matter which reason the project have to do so (it could be because it is a piece of code that is no longer needed… because they want to tied up… it could be for any other reason). It is completely legit, and I am not trying to assign blame or anything like that. What I want to achieve is finding a way to support the transition, and enable the maintainers of pypa/wheel on achieving their long term objectives for the project.

In this sense my previous (probably not-fully-well-thought) use of the words how to react if pypa/wheel drop macosx_libfile was intended to be merely factual: I know that there is a wish to remove it from the code base (and again I think it is completely legit), so what are the options that we have in setuptools.

9

First a nitpick in the naming. If we’re moving this module, let’s rename it from MacOS X to macOS (or its lowercase equivalent). I’d be tempted to call the package and distribution platforms.macOS.libfile, although I recognize not a lot of the ecosystem utilizes namespaces and platforms is already semi-taken.

I’m happy to see this module incorporated into Setuptools, especially if it’s used privately. If it needs to be exposed for public consumption, it should be in a separate package. Setuptools would like to move away from being used as anything but a build backend.

As for ownership/maintenance, I welcome Anderson to own it, within PyPA or not. Personally, I’d rather see it within some organization than hosted by a single individual, and given that it’s already hosted by pypa, it seems a reasonable step to keep it hosted here, reducing the bus factor. I’d also be happy to host it under coherent-oss or see it added to jazzband if PyPA doesn’t want it.

My preference is for it to remain under PyPA whether in Setuptools or as a separate package.

10

Seconded. If no project wants to expose it in their public API, then Setuptools adopting it is the preferred next choice.

1 Like