Draft PEP: The "local" wheel tag

Packaging
1

The discussion from Draft pep: Disabling Manylinux concluded that creating a PEP for a “local” tag would be the logical next step. I think I’ve produced something workable, but I think there’s definitely bits which need the input of those who actually write and are familiar with the current packaging ecosystem. I think I’ve also linked to all the relevant discussions that have happened in this space, so that if someone wants to extend this, or come up with further ideas for tagging wheels, this can act as a starting point. The PEP is below (included inline now, unlike the no-manylinux PEP).

PEP: 9999
Title: The “local” wheel tag
Version: Revision
Last-Modified: Date
Author: James Tocknell aragilar@gmail.com
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 2020-01-29
Python-Version: [M.N]
Resolution:

Abstract and Motivation

Manylinux has been successful in allowing users to install non-pure-python
python packages, without the need of a compiler (or compliers) or the need to
install other shared libraries. This has increased usability of the python
packaging ecosystem, and reduced time waiting on CI builds. However, manylinux
has increased the issues some users face, therefore these users wish to opt out
of using manylinux wheels. They may be encountering issues with projects
producing invalid manylinux wheels [#manylinux-tensorflow], wanting to
experiment with different BLAS/LAPACK implementations, or a variety of other
reasons [#pip-no-manylinux]. Additionally, locally build wheels on Linux use the
linux tag, which is more general than the manylinux tag [#linux-manylinx].
Creating a new “local” tag would flag whether a particular wheel was locally
built (and therefore used as a caching artefact), or being used as a
distribution artefact. Whist the hasn’t been the same level of discussion of
tagging on other OSes (likely due to the existence of a single platform tag),
being able to distinguish between a locally built wheel and wheel used on PyPI
to distribute code is a useful debugging aid. The motivation and early
discussion for this PEP came from GitHub issues on PyPA GitHub organisation
projects [#issue-motivation].

Issues not in Scope

This PEP treats how “local” wheels are built as out of scope, as well as how to
build wheels with a specific configuration. Whilst the likely audience of this
PEP would have opinions on how to “configure” wheels, we deem any more advanced
tagging system to require a separate discussion [#wheel-configure].

Specification and Rationale

A new platform tag [#pep-425] is to be added local_<platform>, where
<platform> is a platform tag as computed by PEP-425. Examples of this new
“local” tag are:

  • local_win32
  • local_linux_i386
  • local_linux_x86_64

Wheels tagged with such a platform tag must be treated as having a higher
priority than any other wheel with a different platform tag (e.g. win32,
manylinux1).

Interaction with other wheels with no appropriate sdist

This case occurs when either there is no sdist available across all versions of
the package, or where the package installer has been instructed to ignore all
sdists.

There are three possible cases:

  1. the highest version of the package has a non-local wheel, and there is a
    version of the package with a local wheel
  2. the highest version of the package has a local wheel, and there is a
    version of the package with a non-local wheel
  3. the highest version of the package has a local and non-local wheel

In cases 2 and 3, the higher priority of the local wheel means that the local
wheel should be installed.

In case 1, we come to an conflict, which can resolved with three options:

  1. the non-local wheel is installed
  2. the local wheel is installed
  3. an error is thrown

Option 1 is the simplest in terms of implementation: find the highest valid
version, and install the wheel of that version with the highest priority.
However, those who wish to use local wheels would likely object to the use of
the non-local wheel, which would rule out this option.

Option 2 and 3 require scanning the whole version history, looking for a
possibly non-existent local wheel. The cost of this scan is unclear
[#local-lookup], but is
presumably greater than option 1. Option 2 then means the user installs an older
wheel (possibly what the user was after, but possibly not), whereas option 3
means that it is not possible for a user to install an older local wheel until
the newer wheel is removed from the search path. The choice between option 2 and
3 comes down to a philosophical difference about what should be in the case of
ambiguity. Leaving implementers to choose between options 2 and 3, and allowing
them to develop appropriate UIs around the options (possible with warnings or
configuration options), seems the best course.

Interactions with other wheels with sdists

This case is the more complicated version of that above, in that we now have
three possible outcomes: install the sdist, install the non-local wheel, and
install the local wheel.

The possible cases are (where ordering refers to the highest version of the
distribution artefact):

  1. sdist <= non-local wheel <= local wheel
  2. non-local wheel <= sdist <= local wheel
  3. sdist <= local wheel <= non-local wheel
  4. non-local wheel <= local wheel <= sdist
  5. local wheel <= sdist <= non-local wheel
  6. local wheel <= non-local wheel <= sdist

Cases 1 and 2 are equivalent to cases 2 and 3 above: the local wheel is the
obvious choice.

Case 3 is case 1 above (we can ignore the presence of the sdist).

Case 4 is the first new case, however as we are choosing between the local wheel
and a sdist, the usual rules of sdist wheel ordering apply (so there is no
addition logic required here).

Case 5 is a variant of case 1 above, in that we should [#building-sdists] be
able to go from an sdist to a local wheel, which reduces to comparing a local
and non-local wheel. One additional bit of logic that some installers may want
to provide is the fallback to the local wheel in the case that building the
sdist fails to build.

Case 6 can be any of the cases above, as case 1 may result if the sdist fails to
build. Using the logic handling of the previous cases (cases 2 and 3 install the
local wheel, case 1 provide an appropriate UI), as well as case 5 in the case
that the sdist building fails, we can cover all possible subcases.

What happens in the absence of a local wheel?

If we have no local wheel, and a non-local wheel, we have two options:

  1. install the non-local wheel
  2. don’t install the non-local wheel

Whilst case 2 seems incorrect at first glance, there are enough cases where
people have asked to disable manylinux, that to dismiss the second option out
of hand seems to ignore community unhappiness with the current status
[#no-local]. Therefore, it is worth thinking about when option 2 is the correct
option.

The first situation that comes to mind is where there is a newer sdist than the
non-local wheel. Currently pip will build the sdist unless the
--prefer-binary option is used. Note in the case of manylinux, a cached local
wheel will override the use of a manylinux wheel uploaded later, which would not
have been the case for a previously created linux wheel (other OSes would have
always used the cached, locally build wheel, even when a different wheel became
available).

In the situation where there is a non-local wheel where the sdist is not newer,
or when an option like --prefer-binary is used, currently the non-local wheel
will be installed. Taking that users of the local PEP are likely those who would
wish to prevent the install of the non-local wheel, at least in some cases, the
question then becomes what use cases do we want to support. Manylinux has
mechanisms to control whether a specific tag should be used on a specific system
[#manylinux-peps], but there is currently no equivalent for any other tags
(which, given the absence of a local tag, would not have been distinct from
avoiding wheels). Given the option to choose between local and non-local wheels
is now possible, installers having an option of only choosing local wheels (e.g.
via a --local-wheels-only option or similar), seems more reasonable than
having manylinux-only options.

What is a “non-local” wheel?

Whilst we have defined what a local wheel is, we haven’t defined what exactly we
mean by a non-local wheel. The simplest option is any wheel whose platform is
not a local platform is a non-local wheel. However, the implication of this is
that wheels with platform any, that is pure python wheels, are non-local
wheels. Therefore, users of an option like --local-wheels-only would be
building any platform wheels repeatedly (and, unless either the pure python
wheels produced were tagged with local tag, or a sufficiently smart installer
knew which any wheels were the ones it produced, the any wheel would be
rebuilt every time). Therefore, the sensible definition would be that a
non-local wheel is any wheel that has a system tag which is not any or a local
tag i.e. there are three types of wheels, pure python (any), local and
non-local.

Backwards Compatibility

As only installers with support for local tags will treat the local wheels with
higher priority than other wheels, older installers will likely be a source of
unhappy users (though it is doubtful that anything can be done about that).
However, this PEP will not make things any worse for those users.

The change in priority of locally-build wheels on linux vs manylinux wheels
could be a source of additional bug reports, but is unlikely to be greater than
other issues related to caching.

Security Implications

The security implications of the PEP are those related to running an older
version of software—it’s possible if sdists for a particular project stopped
being published, or if the newer sdists failed to build (and the UI around
handling this was insufficient), that users could be stuck on an older version.
The former case however would likely have some level of discussion on various
news sites (as has happened recently with the license change of some projects),
whereas the latter comes down to design choices of installers.

Reference Implementation

There is currently no reference implementation. This PEP would likely also
require the coordination of multiple different projects to fully implement, so
the creation of a reference implementation is not currently planned.

Rejected Ideas

No Manylinux

There has been previous discussions of manylinux specific options to control
the install of wheels, which helped spawn the creation of this PEP, but we
reject these as being both too specific to manylinux, and being insufficient for
other Linux systems on non-x86 platforms [#no-manylinux].

Possible future platform tags

There has been the suggestion of more specific platform tags (e.g.
ubuntu_16_04_x86), which may exist in the future. Whilst we could have added
these additional tags, or at least specified how they interacted with any local
vs. non-local selection, we leave that to a future PEP (if one is created), for
any decisions on how such an interaction would work.

Similarly, custom tags (which would likely have a lower priority than local
tags) are left to future PEPs to be decided on.

Singular local tag

Having a single local tag (as opposed to one including OS and architecture)
might slightly simplify implementation of this PEP, but given the existence of
multilib/multiarch systems which run 32 bit and 64 bit code on the same system
(or tools like QEMU which can emulate many architectures), including the system
tag as part of the local tag helps avoid weird cases where non-matching wheels
are installed.

References

… [#building-sdists] This assumes that we can build a wheel from the sdist,
which is not always true (e.g. no compiler installed). If we can’t build
from a sdist, then we can ignore the sdist, and only focus on the local and
non-local wheels.
… [#issue-motivation] The “local” tags origins appear to be in
https://github.com/pypa/pip/issues/6523 and
https://github.com/pypa/packaging/issues/239, however the are a number of
other issues which precipitated these discussions, which are linked to in
the appropriate places within this PEP.
… [#linux-manylinux] The ordering of linux vs the manylinux tag is one source
of conflict about manylinux, with debates about the choise of ordering
chosen by specific projects occuring at
https://github.com/pypa/packaging/issues/160,
https://github.com/pypa/packaging/pull/223,
https://github.com/pypa/pip/pull/3921.
… [#local-lookup] The costs of doing such a scan would depend on the protocol
used to transmit the available versions, and any additional metadata used to
choose the artefact to download and install.
… [#manylinux-peps] PEPs 513 (https://www.python.org/dev/peps/pep-0513/), 571
(https://www.python.org/dev/peps/pep-0571/), 599
(https://www.python.org/dev/peps/pep-0599/) and 600
(https://www.python.org/dev/peps/pep-0600/) cover all current manylinux
standards.
… [#manylinux-tensorflow] Tensorflow produced wheels that weren’t compliant
with the manylinux specification, see
https://github.com/tensorflow/tensorflow/issues/8802. There is discussion of
validation of manylinux wheels via auditwheel uploaded to PyPI planned at
https://github.com/pypa/warehouse/issues/5420.
… [#no-local] This unhappiness can be seen in issues on the PyPA issue tracker
on GitHub (some of which are refered to within this PEP).
… [#no-manylinux] The discussion contained within
https://github.com/pypa/warehouse/issues/3668 highlights that this is not
just a manylinux vs. linux tag discussion, and that being able to
differentiate between cached/locally built wheels and wheels distributed on
PyPI is a more general problem.
… [#pep-425] https://www.python.org/dev/peps/pep-0425/
… [#packaging-pip-issue] https://github.com/pypa/pip/issues/7648 was the
motivating issue which started the creation of this PEP.
… [#pip-no-manylinux] https://github.com/pypa/pip/issues/3689 is an issue on
the pip issue tracker requesting that pip provide some way of opting out of
using manylinux wheels, with some additional use cases mentioned.
… [#wheel-configure] Suggestions for either including more configuration
metadata within wheel tags or other metadata (see
Optional C Extension handling,
PEP 517 and projects that can't install via wheels,
https://github.com/pypa/packaging-problems/issues/264), or creating linux
distribution level tags (see
https://github.com/pypa/packaging-problems/issues/69), fall under this more
advanced tagging system (as do proposals like
https://github.com/pypa/packaging/issues/161 or
https://github.com/pypa/pip/issues/6468). Additional logic about how
the wheel is built (see
https://github.com/pypa/packaging-problems/issues/66,
https://github.com/pypa/packaging-problems/issues/25,
https://github.com/pypa/packaging-problems/issues/4,
https://github.com/pypa/pip/issues/3938) is also out of scope.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

…
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:

2

I’ve put up the idea before (mostly to @brettcannon while he was building packaging.tags) to have a configuration file that specifies the platform tag for a Python install (as opposed to inferring it from the system). So it would be precisely one custom tag for an install, can be anything you want, and would be the highest priority tag (which is what distutils will use and theoretically what any packaging tools would prefer). They wouldn’t be supported on PyPI, but any private index should allow any platform tag you like. This would allow much more future flexibility (e.g. the Linux distros could tag their own builds and allow installing them via pip and their own index server), and should also solve at least one of the underlying issues here.

But that suggestion is only relevant if we really need a shared standard. If the underlying issue is just “pip prefers PyPI wheels over wheels in its cache,” then there doesn’t need to be a new standard to fix that problem - just prefer the cache if it still meets version requirements (or some flag is passed… --upgrade?). Agreement upon a design/behaviour is important, but when there’s only one tool implementing it there’s no need for a PEP.

At a minimum, clarifying the need for this proposal would be helpful. It’s not entirely clear from what’s written in the draft, and the motivation is always the most important part for helping people get up to speed.

1 Like
3

The issue (if indeed it is an issue) is that pip doesn’t guarantee any preference, and we don’t really want to start doing so. The principle that any compatible distribution file for version X.Y of package A should be equally usable is important (IMO) - it’s behind things like the resolution algorithm, dependency specifications, etc. Once we start saying that A version X.Y from index I is “better” than A version X.Y from index J, then we add a whole new dimension to the dependency resolution process.

4

Right, but the usual way is to take it from the primary index and then ignore other indices, whether they are “better” or not (assuming the first one meets the constraints).

You get to assume that the user is somewhat in control of the primary index, and if they wanted a newer version of a package they could put it there, and equally if you tell them what’s there can’t satisfy their request then they can fix it.

The wheel cache is a bit special, because it can’t necessarily be seeded by a user, so you just have to decide how to handle that. Presumably you want to use the cache even under no-binary if that’s how it was filled.

All of this is still pip design and doesn’t need a PEP. But is it the problem this PEP is about? Or am I not getting the real rationale from the text?

1 Like
5

In terms of implementation, this would need modifications in both pip and packaging at a minimum (not to mention the other installers that exist), so this isn’t a pip-only issue.

In terms of other tagging schemes, there’s a bunch been thrown around (I doubt I’ve linked to all the suggestions, but I’ve tried to), but a) they require more configuration and b) no one has actually suggested a PEP for them (the two interaction sections was me sitting down and working through the possible cases I could see, I haven’t seen that level of detail in any of the other schemes). I do think having distro specific tags would be a good thing (though you’d want to specify how it works with rolling releases, or Ubuntu’s main vs universe split), but with all the schemes you need to specify how they interact with existing wheels.

6

Sure, though even that doesn’t necessitate creating a new standard unless the plan is to distribute those tagged wheels on PyPI (or to force the other projects to do it your way rather than convincing them on the severity of the problem and the merits of the approach).

It seems like you want to override version ordering in some way by using a particular platform tag. But if this way requires a private index, then the way to override version ordering is to omit the versions you don’t want from your index (and if your index management software doesn’t make this easy, file requests with them).

I know there was another thread where your scenario was discussed extensively, but the summary needs to be in the PEP. You can’t expect reviewers to read an entire old thread.

7

The first part of the motivation for this PEP, the Disabling Manylinux PEP, and the issues I’ve linked to on the PyPA issue tracker (whether as complaints from users, or ideas being thrown around) can probably be summarised as there are specific wheels which come from PyPI which for whatever reason are not what users want to use (manylinux wheels are the example which is easiest to point out, but pywheels is another possible example). I guess the first question is do people agree with this statement, or do we want to discuss specific cases (and/or if this isn’t clear, suggestions about how to word/approach this)?

The second part is that if we take the statement about the wheels on PyPI being true, then what do we do? Running things like auditwheel on upload can help reduce the problem, but that only fixes/blocks new wheels, not existing ones (and auditwheel can only be reactive to problems, so can’t fix issues where a wheel passed when uploaded but would now fail). Users can build wheels from source, but at least in the case of manylinux, the currently built wheels are given a lower priority, so users need to go to extra effort to block all manylinux wheels, even it it’s only specific wheels that are the problem. Users can do things with devpi (or similar) to filter projects on PyPI, but unless we want to start adding blacklist support for specific artefacts (in whichever project, whether pip, devpi or other installers), getting the result users want requires an intersection of these solutions. I currently have a setup with devpi+_manylinux.py which works for me now, but the for now bit concerns me, as the reason I started with the PR against packaging was things broke, and I was looking for how best to prevent that in the future. The local wheel tag idea makes it simple to effectively blacklist a specific problematic wheel, by simply building a wheel which has a higher priority than existing wheels, and avoids the need to build blacklists into every project (or doing something else as significant). It also means that users don’t have to choose between using manylinux wheels (and having issues with a small subset of them), and building everything from source, and blocking manylinux wheels.

As an aside to this discussion, I wrote this PEP, and the Disabling Manylinux PEP, because in both cases writing a PEP was viewed as the next step: https://github.com/pypa/packaging/pull/262 for the Disabling Manylinux PEP; https://github.com/pypa/packaging/issues/239 for this one. I’m happy to not write/withdraw this PEP if it’s decided that a PEP isn’t needed, but given PEPs were asked for/desired by project members, it would be good to have clarity about what needs a PEP and what doesn’t in order for patches/PRs to be considered (this isn’t a dig at you Steve, this is me trying to understand how I should be proposing to modify how Python packaging works across projects).

1 Like
8

How much would be solved if manylinux wheels were just prioritised lower than the range of potential other tags (which are not permitted on PyPI)?

It seems like that idea was dismissed at some point, and so adding another specially defined tag may well justify a PEP. But I’m not seeing the need for the special tag yet.

9

I think having linux wheel have a higher priority than manylinux wheels would make things easier—the local wheels do have some other advantages, in that we split caching based wheels (which would only work on systems similar to that the wheel was built on), from wheels which have their non-python dependencies bundled/managed—but I agree that change would solve the majority of the issues I encounter.

I’m not sure when the decision about the ordering was made, the best I can do is give an ordering of discussions that respond to the decision (I think Paul is the only participant in those threads who has responded to this PEP, I guess I should post a link to the PEP there, as others may not be subscribed to discourse). I’ve listed the ones I know about in chronological order (starting from the oldest) below, which imply to me that switching the order of the wheel tags isn’t going to happen:

10

We need to separate the caching question from the custom local tag question, as wheels implicitly made from sdists using the default build options will NOT be changing behavior. (The fact that people assume it will be suitable for use in caching is my biggest concern with the “local” name, and hence I would prefer something like “custom” or “override” that is clearly not appropriate for an implicit build cache)

The new tag is intended for the case where the user wants to mostly use PyPI provided wheels, but also wants to override specific wheels in a generic way, rather than having to spell out the list of overrides in the installation command.

11

As for changing the relative priority of “linux” and “manylinux”: we don’t want to do that, as the “linux” platform tag just means “this was built on a machine that runs some flavour of Linux”, whereas the manylinux specifications define a more complete ABI.

12

There’s also an idea for using local_{unique-machine-id}, which would permit these files to be cachable, and make it clear that these are machine-specific (if the ID is crafted in an obvious manner).

13

I don’t get the impression this needs to be as magic as a machine id - make it customisable in a configuration file somewhere and the people who need this can put in a UUID if they want. Or maybe they’ll embed a numpy version number, or a GPU/CPU option specific to that particular install.

Provided the user-specified platform tag beats manylinux and Linux tags, we’re done, right? No need to work any harder :wink:

14

I’d avoid unique IDs, as then you need to track the ID in some way. I suggested local_{system_name}, as it avoided the need for creating more configuration files/options around (reducing bugs and the level of setup required to start using local wheels). It also resulted in a single additional tag (per platform), rather than many.

A custom platform (or platforms, as Archspec: a library for labeling optimized binaries could be useful in order to specify levels of features used) would be much more complex, as there’s choices about how to configure what platforms can be used, and an ordering of preferred platforms.

Local wheels seem to me to solve the quick override of a broken wheel (and quickly replaced when there’s an issue e.g. an ABI bump in a dependant happens), whereas custom platforms would solve the build a coherent and consistent platform for non-PyPI distribution (and could be used as a testing ground for new tags becoming available on PyPI, such as a musl/alpine tag). I’d suggest local wheels would have a higher priority than custom wheels (as local wheels are more specific than a custom platform wheel), but I’m happy to be convinced otherwise.

15

This gets us closer to seeing the problem to be solved, though I would expect there to be multiple ways to override a broken wheel. What if there was an install option (or environment variable) that takes a list of explicit package-link pairs, so you could specify a full path for just the wheel you needed (which is presumably a dependency rather than one you’re directly specifying)? Or what if you install directly from your fixed wheel, then shouldn’t it skip the broken version later (and if not, can we fix that)?

These are the kinds of things that should be written up in the PEP. You don’t have to have 100% coverage, but right now you have discussion of rejected spellings but not other ideas.

A single configurable platform tag for each install still looks good to me, especially if the default is “manylinux” (with special handling of the versioning scheme). Or perhaps an explicit list of valid platforms that can be user-edited to exclude manylinux if that’s what you want.

16

It’s unfortunate that the manylinux tag ordering came out the way it was, I certainly didn’t realize until this bug caused problems for my own deployments where it was important for a local wheel to be preferred over a (broken) pypi manylinux one. The linux tag was meant to take precedence over manylinux for the same reason that a pure Python tag would be less preferred than a wheel that did contain an optional compiled extension. Since we don’t distribute the linux wheels between machines they already mean “most specific to this machine”.

2 Likes