The discussion from Draft pep: Disabling Manylinux concluded that creating a PEP for a “local” tag would be the logical next step. I think I’ve produced something workable, but I think there’s definitely bits which need the input of those who actually write and are familiar with the current packaging ecosystem. I think I’ve also linked to all the relevant discussions that have happened in this space, so that if someone wants to extend this, or come up with further ideas for tagging wheels, this can act as a starting point. The PEP is below (included inline now, unlike the no-manylinux PEP).
Title: The “local” wheel tag
Author: James Tocknell firstname.lastname@example.org
Abstract and Motivation
Manylinux has been successful in allowing users to install non-pure-python
python packages, without the need of a compiler (or compliers) or the need to
install other shared libraries. This has increased usability of the python
packaging ecosystem, and reduced time waiting on CI builds. However, manylinux
has increased the issues some users face, therefore these users wish to opt out
of using manylinux wheels. They may be encountering issues with projects
producing invalid manylinux wheels [#manylinux-tensorflow], wanting to
experiment with different BLAS/LAPACK implementations, or a variety of other
reasons [#pip-no-manylinux]. Additionally, locally build wheels on Linux use the
linux tag, which is more general than the manylinux tag [#linux-manylinx].
Creating a new “local” tag would flag whether a particular wheel was locally
built (and therefore used as a caching artefact), or being used as a
distribution artefact. Whist the hasn’t been the same level of discussion of
tagging on other OSes (likely due to the existence of a single platform tag),
being able to distinguish between a locally built wheel and wheel used on PyPI
to distribute code is a useful debugging aid. The motivation and early
discussion for this PEP came from GitHub issues on PyPA GitHub organisation
Issues not in Scope
This PEP treats how “local” wheels are built as out of scope, as well as how to
build wheels with a specific configuration. Whilst the likely audience of this
PEP would have opinions on how to “configure” wheels, we deem any more advanced
tagging system to require a separate discussion [#wheel-configure].
Specification and Rationale
A new platform tag [#pep-425] is to be added
<platform> is a platform tag as computed by PEP-425. Examples of this new
“local” tag are:
Wheels tagged with such a platform tag must be treated as having a higher
priority than any other wheel with a different platform tag (e.g. win32,
Interaction with other wheels with no appropriate sdist
This case occurs when either there is no sdist available across all versions of
the package, or where the package installer has been instructed to ignore all
There are three possible cases:
- the highest version of the package has a non-local wheel, and there is a
version of the package with a local wheel
- the highest version of the package has a local wheel, and there is a
version of the package with a non-local wheel
- the highest version of the package has a local and non-local wheel
In cases 2 and 3, the higher priority of the local wheel means that the local
wheel should be installed.
In case 1, we come to an conflict, which can resolved with three options:
- the non-local wheel is installed
- the local wheel is installed
- an error is thrown
Option 1 is the simplest in terms of implementation: find the highest valid
version, and install the wheel of that version with the highest priority.
However, those who wish to use local wheels would likely object to the use of
the non-local wheel, which would rule out this option.
Option 2 and 3 require scanning the whole version history, looking for a
possibly non-existent local wheel. The cost of this scan is unclear
[#local-lookup], but is
presumably greater than option 1. Option 2 then means the user installs an older
wheel (possibly what the user was after, but possibly not), whereas option 3
means that it is not possible for a user to install an older local wheel until
the newer wheel is removed from the search path. The choice between option 2 and
3 comes down to a philosophical difference about what should be in the case of
ambiguity. Leaving implementers to choose between options 2 and 3, and allowing
them to develop appropriate UIs around the options (possible with warnings or
configuration options), seems the best course.
Interactions with other wheels with sdists
This case is the more complicated version of that above, in that we now have
three possible outcomes: install the sdist, install the non-local wheel, and
install the local wheel.
The possible cases are (where ordering refers to the highest version of the
- sdist <= non-local wheel <= local wheel
- non-local wheel <= sdist <= local wheel
- sdist <= local wheel <= non-local wheel
- non-local wheel <= local wheel <= sdist
- local wheel <= sdist <= non-local wheel
- local wheel <= non-local wheel <= sdist
Cases 1 and 2 are equivalent to cases 2 and 3 above: the local wheel is the
Case 3 is case 1 above (we can ignore the presence of the sdist).
Case 4 is the first new case, however as we are choosing between the local wheel
and a sdist, the usual rules of sdist wheel ordering apply (so there is no
addition logic required here).
Case 5 is a variant of case 1 above, in that we should [#building-sdists] be
able to go from an sdist to a local wheel, which reduces to comparing a local
and non-local wheel. One additional bit of logic that some installers may want
to provide is the fallback to the local wheel in the case that building the
sdist fails to build.
Case 6 can be any of the cases above, as case 1 may result if the sdist fails to
build. Using the logic handling of the previous cases (cases 2 and 3 install the
local wheel, case 1 provide an appropriate UI), as well as case 5 in the case
that the sdist building fails, we can cover all possible subcases.
What happens in the absence of a local wheel?
If we have no local wheel, and a non-local wheel, we have two options:
- install the non-local wheel
- don’t install the non-local wheel
Whilst case 2 seems incorrect at first glance, there are enough cases where
people have asked to disable manylinux, that to dismiss the second option out
of hand seems to ignore community unhappiness with the current status
[#no-local]. Therefore, it is worth thinking about when option 2 is the correct
The first situation that comes to mind is where there is a newer sdist than the
non-local wheel. Currently pip will build the sdist unless the
--prefer-binary option is used. Note in the case of manylinux, a cached local
wheel will override the use of a manylinux wheel uploaded later, which would not
have been the case for a previously created linux wheel (other OSes would have
always used the cached, locally build wheel, even when a different wheel became
In the situation where there is a non-local wheel where the sdist is not newer,
or when an option like
--prefer-binary is used, currently the non-local wheel
will be installed. Taking that users of the local PEP are likely those who would
wish to prevent the install of the non-local wheel, at least in some cases, the
question then becomes what use cases do we want to support. Manylinux has
mechanisms to control whether a specific tag should be used on a specific system
[#manylinux-peps], but there is currently no equivalent for any other tags
(which, given the absence of a local tag, would not have been distinct from
avoiding wheels). Given the option to choose between local and non-local wheels
is now possible, installers having an option of only choosing local wheels (e.g.
--local-wheels-only option or similar), seems more reasonable than
having manylinux-only options.
What is a “non-local” wheel?
Whilst we have defined what a local wheel is, we haven’t defined what exactly we
mean by a non-local wheel. The simplest option is any wheel whose platform is
not a local platform is a non-local wheel. However, the implication of this is
that wheels with platform
any, that is pure python wheels, are non-local
wheels. Therefore, users of an option like
--local-wheels-only would be
any platform wheels repeatedly (and, unless either the pure python
wheels produced were tagged with local tag, or a sufficiently smart installer
any wheels were the ones it produced, the
any wheel would be
rebuilt every time). Therefore, the sensible definition would be that a
non-local wheel is any wheel that has a system tag which is not
any or a local
tag i.e. there are three types of wheels, pure python (
any), local and
As only installers with support for local tags will treat the local wheels with
higher priority than other wheels, older installers will likely be a source of
unhappy users (though it is doubtful that anything can be done about that).
However, this PEP will not make things any worse for those users.
The change in priority of locally-build wheels on linux vs manylinux wheels
could be a source of additional bug reports, but is unlikely to be greater than
other issues related to caching.
The security implications of the PEP are those related to running an older
version of software—it’s possible if sdists for a particular project stopped
being published, or if the newer sdists failed to build (and the UI around
handling this was insufficient), that users could be stuck on an older version.
The former case however would likely have some level of discussion on various
news sites (as has happened recently with the license change of some projects),
whereas the latter comes down to design choices of installers.
There is currently no reference implementation. This PEP would likely also
require the coordination of multiple different projects to fully implement, so
the creation of a reference implementation is not currently planned.
There has been previous discussions of manylinux specific options to control
the install of wheels, which helped spawn the creation of this PEP, but we
reject these as being both too specific to manylinux, and being insufficient for
other Linux systems on non-x86 platforms [#no-manylinux].
Possible future platform tags
There has been the suggestion of more specific platform tags (e.g.
ubuntu_16_04_x86), which may exist in the future. Whilst we could have added
these additional tags, or at least specified how they interacted with any local
vs. non-local selection, we leave that to a future PEP (if one is created), for
any decisions on how such an interaction would work.
Similarly, custom tags (which would likely have a lower priority than local
tags) are left to future PEPs to be decided on.
Singular local tag
Having a single local tag (as opposed to one including OS and architecture)
might slightly simplify implementation of this PEP, but given the existence of
multilib/multiarch systems which run 32 bit and 64 bit code on the same system
(or tools like QEMU which can emulate many architectures), including the system
tag as part of the local tag helps avoid weird cases where non-matching wheels
… [#building-sdists] This assumes that we can build a wheel from the sdist,
which is not always true (e.g. no compiler installed). If we can’t build
from a sdist, then we can ignore the sdist, and only focus on the local and
… [#issue-motivation] The “local” tags origins appear to be in
https://github.com/pypa/packaging/issues/239, however the are a number of
other issues which precipitated these discussions, which are linked to in
the appropriate places within this PEP.
… [#linux-manylinux] The ordering of linux vs the manylinux tag is one source
of conflict about manylinux, with debates about the choise of ordering
chosen by specific projects occuring at
… [#local-lookup] The costs of doing such a scan would depend on the protocol
used to transmit the available versions, and any additional metadata used to
choose the artefact to download and install.
… [#manylinux-peps] PEPs 513 (https://www.python.org/dev/peps/pep-0513/), 571
(https://www.python.org/dev/peps/pep-0599/) and 600
(https://www.python.org/dev/peps/pep-0600/) cover all current manylinux
… [#manylinux-tensorflow] Tensorflow produced wheels that weren’t compliant
with the manylinux specification, see
https://github.com/tensorflow/tensorflow/issues/8802. There is discussion of
validation of manylinux wheels via auditwheel uploaded to PyPI planned at
… [#no-local] This unhappiness can be seen in issues on the PyPA issue tracker
on GitHub (some of which are refered to within this PEP).
… [#no-manylinux] The discussion contained within
https://github.com/pypa/warehouse/issues/3668 highlights that this is not
just a manylinux vs. linux tag discussion, and that being able to
differentiate between cached/locally built wheels and wheels distributed on
PyPI is a more general problem.
… [#pep-425] https://www.python.org/dev/peps/pep-0425/
… [#packaging-pip-issue] https://github.com/pypa/pip/issues/7648 was the
motivating issue which started the creation of this PEP.
… [#pip-no-manylinux] https://github.com/pypa/pip/issues/3689 is an issue on
the pip issue tracker requesting that pip provide some way of opting out of
using manylinux wheels, with some additional use cases mentioned.
… [#wheel-configure] Suggestions for either including more configuration
metadata within wheel tags or other metadata (see
Optional C Extension handling,
PEP 517 and projects that can't install via wheels,
https://github.com/pypa/packaging-problems/issues/264), or creating linux
distribution level tags (see
https://github.com/pypa/packaging-problems/issues/69), fall under this more
advanced tagging system (as do proposals like
https://github.com/pypa/pip/issues/6468). Additional logic about how
the wheel is built (see
https://github.com/pypa/pip/issues/3938) is also out of scope.
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.