Python Packaging Strategy Discussion - Part 1

(Maintainer of: PyPA: build, cibuildwheel; Scikit-build: scikit-build-core, scikit-build, cmake, ninja, a few others; also pybind11 and it’s examples, bunch of Scikit-HEP stuff, also plumbum, CLI11 (C++), and other stuff, also frequent contributor to nox, also some conda forge and homebrew recipes).

First point: I don’t think the current situation is terrible - I think it’s a great step forward from the past setuptools/distutils monopoly, especially for compiled backends[1]. Making extension modules with setuptools was/is really painful, and requires up to thousands of lines (14K in numpy distutlis, IIRC) to work, and is very hard to maintain. Setuptools/distutils supports extension builds more from necessity and its original use building CPython, not because it was designed to build arbitrary[2] user extensions originally. We are just now starting to see good options for extension building backends built for PEP 517 (scikit-build-core & meson-python are recent additions that wrap two of the most popular existing build tools, cmake and meson). I don’t think finally seeing multiple usable options for build backends is bad!

On unification: I think unifying interfaces and providing small, modular libraries to help in that goal is a fantastic step forward. Certainly, in the compiled space, many/most users will want a build system like CMake or Meson - building a compiled extension from scratch is really hard, and not something I think we want to compete on. Reusing the years of work and thousands of pre-existing library integrations is valuable. I’d love to see more helper libraries though - the public API for wheel would be really useful, for example. packaging and pyproject-metadata are great; I’d like to see a bit more of this sort of thing, it would make building custom backends easier. I’d also love to see more usage unification; config-settings in pip matching build for example (at least for -C and lists, --config-setting vs. --config-settings unification might be too far gone).

On extensionlib: In my opinion, this must be an “extensions” PEP. I want both meson-python and scikit-build-core to work as PEP 517 builders first, so we have a good idea of everything required to make an “extensions” PEP. I also think we ideally should have a proof of concept (in extensionlib or as a hatch plugin) of the idea. Also for some projects, a native PEP 517 builder will probably remain ideal even after this. If your code is mostly (or in some cases, entirely) a compiled extension/library/app, then it likely would be best to just use the PEP 517 backend provided by your tool of choice. However, if you do have a mixed project, especially one that mixes compiled extensions (Rust compiled with cargo and C++ compiled with cmake or meson, for example), then being able to use these tools per extension would be highly valuable. It also allows the author to take advantage of things like Hatch’s pretty readme plugin or vcs plugins, etc. Source file collection is not unified, so it someone already knowns hatchling, reusing hatching and just adding a compiled extension via the extensions system would be nice. The key issue is handling config-settings - this would probably be the bulk of the PEP; for the toml settings, this is pretty easy, but we’d need a good way to pass through extension settings. You’d not pass in a list of files; you’d get out a list of produced artifacts and maybe a list of consumed files (for SDists). Things like cross-compiles are handled by the extension backend; it’s no different than cross-compiling as it is today. Another important thing to handle is get_requires_for_build_*, which is very important for compiled extension building, as they often have command-line app requirements that optionally can be pulled from PyPI.

On conda vs. PyPI: I think both approaches have merits, and I don’t think one should be jettisoned in favor of the other, but we should do what we can to help these work together, and maybe learn from each other. Giving the library author the ability to produce their own wheels has benefits, such as better control over your library, and rapid releases - sometimes conda releases get stuck for a while waiting for someone. Providing good tools to do it (like cibuildwheel & CI) has been huge, and I think the situation is better than Conda’s layers of tooling that makes tooling that injects tooling that duplicates tooling into tens of thousands of repositories. This has been patched so many times that it’s really hard to fix things that are clearly broken, like CMAKE_GENERATOR, which is set to “Makefiles” even if make is not installed and Ninja is, etc. Also, I spent several days trying to get the size of a clang-format install under some amount (500 MB, I think?) so it could be run with pre-commit.ci’s limits - and then I found the other pybind11 maintainers had deleted conda a year or two ago and had no intention of reinstalling it. Then someone produced a scikit-build/cibuildwheel binary for clang-tidy for PyPI - it was 2 MB and installed & ran pretty much instantly, and didn’t require conda preinstalled. The CMake file was less than a page, and the CI file was less than a page. Also, due to the custom compiler toolchain, if a user wants to build compile something locally, conda’s a mess. We get a pretty regular stream of users opening issues on pybind11 just because they are using the conda Python and don’t know why they can’t compile their own code. Conda’s designed to be pre-build via conda-build, and not build on the user’s system via standard tools. On the flip side, Conda can package things that can’t be done as wheels (at least as easily), it can handle shared libraries without name mangling, and it has a uniform compiling environment (mostly). And the central nature does allow central maintainers to help out with recipes a bit more easily. (Though, I should mention that many of the “thousands” of maintainers are really just the original package submitters, just like PyPI).


  1. Even for non-compiled backends, we wouldn’t have things like hatchling if the playing field hadn’t been opened up to multiple backends so the best could win out. And there’s a clear use case for flit-core, too, for building things that hatching itself depends on, for example. ↩︎

  2. It was “able” to because it had to be - there was no way to compete, but wasn’t intended to be full featured. Things like selecting a C++ standard are missing. ↩︎

8 Likes

Thanks for clarifying a few things @henryiii. Regarding this particular point, I suspect it’ll be pretty niche - only a handful of users probably. You’d have at least two more solid options that avoid mixing multiple build systems together. Build the Rust and C++ parts as separate wheels (one with Maturin, one with scikit-build-core/meson-python). Or you can just use Meson for everything, it supports Rust too.

Your main use case / audience for this is probably still “was pure Python, now wants to add a little bit of Cython, don’t want to move build systems”. Either way, it’d be good to see a prototype at some point. A PEP feels quite premature at this point, you can just build it if you want and find some early adopters.

5 Likes

(Mostly off-topic bit of history here, intended for context rather than contributing anything concrete to the discussion. Also, this is from my memory of events, so I may be misremembering things - if the details matter, please check the mailing list history directly).

That’s not actually true. Distutils was originally developed specifically to replace the various (non-portable) custom makefiles and build scripts that were previously used to build C extensions for Python. Being able to build and install pure Python libraries in a standard way was a side benefit, but I suspect that if compiling C hadn’t been involved, people would have been pretty happy with "just put your code on sys.path for a lot longer.

In fact, distutils wasn’t used by Python itself to build core C extensions initially - that was added later, I think because it seemed silly to have an extension-building library and still build the stdlib extensions by hand.

In addition, distutils was developed at a point when most people did build for themselves from sources. That probably alleviated a lot of the complications we have now, as the same build stack gets used for everything (and distutils handled the basic details of how to find the compatible C compiler, and pass it the right settings). That got us quite a long way, but when we add publishing binary builds, and far more complex C extensions, we now start to see the cracks showing :slightly_frowning_face:

6 Likes

I wouldn’t say that PEP 517 solved “the problem”. It is a key enabler of future solutions to many problems by removing the necessity for project authors to use setuptools just so that users/downstream can install/build from source. PEP 517 makes it possible to use alternatives to setuptools but doesn’t actually provide those alternatives and does not directly solve any of the problems that were difficult to solve while still using setuptools. The backend side of the PEP 517 interface was deliberately left as a Wild West at the time but that doesn’t mean that there isn’t any potential benefit from future standards and interoperability in the things that backends do.

I think it’s important to recognise the limited (although not unimportant!) nature of the problem that PEP 517 did solve. It specifically concerns the way that a tool “like pip” will interact with source packages in a future where projects might use build systems that pip doesn’t know about. While that is crucial for projects on PyPI and elsewhere to be able to use different build systems it also doesn’t really address the other contexts where we might want to do different things especially on the development (rather than distribution) side or perhaps on the more “manual” rather than “automatic” interaction with source code.

The premise in many comments above seems to be that the PEP 517 interface makes it possible to have a unified frontend that is completely agnostic about backends. The purpose of PEP 517 was precisely to enable backend-agnostic frontends but specifically for automated consumers of source code. When I imagine my ideal frontend for development use it absolutely needs to have better knowledge of what’s going on in the backend than PEP 517 affords. I would probably want it to understand the relationship between my extension modules and source code, to make something like editable installs, to have some support for managing C dependencies, choosing between different toolchains and so on. I can see why that’s all out of scope for PEP 517 but many of the problems to be solved are still there.

1 Like

Agreed, but I don’t think there’s any benefit from making this a unified frontend. Once you’re in this level of development mode, it’s totally fine to use the backend directly (AFAIK, all the major/active ones have their own interfaces).

They’re all going to have their own configuration formats, or even just their own quirks, which means you can’t develop the project without knowing about your particular backend. Trying to optimise this away feels like an unnecessary unification project.

I’m somewhat more sympathetic to the “I had a complex pure-Python project already defined in a backend that can’t do native modules and I don’t want to rewrite it into another backend just to add a single native module”, but I’m not convinced it outweighs the ability of backends to innovate in this space.

Basically, I think unifying the definition of builds is a distraction and we shouldn’t invest in that yet. Let’s flesh out the functionality that’s actually needed in a range of backends, then let usage gravitate towards the “best” option and eventually that one will expand to handle all the things that matter.[1] Trying to design that interface preemptively really isn’t possible yet.


  1. I’m aware as I say this that it means we’ll likely converge to a thin wrapper around an existing tool, and I’d personally bet on CMake. ↩︎

1 Like

I agree with both Oscar’s comment and Steve’s reply. With the minor note that I don’t think build system usage will ever converge. It’s conceivable build backends do though, since they’re a pretty thin layer in between a couple of pyproject.toml hooks and invoking the actual build system. So it’s not inconceivable that, for example, scikit-build-core and meson-python would merge in the future and have a configuration option for whether to use CMake or Meson.

Overall we’re in decent shape here - there’s work to do on build backends and build systems, but nothing in the overall Python packaging design for that is currently blocking or in clear need of changes.

Getting back to the big picture strategy discussion, here is that blog post: Python packaging & workflows - where to next? | Labs. It’s an attempt at a comprehensive set of design choices and changes to make yes/no. There’s a long version, and a short version with only the key points. I’ll post that short version below.

The most important design changes for Python packaging to address native code
issues are:

  1. Allow declaring external dependencies in a complete enough fashion in
    pyproject.toml: compilers, external libraries, virtual dependencies.
  2. Split the three purposes of PyPI: installers (Pip in particular) must not
    install from source by default, and must avoid mixing from source and binary
    features in general. PyPI itself should allow uploading sdist’s for
    redistribution only rather than intended for direct end user use.
  3. Implement a new mode for installers: only pure Python (or -any) packages
    from PyPI, and everything else from a system package manager.
  4. To enable both (1) and (3): name mapping from canonical PyPI names to other names.
  5. Implement post-release metadata editing capabilities for PyPI.

Equally important, here are the non-changes and assumptions:

  • Users are not, and don’t want to become, system integrators,
  • One way of building packages for everything for all Python users is not feasible,
  • Major backwards compatibility breaks will be too painful and hard to pull
    off, and hence should be avoided,
  • Don’t add GPU or SIMD wheel tags,
  • Accept that some of the hardest cases (complex C++ dependencies, hairy native
    dependencies like in the geospatial stack) are not a good fit for PyPI’s
    social model and require a package manager which builds everything in a
    coherent fashion,
  • No conda-abi wheels on PyPI, or any other such mixed model.

On the topic of what needs to be unified:

  • Aim for uniform concepts (e.g., build backend, environment manager, installer) and a multitude of implementations,

  • Align the UX between implementations of the same concept to the extent possible,

  • Build a single layered workflow tool on top (ala Cargo) that,

    • allows dropping down into the underlying tools as needed,
    • is independent of any of the tools, including what package and environment
      managers to use. Importantly, it should handle working with wheels, conda
      packages, and other packaging formats/systems that provide the needed
      concepts and tools.
5 Likes

The discussions about “integrator” always seem a bit vague and make me worry it means in the future there will be even fewer binary wheels on PyPI, and you will be forced to use Conda to use pytorch etc. Is that really what it means? That sounds very undesirable to me.

Users that are happy with PyPI as-is don’t have to change a thing, and are unlikely to be affected by the hardest to build packages no longer providing wheels.

Doesn’t this propose they will now be forced to build the hardest to build packages themselves or forced to use Conda?

I’m not sure if I understood what is intended by this or not. Concretely would this mean that if I’m on Ubuntu and I do pip install stuff then pip might install some things using apt-get and some things from PyPI?

(If the answer is yes then I have many more questions about how that would work in general but perhaps that’s for another thread somewhere.)

The alternative is getting wheels that don’t work, either in obvious ways (fails to import) or subtle ways (crashes at runtime, silent data corruption, etc.).

Trust me, if there was a way to solve all of these problems within the current system, we would have. We’ve been trying, and we’ve gotten so close that a lot of people have the same concern as you do, believing that wheels are sufficient, but the problem is that the remaining gap appears to be uncloseable.

Uncloseable, that is, unless you have a complete stack that has been built in a known, consistent environment. Which either means you built everything on the target machine, or you built everything on another machine that is compatible with the target machine.[2]

What it excludes is the possibility of building individual parts on different machines and trying to bring them together later on. But this is the social model of PyPI - people publish their own packages - and we don’t want to change that. It’s actually really important for distributors that PyPI exists and operates the way that it does… for sdists.

But the job that distributors are taking on is to provide a coherent set of binaries so that when their users install things, they are going to work. Mixing in binaries built without that coherence is what hurts users.

That is one possible way to make this work. I proposed other ways earlier to achieve the same result.

I think this thread is the place to agree that when a distributor has prebuilt binaries for their build of Python, those should be preferred over unmatched binaries. If we agree that the default workflows should somehow support that, we can do details in new threads.


  1. I substituted “a distro” in the quote, because I know that’s what you mean, but I really want to avoid framing this as “PyPI vs Conda” when it’s fundamentally “individual pieces built separately vs everything built all together”. ↩︎

  2. Initially, only Windows wheels were allowed on PyPI, because it was the only consistent enough environment. There’s been a huge amount of work to define consistent-enough environments for other platforms in order to allow them on, because fundamentally, there’s no way to predict how the build needs to be done when we don’t control the entire stack. ↩︎

2 Likes

That’s a nice and concise way of expressing a design principle on how to better support system integrators. It matches what I had in mind with the “new opt-in installer mode” - which is indeed one of a few possible ways to implement that principle.

1 Like

No, the alternative is that people currently building wheels which work for “most” users, stop building them (because it’s hard to even do an imperfect job) and direct people to get their binaries from a distributor (or build them themselves, but see above re “hard”…)

However, “get your binaries from a distributor”, as has been mentioned a number of times here, tends to include “get your Python from a distributor”. And that means “switch to a different stack”, which for many users is simply not acceptable (especially if they don’t know this is going to happen in advance, which they typically don’t).

If we were talking about distributors/integrators shipping binaries which worked with whatever Python installation the user had, there would be a lot less heat in this discussion. If I could install the Windows Store Python, and then find I needed to get (say) scipy from somewhere other than PyPI, and there was somewhere that shipped binaries that worked with the Windows Store Python, I wouldn’t have the slightest problem (well, the UI of “pip fails, find the right command” needs to be worked out but that’s a detail). But I’ve seen no sign of anyone offering that.

Maybe I’m being pessimistic, and we should assume the existing sitiuation will continue (with most packages having binary wheels on PyPI, and only specialist outliers opting out). But maintainers are human, and volunteers, and having the “Python Packaging Strategy” explicitly normalise the idea of expecting people to get hard to build packages from a distribution seems to me like a huge temptation to just say “sorry, we don’t support binaries from PyPI”.

What I’d like to see at a strategy level is a statement that Python packaging is for everyone, and our goal is to make it as easy as possible for every user, no matter where they got their Python installation from, to have access to all published packages (on PyPI - privately published packages are a different matter). And yes, there will always be exceptions - the point is it’s a goal to aspire to.

4 Likes

That’s because this is the bit that doesn’t work :slight_smile: Windows is a poor example here, because the ABI is much more reliable than other platforms (and the nature of DLL Hell on Windows is somewhat more amenable to these problems than on other platforms).

The reason that binaries don’t work with “whatever installation of Python the user has” is because you need to know about the installation in order to produce the binaries.

To fix that, we’d need to define a complete ABI (down to “how many bits in int and which order are they and how does it get written to memory when calling another function”) and then enforce every single library to also use that. And that only fixes it because “whatever installation” now only has a choice of one - the one that perfectly uses our ABI. This approach is never going to work out.

Most of the time on many modern desktop and server machines, the ABI is usually close enough that you can squint and get away with it (“wheels which work for “most” users”). Once you get into the territory where that doesn’t work, you do need to know all the details of the ABI in order to build a compatible binary for that interpreter. The easiest way to do this is to get the interpreter from the same builder you get the binary from, because they’ll have ensured it.

If it helps take some of the heat out of the discussion, I’m not in any way saying that package developers are doing things wrong, or doing a bad job. They’re trying to do a job that is literally impossible, because their own scope is too restricted to be able to do it. And many of them know it, or at least sense it, which is why they’ll feel frustrated. But it’s not their fault, and it’s not on them to solve it themselves. They’re doing an incredible job of making the most of the impossible situation they’re in, and all we want is to make that situation less stressful, firstly by acknowledging that they don’t have to carry the entire burden of solving it (and indeed, they can’t), and then by better connecting those who can solve it with those who need the solution (and who might be the ones applying the pressure to the original developers).

2 Likes

No one is offering this because it’s not possible, for very fundamental reasons (ABI, toolchains, etc.). It really is not one of the options. It’s the status quo, or some version of what Steve and I suggested, or something more radically different like actually trying to merge the conda-forge and PyPI approaches into a single new thing.

I think that has been the strategy of the crowd here until now, and it’s not working. I agree with Steve’s assessment that the remaining gap is uncloseable.

To rephrase your terminology, Python usage should be for everyone -with a more unified experience.

1 Like

It’s worked incredibly well, actually :slight_smile: But every strategy has its limits, and this is it.

4 Likes

I’d like to make two points in response to this:

  1. I am not worried that there will be a major drop in packages providing wheels, at least popular packages providing wheels for popular platforms. There’s enough maintainers who care, and enough users who care. So if it works today, it is highly unlikely that folks will pull the plug tomorrow (or in 2 or 5 years from now).
  2. There are lots of OS’es, platforms, Python interpreters, etc. for which PyPI allows wheels but no one is building them today anyway. A few examples: PyPy, PowerPC, armv7 (lots of Raspberry Pi users), musllinux (Alpine Linux users, very popular in Docker), etc. Making things better for system integrators will make things a lot better for any of these user groups.

OK. If that’s true, and if the expectation is that there will be little practical change in the availability of binaries on Windows, then I have no problem. I don’t know enough about the other platforms we’re discussing to say anything beyond broad generalisations. So I’m happy to leave the “not-Windows” side of the problem to the experts in those environments.

Quote from @zooba in a footnote[1].

I substituted “a distro” in the quote, because I know that’s what you mean, but I really want to avoid framing this as “PyPI vs Conda” when it’s fundamentally “individual pieces built separately vs everything built all together”.

The problem is that for Windows users, there simply isn’t another example of a “distro” apart from Conda. So in that context, at least, it genuinely is about “PyPI vs Conda”. I understand, and agree with, the principle of not making this confrontational, but I think we should be very explicit about what options actually exist right now, in a practical sense. I’ll refrain from doing any sort of detailed “PyPI vs Conda” comparison, because that’s what we’re trying to avoid here, but IMO it’s really important to understand that when people with a purely Windows background see “you must use a distro”, they really only have “being forced to use conda” as a mental model to inform their views.

But that’s not the difficult part of the question here. (FWIW, I’d be fine with agreeing to that statement). The difficult question is when a distributor doesn’t have prebuilt binaries, what should happen? For example, a library that the distributor doesn’t package. Or a newer version of a package that the distributor has, but hasn’t upgraded yet. Or an older version of a package (because my script was written for the v1.0 API).

And does the answer change depending on whether the code is pure Python or not? Or C with no dependencies? Where do we draw the line?


  1. As an aside, it’s really hard to quote footnote text in Discourse… ↩︎

2 Likes

I’m mostly care about Windows. What is a distro on Windows?

The only problem I run into regularly is missing wheels. If they exist they work; maybe I’m lucky, and/or relying on heroic efforts of the people providing them.

Or maybe that explains it.

Does that mean “of course on Windows everything will keep working as it does now” is silently omitted / implied?

2 Likes

What should I think of when I read “system package manager” here? Is it something that already exists (if yes, what are examples) or something that still needs to be created? Are we talking about things like apt on Debian, winget on Windows, homebrew on Mac?

Perhaps part of the problem with the perception that the ABI is
“more compatible” on Windows is that there aren’t hundreds of
different companies and communities providing their own versions of
Windows. There is one: Microsoft.

The ABI incompatibility across different Linux distributions and
different UNIX derivatives is only an incompatibility if you think
of, say, “Linux” as an operating system. Debian GNU/Linux is an
operating system, Red Hat Enterprise Linux is another operating
system, maybe sometimes you can run the same binaries on versions of
both, but expecting them to be “consistent” is like expecting to run
Mac OSX/Darwin binaries on Windows. If anything, Linux distributions
have a more compatible ABI with each other than Windows does with
any other operating system (emulation layers aside, of course).

2 Likes

My hope is that the change will be greater availability on Windows, because we’ll solve the common-native-dependency problem (e.g. making sure two separate packages can agree on a single copy of libpng, or libblas, or whatever they can’t agree on), and/or because we’ll have more people involved in actually building stuff who can help support the upstream projects too (like Christoph Gohlke did).

But yeah, I doubt everyone who’s invested massive effort into getting their stuff to build will suddenly drop what they’re doing. The main problem is initial availability (getting it to build in the first place) rather than it breaking randomly once it’s (successfully) installed.

ActiveState, WinPython, Python(xy) (maybe still) are fairly siginificant distros. Arguably Cygwin too. Blender distributes their own build of CPython as part of their app (as do some other apps), and may benefit from deliberately compatible package builds.

But yeah, they’re not as prominent as Anaconda, who more-or-less started specifically to solve this problem on Windows. I’d love to see more distros available on Windows, but then the challenge with that kind of messaging is that basically nobody wants to see more distributors for Linux :smiley: It’s tough to walk the line.

The way this currently works for me is that I go to the distributor, remind them about all the money we paid them, and ask them politely to add the package :slight_smile: Failing that (and tbh, it hasn’t failed yet), we build from source, using the distributor’s environment as the target, so that everything is compatible with it.

The key part is using the distributor’s environment. Right now, PyPI implicitly requires builders to use the python.org downloads as the base environment - it’s no good building against Cygwin’s Python and publishing that to PyPI, because it won’t work when someone installs it. But it would be totally fine to set up a separate index specifically for Cygwin and require all packages published there to use that as their target.[1]

I don’t think the answer actually changes based on native/pure packages at all, because a distributor’s repository is valuable for more reasons than “is it a wheel or an sdist”. But I agree with your point, that users will reach for a thing that seems to work no matter the source, and won’t buy the “if you want support you need to get it through the supported channels” answer (and we don’t have to sell them that line either - distributors need to self-promote). Same deal as if you download any other app for free - you get what you get.

But any way we make it easier for distributors to bring in new versions of packages makes it easier for their users to get the newer versions, and they’ll be more likely to ask for them in future, and the whole process gets better for everyone.

Definitely not. winget is an installer, not a package repository.


  1. Platform tags are basically a way of doing this within a single index. ↩︎