Python Packaging Strategy Discussion - Part 1

[steve.dower]

@pf_moore

if the expectation is that there will be little practical change in the availability of binaries on Windows, then I have no problem

My hope is that the change will be greater availability on Windows, because we’ll solve the common-native-dependency problem (e.g. making sure two separate packages can agree on a single copy of libpng, or libblas, or whatever they can’t agree on), and/or because we’ll have more people involved in actually building stuff who can help support the upstream projects too (like Christoph Gohlke did).

But yeah, I doubt everyone who’s invested massive effort into getting their stuff to build will suddenly drop what they’re doing. The main problem is initial availability (getting it to build in the first place) rather than it breaking randomly once it’s (successfully) installed.

@pf_moore

The problem is that for Windows users, there simply isn’t another example of a “distro” apart from Conda.

ActiveState, WinPython, Python(xy) (maybe still) are fairly siginificant distros. Arguably Cygwin too. Blender distributes their own build of CPython as part of their app (as do some other apps), and may benefit from deliberately compatible package builds.

But yeah, they’re not as prominent as Anaconda, who more-or-less started specifically to solve this problem on Windows. I’d love to see more distros available on Windows, but then the challenge with that kind of messaging is that basically nobody wants to see more distributors for Linux It’s tough to walk the line.

@pf_moore

The difficult question is when a distributor doesn’t have prebuilt binaries, what should happen? For example, a library that the distributor doesn’t package. Or a newer version of a package that the distributor has, but hasn’t upgraded yet. Or an older version of a package (because my script was written for the v1.0 API).

And does the answer change depending on whether the code is pure Python or not? Or C with no dependencies? Where do we draw the line?

The way this currently works for me is that I go to the distributor, remind them about all the money we paid them, and ask them politely to add the package Failing that (and tbh, it hasn’t failed yet), we build from source, using the distributor’s environment as the target, so that everything is compatible with it.

The key part is using the distributor’s environment. Right now, PyPI implicitly requires builders to use the python.org downloads as the base environment - it’s no good building against Cygwin’s Python and publishing that to PyPI, because it won’t work when someone installs it. But it would be totally fine to set up a separate index specifically for Cygwin and require all packages published there to use that as their target.^[1]

I don’t think the answer actually changes based on native/pure packages at all, because a distributor’s repository is valuable for more reasons than “is it a wheel or an sdist”. But I agree with your point, that users will reach for a thing that seems to work no matter the source, and won’t buy the “if you want support you need to get it through the supported channels” answer (and we don’t have to sell them that line either - distributors need to self-promote). Same deal as if you download any other app for free - you get what you get.

But any way we make it easier for distributors to bring in new versions of packages makes it easier for their users to get the newer versions, and they’ll be more likely to ask for them in future, and the whole process gets better for everyone.

@sinoroc

winget on Windows

Definitely not. winget is an installer, not a package repository.

---

1. Platform tags are basically a way of doing this within a single index. ↩︎