Encouraging 3.10 wheels on PyPI

Hello all, especially the release crew and PyPA!

How can we state more clearly that with 3.10rc1 out, projects should now release binary wheels for 3.10 on PyPI? (I’m assuming we do want to encourage that, of course.)

See the discussion for numpy, whose maintainers initially put 3.10 wheels on a different package index to clearly communicate that they are experimental. IMO that’s a mistake, and @hroncok’s write-up which convinced them is roughly something we should communicate somewhere to help 3.10 hit the ground running.

More broadly, would it be helpful to cover the development cycle and stability expectations in user-facing docs? Currently I could only find a FAQ entry, but I could expanded it to a whole page.

7 Likes

Strong +1 on me for encouraging more projects to publish binaries earlier in the release cycle.

As to how to get the message out, I don’t really have any good ideas. If something like the write-up you linked to was posted somewhere, it would probably be good to link to it in the release announcements, but I don’t know where would be the best place (for general visibility) to post such an article.

Maybe it would be worth surveying projects like numpy/scipy/matplotlib, and asking them what would have got their attention?

Perhaps this is a topic/guide that could live at https://packaging.python.org/?

3 Likes

In similar spirit I started a discussion about building conda(-forge) packages for Python 3.10 before the final release: Start Python 3.10 migration with the release candidate · Issue #1499 · conda-forge/conda-forge.github.io · GitHub This would enable the same experience for conda users as for wheel users on release.

2 Likes

cibuildwheel builds 3.10 wheels as a default since 3.10.0rc1 availability: Release v2.1.1 · pypa/cibuildwheel · GitHub

2 Likes

Currently, scipy is blocked on MacPython/scipy-wheels, which is blocked on MacPython/numpy-wheels, which seems blocked on Azure adding Python 3.10 (for Windows and Mac).
I don’t think I can help here much, but maybe one of the other people in this topic have a better idea how to help them move forward.

In my experience, putting up a single canonical webpage with clear authoritative info on the topic is pretty helpful – starting with a short and very-easy-to-read intro paragraph, that’s followed by a detailed tutorial/guide and then an FAQ and an email address/contact link to get further advice.

Then you have to advertise the news and link to that page, else people won’t necessarily look for it and find it; this can take anywhere from 30 minutes to weeks, depending on how crucial it is to get the word out and how much time you’re willing to spend on marginal utility. Some of the people you’re trying to reach pay attention to email newsletters, some to podcasts, some to Discords and other fora. Think about whom you’re trying to reach and what other hobby or professional groups/interests they may share – maybe in this case, for example, many of the projects you particularly want to reach are doing scientific/numerical stuff, so a bunch of of them are part of NumFOCUS or the SciPy and PyData conferences/meetups, so you can ask someone who’s in that world to forward stuff there. Or maybe there’s a cluster you’d like to reach in fintech, so you look for pre-existing interest groups there to send your link to.

Some places to start:

@smm heads-up in case any of this is useful to you as you construct more systematic playbooks for communicating out to the users of Python package distribution tools!

@encukou Apologies in advance if any of this is redundant and you knew it already. :slight_smile:

1 Like

FYI, for NumPy we’ll move over to cibuildwheel on GitHub Actions in the coming months (at least for Linux/Windows/macOS), which has new Python releases available earlier than other CI providers usually. CI availability is still the bottleneck though - the request to start building wheels early made sense, but it’s just a lot of work to get a Python install from somewhere custom instead of using the built-in support for Python versions that one normally uses for building wheels (and CI jobs in general).

True in general, and a good idea to do in addition to the “file issues about 3.10 wheels on the repos of many important projects”, but that’s not the problem here. The actual problem is that there is no good recipe/answer currently for where to get Python 3.10 in a way that is easy to integrate in the wheel-build CI machinery across TravisCI, Azure DevOps, Appveyor.

Setting up an environment is not really technically difficult; Python is quite straightforward to compile (and you can cache the artifact), and installation on Windows is even easier, but most package maintainers don’t know how to do it, and virtually no maintainer would accept a PR with a CI vector with custom CPython compilation out of the blue since they wouldn’t have any confidence the setup would actually work, and would very reasonably rather wait for “official” CI support. So a canonical page would still help massively here, since the PR can say “hey I just copied this CI configuration from python.org” and skip most of the persuasion required.

A big advantage of using GitHub Actions is they have new versions available the fastest of all the CI providers, 3.10 was ready within 12 hours. In fact, GHA already has the 3.11 alpha available as 3.11-dev.

(Azure Pipelines had 3.10 ready on some OS last week and the rest this week, AppVeyor will have it in their next image release, Travis CI haven’t reacted to the ticket or in fact any other ticket on their tracker).

We released 3.10 wheels for Pillow during the 3.10 RC using GHA (and multibuild).

We’d happily accept such a PR, it’s just a pain to write from scratch even if not technically difficult - if no maintainers use Windows nor are very fluent in Powershell, there’s a lot of “debug via CI” involved. If you have a concrete example of a CI job that takes this approach, can you please share a link?

Azure apparently won’t provide 32-bit Python 3.10, so building Python from source may still be useful for a while longer.

If it’s only Windows you’re concerned about with 32-bit, the pythonx86 package from Nuget should help. There are some install commands listed, but you’re more likely to want:

nuget install pythonx86 -Version 3.10.0 -ExcludeVersion -o <directory>
$env:PATH += "<directory>\pythonx86\tools"
python ...

(Nuget should be available by default…)

2 Likes

We seem to be collecting lots of interesting bits of advice, but the point of the original post (and many of the comments) was whether we should have a central document for people.

I suggest that someone writes a PR for packaging.python.org, as originally suggested by @dustin above. It doesn’t have to be complete, just a starting point for people to add their own expertise. In my experience, it’s much easier to get people to “fix” an existing document than to start a new one. (And yes, I do understand the irony of what I just suggested :wink:)

I’d happily add any clarifications I can offer, and give help with Windows/Powershell quirks. I’m sure @steve.dower would be willing to add details on how to get builds via nuget. The cibuildwheel folks would probably be able to provide a sample config as part of a “how to get started for a new project” section. We could add sections covering how to add wheel builds to existing CI setups - projects using the various CI providers could add their experiences.