Ah, okay, I see. Yes, there have been tons of problems reported in the Python Help section because various PyPI packages simply haven’t released wheels for 3.12. But I wouldn’t call “the user has 3.12” a problem with the environment, per se - unless Python was specifically installed for this project that needs whatever package.
Trying to think about it more broadly: what if there were some kind of machine-readable metadata that could be generated for a project, that specifies the compatibility matrix? I’m sure that requires a PEP, but - PyPI could generate it from its inventory of wheels and sdists, and render it into a table that could be displayed on the site; and tools like Pip could use that same metadata up front to avoid the need to backtrack and search through releases. Seeing right up front “there are no 3.12 wheels” while investigating the project would be helpful, and e.g. Pip in --only-binary mode could save a lot of time in the failure cases. Of course, setup.py can impose arbitrary restrictions, but it’d be really nice to be able to distinguish at a glance “wheel available”, “sdist only, but support explicitly advertised by trove classifier”, “excluded by requires-python in pyproject.toml”, and “might work but you’re on your own” cases. (It’d be easier to present that, of course, if there were only one Windows, one Linux and one Mac platform; but.)
This is a very common scenario, though. Someone wants to install the requirements for a specific project: let’s say they are trying to reproduce the results that someone else published. They start by installing the latest Python, but there is a window of time (shorter than it used to be!) when many popular packages aren’t available as wheels yet.
This is partially solved (pin the python version in your conda env file!) but a pinned version restricts future compatibility.
I mean system dependencies, not python extension modules. pip install scipy installs numpy as a dependency (from sdist) automatically on GraalPy, but of course if you don’t have a C compiler installed, it’ll just fail.
I’m not sure I agree with this - is it wrong to use PyPy, GraalPy, or just the latest Python release before packages have released wheels for them? But maybe for the kind of users this discussion is about it is wrong.
If something like this would be the default, maybe it’d be a good place to link to some doc (or have a sentence inline) to educate the user as to why this might be, so they know how to fix their problem? Something along the lines of:
You may be running a version of Python that is newer or older than the range supported by the package you are trying to install, or using an operating system that the package maintainers do not support. Please check pypi.org/project/XYZ/path/to/generated/support/matrix to see which versions might be supported. If you know what you are doing and wish to build the package from source, please run pip with the --build-from-sdist flag.
N.b., for GraalPy we would probably just patch the flag to default to true when we ship pip to always install sdists anyway, and just point users to us to ask for support (so users don’t spam project maintainers with requests for GraalPy wheels)
It’s not wrong, but it’s probably wrong for them to do it without realising that they are using something that doesn’t have wheels released for it. Most common for those using CPython 3.?.0, compared to those on PyPy or GraalPy (who clearly already have enough awareness of the ecosystem to choose an alternative runtime that they likely realise they’ve chosen an alternative).
In my experience, the usual response is to switch to a release that does have prebuilt wheels. What we hope to do here is trigger that response within seconds of the first pip install <whatever>, rather than after an hour of trying to resolve each compiler error one-by-one. (And that’s not a strawman - I’ve seen plenty of engineers who know how to solve each error, and needed to be told “these aren’t yours to solve, just switch to 3.(n-1)”).
Perhaps another option for these projects would be to push a -none-any wheel that contains a helpful error on import? Possibly with metadata that installers could one day recognise and provide the helpful error at install.
It would require users to specify --no-binary <package> in order to get the sdist, which is fundamentally not that different from some new switch to opt-in to building sdists.
And it’s achievable today, without any other modifications, though I expect projects will balk at the idea of having some wheels built from different sources that the rest (I’ve had that reaction before when suggesting that not all wheels need to be “identical”).
Yes I meant that something was set up wrong (i.e. misconfigured) for their intended use case. Not objectively wrong or morally wrong in some cosmic sense.
One thing I’d like to mention: There’s - quite rightly - a focus on the ‘big’ packages like NumPy & Scipy, which have complex build processes and may not ‘just work’ from source (although I’ve been surprised sometimes that unwittingly building NumPy has worked). But there’s also a long tail of smaller packages which might contain a Cython module implementing the tight-looping inner core of an algorithm. Numpy and Cython can be specified as build dependencies, so it’s not unusual that the only external thing they need to build is a C compiler. Here and here are two examples connected to my work that happen to be in my mind - what they do isn’t important right now, but they fit this pattern. If we create wheels for these, there’s a good chance that they’ll be only for Linux, and for relatively few Python versions, and they won’t be updated often.
So any change to ignore sdists by default will break some real scenarios that currently work. A C compiler might seem like a high bar already, but if you’ve installed Anaconda, or you’re working on an HPC cluster, it’s part of the scenery, and installing stuff like this will often work.
As a maintainer of an intermediate complexity package (h5py) I’m ambivalent about the idea. Building from source is unlikely to work without preparation, but any demotion of sdists is going to intensify calls to provide wheels for a wide range of platforms and architectures, which is already a chore.
Doing it that way does seem annoying and un-Pythonic.
But I also don’t think it’s necessary. Presumably, a common source code base could check at startup for the existence of whatever system library, and/or an expected compiled artifact (like a .o or .so file in some specific location, or whatever gets produced by a Setuptools Extension call).
Someone could even make a helper library for that and put it up on PyPI.
I may be missing something, but wouldn’t that mean that we (GraalPy) cannot make getting sdists a simple default for the pip shipped with GraalPy? Some packages may offer proper GraalPy wheels, and I don’t want to force source builds there, but if no wheels are available, I don’t want users to have to specify that they want to build from source? That is to say, a new flag --allow-build-from-sdist would be preferable for me personally as GraalPy maintainer, because we would just always set this to true in the pip we ship, so people get wheels for GraalPy if they exist, and automatically get builds from source if they do not.
This is a key (pain) point. You should not be happy about this at all for production use cases. While failing builds are a problem, builds that succeed and silently result in 10x - 100x performance regressions are worse - a lot worse. Users may spend tons of time trying to optimize their code or to figure out why everything is running super slow. I believe I can confidently say on behalf of maintainers of the libraries you mention that this is very much undesirable, also for alternative interpreters (most of which have been built for performance reasons).
What I see as the biggest issue with potential workarounds or changes in default behavior that have been discussed is this:
In production scenarios [1], we need reliability and optimal builds as intended by the package authors/distributors. The --only-binary behavior is desirable.
In development scenarios [2], we need things to “just work”. The fallback to sdist is very convenient.
We can’t have it both ways and do the correct thing for both (1) and (2) by simply choosing a default either in pip & co or per package. And to make that split between production and development a bit more fuzzy: for development one mostly but not always wants things to just work - e.g., when benchmarking your code, you really need production versions of your dependencies.
As for a switch per package: we actually tried that for numpy 1.26 last September, by making builds from source fail when BLAS was missing (to avoid large performance regressions). This made some people very happy, and also it rained complaints from folks where we broke CI pipelines - and having each downstream project that happened to be building from source have to add -Csetup-args="-Dallow-noblas=true" to a pip install numpy call was too disruptive. So we reverted it for the time being (xref numpy#24200).
The lack of production/development split would not be an issue if PyPI/wheels were primarily used for development, and end users used a binary-only distro for production. But that is not the world we live in - PyPI is the most popular choice for end users today, and that seems unlikely to change. So what do we do? Saying “pip is for development, use another frontend for production” isn’t realistic I think - users won’t switch. To me, the most desirable end state is:
pip (and uv, and other frontends that aim to cater to end users) default to what is best for end users / production.[3]
There is a very easy and standard way to indicate to frontends and backends that a development install is requested.
(2) will allow frontends to try building sdists by default, and it allows performance-sensitive packages to allow builds without accelerators and provide a “just works, may be slow” build.
That does indeed seem like a risk to me too, if there is not an easy enough way to switch back. I think it’s mainly a matter of keeping this in mind though, and trying to mitigate it as best we can. Between the load of failing builds, issues with silent performance regressions, and the wheel-only security benefits, improving the default behavior is too compelling to make this risk a blocker imho.
I believe @pradyunsg suggested somewhere phasing this in slowly by only changing behavior for new Python versions. No change is painless - but that may be a good enough way? Other than that, +1 to all thoughts in your post @dstufft.
that includes individual developers, scientists, etc. running their apps, data pipelines and even simple analysis scripts ↩︎
That includes CI to local development where one just needs any working version of a dependency of the package one is working on. ↩︎
There are way more end users than developers, and end users are less able to deal with potential issues, whether that is a failing build/install or a silent regression. So “end users first” seems like a fairly obvious choice to me. ↩︎
Yea that might be fine? Honestly I don’t really know!
What might be interesting is to look at what % of downloads from PyPI for an sdist are from pip to act as a proxy to figure out “how many people is this going to break” [1].
Of course, real metrics inside of pip would be even better, but given we don’t have that, this is the best we can do. ↩︎
But that’s on the maintainers. They’re the ones who would have had to implement a “install a slow Python version if the C version can’t compile”. If that’s such a footgun that users would never want it, that should be reported as a bug to the library. It has nothing to do with whether the sdist should be available and used, assuming the sdist will fail rather than install a slow version.
MarkupSafe has such a fallback, but in our case the speedup is significant but still very small, and there are other factors we consider. We build 59 platform wheels which cover most installs, and only require a C compiler on other platforms. And my understanding from the maintainers of PyPy and GraalPy is that we’re faster without the speedups there anyway. But for another library where the C is required for any performance whatsoever, it’s not clear why they decided to allow a fallback build.
Removing a footgun fallback, and then receiving angry feedback, sucks and should not be acceptable. But that’s a conversation about dealing with those types of users in an open source community and setting expectations. It’s not necessarily a reason to change the way packages are installed in general.
I can assure you it isn’t. It’s not Python vs. C, it’s BLAS vs. no BLAS (or CUDA vs. no CUDA, or whatever). Those are build dependencies that cannot even be expressed in metadata today (xref PEP 725), and are very difficult to satisfy (almost impossible on Windows, even for some maintainers).
Saying that it’s the maintainers fault for offering a footgun and that they must choose to make their sdist pretty much uninstallable isn’t helpful. There are two modes:
production - very hard to build, but performant
development - easier to build (just needs a C/C++ compiler), has a footgun that doesn’t matter for development
We need both, and we can’t read the mind of users who invoke the install command.
If BLAS or CUDA are required, why does compiling succeed if it’s they’re not there? Sure, you can’t express that they’re required in metadata, but that shouldn’t stop the compilation from failing if they’re not there.
I’m not seeing the same distinction you are. Either the user gets a pre-built wheel because they’re on a “tier 1” supported platform for whatever library they’re using, or they see a failure and need to set up their platform with the required dependencies, whatever they are. In MarkupSafe’s case, that’s a C compiler and Python headers. In another library, it might require some other things. Maintainers can try to support more platforms with wheels, or can provide instructions about what’s required for the build.
Whether those instructions are hard shouldn’t matter. We should definitely try to improve the situation, such as by adding metadata that would provide better feedback at install time about the missing non-Python parts (or help install them). But I don’t see why that would mean building from sdists should be prevented.
In the NumPy example I gave, there is a pure C fallback for BLAS/LAPACK functionality, It’s still 100x faster than pure Python, but 10x-100x slower than an optimized BLAS library.
Making most builds fail and forcing all development use cases either deal with BLAS directly or invoke builds only with a complex flag like pip install numpy -Csetup-args=xxx is not only a matter of breakage in a transition. It’s just a bad end state. The ergonomics of using --config-settings via a build frontend are not good, and almost no one can remember what such a non-default flag is.
I’m afraid you are lacking some context - MarkupSafe is too simple to serve as a useful comparison here. We have tried the experiment, and it didn’t work (it wasn’t about angry reactions, it was “this is too hard to be workable for use cases like CI”). Also based on many years of experience, I am certain that it is not possible to ask from all downstream folks who build from source (e.g. to try alpha/beta versions of CPython) to deal with BLAS. It’s the same reason why even more complex projects like PyTorch don’t upload an sdist at all - there is no point when most builds fail. But we can’t afford to not upload sdists for NumPy, that would be a major regression and in general an undesirable outcome. Popular projects uploading sdists is a good thing. We should try to increase the number of projects that upload sdists, by making it safer to do so.
So if we try switching the default again, we need at least an ergonomic way to say “give me the easy slow-ish version”.
I don’t really know what to say, aside from that it does matter to me and will matter to downstream usage. I take it as self-evident that it matters whether instructions are hard or easy.
The OP linked to Unsuspecting users getting failing from-source builds - pypackaging-native which contains a long list of reasons for getting from source builds. It’s not as simple as tier 1 vs not tier 1. Random example: you’ll be making the life of CPython core devs harder if they would try building CPython main against NumPy. When they do that, they usually don’t care about performance, they want to check some CPython C API related thing.
I on purpose did not go into detail about the full final solution, or say that all builds from sdists for all packages must be prevented. One of the main ideas in this whole discussion is a per package default of some sort. Your preference may be to keep from source builds as the default for MarkupSafe. I think there’s a class of packages for which this will be true - “needs a C or C++ compiler, and nothing else”. For more complex packages, I think the common preference will be to default to no sdist builds.
