That’s because this is the bit that doesn’t work 
Windows is a poor example here, because the ABI is much more reliable than other platforms (and the nature of DLL Hell on Windows is somewhat more amenable to these problems than on other platforms).
The reason that binaries don’t work with “whatever installation of Python the user has” is because you need to know about the installation in order to produce the binaries.
To fix that, we’d need to define a complete ABI (down to “how many bits in int and which order are they and how does it get written to memory when calling another function”) and then enforce every single library to also use that. And that only fixes it because “whatever installation” now only has a choice of one - the one that perfectly uses our ABI. This approach is never going to work out.
Most of the time on many modern desktop and server machines, the ABI is usually close enough that you can squint and get away with it (“wheels which work for “most” users”). Once you get into the territory where that doesn’t work, you do need to know all the details of the ABI in order to build a compatible binary for that interpreter. The easiest way to do this is to get the interpreter from the same builder you get the binary from, because they’ll have ensured it.
If it helps take some of the heat out of the discussion, I’m not in any way saying that package developers are doing things wrong, or doing a bad job. They’re trying to do a job that is literally impossible, because their own scope is too restricted to be able to do it. And many of them know it, or at least sense it, which is why they’ll feel frustrated. But it’s not their fault, and it’s not on them to solve it themselves. They’re doing an incredible job of making the most of the impossible situation they’re in, and all we want is to make that situation less stressful, firstly by acknowledging that they don’t have to carry the entire burden of solving it (and indeed, they can’t), and then by better connecting those who can solve it with those who need the solution (and who might be the ones applying the pressure to the original developers).