While I’m not sure I’m in favour of dropping the ABI, surely the answer to this specific question is just “open source”? As in, someone else simply picks up the code and provides new builds as needed?
Depends on the extension. And you can leave that decision to the extension maintainers: they can build for Stable ABI and also compile for particular versions to speed them up.
Yes, for sure, open source is an essential aspect of any system that would entail guaranteed periodic re-compiling of code by extension authors. However, please allow some further elaboration by one who is admittedly not at all involved with the Core Development group.
If an outcome of this discussion is that periodic re-compilation will, in fact, be required, then perhaps a table of successor authors could be maintained, with each extension listed as a key in the table associated with a list of volunteers, who either agree to pick up and occasionally enhance the extension, if it becomes appropriate, or simply just accept the responsibility to re-compile it whenever necessary. When a extension becomes near due for attention, a message could go out to the current author. If there is either a negative reply or none at all, then the voluntary successor authors would get contacted, with ones who agree to offer upgrades, rather than simply re-compilation, given first priority.
The above might smooth the bumps when authors become permanently unavailable, and could potentially identify volunteers to continue to supply updates.
Not sure. Does anyone have data on how many
abi3 wheels there are on PyPI compared to minor version-specific wheels to measure the impact? We all know the Qt examples like PySide6, but without at least an initial measurement we would be guessing at impact with zero guidance.
I’m strong -1 on removing the stable ABI. It would impose a huge maintenance burden on PyPI packagers with binary extensions.
A good example is cryptography · PyPI . The package provides binary wheels for several combinations of CPU archs, platforms, manylinux, and musllinux targets. There are currently 11
cp36-abi3 binary wheels for each release. Without stable ABI @alex_gaynor and @reaperhulk would need to build 66 binary wheels for Python 3.6 to 3.11. And they only have wheels for X86_64, aarch64, and one win32 wheel for x86. The package does not provide binary wheels for PPC64 and LoongArch.
Yes, to state explicitly: in the absence of abi3 we would likely have to constrict our support window.
And we’d sooner drop Python 3.11 than we would 3.6: 3.6 is way more used for us.
abi3 is necessary to enable forward momentum in the ecosystem as a whole – it ensures that wheels are available and usable at the time of a Python release.
sqlite> select project_name, count(*) from project_files where filename like '%.whl' and filename like '%-abi3-%' group by project_name order by 2; aymara|1 cryptacular|1 pifacecam|1 procmaps|1 pybuild|1 yelp-cheetah|2 css-inline|3 diffdart|3 pybddisasm|3 pyobjc-framework-coremedia|3 pyobjc-framework-coretext|3 pyobjc-framework-security|3 pyomexmeta|3 quizdown|3 raptorq|3 safelife|3 sonicparanoid|3 lightstep-streaming|4 pyffmpeg-bin|4 pyobjc-framework-qtkit|4 pyradamsa|4 pywpsrpc|4 refl1d|4 ffilupa|5 rdp-rust|5 pnumpy|6 python-bsonjs|6 raylib|6 dockerfile|7 pyhyphen|7 setuptools-golang-examples|7 brotlipy|8 fast-histogram|8 libsass|9 light-curve-python|9 mdf-iter|9 pep272-encryption|9 qsrs|10 eclipse-zenoh|12 sunpy|12 ligo-skymap|14 pyobjc-framework-spritekit|14 scalene|15 argon2-cffi|16 pyzsf|16 cramjam|17 ncollpyde|18 datafusion|19 libimagequant|19 pynacl|19 tangram|19 milagro-bls-binding|20 nexxt|20 peak-engines|20 tornado|20 pyfast|21 entab|24 tdl|25 libtcod-cffi|26 skytools|26 sip|29 pyobjc-framework-automaticassessmentconfiguration|32 pyobjc-framework-gamecontroller|32 pyqtwebengine|32 adblock|35 portmod|35 pyobjc-framework-addressbook|35 pyobjc-framework-authenticationservices|35 pyobjc-framework-avfoundation|35 pyobjc-framework-avkit|35 pyobjc-framework-cfnetwork|35 pyobjc-framework-contacts|35 pyobjc-framework-contactsui|35 pyobjc-framework-coreaudiokit|35 pyobjc-framework-corebluetooth|35 pyobjc-framework-coredata|35 pyobjc-framework-corelocation|35 pyobjc-framework-coremediaio|35 pyobjc-framework-coreml|35 pyobjc-framework-coreservices|35 pyobjc-framework-corespotlight|35 pyobjc-framework-corewlan|35 pyobjc-framework-cryptotokenkit|35 pyobjc-framework-discrecording|35 pyobjc-framework-externalaccessory|35 pyobjc-framework-fsevents|35 pyobjc-framework-gamecenter|35 pyobjc-framework-gamekit|35 pyobjc-framework-gameplaykit|35 pyobjc-framework-imagecapturecore|35 pyobjc-framework-imserviceplugin|35 pyobjc-framework-inputmethodkit|35 pyobjc-framework-intents|35 pyobjc-framework-mapkit|35 pyobjc-framework-mediatoolbox|35 pyobjc-framework-metal|35 pyobjc-framework-metalkit|35 pyobjc-framework-modelio|35 pyobjc-framework-multipeerconnectivity|35 pyobjc-framework-network|35 pyobjc-framework-networkextension|35 pyobjc-framework-notificationcenter|35 pyobjc-framework-oslog|35 pyobjc-framework-photos|35 pyobjc-framework-photosui|35 pyobjc-framework-pushkit|35 pyobjc-framework-safariservices|35 pyobjc-framework-scenekit|35 pyobjc-framework-screensaver|35 pyobjc-framework-scriptingbridge|35 pyobjc-framework-securityinterface|35 pyobjc-framework-speech|35 pyobjc-framework-storekit|35 pyobjc-framework-syncservices|35 pyobjc-framework-systemconfiguration|35 pyobjc-framework-systemextensions|35 pyobjc-framework-usernotifications|35 pyobjc-framework-videotoolbox|35 pyobjc-framework-vision|35 pyobjc-framework-webkit|35 opencv-contrib-python|36 opencv-contrib-python-headless|36 opencv-python|36 opencv-python-headless|36 protobuf|39 nionswift-tool|40 shiboken2|40 bcrypt|41 nionui-tool|42 pyqt3d|42 pyqtdatavisualization|42 pyqtpurchasing|42 watchfiles|44 pyqtchart|46 onigurumacffi|47 qscintilla|48 reasonable|50 nsmblib|51 pyside2|52 pyoxigraph|63 dss-python|66 pyqt5|66 pycryptodomex|71 pycryptodome|72 pyffmpeg|76 murmurhash2|80 rjieba|89 hdbcli|118 pantsbuild-pants|204 glean-sdk|241 cryptography|248 tcod|263 anki|281 advent-of-code|409
That would have a serious negative performance impact on reference counting. I think requiring re-compilation is the better option.
HPy seems to have this in their design already. With their universal mode, you get ABI compatibility, and with their non-universal mode you get runtime-specific performance features. This lets extension authors develop against one API/ABI but reap the benefits of both worlds: all they need to do is decide which builds to make available.
+1 on removing the concept of the stable ABI in its current form. We should still keep changes to the ABI to a minimum (as we’ve always done) and be careful with changes that require a recompile, but not outright ban such changes going forward.
As a compromise, perhaps we could mandate keeping a subset of the ABI stable for (at least) 2 minor releases and then allow changes when progressing to the next stable ABI version.
Also note that automatic build platforms are quite common nowadays. We still don’t have one for PyPI, but in the conda world, conda-forge is doing great and does away with having to upload tens of binaries for every release.
For wheels, we package authors can use e.g. Github to create automated builds. A platform similar to conda-forge for wheels would be nice to have, though, and also allow for more security when it comes to installing wheels from PyPI.
That’s not abi3, isn’t it? I’m pretty sure HPy has its own abi that is incompatible with abi3.
@markshannon Would it help you if we would to relax PEP 384 and introduce a new stable ABI? PEP 384 promise that
abi3 will be supported throughout the entire life cycle of Python 3. The PEP is from 2009 and targets Python 3.2. Back then we did not anticipate that Python 3 will ever have two digit minor release.
We could consider to introduce a new stable ABI that would be valid for another 5 to 10 years.
I think that PEP 384 promise should remain. If the stable API is broken it surely mandates for a Python 4 (it may not be a big breakage such as Python 2->3, but it’s definitely not to be taken lightly either as I can definitely see lots of added burden on extension authors due to this change).
Particularly for my use case (pydevd/debugpy debugger), the debugger does an attach to process which will load symbols from the CPython DLL and call those as needed. If symbols from the Python DLL (stable ABI) change it would definitely break the debugger attach (for the debugger use case at least not all the needed functionality is available in the limited ABI) – note that even changes as reordering fields would break the debugger and it’d need to adapt specially (just recompiling is not an option in this case).
Note that I’m not particularly against the breakage on the stable ABI as long as it’s properly made visible and that core developers do provide a migration for the cases where the limited ABI isn’t enough at this point (I know a debugger is probably a special case and I’m OK adapting it between versions but I wouldn’t be OK in not having a way to do something that is available now).
My hope/dream is that we will sit down as a group and figure out what kind of C ABI/API we would want if we were to start from scratch (I know HPy has been held up as that API, but I don’t think we have made that as a statement instead of an idea). With that identified, we make that API available. Once we have that, we set a very long time frame (e.g. 5 years minimum, possibly a decade) for projects to migrate over to this ideal ABI/API and we provide as much help as possible in migrating. After that time frame, we drop the old ABI/APIs.
Whether we call the final result when we cull the old APIs Python 4 or not I personally don’t care, but it’s effectively what I’m suggesting we work towards.
I believe we could even support two ABIs at once for a transition period as well.
I agree… the way that I’d probably go with it is probably having the API needed and then at each new release cut some part from the stable ABI and ask the community to upgrade accordingly.
The main work is definitely on the 3rd party maintainers side and managing expectations, but cutting up the API while providing alternatives at each new release and doing it in small parts vs having it in full and asking for a full migration may ease some of that pain…
As a note, given that the problem in the stable ABI is that it ossified things and makes it hard to transition, I don’t even see as bad leaving calling the stable ABI semi-stable ABI and just leave things that don’t need to be changed up until that’s not feasible anymore because some enhancement needed to change it (so, it’d still be stable until some feature/enhancement needs to actually change it and do a lighter breakage over time).
One problem I view is that a range of tools (such as debuggers and profilers) may not be able to use a wrapper API and need access to the actual symbols that are in the CPython dll with a known memory layout for the data it needs to access.
It’s the case of the attach to in a debugger without any previous work to make the attachment work or a profiler that’s done as an external tool (i.e.: GitHub - benfred/py-spy: Sampling profiler for Python programs) – maybe HPy fits the bill, but at the minimum the API symbols must be in the CPython dll. Doing it as a third party library doesn’t cut it in this case.
Having to build binary wheels for many platforms and Python versions is super-exhausting for package maintainers (and that’s even with tooling like
cibuildwheel). I just spent a significant amount of time to port nanobind to the stable ABI so that I won’t ever have to do think about this again in the future. The API/ABI lacked features that painstakingly had to be added for this to work.
The combination of limited API + stable ABI constitutes a promise made by the Python community. It says “If you put in extra work to implement your project with this smaller set of functions, we guarantee that it will run in future Python versions without recompilation.” It therefore makes me sad to see a discussion about intentionally breaking such an important promise here.
While it’s easy to say “I have grown weary of this agreement … let’s get rid of it!”, such a course of action will cause an untold amount of tedious work downstream, and it will break the contract of stability (i.e. the whole point of this API/ABI in the first place).
In response to @pf_moore: ABI3 has just become good enough for many projects. For my projects, Python 3.12 will be the first release where the limited ABI is finally complete enough so that I can make the switch and avoid the fragility of so having many builds. Now @markshannon wants to tear it all down. This is so frustrating!
I think that the very important middle ground here between “ABI change for every feature version” and “stable for eternity” (which I don’t think anyone ever seriously intended to promise, esp. at a time when a widespread assumption was that 4.0 would follow 3.9) is what @brettcannon mentioned: the stable ABI is stable for a long time (e.g. 5-10 additional years), but can still evolve eventually to
abi4 or whatever it ends up being called.
Without taking sides, it’s probably also frustrating to try to evolve python (a very important task, I think we can agree - the alternative is stasis and eventual decay), when so much that ended up being covered by an old promise that keeps getting extended makes that nigh-impossible.
the issue is that stable ABI promised stability while it did not really solve the reasons for instability: the need to evolve Python internals, because the ABI still exposes internal details.
This can go on and on in circles. The real solution is, IMHO, to have a different API (and really stable ABI) for extensions that abstracts and hides the implementation details and is different to what CPython uses to implement itself. How did Java or C# and others manage to innovate so much in last decades while keeping binary compatibility with existing extensions all that time?
I’m confused. Stable ABI or limited API? How can the stable ABI have only just become good enough? The whole point is that it hasn’t changed… I’m guessing that other parts of the ecosystem (tooling, or the limited API, maybe?) have made it easier to build ABI3 binaries. That’s good - the stable ABI isn’t useful if people can’t use it easily - but it’s a different issue.
I don’t think we should remove the stable ABI. I think it’s clear from this discussion that it adds value for extension authors. But I am in favour of @tiran’s suggestion:
I think that if the design of the stable ABI is limiting the evolution of Python, because it exposes too many internal details, then it should change. Improving performance and other internal enhancements are important work. They just shouldn’t happen without considering the impact on extension authors. So we have to balance - but “the stable ABI will never change” goes too far in one direction, just as “let’s get rid of the stable ABI” goes too far in the other direction.
Fully agreed on that. The key point is that we stabilized something that wasn’t yet ready for the very very long haul (and that’s OK!).
I didn’t state how long a putative
abi4 should stay untouched, just that
abi3 isn’t future-ready enough to not be changed anymore, ever.
Stable ABI and limited API are intertwined. The stable ABI is stable, but not fixed. Minor releases of Python can add new features to the stable ABI. Therefore I see the stable ABI as an extension of the limited API.
An extension that uses limited API of Python 3.11 will also compiled with any Python version >= 3.11, < 4.0. An extension module compiled with limited API and stable ABI of Python 3.11 can be imported by Python >=3.11, < 4.0.