I’m not an expert in these matters, so I’m hesitant to own the implementation of these primitives, especially where there is code specific to certain compilers and platforms.
Wouldn’t it help the community as a whole for CPython to provide even a minimal set of atomic primitives? If we have to implement them ourselves, we’ve spread compiler-specific details through many projects. What happens when CPython adds support for a new compiler in the future? Wouldn’t everyone then need to discover the problem, find the solution, and copy it into their own project?
Is there a reason not to provide basic atomic int operations? They seem fundamental to correct support for free-threading.
They’re part of modern languages (C11, C++11, Rust). You need /experimental:c11atomics for MSVC, but you can add that to your project.
For CPython, we want to add atomic operations without changing data types in certain structs, but this isn’t a consideration for most projects.
We also want to build CPython without adding the MSVC /experimental:c11atomics flag, but this may change in the near future if MSVC finally gets their act together (or we ever switch to clang-cl). APIs are commitments for the long term. We don’t want to add an API that is likely to get deprecated in a year or two, especially when the alternatives are not too difficult
For you it’s just atomic_load_int and atomic_store_int, but for someone else it’s going to be pointer operations or compare-exchange, or those things plus just one more data type. I don’t think a hodgepodge public API is a good idea – that seems like a recipe for disappointment (for users) and frustration (for maintaineance).
I understand the slippery slope argument that I only need two, but others will need more. But your other three bullets amount to, “compilers vary a lot, so this will all have to change”: that’s exactly the reason I don’t want to have things like #if defined(_M_X64) || defined(_M_IX86) in my code: I don’t know what this means, I don’t know how the compilers are changing or when, and I don’t know what I will need to change them to. CPython is tracking those things and could solve the problem for extension writers.
I haven’t chosen compilers. I use setuptools.Extension, and it builds my code. It would be great if we don’t have to complicate things for all extension authors.
Is there perhaps something already exposed in the CPython APIs that I could use instead?
But your other three bullets amount to, “compilers vary a lot, so this will all have to change”
My point is nearly the opposite. They are a standard part of the languages that people use to write Python extensions.
It would be great if we don’t have to complicate things for all extension authors.
My impression is that most extensions are written in C++ or Rust and not pure C these days. So it’s a small, extra complication for a subset of extension authors.
I’m trying to provide a summary of the reasons not to provide atomic operations as part of the Python C API and why the reasons we do something internally in CPython aren’t applicable to most extensions.
I don’t feel particularly strongly about making the APIs public or not. Since the other thread I learned that it’s not worth my time to fight the battle of making them public and probably not worth your time either.
Since you’re pointing it out, I should say that I meant to publish those functions as a separate header-only package, exactly for a use case like yours.
(sorry the issue doesn’t say much)
Right now there’s no support for MSVC, but I was working on that.
Unfortunately, I cannot really give you a timeline for this because I’ll be on vacation soon. I’ll see if I can work on it this weekend.
Are load and store functions even the right API?
If you use _Atomic int for the variable, then normal loads and stores will be atomic. For more complex operations there are functions in <stdatomic.h>.
Unfortunately, on Windows this needs /experimental:c11atomics. It’s been “experimental” for almost 3 years now.
Perhaps setuptools should add the flag automatically?
Documenting exactly how to do this for the most popular build backends, probably at Updating Extension Modules - Python Free-Threading Guide would be great. It’s bad form though for build tools to add maybe-needed flags automatically - it causes log pollution, confusion and some risk of breakage with no upside for packages that don’t need it.
It’s easy enough for package authors to do this themselves when they find out they need it, especially if the to-be-written docs contain the expected compiler error so it’s easy to search for. Only a small subset of all packages with extension modules will actually need this, or this would have come up more often already over the past year.
Yes, that’s true. I had a couple of requests like this in the past, but the requester eventually found a way that was more convenient for their specific project.
The main issue is cross compatibility with different compiler toolchains. When you distribute sources that can be compiled on the user’s machine, then you may need to support a lot of compilers.
But, at least from what I know, this has not been such a substantial issue in practice. Partly also because a lot of C extensions do publish wheels, so they just need to support the toolchains in their own CI.