In Python 3.13 alpha 1, I removed many private functions. In Python 3.13 alpha 2, we decided to revert around 50 private functions which caused most troubles. In the meanwhile, many public functions were added to Python 3.13 and 3.14 to replace private functions. A few examples: Py_HashBuffer(), PyLong_GetSign(), PyTime_Time(), PyDict_Pop(), etc. I added these functions to pythoncapi-compat. So it’s possible to use these new functions on Python 3.6 and newer right now using pythoncapi-compat.
What do you think of removing in Python 3.14 (alpha 2) the private functions which now have public functions to replace them?
Should it be decided on a case by case basis depending on how easy it is to upgrade the code, and how many times the function is used (e.g. in PyPI top 7,5000 projects)?
Should we keep the private functions forever?
Should we deprecate private functions first, add Py_DEPRECATED() and maybe even emit DeprecationWarning at runtime when possible? In previous discussions, this idea was not popular.
My motivation to remove private functions is to:
Clarify the scope of the C API: define better what can be used outside Python, and which API is really “internal”.
Provide better APIs, less error-prone, well tested, documented. For example, public PyDict_Pop() has a better API than private _PyDict_Pop(), the public API is easier to use and less error-prone.
Make the C API smaller, and so easier to maintain in CPython, and easier to implement in other Python implementations.
The alternative of removing private APIs for everybody is the PEP 743 – Add Py_COMPAT_API_VERSION to the Python C API which offer a way for opt-in option for a cleaner and smaller C API. It has the advantage of not affecting the default API, and only users who choose the opt-in.
I’d vote for case-by-case cleanup based on those that would actually reduce our maintenance burden.
We’ve had one request already to remove functions asap, so let’s honour that, but if the function still works and isn’t costing us anything then we may as well leave it there.
Now that we have public alternatives, we don’t have to be concerned about breaking them if we need to change something. But I see no reason to break people just for the fun of it.
I second Steve’s comment. Why risk breaking existing code needlessly? If existing non-public C-API functions get in the way, consider breaking them, as always. If they don’t hurt anyone, wait with bothering the world.
I agree with victor. In fact, too many private APIs will make public APIs that reuse these private APIs more weird. I think removing useless private APIs can make internal APIs more unified. I think necessary APIs should be split into minimal implementations, which can make reused code more flexible and easier to debug.
I think it is a good idea to use Py_COMPAT_API_VERSION, which should be similar to LINUX_VERSION_CODE. This can make some old APIs optional, depending on whether the user needs the feature.
That would be my preference. Be explicit about APIs we don’t like, and guide people toward best practices, but don’t break existing code unnecessarily.
How is providing a public macro any kind of alternative to removing private APIs? Are they private, or are they public?
Can we please try to be a bit consistent here, rather than either pushing on pet projects for unrelated reasons or trying to make all APIs treated as public by stealth?
IMO, over the decades, our messaging has been inconsistent. We’re making it consistent – you really shouldn’t use underscored functions – but new messaging is not helpful for people who inherited existing code.
I would like to break them as little as possible. When we can use public-API mechanisms (like a deprecation period or just leaving the thing in), that is the friendly thing to do.
That depends which problem solve by removing private APIs.
If we’re doing it for users, IMO the best we can do is close to what a linter does: a way to let them know they’re using API that can be removed without notice. But, not breaking their code now.
PEP 743 does that mainly for “unsafe” or hard-to-use API. (It wants to provide a machine-readable list of easily-replaceable soft-deprecated C API, at which point its main user-visible change of blocking such API becomes rather trivial, and has some advantages over leaving everything to linters.)
It would be possible to extend it to most private API (which should also trigger linters, and is “unsafe” in a slightly different way, but doesn’t need an explicit list for third-party linters). But, we’d probably need a new PEP to sort out the details.
If we’re doing it for ourselves, to reduce maintenance burden, IMO we should just leave public headers be until there’s a reason to change them.
It must be decades, because for the last decade since I’ve been a core dev it’s been very consistent. And I suspect anyone who inherited code from more than ten years ago has likely already experienced breakage of some kind, if not from us, then from their OS.
Inventing a new mechanism for warnings doesn’t help people who are ignoring their inherited code and hoping it keeps working.[1] If they’re not ignoring it, then it’s very easy to search for _Py whenever they like - arguably easier than modifying compiler settings (again, for code that has worked since the 2000’s).
I think Petr and I keep landing at the same place, which is making the minimal possible changes, and only making them for our own benefit. I just can’t rationalise going through and modifying every function definition so that it is included/excluded based on brand new optional macros as “minimal” or “our own benefit”.
I don’t know of anyone like this, but I have to assume they exist because we keep trying to support them. ↩︎
Yeah. If we need to do something, let’s not break people who didn’t ask for it. But, we don’t need to do something.
You’re right – that is intended to help other users – the kind that sets up linters to yell about wrong amounts of whitespace. (Interestingly, that kind of “code quality” tool seems more common in C-and-Python projects than grepping the C code for \b_Py. Why is that?)
That extreme is, IMO, most useful as the persona to think about when writing What’s New notes & deprecation notices, so that these docs work for everyone.
(Writing those docs is hard, and making them required might help stability a lot.)
+100 (also porting notes). I’m very much in favour of making it harder for us to change things that impact existing users at all, though preferably harder in beneficial ways such as this (and not by making people manage hundred-post long discourse threads…)