In Python 3.13 alpha 1, I removed many private functions. In Python 3.13 alpha 2, we decided to revert around 50 private functions which caused most troubles. In the meanwhile, many public functions were added to Python 3.13 and 3.14 to replace private functions. A few examples: Py_HashBuffer(), PyLong_GetSign(), PyTime_Time(), PyDict_Pop(), etc. I added these functions to pythoncapi-compat. So it’s possible to use these new functions on Python 3.6 and newer right now using pythoncapi-compat.
What do you think of removing in Python 3.14 (alpha 2) the private functions which now have public functions to replace them?
Should it be decided on a case by case basis depending on how easy it is to upgrade the code, and how many times the function is used (e.g. in PyPI top 7,5000 projects)?
Should we keep the private functions forever?
Should we deprecate private functions first, add Py_DEPRECATED() and maybe even emit DeprecationWarning at runtime when possible? In previous discussions, this idea was not popular.
My motivation to remove private functions is to:
Clarify the scope of the C API: define better what can be used outside Python, and which API is really “internal”.
Provide better APIs, less error-prone, well tested, documented. For example, public PyDict_Pop() has a better API than private _PyDict_Pop(), the public API is easier to use and less error-prone.
Make the C API smaller, and so easier to maintain in CPython, and easier to implement in other Python implementations.
The alternative of removing private APIs for everybody is the PEP 743 – Add Py_COMPAT_API_VERSION to the Python C API which offer a way for opt-in option for a cleaner and smaller C API. It has the advantage of not affecting the default API, and only users who choose the opt-in.
I’d vote for case-by-case cleanup based on those that would actually reduce our maintenance burden.
We’ve had one request already to remove functions asap, so let’s honour that, but if the function still works and isn’t costing us anything then we may as well leave it there.
Now that we have public alternatives, we don’t have to be concerned about breaking them if we need to change something. But I see no reason to break people just for the fun of it.
I second Steve’s comment. Why risk breaking existing code needlessly? If existing non-public C-API functions get in the way, consider breaking them, as always. If they don’t hurt anyone, wait with bothering the world.
I agree with victor. In fact, too many private APIs will make public APIs that reuse these private APIs more weird. I think removing useless private APIs can make internal APIs more unified. I think necessary APIs should be split into minimal implementations, which can make reused code more flexible and easier to debug.
I think it is a good idea to use Py_COMPAT_API_VERSION, which should be similar to LINUX_VERSION_CODE. This can make some old APIs optional, depending on whether the user needs the feature.
That would be my preference. Be explicit about APIs we don’t like, and guide people toward best practices, but don’t break existing code unnecessarily.
How is providing a public macro any kind of alternative to removing private APIs? Are they private, or are they public?
Can we please try to be a bit consistent here, rather than either pushing on pet projects for unrelated reasons or trying to make all APIs treated as public by stealth?
IMO, over the decades, our messaging has been inconsistent. We’re making it consistent – you really shouldn’t use underscored functions – but new messaging is not helpful for people who inherited existing code.
I would like to break them as little as possible. When we can use public-API mechanisms (like a deprecation period or just leaving the thing in), that is the friendly thing to do.
That depends which problem solve by removing private APIs.
If we’re doing it for users, IMO the best we can do is close to what a linter does: a way to let them know they’re using API that can be removed without notice. But, not breaking their code now.
PEP 743 does that mainly for “unsafe” or hard-to-use API. (It wants to provide a machine-readable list of easily-replaceable soft-deprecated C API, at which point its main user-visible change of blocking such API becomes rather trivial, and has some advantages over leaving everything to linters.)
It would be possible to extend it to most private API (which should also trigger linters, and is “unsafe” in a slightly different way, but doesn’t need an explicit list for third-party linters). But, we’d probably need a new PEP to sort out the details.
If we’re doing it for ourselves, to reduce maintenance burden, IMO we should just leave public headers be until there’s a reason to change them.
It must be decades, because for the last decade since I’ve been a core dev it’s been very consistent. And I suspect anyone who inherited code from more than ten years ago has likely already experienced breakage of some kind, if not from us, then from their OS.
Inventing a new mechanism for warnings doesn’t help people who are ignoring their inherited code and hoping it keeps working.[1] If they’re not ignoring it, then it’s very easy to search for _Py whenever they like - arguably easier than modifying compiler settings (again, for code that has worked since the 2000’s).
I think Petr and I keep landing at the same place, which is making the minimal possible changes, and only making them for our own benefit. I just can’t rationalise going through and modifying every function definition so that it is included/excluded based on brand new optional macros as “minimal” or “our own benefit”.
I don’t know of anyone like this, but I have to assume they exist because we keep trying to support them. ↩︎
Yeah. If we need to do something, let’s not break people who didn’t ask for it. But, we don’t need to do something.
You’re right – that is intended to help other users – the kind that sets up linters to yell about wrong amounts of whitespace. (Interestingly, that kind of “code quality” tool seems more common in C-and-Python projects than grepping the C code for \b_Py. Why is that?)
That extreme is, IMO, most useful as the persona to think about when writing What’s New notes & deprecation notices, so that these docs work for everyone.
(Writing those docs is hard, and making them required might help stability a lot.)
+100 (also porting notes). I’m very much in favour of making it harder for us to change things that impact existing users at all, though preferably harder in beneficial ways such as this (and not by making people manage hundred-post long discourse threads…)
I created PR gh-128864 to deprecate 7 private C API functions which now have a public replacement:
_PyBytes_Join()
_PyDict_GetItemStringWithError()
_PyDict_Pop()
_PyThreadState_UncheckedGet()
_PyUnicode_AsString()
_Py_HashPointer()
_Py_fopen_obj()
The PR documents how to replace these private functions with public functions. The documentation points to the pythoncapi-compat project to get replacement functions on Python 3.13 and older.
The PR also schedules the removal of these functions in Python 3.18.
Update: The removal is scheduled in Python 3.18 instead of Python 3.16.
// Alias kept for backward compatibility
#define _PyBytes_Join PyBytes_Join
There is no maintenance burden for this function. There is no danger of us changing the signature or semantics – such a change would need to apply to the public function as well. If the public version is deprecated or removed, that will automatically apply to the private one too.
There is no one who will be helped by deprecating or removing this function. All the deprecation does is force users [1] to change working (though not ideal) code, for no reason that I can see.
I’d much rather have users focus their time and attention on necessary changes.
Let’s look at another one, removed just now: _PyLong_FromDigits. With this one, there is a maintenance burden to keep working.
However, the removal added no porting notes: the What’s New notice only says “_PyLong_FromDigits and _PyLong_New: use PyLongWriter_Create.”
And that’s the case for all functions removed so far: there is no mention of whether the replacements are different, and how to actually switch.
now or in the future, depending on whether they treat deprecation warnings as actionable ↩︎
In Python 3.13 and before, private functions have been removed immediately and the removal was not documented or announced. In Python 3.14, private functions are only deprecated, with a delay a 4 years to update impacted code. Moreover, only private functions with a public replacement have been deprecated, and the deprecation is documented with hints on how to update the code. So I don’t consider that this discussion has been ignored.
The maintenance burden of the C API is not the only motivation to deprecate private functions. See my first message in this thread.
You’re right, the documentation can be enhanced and completed. Contributions are welcomed
Could you be more specific? For example for _PyBytes_Join:
Clarify the scope of the C API: define better what can be used outside Python, and which API is really “internal”.
With porting notes asking people to move from _PyBytes_Join to PyBytes_Join, this should be
Provide better APIs, less error-prone, well tested, documented. For example, public PyDict_Pop() has a better API than private _PyDict_Pop(), the public API is easier to use and less error-prone.
This is a fine goal, but I don’t think it’s related to removing existing API (which might be harder to use and more error-prone, but it currently works in existing software.)
Make the C API smaller, and so easier to maintain in CPython, and easier to implement in other Python implementations.
For making the API easier to implement, it does helps to remove things, but I’m not sure if the benefits are worth the cost.
As for “make the C API smaller”, that seems like another way of saying “remove things”. I don’t think it works as motivation for removing things.
I don’t think it should work this way. It feels like you are creating work for others, at scale.
Do you think it would be fair to ask you to not merge or approve PRs that deprecate/remove functions without complete porting notes?
It seems that for some of the APIs, the main reason to remove them is to allow us to change the internals – to add new features, optimizations, or simplifications.
IMO, that’s a good reason to remove them at the time when we need to make the change.
I don’t think pre-emptively deprecating these, with a removal date set a few releases in the future, works well. If the internals need to be changed sooner, we’re either blocked or break the API before the documented time. If we don’t get around to changing the internals, but remove the API anyway, we break users without a good reason.
Would it be better to mark them as deprecated, but without a planned removal date? Then, they can be removed when necessary. (Like any private API, of course – but with a warning and with porting notes. Good porting notes in particular should help to ensure that it’s possible to switch to supported APIs, and make the switch as painless as possible.)
