In PEP 689, I wrote – foolishly, I now realize – that:
Any C API with a leading underscore is designated internal, meaning that it may change or disappear without any notice.
I failed to make the distinction between advice for users, and advice for core developers. I should have been much more explicit about that. This one was meant for users.
The devguide contains a (hidden and weak) hint:
Note that historically, underscores were used for APIs that are better served by the Unstable C API:
- “provisional” APIs, included in a Python release to test real-world usage of new APIs;
- APIs for very specialized uses like JIT compilers
That is, underscored API could be private, or it could just be from a time when the leading underscore wasn’t clearly defined as a “private” marker (which was, like, a year ago). It’s not easy to tell.
Unfortunately, it is quite easy to mass-remove underscored API, breaking all its users. Especially if you don’t wait for a review. It is much harder to research if each one should be removed, and harder still to revert a removal.
I argue that we should be careful and deliberate when cleaning stuff up, even if it’s more work. Partly because I care about users (some of which will properly report breakage and argue+wait for a revert, but others will get fed up and leave). And partly because I spend a lot of my time fixing avoidable breakage, and I’m frankly fed up. (And a lot of this is not volunteer work, so I can’t easily just leave and say it’s not my problem.)
Mass changes to the API leave a disproportionate amount work for other people.
I argue that for removing API, there is no rush. It would be nice to make the API cleaner, but it’s not a goal we need to reach ASAP.
It’s fine to leave an old untested function in, until someone finds the time to remove it properly.
So, let me channel my frustration into a radical draft guidance for core devs. How does this sound?
Treat any API in public headers as public.
There are exceptions, but they aren’t clear-cut. Consider them carefully. Some common exceptions include:
- Underscored functions added for 3.12 or later (but be careful about these too – e.g. the underscore could be there just to match surrounding declarations)
- A note in the docs marking it unsupported (but check the history – the note was after the API, some users haven’t seen it)
- A proper public function nearby, added at the same time or before, that’s a straightforward “frontend” to the private one
- Internal or impl in the name
- API in
Py_BUILD_CORE(fwiw, we should tack an underscore onto that macro)
Search the docs and the Internet. If you find that the API is documented, or used in a public project/tutorial, treat the API as public (i.e. deprecate it, or leave it in).
When adding a private function to a public header, let’s put
_impl in the name. Make it extra ugly so people know not to touch it.
The advice for users is still to avoid all underscored names, and report cases where it’s the only way to do something.
But, while it would be great for us if they all dropped what they’re doing and fixed their API usage now, let’s not force them to do that.