The SC accepted PEP 689 (Unstable C API tier) with a catch:
We discussed having leading underscores for this API tier, and it was decided that a leading underscore was preferred.
This subtle point puts me in the unfortunate position of authoring a PEP I don’t agree with. That would be fine – I can compromise – but additionally I don’t fully understand what the leading underscore is supposed to mean, and what the desired end state is. It turns out that we don’t have a project-wide agreement on this.
Since this was only discussed privately and verbally, I’d like to rehash/clarify some of the arguments here. I’ll frame what I originally thought was status quo as a proposal. If it sounds as status quo to you… I don’t think you’re wrong.
This was also discussed on capi-sig, with limited engagement.
My proposal is:
Underscore-prefixed names in the C API are “hands-off” for third-party code. They may change at any time. They might be useful in today’s debugging session, but may disappear tomorrow. If you need itheir functionality, ask CPython to expose it
We will not break underscored APIs unnecessarily, especially in patch releases, and especially for ones that are used or weren’t changed in a long time. But they can change in incompatible ways if necessary.
If something with an underscore is documented (or meant to be used externally), that is a bug to fix: the underscore should be removed.I this is formalized, you can grep your codebase for
'\b_Py'
to ensure you’re playing by the rules. (Or your linter can do it. Or a CPython dev looking for useful features to expose.)Internally, CPython devs should try to be friendly even to people/projects that “break the rules”. The rules should provide third-party code an opportunity to be more future-proof, not be used as an excuse to break things. Not much should actually change in CPython development.
One example (of many possible exceptions): If someone asks to expose underscore-prefixed API and we agree it’s a good idea, the old API should be treated as public/frozen in all the old versions where it appears, so it can be used with appropriate#ifdef
s (or pythonapi-compat). We should also consider keeping the old name around as a courtesy (esp. for code that predates this convention): an alias has very low maintenance overhead. (Other things being equal, it’s good to keep code working even if it doesn’t play by the rules.)
Now, what are other proposals (or versions of status quo)?
A common one (as far as I know – correct me!) is that a leading underscore is a “warning”, meaning roughly that “all authors/reviewers should read the documentation”. But the documentation for underscored functions is often missing, both for functions you can use and ones you shouldn’t.
Consider questions like:
_PyCode_GetExtra
is mentioned in PEP 523, but not documented on docs.python.org. Can I use it?_PyImport_AcquireLock
is mentioned in a StackOverflow answer, but not on docs.python.org. Can I use it?- As a core dev, I find that (say, hypothetically)
_PyArg_UnpackStack
is no longer necessary in CPython. Where do I look to see if I can remove it or change its signature?
This situation can’t be fixed easily: there are hundreds of underscored functions exposed in the public headers (and used in the wild), and no good way to prevent adding new ones (of either kind).
Some of them are exposed for technical reasons: e.g. _Py_NewRef
needs to be exposed even though we’d like users to never use it (though this particular function is not too dangerous to use.) But there’s no good way for CPython to mark something in a public/unstable API header as private.
What are your thoughts?
Back to PEP 689 (Unstable C API tier): my issue is that if a simple underscore means “warning, read the docs”, then PEP 689 is overengineered. There’s no need for an elaborate tier concept and opt-in mechanism.
(I’m aware of recent suggestions to throw away the current API and replace it by a new one. That would, of course, make all issues with the current C API and its docs, including this one, moot. Please consider that proposal off-topic in this thread.)
(And rather obviously: I’m speaking as myself here, not on behalf of the SC.)