PEP 387 (Backwards Compatibility Policy) updates/clarifications

I’d like to propose the following changes to our backwards compatibility policy. I see them as clarifications and/or codifying the status quo, but, you might disagree.

Thank you to other members of the 2022 Steering Council for lots of discussion on this. The remaining bad ideas are mine.

I sent this as a draft PR, see there for the current version.
The initial proposal, with more rationale, is below:

Explicit policy for undocumented things

 - Anything documented publicly as being private.
+  Note that if something is not documented at all, it is *not*
+  automatically considered private.

Our documentation doesn’t cover everything. When it’s lacking, people reach for the source code or other sources, from StackOverflow to blogs. I’m all for better stability expectation management, but for existing stuff (which might be decades old), by default we should assume that someone uses it.

Examples for extended deprecation period

-   It's fine to wait more than two releases.
+   It's fine to wait more than two releases, for example:
+   - If the expected maintenance overhead and security risk of the
+     deprecated behavior is small (e.g. an old function is reimplemented
+     in terms of a new, more general one), it can stay indefinitely
+     (or until the situation changes).
+   - If the deprecated feature is replaced by a new one, it should
+     generally be removed only after the last Python version
+     *without* the new feature reaches end of support.

It was too easy to misread the PEP as saying that Python deprecation periods are “two releases”, or “two releases by default”. Rather, I see the two releases as the minimum, a line that needs a project-wide decision to cross. (The project-wide decision is implemented/approximated as a SC ruling.)


-   [...] Compiler warnings are also acceptable. [...]
+   For C API, a compiler warning generated by the ``Py_DEPRECATED`` macro
+   is also acceptable.

The Py_DEPRECATED macro doesn’t contain the info a deprecation is supposed to have (“the release the incompatibility is expected to become the default and a link to an issue that users can post feedback to.”) It also relies on compiler extensions, so users with less common compilers won’t see it.
Ideally that should all be fixed in Py_DEPRECATED some day, but in the meantime, let’s explicitly say that Py_DEPRECATED is fine.


-2. Add a warning. [...)
+2. Add a warning to the current ``main`` branch. [...]
-  [...] (e.g. for a warning in Python 3.10, you either wait
+  [...] (e.g. for a warning in Python 3.10.0, you either wait
    until at least Python 3.12 or Python 4.0 to make the change).

Generally, adding a deprecation warning is not a bugfix so it shouldn’t go in a stable release. These changes should nudge people to not worry about older branches.


The PEP should also get a Changelog section with a link to this discussion. That’s not in the PR yet.


Someone probably does, but I’m not sure if that should extend the deprecation policy to undocumented APIs. How would I then document that an API is private or should be considered unstable? Do I have to do that with every method and variable in my code (probably at least narrowing out leading-underscore names)?


Probably leading underscore or PEP 689 – Unstable C API tier | .


Right, use a leading underscore. That’s the first thing on the PEP’s list of what’s private.


Sure, of course I get that, and of course do the same in my own code. But what about existing stdlib code that this PEP covers? Do we go a-rewriting to ensure that stdlib APIs conform to this convention, otherwise they are now all considered public? And if we do, then doesn’t that effectively break the backward compatibility constraints proposed in the PEP’s revision?

Assuming that module has actual stewards who care, you probably could.

Only if you don’t follow the deprecation policy for the old names. :wink: You can probably do a redirection where the old name raises the warning.


It wouldn’t make sense to do that pre-emptively, but if a future refactoring wanted to make changes to that “intended to be internal but not marked as such” API, that would be the time to add the missing underscore.

The old unintentionally public API would gain a code comment explaining why it still existed as a compatibility wrapper around the new clearly private API. Whether it was actively deprecated or not would depend on the specific situation.