Formalize the concept of "soft deprecation" (don't schedule removal) in PEP 387 "Backwards Compatibility Policy"

Core Development
1

Hi,

I would like to specify clearly a new “soft deprecation” status in PEP 387: Backwards Compatibility Policy for functions which should be avoided for new code since these functions will not be developed further, but it’s ok to keep them in old code since their removal is not scheduled.

Over the least years, whenever I saw anything marked as deprecated for at least 2 Python releases, I eaggerly removed it as soon as possible. In Python 3.13, I started to dig deeper and found many “inactive” deprecations: deprecations only in documentation or code comments: mostly invisible to users.

Example in code (ctypes):

# XXX Deprecated
def SetPointerType(pointer, cls):
    ...

Example in the doc (C API), comment not visible in rendered HTML documentation:

.. XXX deprecated, will be removed
.. c:function:: int PyArg_Parse(PyObject *args, const char *format, ...)

Another example, I was surprised that the optparse module got deprecated in its documentation as soon as the new shiny argparse module was added to Python 2.7, whereas the getopt was not deprecated. I like how optparse is deprecated in the documentation:

Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

No removal is planned. There is no urgency for users to upgrade their code to argpase. This deprecation is there since Python 2.7 (2010) and Python 3.2 (2012): for 13 years. I proposed a PR to deprecate getopt, but CAM doesn’t seem to fully embrace the idea of “a deprecation” (even if I don’t schedule its removal).

Another tricky deprecation is the PyObject_SetAttrString(obj, attr_name, NULL) C API which can be used to delete an attribute, instead of using PyObject_SetAttrString(obj, attr_name). Serhiy Storchaka explained that this API is error-prone since the 3rd argument can be NULL because a function call. See the issue and my PR discussion for details. PyObject_SetAttrString(obj, attr_name, NULL) should be deprecated, but it’s unclear how since currently it’s valid code for 30 years.

I dislike this gray area of things not really deprecated but still deprecated. It would be nice to have a well defined status “deprecated with no plan to remove it” in PEP 387. It doesn’t mean that this function can no be removed anymore. It’s a way to announce to users that they should avoid it for new code, it’s ok continuing to use it in existing code, and there is no urgency to migrate existing usage.

Deprecation is usually implemented at different places:

  • Just a sentence in the documentation without markup
  • .. deprecated:: 3.x markup in the documentation
  • Just a sentence in the function docstring
  • DeprecationWarning emitted in the code

I propose to define a “soft deprecation”:

  • .. deprecated:: markup in the documentation, or maybe (better?) a new .. soft-deprecated:: markup in the documentation (which would link to what it means in details)?
  • Emit PendingDeprecation in the code
  • Maybe be explicit that it’s “soft deprecation” and that the feature is not scheduled for removal

For me, each deprecation change should require a new discussion:

  • not deprecated => soft deprecation: it should only impact users who check for warnings in tests, but PendingDeprecationWarning is safe to ignore, no?
  • soft deprecation (PendingDepecationWarning) => “hard” deprecation (DeprecationWarning): so far, all “hard” deprecations that I saw were always be followed by removal. So we must think ahead: how to have a smooth migration plan, how to write
  • “hard” deprecation => removal (usually, this one is easy, but not always)

Today, the most common case is to move from “not deprecated” to “hard” deprecation immediately. IMO it’s fine for most cases, especially when the code is known be error-prone, caused security vulnerabilities, is rarely used, etc.

At each status change, we should investigate how the code is used in CPython itself (remember asyncore/asynchat or imp module in the Python stdlib and test suite!), in PyPI top 5,000 projects, look for usage in the Internet (code search, questions on forums, etc.), check also for issues in the Python bug tracker, etc. It’s not because a function is still widely used that i cannot be removed. It should be a consensus.

What do you think? Should we keep this “gray area”? Is it worth it to specify a “soft deprecation”?

See also my issue: Keep deprecated functions in Python 3.13: remove this deprecation.

Victor

6 Likes
2

I like the concept of “soft deprecating”, but please don’t use PendingDeprecationWarning for it.

As a project maintainer, I want to easily distinguish between “my code will break in 2 releases, I must fix it now” and “my code uses soft deprecated APIs, it could be a good idea to update it”. Why not a third warning type, SoftDeprecationWarning (bikeshed: DiscouragedWarning?).

3 Likes
3

I noticed that more and more projects are broken when a new warning is emitted by Python stdlib, since they treat warnings as errors in their test suite.

Maybe the whole concept of PendingDeprecationWarning doesn’t work in practice, or users have to learn… how to ignore them?

I’m fine with not emitting any new warning and only implementing the soft deprecation with… documentation, that’s it.

1 Like
4

As a maintainer of libraries, I explicitly want this behavior. I want to be notified as soon as something gets deprecated, even for soft deprecations. This keeps me up to date. I always have the option (and use) the ability to file an issue and selectively ignore warnings in my tests fairly easily.

6 Likes
5

Agreed. I don’t think soft deprecations should emit any sort of warning, they should be “documentation only”.

Pip uses optparse. We cannot change to argparse, because it doesn’t support some CLI syntaxes that we use (don’t ask me what, I can’t recall). We have talked about switching to a more modern CLI parser like click, but it would be a significant job, and it’s very low on our priority list. People use pip with warnings turned into errors. We don’t necessarily like this, but they do, and we get bug reports as a result. We could of course suppress any warning that gets added, but that’s just busy-work.

After all, a “soft deprecation” warning isn’t actionable. The end user can’t do anything, and the developer has no required timescale to make any change. So what’s the action that the warning would be prompting? The only impact of a soft deprecation is “don’t use in new projects”. But it’s not possible to determine programmatically if a use is “in a new project”, so it’s not possible to make an appropriate choice about issuing a message.

By all means, let’s create a clear definition of what “soft deprecation”[1] means. And having a standard paragraph that can be used in the docs for anything that is soft deprecated seems like a good idea, as it gives a very consistent message. But please, let’s not alter the behaviour of running code that uses features that are only still there to avoid breaking running code in the first place!

  1. I don’t like that term, but let’s bikeshed later. ↩︎

6 Likes
6

If the API will be removed in two releases, it must use DeprecationWarning. not PendingDeprecationWarning.

If we don’t use PendingDeprecatedWarning for soft deprecation, we should soft deprecate PendingDeprecationWarning.

3 Likes
7

In the past, I saw some “hard” deprecations being planned over 3 or more Python releases:

  • Python 3.N: PendingDeprecationWarning
  • Python 3.N+1 and 3.N+2: DeprecationWarning
  • Python 3.N+3: remove the code
8

I like the idea of a soft deprecation. It would be useful for a number of objects in the typing module that are currently deprecated without an explicit removal timeline (typing — Support for type hints — Python 3.12.0b2 documentation). I would also prefer not to emit any warnings for soft-deprecated features.

9 Likes
9

Is it something that linters should/could react to? So that the warnings appear to the maintainers instead of the end users of the applications or scripts. I must say I do not know what the current state is. Do linters maintain their own lists of deprecated things?

10

Speaking only for pip (but I imagine we’re similar to anyone else using a deprecated-but-won’t-go-away module), we don’t want anyone telling us optparse is deprecared. We know. There simply isn’t anything we need or want to do about it.

We can of course suppress the relevant linter warning, just like we can suppress a runtime warning. But just like a runtime warning, why should we have to?

Should someone decide to rip it out, then we’d want to know. And I’d hope that any such removal would be done on the same timescales as an actively maintained module like argparse would have been - if “soft deprecated” actually means “we can decide to rip this with no warning because you shouldn’t be using it”, that’s a very different promise than the current one, and I’d expect it to be communicated as such.

11

There’s a few things being brought up in this thread.

(A) The documentation issue: Yes I think it is worthwhile making it more clear what type of “deprecation” something is. Right now the term is overloaded with a lot of meanings, so adding clarity to differentiate between “no longer recommended” (APIs we don’t recommend people to use for many possible reasons, but don’t believe it is worth planning to remove) and “stable, but we’ll never remove it and are unlikely to fix bugs or enhance it in the future” (getopt, optparse) and “actual” deprecations where we’ve got a scheduled version in which we intend to make the thing go away (ala PEP-594).

(B) The (f)utility of PendingDeprecationWarning - this topic has come up before: PendingDeprecationWarning is really useful? - our current “wisdom” is basically “don’t bother using that warning, pretend it doesn’t exist”. (hat tip to the lets soft-deprecate PendingDeprecationWarning comment above, we effectively have).

A warning being emitted should always have a justifyable reason for the action it suggests taking to avoid future breakage becoming someones priority to fix in their code.

Having code in place to emit warnings at runtime about things we already know we’ll never break does a disservice to everyone and just cries :deprecated-wolf: when there is no reason for code to be changed. That erodes trust in warnings being meaningful.

So lets not promote the addition of warnings on soft deprecations. Just guide that any form of warning should only be emitted if acting on it clearly improves the future health and longevity of the code triggering it.

For example: We’ll never warn about getopt and optparse. Those are effectively long term stable/stale modules. It isn’t even clear that they should even be called “soft-deprecated”, more that “hey, there’s a fancier thing over here in argparse - but use whichever floats your boat”.

We obviously warn about specific planned date API removals (good riddance asynchat). And we may choose to warn on APIs that “work” but we consider harmful because they’re too easy to use wrong, especially where we have an alternate API with less sharp edges. We may never remove them, thus soft-deprecation, but peoples code is healthier by not using them.

7 Likes
12

Hello Victor,

I like the concept and your proposal, with the exception of the name. What is a “soft” deprecation? A deprecation that does not issue a warning? A drepecation that will be enforced after several releases, not a couple? A deprecation that doesn’t harm you if it falls on your foot? :stuck_out_tongue:

Let’s find a better name. What about “discouraged”?

Regards,

13

My concern with skipping any sort of warning is people who discover via the REPL or copy some code from the internet which is “soft” deprecated and then come wanting a fix or change and get upset we didn’t actively notify them that the thing they were using was abandoned code.

3 Likes
14

Thanks so much, Victor, for bringing this up and for the comprehensive summary! IMO, formalizing the concept of a “soft” deprecation and a consistent, machine-readable way to explicitly document that would be a major improvement to the status quo. It would also address not only my own specific concerns around the getopt deprecation, but also the more general ones expressed on python/cpython#92564 by allowing us to provide additional clarity and specificity with deprecations and their messaging to reduce user and tool uncertainty, but without requiring committing to a specific removal version for cases where it isn’t appropriate.

As a result of your previous PR, I’d actually been considering proposing an extension to the current directives to explicitly mark a deprecation as a “soft” deprecation, and thinking about it some more in light of your comments above, ISTM that the simplest way to implement that for both user and programmatic consumption would be extending our existing custom deprecated-removed directive to support one or more special string values for the second argument in place of the removal version.

For example, we could use .. deprecated-removed:: 3.14 notplanned <message> to signal a “soft” deprecation where the API is deprecated but there are no plans to remove it for the immediate future. This would automatically modify the message to read Deprecated since version X.Y, removal not planned: <message>, with not planned being a dotted-underline tooltip (:abbr:) providing additional clarification, e.g. There are no immediate plans to remove this, but it is no longer actively developed and should be avoided in new code.

We could also add a few other special values if needed, e.g. notdecided for things that may be removed but the removal timeline is not yet decided upon, with a similar clarifying tooltip.

For distinctions as to the reason something was deprecated, e.g. because it is superseded/no longer developed (optparse) as opposed to actively bad to use (PyObject_SetAttrString(obj, attr_name, NULL)), because it is a deprecated name alias, etc., I agree this would be valuable to be explicit about in the semantic markup (which could be both read programmatically and generate an appropriate, consistent message for humans).

While we could shove in more special values to the above, after some thought, it would be a lot cleaner, consistent with current usage and more generally useful to add a :reason: option, so it would be usable with both planned and unplanned removals. It would be set to one of a few option values (e.g., notdeveloped, unsafe, alias, obsolete, superseded, etc.), and so:

.. deprecated-removed:: 3.2 notplanned MESSAGE
   :reason: alias

could render as, for instance, “Deprecated since version 3.2, removal not planned (legacy alias): MESSAGE”, with legacy alias having an automatic mouseover text describing what that means in more detail.

It wouldn’t be hard to add a :replace: option too, so you could do

.. deprecated-removed:: 3.2 notplanned
   :reason: alias
   :replace: Logger.warning

And get automatically “Deprecated since version 3.2, removal not planned (legacy alias of Logger.warning)”. We could add others, too, which would be very helpful for automation, but that’s getting a bit off topic here.

If we’re going to emit a warning where we aren’t already for these types of things, IMO it shouldn’t be the existing, established PendingDeprecationWarning (which implies it will be removed at some point in the forseeable future) but rather something new that clearly communicates the intent, and that devs can easily individually enable/disable if they want it

As for whether there should be some kind of warning in the code, I don’t have particularly strong opinion either way. It would be nice to give developers some form of runtime indication, but it could of course be noisy enough that it is more of a net nuisance than a help. Perhaps it should depend on the type of deprecation like those @gpshead mentioned — the relative cost-benefit of warning seems a lot better for “not recommended APIs” vs. just “superseded and no longer actively developed” ones, with things like name aliases falling in between—they are easier for users to find and fix than either, but are also less directly harmful.

With appropriate docs markup, though, linters could handle the warnings, which are more dev-focused, “softer” and easier to silence than warnings. I’m (slowly) working on some tooling that would allow parsing deprecation and removals out of the docs programmatically (as well as other things like the APIs were added, and other structured metadata), and in turn could then be used to detect (or in some cases even autofix) these issues. It’s not perfect, but it might be a workable middle ground between a runtime warning and a docs message (which unfortunately most devs will probably never read directly), and the proposed docs markup will substantially aid that.

15

From my point of view deprecations in general should be caught by linters, rather than by the Python interpreter at run-time. That is for the general case, and the other hand I am sure there are plenty of cases where we want the Python interpreter itself to react to deprecations.

It feels to me like it is important to get in touch with authors of linting tools to see what would make sense. I guess one important feature here is that deprecations should be easy to filter out (or in) individually or in sub-groups, over the whole project level or on a case-by-case basis per line of code.

16

For catching deprecations in linters, let me just link to my PEP 702 – Marking deprecations using the type system | peps.python.org, which is currently pending before the Steering Council.

2 Likes
17

For developers that rely more on unit testing than on static analysis [1], some kind of run-time warning of deprecations is important (even if it’s disabled by default, like DeprecationWarning is).

  1. In my modest experience, that’s a whole lot of people. ↩︎

3 Likes
18

Apparently, the “soft deprecation” expression is bad since it’s too close to “regular hard deprecation”, and usually “deprecation” is associated with a short-term or long-term plan to remove a feature. Is there a better term to say “you should not use this function but there is no plan to remove it”?

It seems like this concept should be dissociated from deprecations, so reusing the existing PendingDeprecationWaring is a bad idea since PendingDeprecationWaring means indirectly “pending removal” which is not the case here.

1 Like
19

Abandoned?

20

A possibility would be: “This function is obsolete” or perhaps “obsolescent”. The latter is much less common, and means “Being in the process of passing out of use or usefulness; becoming obsolete.”; it’s used by the C standard for example, which says:

3 Likes