Deprecating importlib.resources legacy API

Only after a hard date is determined should you start emitting the always-spams-the-wrong-people stderr by default warning for the requisite policy’s n-release cycles.

I was expecting this error not to be emitted by default but to be flagged in tests.

And in fact, the docs still give me that impression:

DeprecationWarning [is a] Base category for warnings about deprecated features when those warnings are intended for other Python developers (ignored by default, unless triggered by code in __main__ ).

In my experience, a DeprecationWarning is an ignorable warning in the short term, but a call to action (not a call to arms) to address a needed change. By default, a project under pytest will emit these warnings after the test suite run, but won’t block development. I’ll frequently use a revealed DeprecationWarning to identify the project implied in the deprecated behavior to locate or file an issue to correct the behavior and if it’s a slow-moving project, I’ll suppress the warning with a link to the filed issue to avoid being spammed about a known issue. This workflow works really well and it seems to empower projects to make progress in this complex world.

Here I don’t think there’s a one-size-fits-all solution. With the recent deprecation in importlib_metadata, it was the case, pointed out by the SQLAlchemy project (but relevant to others), that the tight constraints of allowed versions were too tight to be reasonably adopted by a library (SQLAlchemy might request >=2.0, but another library might require <2, a true conflict regardless of pip’s handling of it). I acknowledged this concern and provided a specialized workaround to help ease adoption

In the case of this deprecation, importlib_resources 1.3 is the minimum requirement, which has been out for a long time, has Python 2.7 support, and contains no backward-incompatible changes. I’d be shocked if there is a library out there that would encounter a conflict on importlib_resources>=1.3.

I suspect this is a case of over-generalizing the challenges of dealing with backports.

Unfortunately, I don’t have a crystal ball, so I can’t be certain. When it was written and published, it was presumed to be stable and dependable. It was known not to cover all known use cases, but was expected to cover most relating to resources. As it turns out, that assumption was wrong. But even then, it’s not obvious that an incomplete solution couldn’t be extended in a compatible way. It wasn’t until users reported shortcomings and we delved into the problem that we found the existing design to be inadequate for the use-case.

Barry asked me the same question about importlib_metadata, “is it stable, complete?” And my answer was the same - except for the known reported issues in the trackers, it’s stable and complete and I don’t anticipate any more backward-incompatible changes, but I don’t know what important use-cases might emerge that will drive a need to rethink the interface.

I’d like to point out that although the (former) provisional status of importlib.metadata has been raised a few times during this conversation, that status was never used to justify any change.

Is there more that’s not covered? In python/importlib_metadata#38, I did a survey of all of the usage of pkg_resources in setuptools itself and found there were quite a few usages that weren’t met directly. I suspect that the remaining cases can be dealt with in a one-off manner, but I haven’t yet had the time to enact the migration, which may reveal some other important weaknesses that one or both of the libraries should reasonably be expected to address. Still, I’d expect that a backward-compatible solution will be developed unless there’s a strong reason not to do so.

2 Likes

In my mind, I’ve gone back and forth on this possibility. And I realize I left out two crucial considerations.

First, leaving the code in place indefinitely leaves a maintenance burden of that code. It remains a surface vulnerable to bugs and security risks and simple user requests. Its mere presence adds a cognitive burden to inspecting the code. And since the code was intended to be deprecated, it’s considered frozen and allowed to be unprotected by tests.

Second, because the backport and CPython implementation are kept in sync, choosing not to remove the functionality from CPython would imply also not removing it from the backport or maintaining divergent implementations.

The goal in deprecating this functionality was to set it on the path to retirement so that what remains after some years is a clean implementation without the burden of legacy code.


Since the steering council has not given any guidance either here or in the relevant pull request, here’s my plan:

  1. Merge the pull request to introduce the deprecation warning on Python 3.11. This will give four Python versions (3.9-3.12) of cross compatibility, such that only 3.8 and 3.13 are incompatible (and thus honoring compatibility across all supported Python versions), a very generous window for functionality that’s available for Python 2.7 (via the backport).
  2. If someone volunteers to carry the maintenance burden of the legacy module (documentation, bug triage, tests, etc) and any divergence in CPython, I’ll allow that person to decide if and how to deprecate the module (including backing out the existing deprecation to a PendingDeprecationWarning or removing the warning completely).

To be clear, you didn’t ask the SC for guidance, just Barry and me on that PR. To make a formal request you should open an issue at GitHub - python/steering-council: Communications from the Steering Council or email the SC.

1 Like

Good point. Acknowledged. I was imprecise in using SC as shorthand for “members of the SC”. My presumption was that since both Barry and Brett developed the original API, they would have the best familiarity with the implementation but also would have the best instincts as to whether this issue should be brought before the SC.

I’ve identified that importlib.resources in this case is in fact meeting the more conservative standard I paraphrased above:

The backward-incompatible change should not be made until the replacement is available in all supported Python versions, and the DeprecationWarning should not be introduced until as late as possible (two minor releases prior to the removal).

Since this approach meets that conservative standard, the change under consideration isn’t even requesting an exemption from a higher, unpublished standard. I’m uninterested in driving the adoption of a stricter standard for deprecations in the stdlib, which is why I haven’t pushed to get answers to my questions above.

Because this issue has caused a good deal of consternation and hostility, I would like to get feedback from the SC, so I’ve filed a request.