In 2018, users reported a substantial deficiency in importlib.resources API that prevented a common use-case previously supported by pkgutil and pkg_resources - namely, allowing for a directory (or tree) of resources.
The solution to this issue could not be reached in an API-compatible way and required a difficult redesign of the API (the so-called “traversable” or “files()” API), which landed in importlib_resources 1.3 and Python 3.9. This new API provided a simpler design and completely superseded the previous functions and was reviewed by one or two Steering Council members.
In March of 2020, the project announced the intention to deprecate the legacy functions and in May of this year began work to make that possible.
Originally, I’d hoped the deprecation would land in Python 3.10, but time passed and Python 3.10 beta was released before the deprecation could be introduced, so the plan was modified to introduce the deprecation in Python 3.11, with the removal to occur (per policy) in Python 3.13.
This week, the
importlib_resources project released 5.3 introducing the deprecation warnings.
Some have argued that:
mass changes are being made again with aggressive deprecation breaking interfaces
I dispute this claim. The changes aren’t aggressive, but are matter-of-fact. The changes are being made slowly and deliberately. The fact of the matter is that users won’t take action until the DeprecationWarnings are raised, which is why I sought to raise these warnings soon to signal to the community that the changes are coming so they can take action.
the new apis are not better than the existing apis
This claim is flat-out wrong. This new API unlocks crucial functionality that the old API could not and does so with much a significantly simpler and more intuitive implementation (re-using pathlib semantics).
the new apis … are creating a lot of churn for developers
Acknowledged. The only alternative is either to not make changes at all or to move more slowly and create more churn for more projects by allowing the legacy implementation to replicate.
importlib_resources project provides a backport of the functionality, it allows users to have more control over the end experience than another stdlib project. For example, instead of silencing the DeprecationWarnings (an acceptable short-term workaround), libraries also have the option to pin to
importlib_resources<5.3 to avoid the deprecation. Moreover, because of the backport, the traversable API is available on Python 2.7+, so libraries and applications supporting older Pythons need only require
python_version < "3.9" in order to have compatible interfaces for all supported and recent-sunset Pythons. It’s a lot of work to maintain this backport, keeping changes in sync across the two projects, straddling the gap between pytest and unittest, de-duplicating documentation, testing everything multiple times, and more.
there is no good migration pathway without reintroducing backport packages which is unacceptable complexity.
In this particular case, the change does expand the Pythons that require backports (from Python < 3.7 to Python < 3.9). It only causes one to reintroduce backport packages where support for Python < 3.7 was already dropped, but most packages still support Python 3.6, so the complexity of the migration is small and basically boils down to:
# setup.cfg - install_requires = importlib_resources>1.3; python_version < "3.7" + install_requires = importlib_resources>1.3; python_version < "3.9"
- if sys.version_info < (3, 7): + if sys.version_info < (3, 9): import importlib_resources as resources else from importlib import resources
And then replace the usage as described in the migration guide.
That hardly seems like unacceptable complexity.
But even if a project wishes not to re-introduce the backport for the intermediate Python versions, it’s possible (and admittedly more complex), but I wouldn’t recommend it. The backport is there to help users avoid that complexity. If a project wishes to refuse the backport, that’s their prerogative and their burden.
Unfortunately, short of waiting until there are no users of Python < 3.9, making this migration is going to cause churn, and delaying the deprecation will only encourage more users to adopt the deprecated behavior.
Most libraries will have had early exposure to the deprecation and future compatibility before Python 3.11 even hits beta. As a result, almost no users are even likely to encounter the deprecation warning in CPython because they will have addressed it in
importlib_resources earlier, where there exists a great deal more flexibility for managing the incompatibility. It’s for this reason that I’d even argue that the two-release window is overly conservative for a library like importlib.resources that has an actively-maintained backport (though I’m not making that request here).
As with incompatible changes introduced in
importlib_metadata and others, I aggressively work to help users work through these important breaking changes to make the transition as smooth as possible. I’m happy to back out deprecations or breaking changes if a suitable workaround cannot be identified.
I’m reluctant to introduce
PendingDeprecationWarnings and to introduce multi-year delays in the deprecation process if all that’s likely to do is delay the inevitable churn, increase adoption of deprecated functionality, and introduce more steps in the process.
I hope this post helps clarify the thought process on this recent change and for other backport-supported functionality. I believe the project is honoring the spirit and the letter of the stdlib policy, and I welcome feedback and discussion.