Will setuptools remove `pkg_resources` module in the future?

Packaging
1

pkg_resource module in setuptools is marked as deprecated now. Will it be removed in the future?

If so, when will it be removed?

It says to migrate from pkg_resource to importlib.metadata and importlib.resource. But I don’t think it is a easy work. There is no drop-in replacement for Requirement and WorkingSet class in pkg_resource, is there any tutorial about it?? or does anyone know how to commit the migration?

2

Specifiers and Requirements in the PyPA packaging project appear to be the replacement for pkg_resources Requirement functionality.

WorkingSet was always rather rudimentary, and was explicitly recommended for quick n’ dirty hacky scripts rather than serious use. resolvelib is a somewhat lower-level but far more sophisticated and robust replacement, which is what pip itself uses at the low-level for resolving dependencies.

@abravalheri , maybe worth adding a mention of packaging in the deprecation notice too, since a number of other pkg_resources APIs have their replacements there as well (e.g. version parsing and handling, various utilities, etc)?

2 Likes
3

I think distlib might also have some overlapping features with pkg_resources.

1 Like
5

I’m not entirely sure how the above adds to the conversation here, as it appears to simply repeat the information in the question and offer very general advice, which is actually not correct—as mentioned, there is a (mostly) drop-in replacement for pkg_resources.Requirement, packaging.Requirement, and I’m not aware of specific migration guides on the packaging.python.org site.

Did you write the above response yourself, or was it generated with an LLM (like ChatGPT)? That would explain the issues here, as the latter can very often write answers that look right to an untrained eye, but can actually be quite incorrect, misleading and unhelpful. We suggest you either avoid doing so, or else use its output as a basis for your own writing after carefully fact-checking it and making sure the response give is actually helpful in the context of the discussion. Thanks.

5 Likes
6

I’m mostly just curious, what do people use WorkingSet for, except for getting installed package metadata? (which is well replaced with importlib.metadata and packaging)

From my experience migrating pip away from pkg_resource, most parts in WorkingSet are in general pretty broken, not usable for anything practical, and only add unwanted overhead, to be honest. But that may very likely just because most people have different use cases from pip.

7

I was once forced to use it due to me appending to sys.path and pkg_resources.working_set having a cache that I had to update because something else was using one of pkg_resources APIs that depended on it but I don’t think that counts :slight_smile:

1 Like
8

Thank you very much for the pointer @CAM-Gerlach. I will try to add this to the docs soon (if any other member of the community wants to beat me to that and submit a PR, please go ahead :smile: )

1 Like
9

Thank you very much for summarising Tzu-ping. I think that is the main point.

Some APIs in pkg_resources like WorkingSet have a lot of side effects and unwanted performance hits. No one seems to have figured out how to implement those features in a clean and performant way. Therefore, the only parts of pkg_resources that are worth migrating to are effectively the ones covered by importlib-resources, importlib-metadata and packaging.

Anything else may need some rethinking.

10

Finding this a year later, but a note:

I am updating a project (that I did not originally write) to remove pkg_resources, and finding it very trick, as I’m not that familiar with pkg_resources, nor this package.

What would help a LOT is a porting guide.

There’s docs saying that you can find most of it in importlib, or packaging, or others? – but no one-to-one mapping that I could find.

Topic at hand, and why I found this thread:

I’m trying to replace pkg_resources.working_set.iter_entry_points – and no idea how to do that.

Plugging away!

Anyway, if there is a migration guide, can someone point me to it? otherwise maybe we could start writing one? Or simply add it to the pkg_resources docs – for each function, a note to how to not use it :slight_smile:

11

Migration guide - importlib_resources 6.4.1.dev1+g6fa0a7a.d20240321 documentation is the migration guide. Not every method in pkg_resources has an equivalent and yours (entrypoints related stuff) is actually one mentioned as not having one and you’ll need an entirely new way.

12

Thanks! Sure wish I’d found that when I started this :slight_smile:

Actually, I did find a solution, and it wasn’t that different:

For the purposed of this archive – the pkg_resources code was:

from pkg_resources import working_set

checkers = working_set.iter_entry_points("compliance_checker.suites")

and the replacement is:

import importlib
checkers = importlib.metadata.entry_points(group='compliance_checker.suites')

The differences are that working_set.iter_entry_points returns an iterable (duh), and importlib.metadata.entry_points is a realized list (which is iterable, so no problem there.

The other difference is that importlib EntryPoint objects have a .load() method, instead of the setuptools EntryPoint objects, which have a .resolve() method.

So at least for that use-case, it was a pretty easy replacement.

Also – I notice that the migration guide is part of the backport of importlib – which is why I didn’t find it. Maybe it could be in the main Python docs? or a link to it? or in the setuptools docs?

So now I don’t know where to post a PR :frowning:

2 Likes
13

Unlike pkg_resources, importlib is part of the Python Standard Library. The importlib.resources documentation in the Python Standard Library documentation does actually link to the migration guide:

NOTE: This module provides functionality similar to pkg_resources Basic Resource Access without the performance overhead of that package. This makes reading resources included in packages easier, with more stable and consistent semantics.

The standalone backport of this module provides more information on using importlib.resources and migrating from pkg_resources to importlib.resources.

The pkg_resources deprecation notice, in turn, links to the importlib.resources documentation, even in pydoc3 form:

$ pydoc3 pkg_resources
Help on package pkg_resources:

NAME
    pkg_resources

DESCRIPTION
    Package resource API
    --------------------

[...8<...snip...8<...]

    This module is deprecated. Users are directed to
    `importlib.resources <https://docs.python.org/3/library/importlib.resources.html>`_
    and
    `importlib.metadata <https://docs.python.org/3/library/importlib.metadata.html>`_
    instead.

I suppose it could link directly to the migration guide as well, but since it’s going away it seems more important that the importlib.resources documentation reference the guide, which it already does.

14

So it does – so chock it up to me being an idiot – not sure how I missed that.

I suppose it could link directly to the migration guide as well

that would be good for brain-dead folks like me, but it’s really not in bad shape now – thanks!

1 Like
15

Mmm, and the hotdoc code I’m currently staring at contains:

def __get_extra_extension_classes(path):
    wset = pkg_resources.WorkingSet([])
    distributions, _ = wset.find_plugins(pkg_resources.Environment(paths))

    for dist in distributions:
        sys.path.append(dist.location)
        wset.add(dist)

    for entry_point in wset.iter_entry_points(group='hotdoc.extensions',
                                              name='get_extension_classes'):
        try:
            activation_function = entry_point.load()
            classes = activation_function()

So it does seem like code referring to WorkingSet objects (constructed or predefined) and the EntryPoints they access is maybe more common than the pkg_resources documentation optimistically assumes:

Note that you will not normally construct WorkingSet instances yourself, but instead you will implicitly or explicitly use the global working_set instance. For the most part, the pkg_resources API is designed so that the working_set is used by default, such that you don’t have to explicitly refer to it most of the time.

…and could perhaps use some explicit coverage in the migration guide. Or a link to the separate importlib.metadata migration guide — really not sure what’s gained by having those as completely separate documents — because it does seem like the two guides could at least acknowledge each other.

The importlib.metadata migration guide at least mentions importlib.resources (though not its migration guide). Unlike the importlib.resources migration guide, which contains no mention of importlib.metadata whatsoever, instead dismissively saying:

projects that use other features (e.g. entry points) will have to find other solutions.

At least for entry points, specifically, that “other solution” is not a mystery — it’s importlib.metadata!

1 Like
16 
import importlib.metadata

installed_packages = [package.name for package in importlib.metadata.distributions()]

print(installed_packages)

That gets installed packages

2 Likes
17

*nod* It took me two stabs at it, but I did eventually get hotdoc migrated from pkg_resources to importlib.metadata, with its extension support intact.

The code I posted above became something like this, in the New Normal:

from backports.entry_points_selectable import entry_points
try:
    import importlib.metadata as meta
except ImportError:
    import importlib_metadata as meta

def __get_extra_extension_classes(paths):
    path_dists = {
        path: meta.distributions(path=[path])
        for path in paths
    }

    for path, dists in path_dists.items():
        entry_points = [
            dist.entry_points.select(
                group='hotdoc.extensions',
                name='get_extension_classes')
            for dist in dists
        ]
        if entry_points:
            sys.path.append(path)

        for entry_point in itertools.chain(*entry_points):
            classes = __load_entry_point(entry_point, False)

Where __load_entry_point() does the same entry_point.load() and returns the results of executing the activation_function(). (It has more error handling than before, so I broke it out.)

18

it’s unfortunate that without pkg_resource functionality there seems to be no straightforward way to do something like

pkg_resources.require('wheel>=0.42')

Yes, one can certainly import wheel, and then manually analyze the value of wheel.__version__ - but this is extra bloat, it really should be standard, no?

19

FWIW, I’ve edited the title here to refer to the correct import name (which is the only name that this code has).

21

Hmm – I think the idea here is that requirements should be checked at install-time, rather than run-time, which is why this functionality has not been reproduced.

https://importlib-metadata.readthedocs.io/en/latest/migration.html#pkg-resources-require

And the “right” way (or, at least, the closest to pkg_resources way) to check the version of an installed package is:

>>> import importlib.metadata
>>> importlib.metadata.version('wheel')
'0.41.2'

If you do think that a runtime check is a common and useful thing, then maybe propose an addition to importlib.metadata, e.g.:

importlib.metadata.require_version(dist_name, version_spec)

(or check_version?)

However, it’s unlikely to be accepted – or it would already be there :slight_smile:

I haven’t looked, but I imagine there’s a PyPi package that does this for you.

1 Like