State of importlib.metadata

Poking here since everyone who might care is likely using Discourse in some form.

I’m trying to figure out how and when pip can move to using importlib.metadata in place of pkg_resources, since I understand that it’s a more modern replacement for entry points and pkg_resources.Distribution.

  • Is there any work to be done, to make it feature complete, especially on the PyPI backport side? (I see that the current release on PyPI, has an “Alpha” classifier)
  • Is there possibly a “one-to-one mapping + caveats” mapping between the interfaces of the two libraries?

/cc @jaraco

/cc @barry

Does this documentation from Python 3.8 and this documentation from the backport help?

There’s also this ticket which explores how best to maintain the backport. I think whatever resolution happens there can also happen for importlib_resources. But to answer the original question, I think @jaraco has sync’d the stdlib and backport versions now with Python 3.8b3.

Relevant quote from the ticket @barry referenced:

the problem is that we’re really trying to kill off pkg_resources and the only way I can see us doing that is to support alternatives all the way back to Python 2.7 (yes, likely even after January 2020).

Given that the context here is the question @pradyunsg asked about pip moving to importlib.metadata rather than pkg_resources, I’d take this as meaning that anything where we can find an importlib.metadata equivalent to something in pkg_resources is fair game for us to convert, and the intention is that ultimately we’ll be able to convert everything.

The question is then one of what the expectation is on when importlib.metadata will achieve feature parity with pkg_resources, because until then pip will either need to vendor both, or stick with pkg_resources. Also, as @pradyunsg mentioned, it would be nice if there was a "migrating from pkg_resources to importlib.metadata document somewhere.

Yep yep.

Additionally, I don’t directly see anything in the API to get information about a dist-info in an arbitrary location. That’s basically required for pip to switch over to using importlib.metadata.

See https://github.com/pypa/pip/blob/ef91b330cbb95998655fde2567ec8f6d707d5105/src/pip/_internal/req/req_install.py#L700-L715, which is the logic in pip 19.2, for creating a Distribution from a directory containing a dist-info or egg-info.

One thing to keep in mind is that pkg_resources is a hodgepodge of functionality. We didn’t want to propagate that approach so we’ve been splitting up the relevant functionality into separate modules. importlib.metadata does part of the work, but so does importlib.resources. We intend for them both to support as much of the pkg_resources use cases as is feasible, so if there’s something missing, then please do file bugs either on Roundup or the individual project backport pages.

One thing we know is missing from importlib.resources is the ability to have a resource in a subdirectory of an imported package. That would require us to allow slashes in the resource names, just as pkg_resources does. Right now the requirement is that the resources must live in the package directory and that the package directory be importable. I actually tried to implement this but it’s more difficult than it seems, so I abandoned it and haven’t gotten back to it. Contributions welcome of course. :slight_smile:

Can you file an issue here so that we can track that?

@barry Done!