Provide information on installed extras in package metadata

From discussion at PyCon.

Unicorn wants to provide optional features, for example WebSocket support.
We do this via an extra, which specifies a minimum supported version of websockets.

Unfortunately if users choose to install webscockets without installing our extra, they can install any version and the only way we can decide to enable our disable our websocket support is by trying to import webscockets, parsing the version, etc.

One solution to this would be for pip and other tooling to somehow enforce versions constraints for extras even if they are not chosen, but this would probably require backwards incompatible changes and have a lot of edge cases.

Another path that would provide some relief and flexibility to package maintainers would be an API to get information about what extras were selected at runtime. This would allow us to disable our websockets feature by checking this API instead of importing websockets and parsing the version (if it is even installed).

cc @uranusjr

1 Like

See the following:

It was mentioned in the Packaging Summit that there’s currently not a way to record installed extras, and the problem we had with this is that we don’t have a way to figure out whether an extra is considered “installed” in situations like

pip install uvicorn
pip install websockets # Is uvicorn[websockets] considered installed?

But from what Adrian and I talked about, uvicorn actually don’t want to consider the websockets extra installed, because the fact that websockets is installed alone does not guarantee websockets features would work in uvicorn—it also needs to check whether the user has an appropriate version. While this is still achievable with currently available means (include the supported websockets version range in code and use importlib.metadata + packaging to verify at runtime), simply making the above question answer no is actually a reasonable and nicer solution.

In other words, we’ve been laving trouble to detect whether a declared extra is logically installed in an environment, which is not easy. However, it might be that packages actually want to know more about their extra is requested, and not detecting logically installed extras may be acceptable—just let user do a uvicorn[websockets] (which would be no-op if the extra is logically installed anyway). This makes the whole “record installed extras” operation much simpler, since now we only need to record whatever the user specified when they install a package.

I’m very interested in how people think of this.

6 Likes

My impression is that we should offer a packaging helper method that provides has_package_extra that takes the package name and the extras. This would check what libraries a given extra for that package requires (based on metadata of the package) and check if all of them are satisfied. Then uvicorn could use that method to see if a feature is enabled or not rather than just check if a module is available or not.

That would be useful to have regardless of recording of user selected extras!

Where would this functionality live? In Uvicorn we wanted to minimize dependencies, and as far as I can tell to parse versions you need either packaging or setuptools, so I’m assuming has_package_extra would depend on / require one of those two.

I think I’ve already implemented that logic here: hatch/core.py at master · ofek/hatch · GitHub

If this checks whether an extra is logically installed (i.e. are its dependencies all satisfied), it needs to go into packaging because that’s needed to parse dependency information. This is why I’m interested in the “record the extra user explicitly requested” idea; that information can be reliably populated on install time and don’t need packaging to parse, and can therefore go into importlib.metadata.

Perhaps it would be a good idea to have discrete functions for these two things so a package can freely choose which approach works better for the use case. If you’re fine telling users they need to explicitly specify foo[extra] for things to work, you can use the stricter is_extra_requested from importlib.metadata; if you want to be permissive and allow a user to populate the extra via other means, you can pull in is_extra_satisfied from packaging and do it that way.

1 Like

I’m not a packaging maintainer, but my worry is that it’s not a good fit for packaging, because it would need to read package metadata and that feels out of scope for packaging (which I understand to be essentially about implementing fundamental data types related to packaging, rather than introspecting actual packages). Ultimately it would be the maintainers’ call, though.

1 Like

I prefer explicitness as well over introspection. However, while I prefer that, it would break using extras in Nixpkgs where we install each package under a separate prefix and compose an environment during build time using PYTHONPATH. It basically means to be able to use an extra we first always need to do an installation with the extra, just to have the metadata, but then, we split dependencies again up potentially breaking that feature.

To solve this issue at our side we would need some kind of script that could create just the metadata needed separate from the wheel build or in a separate wheel so that when a user composes an environment, selects the package and its extra, the metadata gets included. Separate wheel basically meaning some kind of virtual package.

Why not also publish uvicorn-websockets that includes your own indicator (such as a uvicorn/_websocket.py file)? Then you can reference the package from the extra, and you’ll have your own marker.

Presumably the package resolvers can/should be able to handle if uvicorn-websockets also has all the dependencies necessary to make it installable directly instead of uvicorn[websockets].

This seems like a specialised enough case to warrant an additional package. If many/most uses of extras needed this kind of information, then broad-sweeping changes might be worth it, but it seems this one can be solved today.

1 Like

Correct. We have historically stayed away from doing anything related to the active environment in terms of actual stuff that is installed. The most we do is query the interpreter for marker evaluation and tag collection.

I feel like we are now discussing two different features being discussed, which is making opinions a bit hard to follow for me. Paraphrasing the two features that @uranusjr proposed in Provide information on installed extras in package metadata - #7 by uranusjr

  1. A method in importlib.metadata to introspect package metadata to check if users selected an extra
  2. A method for packaging (or some other package, maybe even 3rd party, I don’t think anyone is saying it must be in packaging) that would introspect the environment and see if all of the version constraints are satisfied for an extra

My main point with this is that these could exist separately. I understand the point of view that (2) should not go in packaging, perhaps it could go in some other module. But fundamentally I think having these tools easily available at runtime would be good for package maintainers. Other ecosystems (I’m mainly thinking of Rust) solve these problems at compile time but we can’t, so we need to do it at runtime and, as it currently stands, this is complicated to do (e.g. the suggestion above of publishing multiple packages).

1 Like

When would someone want to use 1 over 2? Isn’t the important question here is if we can activate an extra or not? I’d personally go with 1.

@brettcannon @pf_moore so where would you see makes the most sense for 2 to live?

I have no opinion, beyond what I’ve already stated that as someone who isn’t a packaging maintainer, it doesn’t seem to fit well there. Presumably one option would be that someone could publish this functionality as a new library?

I think option 1 makes more sense out of the two. Otherwise a separate project.

For what it’s worth, option 1 brought me to this thread. There’s not an obvious way (that i’m aware of) to transition a previously required dependency to being an optional extra without an immediately breaking change.

Option 1 would enable one to emit a warning until a person does depend on the extra.

Stumbled on this while trying to debug some dynamic feature activation behavior based on try: ... except ImportError: ... doing unexpected stuff.

From my perspective as a build engineer, it doesn’t make a ton of sense for a package to automagically be activating extras on the basis of dynamic dependency detection, and a package really shouldn’t be trying to dynamically verify a version solution of some sort. To me it makes far more sense to think about packages as having Cargo style features which provide additional constraints to the solver and which when selected leave an activation record of some sort behind for the configured package to flag based off of.

From the perspective of a package author, it would make the most sense for me to be able to import something from as @adriangb suggested either packaging or importlib to determine what features are active for the package containing the current module’s name.

The specific use case here is let’s say I have deps a, b and c. a and b both have optional behavior (arguably features) that use c, but I only want to activate a[c]. From my perspective it would be undesired for b[c] to become active just because the required dependencies exist. It would be much better for packages to be able to introspect their installation configuration and determine an explicitly active feature set.

1 Like