Adding a default extra_require environment

Packaging
1

Hi everyone, first message to this forum. I hope this will be compliant with the expected format.

There has been a number of discussion opened for a long time on this topic:

Somehow related, I opened yesterday an issue on Github with a potential workaround which effectively is not working:

I would like to take this topic as a first issue and to get my hands dirty on this. If I understand things correctly, the main issue is not the technical aspect but more agreeing on how we want this feature to be exposed.

The naive ways (I listed 4, maybe there’s more) are the followings:

setup(
    name='test_pkg',
    version='1.0.0',
    extras_require={
        "None": ["numpy"],           # solution A
        None: ["numpy"],             # solution B
        "": ["numpy"],               # solution C
        ": extra == ''": ["numpy"],  # solution D
    }
)

A few questions:

  • What is the proper to push for this ? Opening a PEP ?

  • How can I help to get the conversation started and converge ?
    Every other attempts stalled and no decision was taken unfortunately.

Thanks a lot,
Jonathan

3 Likes
3

I think there are some technical challenges as well.

I got close to being able to implement this (https://github.com/pypa/setuptools/pull/1503 which specifically was trying to implement the “empty string extra” and is linked to from https://github.com/pypa/setuptools/issues/1139) but I could not find a clean, user-friendly and backwards-compatible way to do it, so ultimately I gave up.

You may want to review that PR and the conversation on it to see why that solution won’t work.

1 Like
4

Will definitely do.
So if I understand correctly, you tried solution C and it wasn’t really possible. Do you think we might have better luck with the other solutions I mentioned above ? If so, how can I get a preference order for the solutions I mentioned above ?

Example (order absolutely random, I have no personal preference. Mayyybe Solution A is my least favorite):

  • Most Preferred: Solution C
  • 2nd Preferred: Solution A
  • 3rd Preferred: Solution B
  • Least Preferred: Solution D
5

The issue is not so much about how the name of the extra is set in setup.py, but more about how it is actually stored in the metadata. I think this means that any empty-string extra is not going to work, and it also means that it doesn’t make sense to set None as the extra (what is the string equivalent in the metadata? how do we ensure the user doesn’t actually want to use it as an extra?).

Even if we get None, "None" or ": extra == ''" to work, these are fairly un-ergonomic – how are we planning to explain this to new users? Is this intuitive?

6

I think storing the metadata is not an issue, since PEP 508 is flexible enough to handle that (see pypa/pip#8686 linked by OP), and installers can be modified to work with it.

The most significant problem is indeed ergonomic though. If we want this to be the “default extra”, foo would need to include that extra, then how do users specify a non-extra-ed version of that package? And does foo[bar] select the default extra as well as bar, or disable the default extra? There are many design issues we need to solve to make this work in a usable way. This will definitely need a PEP to cover all the design problems and record the discussion around them.

4 Likes
7

Fine let’s rule out “None” which indeed is confusing, users could indeed register an extra environment called “None” for whatever reason.

If you look at how pip currently handles package resolution, you literally have extra is None when no extra is specified: https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/markers.py#L107

So IMHO, it is already implemented that way. We just need to allow users to create an environment called None.

For now the following will fail:

from setuptools import setup

extras_require = {
   None: ["numpy"]
}

setup(
    name='test_pkg',
    version='1.0.0',
    extras_require=extras_require,
)

Which leads to:

$ python setup.py bdist_wheel
error in test_pkg setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

Similarly, ":extra==''" is in my opinion perfectly understandable since the following is perfectly legal:

from setuptools import setup

extras_require = {
   ':python_version>"3.4"': ["numpy"]
}

setup(
    name='test_pkg',
    version='1.0.0',
    extras_require=extras_require,
)

Which gives the following METADATA file:

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: numpy ; python_version>"3.4"

UNKNOWN
8

I would add that even if this is perfectly legal:

from setuptools import setup

extras_require = {
   '': ["numpy"]
}

setup(
    name='test_pkg',
    version='1.0.0',
    extras_require=extras_require,
)

We obtain a METADATA file which is not what it should be:

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: numpy

And it should be:

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Provides-Extra: 
Requires-Dist: numpy ; extra == ''

UNKNOWN

This behavior makes the way extras_require inconsistent and hardly predictable.

If you manually modifies the METADATA file and try:

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Installing collected packages: test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed test-pkg-1.0.0

No crash or error, just numpy is not installed as we would like it to be.

The following could also have been possible:

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: numpy ; extra != ''

UNKNOWN

It was clooose to be working but crash:

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl[foo]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  WARNING: test-pkg 1.0.0 does not provide the extra 'foo'
  Ignoring numpy: markers 'extra != ""' don't match your environment
ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 535, in _determine_conflicts
    return check_install_conflicts(to_install)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 118, in check_install_conflicts
    package_set, should_ignore=lambda name: name not in whitelist
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 84, in check_package_set
    missed = req.marker.evaluate()
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 328, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 244, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 225, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
Installing collected packages: test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed test-pkg-1.0.0

See the message: Ignoring numpy: markers 'extra != ""' don't match your environment => It’s good.
But the rest ends up in a crash.

However, it’s important to note that if instead of doing 'extra != "" we want to do: 'extra != "foo", now it works as we expect !

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Provides-Extra: foo
Requires-Dist: numpy ; extra != 'foo'

UNKNOWN

With the scripts:

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl[foo]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  Ignoring numpy: markers 'extra != "foo"' don't match your environment
Installing collected packages: test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed test-pkg-1.0.0

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Collecting numpy; extra != "foo"
  Using cached numpy-1.19.1-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy, test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed numpy-1.19.1 test-pkg-1.0.0

Yeah ! Successfully installed numpy-1.19.1

So far this is the only possible solution to make it work. However, as shown in my issue: https://github.com/pypa/pip/issues/8686. This only work if you want to reject only one “extra”, if you need to reject many extra != "foo" and extra != "bar" this is not working.

With None value it’s a bit more tricky:

Metadata-Version: 2.1
Name: test-pkg
Version: 1.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Provides-Extra: None
Requires-Dist: numpy ; extra == "None"

UNKNOWN

No crash from the point of view of METADATA (if modified by hand), however numpy is completely ignored:

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Installing collected packages: test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed test-pkg-1.0.0

$ pip install -U "dist/test_pkg-1.0.0-py3-none-any.whl[None]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  Ignoring numpy: markers 'extra == "None"' don't match your environment
Installing collected packages: test-pkg
  Attempting uninstall: test-pkg
    Found existing installation: test-pkg 1.0.0
    Uninstalling test-pkg-1.0.0:
      Successfully uninstalled test-pkg-1.0.0
Successfully installed test-pkg-1.0.0

My hypothesis for Ignoring numpy: markers 'extra == "None"' don't match your environment, it’s that python normalize None to none, and therefore not matching.

9

Note that pip3 install trimesh[] is valid (and will currently install trimesh without any extras) and is my first intuition as to what the "" extra is.

Also note that the following fails:

pip3 download trimesh --no-deps
pip3 install trimesh-3.7.14-py3-none-any.whl[]

ERROR: Could not find a version that satisfies the requirement trimesh-3.7.14-py3-none-any.whl (from versions: none)
ERROR: No matching distribution found for trimesh-3.7.14-py3-none-any.whl
1 Like
10

Yeah I feel like we could really take many many different approaches to address the issue. I’m honestly not biased in favor of one or other. If one approach is not easily explainable to users or too hard to implement if even doable we have the luxury in this case to have a wide array of possible approaches.

I’m convinced we can find one that will satisfy everyone and be easy/intuitive to use.

So how shall we proceed?

  • We eliminate approaches one by one?

  • We make a ranking vote and see which one looks better? (If so who should even participate, I guess the 4 people in this conversation is not going to be sufficient).

I don’t really know the process with PyPA and how do you usually deal with these situations. However I’m very eager to learn :upside_down_face:

Sidenote: If someone is willing to mentor me on this PR & PEP I would really appreciate.

11

I would suggest just start writing a PEP and post here. People tend to get involved more when there’s something concrete. Don’t worry about the writing too much now; as long as you can lay out your rational, motivation, and a design that makes sense, the rest can be filled out gradually.

See also: How to propose new specs

Edit: Since setup.py syntax is not a standard (it only concerns setuptools), I would suggest starting with extending PEP 508’ extras syntax and describing how to specify a dependency with default extras.

12

One thing I would strongly suggest you include (which seems to have not been covered so far) is a motivation section that covers the use case for this feature, what a “default extra” would actually mean, and why anyone would want to use it.

Personally, I’m mostly ignoring this discussion at the moment because I can’t work out what a “default extra” is even meant to be :slightly_frowning_face:

2 Likes
13

Oh that’s a fair point. I can think of a few examples.
Thanks everyone. I’ll try to come up with something tomorrow or Wednesday. I’ll reach out if I have a doubt or a question

14

Same here. I am a bit confused about what the point would be. I see the case for maybe an application that can have different backends. Let’s say a text processor (frontend) that can output multiple formats: PDF, HTML (backends).

We could call the installation of either Txt[pdf] or Txt[html] or maybe Txt[pdf,html] and then we have one or both backends installed as well.

I see that at least 1 backend should be available but I would still argue that Txt alone should install none of the backends. User might already have the necessary dependencies for a specific backend in their environment and only want to install the frontend. The proposed default extra might be exactly the one the user doesn’t want, so why force its installation? If we consider the output of a tool like pip freeze>req.txt, then Txt alone would be listed and again pip install -r req.txt would install the default backend that the user doesn’t want. Or am I missing something? What if the user has their own implementation of the default backend, how do they install only the frontend so that the imports don’t clash?

Extras are optional and should stay optional (and also technically they can’t be exclusive or follow any other logic than just an additive list). Again, I see that at least 1 backend should be available to the frontend for it to work properly, but I believe that’s something the authors of the project should specify clearly in their documentation: “Please install either Txt[pdf] or Txt[html] or Txt[pdf,html]”. Additionally If the frontend can’t detect any backend at run-time, then it should fail gracefully with a clear message containing instructions to fix it.

15

Thanks for expressing my confusion better than I could :slight_smile:

To put it another way, if Txt has a “default extra”, how do I request the installation of Txt without that default extra (and without having to pick another, non-default, extra)?

16

Sorry for the confusion! I think part of the problem is that we’re calling it a “default extra” but the issue is that we actually want a “default extra dependency” or a “null extra dependency”.

Here’s an example. Say I have:

name="mypackage",
install_requires=["A"],
extras_require={
    "foo": ["B"],
    "bar": ["C"],
},

so

  • pip install mypackage installs ["A"]
  • pip install mypackage[foo] installs ["A", "B"]
  • pip install mypackage[bar] installs ["A", "C"]

Now, if I wanted to say “install the dependency null_extra_dependency only if no extras are requested”, how do I do it?

I want:

  • pip install mypackage installs ["A", "null_extra_dependency"]
  • pip install mypackage[foo] installs ["A", "B"]
  • pip install mypackage[bar] installs ["A", "C"]

but this isn’t possible with install_requires and extras_require.

1 Like
17

Thanks. That makes a lot more sense, and specifies what we are trying to define here.

But it still leaves open my question of why would you want to do this? The “text processor” example, with different backends as extras, is the closest I can imagine, with a “default” backend if you don’t specify anything. But doing that by having the default backend be included if you don’t specify any explicit extras still leaves the question of how would I just install the text processor with no backends (for example, if I’d developed my own backend)?

This whole discussion suggests to me that we’re trying to be too clever here. What’s wrong with not declaring extras at all, and simply saying in the documentation that users can install backends and the package will pick them up automatically?

Disclaimer: I think this about most actual uses of extras as well…

2 Likes
18

I think there is probably more than one use case that this would solve. Being able to easily swap different but compatible sub-dependencies comes to mind.

I specifically encountered a need for this with pypa/readme_renderer, where we had something like:

name="readme_renderer",
install_requires=["A", "B", "very_heavy_but_optonal_dependency"],

and we wanted a way for users to be able to “opt out” of installing the heavy dependency with an extra:

  • pip install readme_renderer installs ["A", "B", "very_heavy_but_optonal_dependency"]
  • pip install readme_renderer[lightweight] installs ["A", "B"]

We ended up doing the inverse which works but is not exactly the behavior we wanted.

1 Like
19

Thanks.

That sounds like we actually need something that’s conceptually slightly different than “extras” - install-time selectable groups of dependencies. Extras are like that, but only additively. If what we actually need is the ability to compose groups of dependencies in more complex ways, then we should look at that directly, and not just make adhoc adjustments to the “extras” idea.

The reason that I say this is that extras are already a significant additional complexity when processing package dependencies, and bolting on additional capabilities without a proper underlying model will likely result in more bugs and unexpected interactions.

IMO, we need a formal PEP that defines a mechanism for the whole exercise of defining, selecting, and recording variable dependency sets. Things to consider:

  • How do we define a “group” of dependencies?
  • What operations should be allowed when combining groups?
  • How do users state what they want?
  • Can variable dependencies be used in metadata? How do they interact?
  • How do we record what was installed? Tools like pip freeze will need this information.

Etc.

I’m basically -1 on adding any more features¹ to extras without finally biting the bullet and properly standardising their behaviour, or specifying a replacement.

¹ Even features where there’s a clear need.

1 Like
20

Same again. I feel like it’s not extras anymore but something else, and trying to make it fit into extras is risky.

Haven’t read the full thing, but looks like poetry is trying something like that:

21

To kinda-sorta pile on to what Paul is saying, extras are already a complicated special case in pip’s resolver (both old and new) and this will add more complexity to that bit of code. I don’t like that either. :slight_smile:

I think it’d be a useful exercise to give the entire “broader region” of optional/conditional dependencies in Python Packaging a bit of a revisit. I’d volunteered to start us down this path in the Packaging Mini-Summit discussions back in 2019 (in the form of a new PEP for handling extras) but that PEP never got written.