Adding a default extra_require environment

It looks like everything in that response seems to assume that the project and project-lib packages should have their own independent copies of the shared “library” code (or the shared “gpu” code, etc.). If you actually split up your packages so that project-lib contains the library code and project depends on project-lib, then I think you avoid all the problems there and end up with something much easier to reason about than with “default extras”.

1 Like

One practical problem is it requires significant extra work to maintain multiple packages in a repository, and Python packaging lacks a good solution similar to make -C or Cargo workspace projects.

1 Like

Yes, I’d love to see packaging grow better support for multiple projects sharing the same repository, and possibly multiple wheels coming from a single sdist.

4 Likes

Sorry to bump an old thread but I can provide a use case.

I currently have package which provides a collection of ABCs and a number of implementations for each. Each of implementation comes with its own dependencies. I now want to break out each implementation into its own extra because I have users that can’t install some of the implementations, but need access to other parts of the library. So I would like to rewrite my setup.py to have extra implementation specified as an extra:

reqs = ['package1', ...]
a_reqs = [...]
b_reqs = [...]
c_reqs = [...] 

setup(
    ...
    instal_requires = reqs,
    extras_requires = dict(
        A=a_reqs,
        B=b_reqs,
        C=c_reqs,
        ALL=a_reqs + b_reqs + c_reqs
    ),
)

But this would break many downstream users install scripts as they expect package[ALL]. This would not be a problem with a
default_extras_requires (or as I prefer the shed default_requires):

setup(
    ...
    instal_requires = reqs,
    default_requires = a_reqs + b_reqs + c_reqs,
    extras_requires = dict(
        ...,
        MIN=[],
    ),
)
1 Like

Not sure if it is applicable, but we have a usecase when the dependency installation requires some specific external tooling for compiling on some platforms, and in this case, we can work without this dependency. But most other users can quite happily use the default config without thinking.
Not sure how it can be worked around with the current approach?

1 Like

If someone wants to pick this up and write a design document for this, I’d suggest doing the following:

  • Adopt vocabulary of “selecting” extras, for discussing which extras are used.
  • Add a Core Metadata key to allow “selecting” certain extras by default. (Default-Extras: default)
  • Add the ability to “unselect” an extra that is selected by default (package[-default]).
3 Likes

I’m writing a PEP for this. Would a core developer be willing to sponsor the PEP?

7 Likes

I’m happy to.

3 Likes

@pradyunsg any chance you could share a GDoc or something ? I’ll be willing to help writing the PEP.

@pradyunsg I’m also happy to help with writing or giving initial feedback if needed.

I’m slowly coming around to this idea. For Werkzeug, we’re considering adding a dependency on a production WSGI server, so people can in fact run the “dev server” in production without worry. But if they want to use a different server, it would be nice if they could opt out of the required dependency.

For example: pip install werkzeug would install hypercorn as well, but pip install werkzeug[no-server] would not install hypercorn.

1 Like

How would werkzeug[-devserver] work as syntax for this use case?

I wouldn’t be opposed to subtracting sets instead of adding a default.

It would also be nice to have the same extra on Flask and Werkzeug, and disable it on Flask to also disable it on Werkzeug.

After thinking through it, I don’t think that would be directly possible with default-extra negation as @pradyunsg has described it, as I’m not sure there’s a reliable way to do multi-level negation without ambiguity or additional syntax, assuming I understand the proposed semantics correctly (which I quite possibly don’t). You’d either have to make the rather dangerous assumption that -extraname means negating every extra any direct or transitive dependency may have defined with that name, or treat negating an extra specifying a dependency with another extra as negating just the dependency’s extra, not the whole dependency (which would be desired in some cases, but not others).

Rather, I think you’d have to define a no-server extra in Flask which contained werkzeug[-devserver] or similar, or just use [no-server] which depends on werkzeug[-devserver] in both, assuming that was supported.

I don’t think that you’d need to make the assumption that you are describing. Assume that base flask depends on werkzeug[-devserver], and flask[devserver], with devserver being a default extra, depends on werkzeug[devserver]. Deselecting devserver in flask would deselect werkzeug[devserver]; you don’t need an additive proxy extra and extras don’t need to be reflective. But…

You do need to define how extra arithmetic should work however. Logically, a package which depends on a = x[-default] and b = x[default] would depend on the union of the dependencies of a and b, which is just the dependencies of b. Therefore, once you opt into a default extra you can never opt out of it. What this means in practice is that every library which depends on a package with a default extra would be compelled to adopt the default extra pattern above (perhaps multiplicatively, for every dependency with a default extra). That’s not great.

I haven’t come around to this due to life happening and, reviewing things today, it’s now fallen pretty low on my list – if someone else wants to pick this up before me, please feel welcome to!

Adding a default extra_require environment - #127 by pradyunsg has the design/model that I’d be happy to co-author a PEP on. I expect that writing that PEP will be a case of elaborating on the model in the PEP template, and it’s likely a matter of sitting and writing 1-2 pages about it.

@pradyunsg I am also on board with your design/model suggestion and wanted to check whether you have already written up anything? If not, I may have a go at writing a draft PEP.

I think it might be nice in addition to the design you suggested to also have a special case where [] means no default extras, i.e. package[] - this is because if someone wanted to do a minimal install, it would be nice to not have to figure out how what the name of the default extras are in order to deselect them. This was suggested by various people above but I’m not sure if there was a consensus. This wouldn’t actually break backward-compatibility as far as I can see because default extras didn’t exist before anyway, so for packages that don’t use default extras, it won’t change anything.

No PEP written, if you want we can start a draft together.

I’m not 100% sold on the usecase for “no extra” though. Could you give an example ?
Most packages will give dependencies as part of install_requires=[...], the use-case behind extras_require() is more the idea of a “backend” or some optional functionality, you can have the default backend (default extra) or the backend you want to specify (specific extra). But having “no backend” is kinda a weird situation. If we talk about “optional feature/dependency”, then the default is empty.

So I guess, I don’t exactly “picture” a use-case where you would want:

  • multiple extra_deps
  • a default extra_dep
  • possibility to get no extra_dep

Please light up my torch, I’m in the dark regarding such a use-case :slight_smile:

I agree that it’s unclear what “no extras” is for. I get that it’s technically possible and the syntax suggested is currently an no-op and thus available.

However, I’d argue that it’s not all that clear that it’s supposed to mean no optional dependencies. Assuming we have a compelling use case for a deselect-all syntax, I suggest using -* for that, enabling things like package[-*, extra], to allow selecting specific stuff and also clearly indicating that a deselect all is happening.

PS: I just skimmed through the thread and no use cases for a deselect-all have been mentioned/discussed as far as I can tell. We should start with one.

Interesting. Could you present a practical example where you would be tempted to use a “deselect all”. Such a scenario would need to have:

  • Maybe some install_requires (aka. no matter what deps)
  • A default set of extras_require deps
  • A finite set of alternative extras_require which would deactivate the default if used
  • A usecase where you actively want to reject any extras_require including the default.

I honestly can’t picture such a usecase …


Plus you also have the problematic of : [-*, extra] => what does it mean ??

  1. If it’s first deselect all then pick extra, how different is it from [extra] ?
  2. What about [extra, -*] ? We don’t consider the order to be important as of today … So shall it be like 1 (order doesn’t matter)? Or shall it be like [-*] (order does matter)

I sort of feel that [-*] would be better represented as [] (empty set/list) to avoid confusion. If there’s any use case for it