# Proposal: overrides for installers

**URL:** <https://discuss.python.org/t/proposal-overrides-for-installers/23666>\
**Category:** Packaging\
**Created:** [February 8, 2023, 8:25pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666 "2023-02-08T20:25:56Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [February 8, 2023, 8:25pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/1 "2023-02-08T20:25:57Z")

</div>

This is a proposal for a potential solution to some common issues with packaging. The main idea is to let users define overrides for the package installers for the cases where the normal behavior is either broken because of poor package metadata or results in something that is not what the user actually wants.

Currently this proposal touches only on the UX aspect. It is born from questions that I keep seeing (on StackOverflow, here, and other places). And I guess this proposal represents the kind of answers I wish I could give.

I wanted to wait until this is a bit more mature. But there are other discussion threads going on right now, that I feel are closely related, so I thought I might as well publish this now and take advantage of this momentum. This still lacks structure, and looks like a brain dump rather than a fully formed proposal. This is a naive take that does not concern itself much with actual feasibility. And finally this is a work in progress, changes are to be expected.

What do I expect out of this? Personally I am not impacted by the pain points that this aims to alleviate, so in the end my interest is relatively limited. I guess, it was important to me to write down these ideas, whether something comes out of it or not. With all that said, feedback is welcome.

The full (live) document is here:

- [python-package-installers-overrides.md · GitHub](https://gist.github.com/sinoroc/2a9a319dfc6715cadabe73c7389bf2f8)

And the short of it is right below…

* * *

This is a proposal for a solution to help alleviate some frequent pain points with Python packaging. Mainly those things:

- Fix bad dependency requirements in metadata
- Fix bad build system in dependencies
- Provide more installation options per-dependency (`--pre`, `--config-settings`, `--index-url`, `--find-links`, etc.)

```plaintext
python -m pip install Application --overrides overrides.toml

```

where `overrides.toml` looks like this:

```plaintext
[Library]
requirements = [
    "LibraryNightly[cpu]",
]

[LibraryNightly]
index-url = "https://index.internal/simple/"
pre = true

```

In this scenario, the user wants to install `Application`. But in case it has a dependency on `Library` the user wants _pip_ to install `LibraryNightly` with the `cpu` _extra_ instead. Additionally the user wants that _pip_ looks for `LibraryNigtly` on an internal index exclusively, and that _pip_ considers _pre-releases_ of that `LibraryNightly`. Other potential dependencies should be handled as usual,  
including potential dependencies of `LibraryNightly`.

---

<div class="post-metadata">

**Author:** ![mdrissi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mdrissi/32/5427_2.png) [@mdrissi](https://discuss.python.org/u/mdrissi)\
**Post date:** [February 8, 2023, 9:09pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/2 "2023-02-08T21:09:34Z")

</div>

I like the idea of an override mechanism for dependencies especially for case of overly strict upper bound/bad dependency metadata. Given the very long GitHub issue on this [topic](https://github.com/pypa/pip/issues/8076) this is definitely pain point many have experienced.

The other fields (pre, config settings, etc) though feel unnecessary for a lot of common use cases and add fair amount of complexity to proposal. I’m neutral on them as unlikely to use them, but lean for first improvement on allowing dependency overrides, restricting the initial proposal to just overriding requirements would be helpful. It also lowers chance of unexpected interactions if we can view it as solely replacing one dependency constraints by a user provided one.

---

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [February 11, 2023, 7:31pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/3 "2023-02-11T19:31:03Z")

</div>

Yes, in my mind, this whole thing is meant to be split in smaller pieces (although I guess none of them would be small to implement in the installers).

The focus of this proposal is really only an idea of a UX which itself is simple to comprehend from the user’s point of view but would cover a lot of ground. If a new file format were to be added (which seems relatively likely), then I think it should be somewhat multi-purpose while still staying consistent. And I think this proposition achieves that (or at least that is the intention).

A first step could be the notation for dependency overrides for example, and then other features can be added later. Similar to `pyproject.toml`, intially it had `[build-system]` (and `[tool.*]`), then `[project]` was added later on.

---

<div class="post-metadata">

**Author:** ![FFY00](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ffy00/32/4007_2.png) [@FFY00](https://discuss.python.org/u/FFY00)\
**Post date:** [March 1, 2023, 6:57pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/4 "2023-03-01T18:57:31Z")

</div>

IMO, this makes sense. My only worry is how common this use-case is.

The config design for your `overrides.toml` might be a bit tricky, so I wonder if a Python file where you can provide a “transform” method wouldn’t just be a bit better?

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [March 1, 2023, 9:33pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/5 "2023-03-01T21:33:02Z")

</div>

This would be useful for implementing [PEP 708 - Extending the Repository API to Mitigate Dependency Confusion Attacks](https://discuss.python.org/t/pep-708-extending-the-repository-api-to-mitigate-dependency-confusion-attacks/24179), which also needs some kind of override file. @dstufft will have contributions.

---

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [March 1, 2023, 10:20pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/6 "2023-03-01T22:20:09Z")

</div>

> [@steve.dower](#):
>
> This would be useful for implementing [PEP 708 - Extending the Repository API to Mitigate Dependency Confusion Attacks](https://discuss.python.org/t/pep-708-extending-the-repository-api-to-mitigate-dependency-confusion-attacks/24179), which also needs some kind of override file.

Yes. Since I first read the Google document that is now _PEP 708_, I have wanted to investigate if and how this proposal can be rewritten to accommodate for it, maybe take over some ideas from the file format suggested in the Google document. But I have not gotten to it yet.

Also [this](https://discuss.python.org/t/adding-a-global-config-to-specify-package-indexes/8599/) came back up, and I want to re-investigate it as well.

> [@FFY00](#):
>
> My only worry is how common this use-case is.

The proposal contains many links to use cases taken from questions I have seen on StackOverflow. But I do not know if it is a good enough way of measuring how often this proposal would be used if it were to be implemented.

> [@FFY00](#):
>
> The config design for your `overrides.toml` might be a bit tricky

For sure there are plenty rough corners in this first version, I would expect some compromises to be made to keep things approachable.

> [@FFY00](#):
>
> I wonder if a Python file where you can provide a “transform” method wouldn’t just be a bit better?

I see what you mean. My first reaction is that I feel like for this proposal to be useful it should target inexperienced Python users. I feel like more experienced users are able to setup other solutions for the targeted use cases (examples: setup a `simpleindex`, rebuild wheels, fork and fix a poorly packaged library). That is why I feel like a static configuration file is a better fit than a script. But I might be looking at this from the wrong angle.

---

<div class="post-metadata">

**Author:** ![hauntsaninja](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hauntsaninja/32/12451_2.png) [@hauntsaninja](https://discuss.python.org/u/hauntsaninja)\
**Post date:** [March 2, 2023, 8:57am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/7 "2023-03-02T08:57:08Z")

</div>

> [@FFY00](#):
>
> IMO, this makes sense. My only worry is how common this use-case is.

I’ve had this use case. It looks like several users of Poetry have as well: [Ability to override/ignore sub-dependencies · Issue #697 · python-poetry/poetry · GitHub](https://github.com/python-poetry/poetry/issues/697)

---

<div class="post-metadata">

**Author:** ![mdrissi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mdrissi/32/5427_2.png) [@mdrissi](https://discuss.python.org/u/mdrissi)\
**Post date:** [March 2, 2023, 9:22am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/8 "2023-03-02T09:22:18Z")

</div>

Additionally other languages have also had common needs for dependency override mechanism. In Java using maven you have [dependency exclusions](https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html). Javascript’s yarn has similar dependency override [mechanism](https://classic.yarnpkg.com/lang/en/docs/selective-version-resolutions/). Rust’s cargo has override [mechanism](https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html).

I think if diverse set of other languages have commonly found value in this mechanism + that lengthy pip issue is good evidence that there are enough pain points where some override mechanism is useful. My personal experience is most of time I want this it’s because dependency I use has overly strict upper bounds. Occasionally I’ll even see a library uses equality pins for many of it’s dependencies treating it sorta like a lock file instead of all versions it may support. Even major libraries sometimes have done this. For a long time tensorflow had many pins/very strict requirements in it’s setup.py.

At moment my workaround to this problem is download wheel with dependencies I want to override, modify dependency metadata in wheel, re-upload wheel to private company index and intentionally shadow public version. This works but is heavy handed way that at minimum requires using your own index server. As internal index servers are commonly shared across company (or at least department+) this also means my team’s override affects all teams in company. Intentionally shadowing public package like this is also awkward for dependency confusion attacks case and I know security isn’t fond of this approach.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [March 2, 2023, 10:55am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/9 "2023-03-02T10:55:06Z")

</div>

To clarify - I assume that the intention here is for this to be a standard file, supported by _all_ installers, and not simply a pip feature request (because there’s _already_ [a pip issue](https://github.com/pypa/pip/issues/8076) for this capability)? I believe that at least pip, poetry and PDM have their own resolvers, so the intention would be for all of these to support this file? And I think pipenv and pip-tools might need support as well, although this might be easier to add, as I believe they re-use pip’s resolver. What about conda? Would you expect them to support this?

> [@hauntsaninja](#):
>
> I’ve had this use case. It looks like several users of Poetry have as well: [https://github.com/python-poetry/poetry/issues/697](https://github.com/python-poetry/poetry/issues/697)

Reading that thread, the poetry maintainers seem to have significant reservations about this functionality, and I think that mirrors the pip maintainers’ feelings (it certainly reflects mine). The relevant pip issue has already been linked. And that also links to a [further pipenv issue](https://github.com/pypa/pipenv/issues/1921). So the capability is clearly wanted, but tool maintainers have reservations, and no-one has yet managed to implement anything in a workable manner. So while discussing the issue centrally here is useful, people should understand that the lack of an existing capability isn’t just because no-one has proposed it yet…

I will also note that it is possible (not easy, by any means, but possible) to override dependency metadata right now:

1. Download the offending wheel, modify the metadata and save the modified wheel as a local version, and install using that.
2. Use `pip install --dry-run --report`, or something like `pip-tools` or `pip freeze` to do the resolution and report a fully pinned set of projects, then modify the pin for the project you want to override, and install using `--no-deps`.

Neither option is easy, but I’m not sure how easy we want to make a potentially-breaking operation like this. But I do think it’s worth noting that the proposal makes doing this _easier_, rather than allowing something that’s not possible right now.

Other practical points that I think need to be considered here:

1. It’s likely to be a lot of work to implement this, in multiple projects. Assuming that the project maintainers will implement this is unrealistic - given that the poetry and pip issues have been around for _years_, I think something would have been implemented by now if it was easy for the maintainers to do so.
2. It’s not obvious how users will determine _what_ to override - a recurring theme on the pip tracker is that the reporting of conflicts in complex cases is pretty unhelpful. We haven’t been able to work out how to give better diagnostics yet, so this is likely to be an ongoing problem. As a result, I can see us getting into a situation where people are using overrides “because someone on the internet said it might help”, without context or checking. That, IMO, is going to be far more damaging in the long run than the current situation.
3. By overriding, we make it less likely that people will report dependency issues to project maintainers. While we have seen reports that maintainers aren’t always willing to change dependencies, and there’s always a need for a “quick fix” while waiting for any upstream change, nevertheless it’s important IMO that we encourage discussion of the issues around dependency conflicts, and encourage projects to explain _why_ they choose restrictive dependency specifications. Allowing users to just override locally means that problems are much more likely to remain unreported.

> [@mdrissi](#):
>
> At moment my workaround to this problem is download wheel with dependencies I want to override, modify dependency metadata in wheel, re-upload wheel to private company index and intentionally shadow public version.

As I noted above, you could use a local version suffix (which would be seen as later than the PyPI version, and hence preferred). But yes, this is the current workaround here - as you say it’s a bit heavy handed, but it’s good to have confirmation that it works.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [March 2, 2023, 11:06am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/10 "2023-03-02T11:06:16Z")

</div>

Paul sums it up in detail, but for those who value my opinion, this seems like per-installer configuration. I _don’t want_ different installers having the same settings - that’s the point of them being different! Just like I don’t want different virtual environments (necessarily) having the same settings.

This file would be a contract between the installer and the user. It doesn’t need to be specified any more broadly than that - it’s purely UX.

---

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [March 2, 2023, 1:16pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/11 "2023-03-02T13:16:28Z")

</div>

> [@steve.dower](#):
>
> this seems like per-installer configuration. I _don’t want_ different installers having the same settings

In my mind, this is _per-invocation_: `<installer.exe> --overrides overrides.toml`, if that makes any difference. And, if we ever get this far, then yes the intention is to have the file format specified in a standard, so that such a file is portable from one installer to the other.

> [@pf\_moore](#):
>
> I assume that the intention here is for this to be a standard file, supported by _all_ installers, and not simply a pip feature request (because there’s _already_ [a pip issue](https://github.com/pypa/pip/issues/8076) for this capability)? I believe that at least pip, poetry and PDM have their own resolvers, so the intention would be for all of these to support this file? And I think pipenv and pip-tools might need support as well, although this might be easier to add, as I believe they re-use pip’s resolver. What about conda? Would you expect them to support this?

Yes. Installers would choose to support this or not. I believe the abstractions chosen in this proposal match those in use by current installers, at least in the _PyPA_ neighborhood (including _Poetry_, _pip-tools_, and so on). I have not considered _conda_, I do not know enough about it (if I am not mistaken there is a concept of _channels_, I do not know if it is relevant and if yes then I do not know how it would fit in this). But in principle, yes, if it is good for other installers, then why not try to make it work for _conda_ as well.

> [@pf\_moore](#):
>
> It’s likely to be a lot of work to implement this, in multiple projects.

Yes. I am afraid it will not be easy.

> [@pf\_moore](#):
>
> As a result, I can see us getting into a situation where people are using overrides “because someone on the internet said it might help”, without context or checking. That, IMO, is going to be far more damaging in the long run than the current situation.

Yes, true. It would be interesting to investigate how users behave around similar features in other ecosystems.

> [@pf\_moore](#):
>
> By overriding, we make it less likely that people will report dependency issues to project maintainers.

Indeed, that is a risk I have not thought of. Same as for the previous point, it could end up being an [attractive nuisance](https://blog.ganssle.io/articles/2023/01/attractive-nuisances.html) and some education would be needed.

> [@pf\_moore](#):
>
> poetry maintainers seem to have significant reservations about this functionality, and I think that mirrors the pip maintainers’ feelings (it certainly reflects mine). The relevant pip issue has already been linked. And that also links to a [further pipenv issue](https://github.com/pypa/pipenv/issues/1921)

That multiple installers (in Python and other ecosystems) all encounter the same kind of issues reinforces my thinking that collaboration over a standard file format is worthy.

I want to point that this proposal is not only about overriding dependencies. Another goal is to offer the possibility to provide installation options on a per-dependency basis instead of global such as: `--pre`, `--ignore-requires-python`, `--config-settings`, `--find-links`, `--index-url`, and so on. So this could be the “_repository file_” of _PEP 708_, and cover a lot more use cases.

I also want to state again that I am clear on the fact that for some use cases such a feature should be used as a last resort. The right ways would still be to have well behaved packages with clean metadata (including build system), to setup something like `simpleindex`. But for some use cases, I do not recall having seen any better alternative, for example for things like _Core Metadata_’s `Provides-Dist` and `Obsoletes-Dist`.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [March 2, 2023, 1:48pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/12 "2023-03-02T13:48:39Z")

</div>

> [@sinoroc](#):
>
> That multiple installers (in Python and other ecosystems) all encounter the same kind of issues reinforces my thinking that collaboration over a standard file format is worthy.

Collaboration over the format is worthwhile if the functionality is available in multipe tools. But if this proposal doesn’t get buy-in from tools to implement it (and commitment from _someone_ willing to do the work) then IMO it’s premature to define a file format.

From pip’s POV, I’m still at best +0 on having the override feature at all, and I wouldn’t commit to anything being implemented by the pip maintainers, so community resource would need to step up for this, not just to implement it but also to argue the case for the feature, and resolve questions about how it would interact with the _rest_ of pip (not all of which are going to be addressed by a generic standard).

From the sound of what you’re saying here, I think it sounds more like we should be saying “here’s a bunch of functionality that’s already implemented in multiple installers, let’s define a unified file format to control it” rather than trying to define a format for something and then hope it gets implemented (especially as there’s a risk that when we do the implementations, we find that there’s problems with the way the format is defined and need to revisit the spec anyway).

---

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [March 2, 2023, 2:23pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/13 "2023-03-02T14:23:05Z")

</div>

Absolutely. Until there is at least one proof of concept implementation in at least one or two installers, there is nothing to standardize. I guess writing down some kind of somewhat realistic file format makes it more tangible, and easier to reason about the features (at least for me).

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [March 2, 2023, 3:09pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/14 "2023-03-02T15:09:12Z")

</div>

> [@sinoroc](#):
>
> I guess writing down some kind of somewhat realistic file format makes it more tangible, and easier to reason about the features (at least for me).

It does - one thing this has done is make me think about the use cases.

It seems to me that the only valid use case for this sort of override is when building an application. You cannot expect to supply an override when installing a library, as that essentially means that your library is _by design_ not installable based on standard metadata, and I don’t see how that makes any sense. Once a library requires an override, anything using that library _also_ requires that override, all the way up the chain - so the override should be metadata, not a command-line choice. And that’s a whole different discussion.

So we’re talking here about building an application, which should have all of its dependencies pinned, ideally via a lockfile, but we don’t have those standardised yet, so typically with a fully pinned requirements file. The lockfile should be built from the environment that the application was tested in, by something like `pip freeze`, and installed using `pip install --no-deps`. So the only point where you need to override dependency metadata _should_ be when building that initial environment. And you can do that however you want - it’s tricky, because you’re fighting pip’s by-design reluctance to let you create a broken environment, but it’s possible. And honestly, I feel that making it hard is a _feature_, not a bug, precisely because the environment you’re creating is in violation of the installed projects’ declared requirements.

So having thought about this, I’m moving towards being -1 on the whole idea of an override option.

Note: I’m aware that many applications are provided as installable wheels with entry points, and as such do _not_ pin all their requirements. But that’s a compromise, using tools designed for libraries to install applications, and I don’t think we should design around this model any more than necessary. I’d rather see something like pipx adding the ability to install an app from a fully-pinned requirements file, as that allows the entry point model while retaining the idea that an app should have fully pinned requirements, but keeping within the existing design.

---

<div class="post-metadata">

**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\
**Post date:** [March 2, 2023, 3:32pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/15 "2023-03-02T15:32:34Z")

</div>

> [@pf\_moore](#):
>
> So we’re talking here about building an application, which should have all of its dependencies pinned, ideally via a lockfile, but we don’t have those standardised yet, so typically with a fully pinned requirements file. The lockfile should be built from the environment that the application was tested in, by something like `pip freeze`, and installed using `pip install --no-deps`. So the only point where you need to override dependency metadata _should_ be when building that initial environment. And you can do that however you want - it’s tricky, because you’re fighting pip’s by-design reluctance to let you create a broken environment, but it’s possible. And honestly, I feel that making it hard is a _feature_, not a bug, precisely because the environment you’re creating is in violation of the installed projects’ declared requirements.

One bit of note, is that _technically_ you need a lockfile with hashes to be assured that `pip install --no-deps` is going to do the right thing, otherwise you can end up in a situation where a new wheel is added that has different dependencies that would get installed silently.

Hopefully that’s a pretty esoteric situation though!

Personally I’m neutral on this.

I think it’s likely useful to have the ability for end users to “fix” broken metadata or to force otherwise invalid dependency sets to install for a variety of reasons, and I think that have some direct overrides mechanism makes that easier and more streamlined.

But, I also agree with @pf_moore that there is already tools available to accomplish this, albeit in a less user friendly way, so the functionality already exists. So the question then shifts from “should we allow this functionality” to “is this functionality important enough to deserve it’s own feature” as well as “is this functionality too advanced for a streamlined feature that might be a footgun to end users”?

That is where I start waffling on it. Every feature that gets added has an ongoing cost both to the maintainers of pip, and to the user’s of pip, and this starts feeling like something where the cost isn’t worth the reward.

---

<div class="post-metadata">

**Author:** ![cdce8p](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cdce8p/32/3575_2.png) [@cdce8p](https://discuss.python.org/u/cdce8p)\
**Post date:** [March 2, 2023, 4:58pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/16 "2023-03-02T16:58:38Z")

</div>

Just wanted to share some of the experiences we have for [Home Assistant](https://github.com/home-assistant/core).

As an application, we pin all first order dependency about 900 at the moment. With that many it’s (almost) impossible to create an environment without conflicts without resorting to `--no-deps` (or similar). For that reason alone, we haven’t yet been able to update to the new resolver for example and still rely on `--use-deprecated=legacy-resolver`. True it means that some requirements are technically invalid but the overwhelming majority of conflicts are just because of too restrictive library requirements.

As part of each dependency upgrade we run `pip check` and report, even fix, new issue upstream. However, that doesn’t work if the project maintainer don’t want to change the requirements or if the project hasn’t been updated in years. All the while we know that a simple metadata update would be enough.

From that POV it’s sometimes quite frustrating tbh and at least I have wished multiple times that an overwrite option would exist.

Pip already has an option to pass additional constraints to the `install` command. I would envision, overwrites could work in a similar way.

---

<div class="post-metadata">

**Author:** ![oscarbenjamin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/oscarbenjamin/32/1209_2.png) [@oscarbenjamin](https://discuss.python.org/u/oscarbenjamin)\
**Post date:** [March 2, 2023, 5:13pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/17 "2023-03-02T17:13:09Z")

</div>

> [@pf\_moore](#):
>
> It seems to me that the only valid use case for this sort of override is when building an application.

There are other valid use cases than just “building an application”. As you say it should not be possible for one library to change the dependencies of another library so it does not make sense for libraries.

So we have established that library authors cannot use overrides but otherwise they are potentially applicable to any situation where someone is installing things. That could be an application or it could be:

- Building an application
- Setting up the development dependencies for a library author.
- Installing some software for scientific computing.
- Setting up dependencies for a “folder full of scripts”.
- etc.

Basically any time you are installing stuff into an environment but not publishing a library this is potentially applicable. In some of those situations versions might all be frozen but in others it might just be someone installing things piece by piece with pip or whatever.

Looking at the GitHub issues it seems that the people who have the most problems with this are people building applications and I suspect that’s because they like to make use of lots of dependencies and have large dependency chains (library authors are more conservative about adding dependencies). Another group that shows up having problems with this are integrators like WinPython and Nixpkgs who want to try to make a single consistent environment containing the largest collection of packages possible.

The other group that appears are people who are trying to develop libraries and install development versions of different things into a single environment. For example I am working on A and need B as a development dependency but B depends (with some version constraint) on A as a runtime dependency. Now I cannot install B properly in my development environment in order to test the changes I am making to A.

You can imagine more complicated versions of where you are trying to make changes to more than one library at a time like “I want library B from gh-1234 PR and here I have library A checked out in git but B depends on A transitively through C, D, etc”. I guess in some situations manually overriding all dependencies like pip’s `--no-deps` is probably easy enough but in others you really don’t want to have to manually resolve all the _other_ dependencies.

> [@pf\_moore](#):
>
> So the only point where you need to override dependency metadata _should_ be when building that initial environment. And you can do that however you want - it’s tricky, because you’re fighting pip’s by-design reluctance

As you say one way to resolve this is to separate resolving and installing into two phases and then a tool like `pip` can be used for the install step but currently does not help with the resolve step if ultimately no versions exist satisfying all constraints.

The suggestion is for there to be some tool that can help with this situation and in principle it could be something different from pip or poetry that generates the requirements file to be used. Then again pip and poetry are dependency management tools for Python and this is a dependency management problem so it seems that those asking for this don’t see why it should be a separate tool that does this.

I personally have never needed overrides but I can see why some people do. I imagine that if I was in their situation I would want something like an overrides file which is as tightly constrained as possible like:

```auto
[overrides-A]
A == 1.0, B < 2, B < 3
A == 2.0, B < 2.1, B < 3

```

What this means is “I expect the resolver to find version 1.0 or 2.0 of package A which have constraints B \< 2 or B \< 2.1 and I want that particular constraint for A to be replaced by B \< 3 when resolving”. Then I would want a resolver or installer to print warnings about inconsistent environments. Probably the resolver should also warn if you are getting some other version of package A: a new release might make it possible to remove the override or you might need to add a new override for the new version. You would have to maintain the override with each new release of A unless it fixes the constraints to make your desired collection of dependencies consistent.

---

<div class="post-metadata">

**Author:** ![mdrissi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mdrissi/32/5427_2.png) [@mdrissi](https://discuss.python.org/u/mdrissi)\
**Post date:** [March 4, 2023, 10:47am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/18 "2023-03-04T10:47:44Z")

</div>

One related issue [here](https://github.com/pypa/pip/issues/11771). Making it easier to use simpleindex would help with current workaround of rewriting dependency metadata in wheels. That + small tool/script that downloads a wheel and rewrites dependency metadata would make the process easier and not need to live in pip at all.

I think those pieces only thing it’d be nice to have pip/packaging then would be a tutorial page explaining the approach vs current situation that most people probably would not stumble on this idea. Probably makes more sense for packaging documentation with pip just having link to it somewhere in section on dependency resolution common issues.

---

<div class="post-metadata">

**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\
**Post date:** [January 12, 2024, 7:08pm UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/19 "2024-01-12T19:08:50Z")

</div>

> [@Handling of Pre-Releases when backtracking?](https://discuss.python.org/t/handling-of-pre-releases-when-backtracking/40505/22):
>
> > [@Handling of Pre-Releases when backtracking?](https://discuss.python.org/t/handling-of-pre-releases-when-backtracking/40505/21):
> >
> > I see pre-release specifiers used in the wild in three different ways that would not work well with the `-pre` flag Pip offers
> 
> Having some sort of per-project “allow prereleases” option (`--allow-pre-for=numpy`?) would be a perfectly reasonable thing to offer, and well within the spitit of what even PEP 440 says (i.e., it’s not a “versioning v2” type of change)

This seems like it would be good a fit for this idea/proposal.

In general, I think it would solve a bunch of issues if more installation options were selectable per dependency (instead of for the whole installation session).

---

<div class="post-metadata">

**Author:** ![NeilGirdhar](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/neilgirdhar/32/5776_2.png) [@NeilGirdhar](https://discuss.python.org/u/NeilGirdhar)\
**Post date:** [January 16, 2024, 10:17am UTC](https://discuss.python.org/t/proposal-overrides-for-installers/23666/20 "2024-01-16T10:17:20Z")

</div>

A couple use cases that haven’t been mentioned:

- Specifying local (possibly editable) installs: If you wanted to replace the dependency `numpy>1.21` with a local install on your machine. That should clearly not be committed in `pyproject.toml`. It belongs in some kind of local `overrides.toml` that is added to `.gitignore`.
- Passing machine-specific options to tools. For example, to tell linters how many CPUs they should use.

[Next page](https://discuss.python.org/t/proposal-overrides-for-installers/23666.md?page=2)