Idea: Allow installation of package local dependencies

Well, from your side you don’t need to ensure that every dependent
package does everything it is supposed to do for its users; all you
need to ensure is that your application works properly with one set
of dependencies on the platforms you support, which presumably your
unit/integration/functional tests, tox config and CI/CD jobs already do.

I can’t second this enough! Upstream developers should focus on doing
what they are good at: developing the application or library. As long
as “it works on my machine” it can be integrated into downstream
repositories where more rigorous testing can be performed if wanted.

1 Like

This was the scenario I was explaining in the hypothetical. And it’s not that hypothetical when dealing with very large requirements lists. E.g. I was going through the dependencies of homeassistant and there are at least 4 conflicting transitive requirements on the library click shared across 20+ package, there is no easy way to update many of the dependencies because of this “dependency hell” on click sometimes leaving you stuck on packages that are years out of date.

Here is a real world example of the security issue hypothetical: Support force resolving dependency versions a la yarn's "resolutions" support. · Issue #4530 · pypa/pipenv · GitHub . oci didn’t prioritize updating cryptography so if you depended on oci you were stuck on a vulnerable version of cryptography, if another library you depended on exposed the cryptography vulnerability then you are just out of luck and vulnerable.

That said I am largely convinced from this discussion that the idea of having multiple version installs in Python would be very difficult to design without creating other problems outweighing my the problem I am trying to solve. IMO though the security point you’re making though is incorrect and the current situation is worse than a hypothetical Python that allowed installs of multiple library versions, but I’m not a security expert so maybe there’s some nuance or subtlety here I am missing.

I’m now of the opinion that the better solution for the problem I stated would to have installer tools like pip provide functionality that allowed users to override downstream library requirements. That way users can force certain versions of requirements that suited their needs and if it broke their library that’s on them.

The problem here is that you would still be exposing a vulnerable version of cryptography through oci requiring it, so you’d still be vulnerable. With the status quo, downstream dependents are able to specify minimum versions of libraries they require for security reasons, and will be immediately notified if their upstream dependencies don’t support it, so they know if they are vulnerable. At that point, they can assist (or at least pressure) the oci maintainers to upgrade to a secure version, and if the problem persists, they can consider dropping, forking or taking other necessary action to deal with the unmaintained, insecure dependency.

With the proposed feature at the link you included above, packages/users would be able to force-upgrade the transitive dep to a secure version, overriding the pin lower in the stack and gaining a secure version, at the cost of accepting any breakage in the dependency.

On the other hand, if dependencies could simply stay on old versions forever without having to upgrade with the rest of the ecosystem, there would be much less pressure and incentive to upgrade to a secure one, and it could remain on the old version indefinitely, with no real way for the downstream dependent to do anything about it other than fork or drop, exactly the same options they would have had previously (and exactly what allowing packages to depend on different dep versions is supposed to avoid in teh first place). Furthermore, the issue is less obvious as there won’t be an immediate conflict, and will be unable to be directly enforced (or, with the above proposal, overridden) by the requirements specifies of dependents, at least within introducing even more complexity.

Indeed, and that’s exactly what the issue you linked proposes. I’d suggest describing your need and usecase there, or in the issue crosslinked therein, to help further motivate that approach.

By updating cryptography for other packages it still reduces the attack vector to just oci, so at worst it’s an improved security scenario.

But actually in this case (which funnily enough exactly matched my hypothetical) the user wasn’t using the oci functionality that used that used cryptograpgy so updating it for other packages would reduce the attack vector is reduced to zero.

oci didn’t prioritize updating cryptography so if you depended
on oci you were stuck on a vulnerable version of cryptography,
if another library you depended on exposed the cryptography
vulnerability then you are just out of luck and vulnerable.

oci pins every single version of dependency, which is, by definition,
downstream-hostile. Distros will patch the requirement list upon
inclusion and cryptography shall be updated when needed.

My point was, and I think you also recognized it, providing a vendoring
mechanism would only encourage more libraries to do similarly. I think
I haven’t been clear enough, upstream developers shouldn’t be in the
position to choose which specific version; they should only specify
which releases do not work. It’s not up to them to decide which to
upgrade, but try to provide a broader range of compatibility (which,
most of the times, doesn’t require any action).

I’m now of the opinion that the better solution for the problem
I stated would to have installer tools like pip allows provide
some functionality that allowed you to override downstream
library requirements in these cases. That way you could force
certain versions of requirements that suited your needs and
if it broke your library that’s on you.

No user wants to or has the knowledge to hand-pick every dependency
version (source: I use software too). Why don’t you question the value
of strict requirements? What are they for, exactly? If they are to
ensure the software works on someone elses machine along with other
software, they will have to fall in to a common subset through human
communication. If they’re simply for CI/CD stability, they are hiding
the problems instead of fixing. Remember: we use the toolings to serve
the end-users, not the other way around.

FWIW both home assistant and airflow are packaged for NixOS. click
requirements are communicated with upstream and as a last result
are relaxed downstream if in reality there is compatibility. Upstreams
are often anxious on a combination they can’t test, while the
integration must be done downstream any way.

If there were a way to only allow use of such a feature when the user had fully audited their application, and verified that the overrides they applied caused no issues, then maybe I’d agree. But in reality, people will blindly override, just “to fix the install”, and then report bugs to packages and/or pip when their resulting installation fails to work.

I think where we are now: you either follow the dependencies strictly or you opt-out of any/all help is a reasonably good spot.

I can see the appeal of wanting to relaxing dependency constraints. That’s literally an open issue on pip’s issue tracker. Here’s a relevant comment on there: Relaxing / Ignoring constraints during dependency resolution · Issue #8076 · pypa/pip · GitHub

To reiterate, I don’t think we’re adding anything new to this topic that hasn’t been said already.

1 Like

I don’t disagree there are pros and cons, I personally think the pros outweigh the cons and users should be allowed to shoot themselves in the foot if they wish, but it’s a discussion for another thread, I don’t want to confuse this discussion too much by pivoting in to something else. If I have something new to add to the existing discussions I will do it there.

Isn’t that way to generate (i.e. pip-compile) the full list of intended dependencies, manually resolve conflicts in a big requirements.txt file, and then install it with --no-deps? That’s how I deal with this issue when it arises at work.

Perhaps if pip’s resolver could bail out and print the “best reqs.txt we could come up with and it has conflicts marked in it” then more people would discover this option?

3 Likes

I get the point you’re making but it seems more like a philosophical than pragmatical to me. That it’s about ideals about what a library author should do and making decisions based on that about what tools to give them so they can’t hurt themselves as much.

I think I’m coming from the opposite philosophical view point that users should have unsafe tools and if they make a bunch of damage with those tools that’s on them.

Further with the tools library authors have now they are can and do cause dangerous situations for users. and users can’t easily solve because they don’t have access to less safe tools that would solve them.

Maybe that metaphor is extended too far, but I do get what you mean.

I’d be curious how this achieved and if it’s just the core homeassistant or the full install.

I spent quite sometime digging through their dependency tree and found several cases where you can’t match requirements. For example the requirement mycroftapi pins websocket-client 0.44.0 but there are at least 4 conflicting requirements that need a newer version. Also pysmarty pins pymodbus 1.5 but pystiebeleltron requires pymodbus>=2.1.0.

There are further conflicts on aiohttp , websockets, boto3 and others that took me a lot of manual effort to find a solution for.

Good point, yes that would work. And because it needs --no-deps it’s clear (well, as clear as it’s likely to get) that you’ve deliberately chosen not to respect the dependency metadata that the packages provide.

That’s not really how pip’s resolver works - we can reach a point where we know we’re never going to find a solution even though we’ve only looked at a small number of requirements (as an extreme case, A depends on B 1.0, but B hasn’t released a 1.0 version).

I’d be happier to leave this sort of thing to a dedicated external tool, like pip-tools (or maybe a new tool designed specifically for this task), that can just as easily spit out a requirements file, and which can be focused on producing the best “partial resolve” it can - rather than overloading pip with even more functionality that is only useful for a small proportion of users.

2 Likes

I think I’m coming from the opposite philosophical view point
that users should have unsafe tools and if they make a bunch
of damage with those tools that’s on them.

This I can agree on. However, I believe we should not encourage
developers to go in the direction that increasingly push end-users
to dangerous situation.

I’d be curious how […] achieved and if it’s just
the core homeassistant or the full install.

At the first glance it seems to be the full instalation,
though as you may expect it has to be patched here and there. Nix/Guix
are a bit more flexible than most distros in the way that each program
can have its own root, but even so multi-versioning is minimized
for the maintainability I described earlier.

I spent quite sometime digging through their dependency tree […]
There are further conflicts on […] that took me a lot of manual
effort to find a solution for.

What you did was part of a downstream maintainer’s job and this is why I
suggested installing from a distro would best benefit the community
and save duplicated effort. One can choose to maintain a downstream
repository and that’s perfectly fine, but that doesn’t mean that
most people should spend time doing it, and this applies to software
outside of Python ecosystem as well.

Upstream requirements are to communicate the baseline before the package
is tightly integrated in a downstream repo. They are toolings and
conflicts are part of the issue with the toolings. When found,
they should be fixed by working towards compatibility, because we should
not prioritize toolings (what we use) before end-users (who we serve).

Multi-version linking/importing hurts users because it encourage using
unmaintained library releases. While overly strict version requirements
actually shouldn’t hurt users but downstream maintainers, many end-users
install from PyPI (it’s called a warehouse because it offers no
integration warranty) so they are de facto downstream maintainers
of their own machine. Making it easier to do either will encourage
upstream developers to do it (they are lazy, I know because I’m one)
and cause a net negative to the ecosystem.

I agree on not encouraging but that there should be an option for solving arguably worse situations e.g. impossible resolutions or downstream packages pinning vulnerable requirements.

I’ll have a read through this and get a better understanding of this, thanks for the info!

At first look though it seems to be some cut down versions of the full install which are described in these files core/requirements_all.txt at dev · home-assistant/core · GitHub and core/requirements.txt at dev · home-assistant/core · GitHub.

1 Like