Can vendoring dependencies in a build be officially supported?

[pradyunsg (Pradyun Gedam)]

@sirosen

I maintain a package which is a building-block for many of our internal applications and some of our customers’ applications, so maximal compatibility is a goal. This leads to us not wanting to use many libraries, as there are conflicts this would or could introduce with the downstream applications. e.g. I don’t want to use jsonschema because we have applications which use specific versions of it.

The answer here is isolating each of the applications from each other. Don’t make all of them use the same set of Python packages. Use a separate venv/virtualenv for each of them, or utilise a higher level of isolation for each application (Overview of Python Packaging - Python Packaging User Guide talks about this, in terms of “depending on …”).

@sirosen

We depend on a package, call it weird_dep , which has many updates in its main branch but which does releases very infrequently and has not released bugfixes which we need. The main branch of weird_dep is stable, but we don’t want the semantics of a VCS dependency on a branch in our package metadata. e.g. Users can get a version when installing which we might never be able to fetch again. We’d like to release a version of our package which vendors weird_dep@main .

@FFY00

Well, the correct answer here would be to not depend or the main branch,

+1 to what @FFY00 said. That said, I can understand how sometimes the needs of the upstream package can be different from your project.

In those cases still, you can depend on weird_dep@main… the following work:

pip install git+https://github.com/pypa/pip@main
pip install git+https://github.com/pypa/pip@ec8edbf5df977bb88e1c777dd44e26664d81e216

See VCS Support - pip documentation v23.3.2

---

Mixing multiple versions of a single package, in a single Python process, is usually a really bad idea. See the following post, from one of the threads you’ve mentioned:

Allowing Multiple Versions of Same Python Package in PYTHONPATH

To explain a bit why this isn’t as desirable as it might seem: imagine I am using package A which imports NumPy 1.21, alongside package B which uses NumPy 1.16. I call A.load_data() which returns a NumPy array, then I pass it to B.process(arr). Either B thinks it hasn’t got a NumPy array at all (isinstance(arr, numpy.ndarray) is False), or it has an array that may have a different layout than its NumPy library expects (different attribute & method names at the Python level, different memory layo…