# Optimizing installs of many virtualenvs by symlinking packages
**URL:** https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983
**Category:** Packaging
**Created:** [January 9, 2020, 3:48am UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983 "2020-01-09T03:48:54Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@leo](https://discuss.python.org/u/leo)
#### Post date: [January 9, 2020, 3:48am UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/1 "2020-01-09T03:48:54Z")
I’d like to be able to set up a large number (~100) of virtualenvs on a single system. Many of these virtualenvs will share the same dependencies of the same version although some may have different versions of those dependencies. Suppose I have envs A, B and C all using `requests` 2.20.0 and env D using 2.22.0.
If I simply run pip installs for all of the envs, I’ll have three copies of `requests` 2.20.0 on my file system. Is there any standard pattern to avoid that wasted disk space? So for example to have the actual `requests` 2.20.0 sit in some sort of cache and have each of the virtualenvs either symlink to that folder from its `site-packages` or to reference that folder with a line in a .pth file.
Sort of the equivalent of `pip install -e` but against a local cache of packages.
Doing this would also make spinning up a new virtualenv that includes most of the same packages super fast as they wouldn’t need to be downloaded from anywhere, copied or installed.
---
### Author:  [@tgamblin](https://discuss.python.org/u/tgamblin)
#### Post date: [January 9, 2020, 4:16am UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/2 "2020-01-09T04:16:06Z")
FWIW, this is actually how [Spack environments](https://spack.readthedocs.io/en/latest/environments.html) work.
Each package is installed in its own prefix with a unique hash, and packages “in” and environment are symlinked into a common prefix for that environment. There will be only as many installs of each package as you have unique configurations for it.
We also do the virtualenv trick, where we copy the python interpreter (and a few other things) into place so that python thinks it lives in the env, but everything else is just pointers back to the original packages. Practically speaking, that just means you can use pip in a spack env if you want to mix the two.
More on spack [here](https://github.com/spack/spack). See also the [tutorial](https://spack-tutorial.readthedocs.io/en/latest/tutorial_environments.html) on environments.
---
### Author:  [@pitrou](https://discuss.python.org/u/pitrou)
#### Post date: [January 9, 2020, 6:41pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/3 "2020-01-09T18:41:07Z")
Also, FTR, [Conda environments](https://docs.conda.io/projects/conda/en/latest/) use hardlinking to automatically share packages which are installed accross environments.
---
### Author:  [@sinoroc](https://discuss.python.org/u/sinoroc)
#### Post date: [January 12, 2020, 11:22am UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/4 "2020-01-12T11:22:50Z")
I am also interested in this. I believe the following discussion is related:
> [@Proposal - sharing distrbution installations in general](https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-general/2524):
>
> im picking up on [https://mail.python.org/pipermail/distutils-sig/2017-October/031738.html](https://mail.python.org/pipermail/distutils-sig/2017-October/031738.html) Hi everyone, since a while now various details of installing python packages in virtualenvs caused me grief a) typically each tox folder in a project is massive, and has a lot of duplicate files, recreating them, managing and iterating them takes quite a while b) for nicely separated deployments, each virtualenv for an application takes a few hundred megabytes - that quickly can saturate disk space …
---
### Author:  [@RonnyPfannschmidt](https://discuss.python.org/u/RonnyPfannschmidt)
#### Post date: [January 12, 2020, 4:09pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/5 "2020-01-12T16:09:23Z")
ftr - for personal reasons i haven’t worked any further on this
---
### Author:  [@leo](https://discuss.python.org/u/leo)
#### Post date: [January 18, 2020, 6:53pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/6 "2020-01-18T18:53:13Z")
Thanks for the suggestions - Conda and Spack might be overkill for our use case and it sounds like there’s not an easy way to do this with pip ☹
---
### Author:  [@sinoroc](https://discuss.python.org/u/sinoroc)
#### Post date: [January 19, 2020, 11:28am UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/7 "2020-01-19T11:28:00Z")
There’s something I have been experimenting a bit with:
```
$ mkdir /pool
$ python3 -m venv /venv
$ /venv/bin/python3 -m pip install --target /pool pyramid
$ echo "/pool" > /venv/lib/python3.6/site-packages/pool.pth
$ /venv/bin/python3 -m pip list
$ /venv/bin/python3 -c "import pyramid"
```
See [Python’s documentation chapter on “Site-specific configuration hook”](https://docs.python.org/3/library/site.html) for the details on what the effect of the `*.pth` exactly is. In short it adds the location of the `pool` directory to Python interpreter’s `sys.path` list. This way packages installed in this directory can be imported. This file can be added to as many _site-packages_ as one wants, and the pool can be shared between virtual environments. This probably has a lot of limitations. But with some clever wrapping code around _pip_ it could probably get somewhat useful, depending on the exact requirements.
---
### Author:  [@pitrou](https://discuss.python.org/u/pitrou)
#### Post date: [January 19, 2020, 12:42pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/8 "2020-01-19T12:42:11Z")
> [@leo](#):
>
> Thanks for the suggestions - Conda and Spack might be overkill for our use case
Why “overkill”? Conda isn’t really more difficult to use than pip.
---
### Author:  [@bearrito](https://discuss.python.org/u/bearrito)
#### Post date: [February 18, 2024, 9:20pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/10 "2024-02-18T21:20:47Z")
Did this make any progress off this conversation?
My companies maintains and uses [GitHub - locusrobotics/catkin\_virtualenv: Bundle python requirements in a catkin package via virtualenv](https://github.com/locusrobotics/catkin_virtualenv)
This leads to the OP’s scenario where we have 100’s to 1000’s of virtualenv per robot. Most with identical dependencies. This leads to our deployment artifacts being very large.
@sinoroc Did you ever take your example further?
---
### Author:  [@sinoroc](https://discuss.python.org/u/sinoroc)
#### Post date: [February 18, 2024, 10:48pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/11 "2024-02-18T22:48:35Z")
> [@bearrito](#):
>
> @sinoroc Did you ever take your example further?
Not much, no (mostly because it is not an annoyance for my use cases). Many things have changed in Python’s packaging ecosystem since 2020, maybe there are fresh ideas now. You could start a new thread (linking to this one), see if if the state of the art has progressed in 4 years. I do not recall having seen anything new in this specific area recently.
---
### Author:  [@sinoroc](https://discuss.python.org/u/sinoroc)
#### Post date: [February 18, 2024, 10:52pm UTC](https://discuss.python.org/t/optimizing-installs-of-many-virtualenvs-by-symlinking-packages/2983/12 "2024-02-18T22:52:56Z")
@bearrito And, well, just now I see that maybe [uv’s `--link-mode`](https://discuss.python.org/t/uv-another-rust-tool-written-to-replace-pip/46039) is that feature we are talking about.