# What's the deal with the pipx shared venv, exactly? **URL:** \ **Category:** User Feedback\ **Created:** [July 11, 2024, 6:38am UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882 "2024-07-11T06:38:53Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [July 11, 2024, 6:38am UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/1 "2024-07-11T06:38:53Z")
I’ve been experimenting with Pipx the last couple of weeks trying to figure out ways to unleash its full power. I’ve noticed that a shared venv is created that contains `pip`, `setuptools` and `wheel`. Then the per-project venvs have `.pth` files that include the shared venv’s setup, and are created `--without-pip`. So far, so good - now I don’t repeat copies of Pip across my disk just so that Pip can know where to install things. But the details of the architecture have me scratching my head a bit: - Are the `.pth` files actually necessary, given the existence of `--python` and `--target` options for Pip? Pipx presumably always knows where Pip is located, and presumably will be running it in a separate process, and presumably can figure out which executable to use it with and what flags to pass. So I don’t understand why it’s important, or even helpful, to ensure that the `pip` package is on `sys.path` when the application runs. - When do these `setuptools` and `wheel` installations actually get used? I thought that Pip defaults to isolated builds now, so that if Pipx tries to install from an sdist, Pip would be forced to install a fresh `setuptools` and `wheel` into an isolated environment first before installing the built wheel into the real virtual(?) environment. - Supposing I’m right that they’re specifically there in case I explicitly do something like `pipx install pycowsay --pip-args "\"--no-build-isolation\""` \[1\]. What if I instead install something that uses a different backend? I don’t see a supported way to add other backends (or other packages) to the shared environment. - Is there anything else particularly clever or subtle about this setup that I might want to know about? * * * 1. I previously found in testing - [while trying to avoid hitting the Internet for a simple install](https://discuss.python.org/t/why-does-pip-need-an-internet-connection-to-install-from-a-local-cache/57046) - that this weird quoting is necessary; one pair is stripped by the shell and the other is needed to make it a Pip arg instead of a Pipx arg. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [July 11, 2024, 6:49am UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/2 "2024-07-11T06:49:06Z")
> [@kknechtel](#): > > Are the `.pth` files actually necessary, given the existence of `--python` and `--target` options for Pip? The shared lib predates pip’s `--python` flag. The `--target` flag can’t be used as it doesn’t properly support upgrades and uninstalls. Also, some projects (still) expect pip to be installed in every environment. So it’s fixable, but no-one has got round to looking at doing so. > [@kknechtel](#): > > When do these `setuptools` and `wheel` installations actually get used? Same answer - never these days, it’s just out of date code. > [@kknechtel](#): > > Is there anything else particularly clever or subtle about this setup that I might want to know about? No, just the historical context. I guess you could say it’s a clever/subtle solution to a problem that no longer exists 🙂 ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [July 11, 2024, 7:09am UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/3 "2024-07-11T07:09:35Z")
I see. (… Any ideas for forcing Pipx-driven Pip to use a cached backend, when it’s compatible with the package? This is mainly so I can speed up editable installs. With no Internet connection and no build isolation, a Setuptools-powered approach is taking over 3 seconds on my machine for a trivial project. With build isolation and a barely working Internet connection, it can hang pretty much indefinitely.) ---
**Author:** ![bulletmark](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/bulletmark/32/12863_2.png) [@bulletmark](https://discuss.python.org/u/bulletmark)\ **Post date:** [July 11, 2024, 1:26pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/4 "2024-07-11T13:26:35Z")
> [@pf\_moore](#): > > So it’s fixable, but no-one has got round to looking at doing so. Well actually [somebody has](https://github.com/astral-sh/uv)! 😉 ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [July 11, 2024, 1:35pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/5 "2024-07-11T13:35:35Z")
I’m not sure I follow you. Yes, `uv` could be used in pipx as an alternative to `pip` (the performance of `uv` makes this attractive), but nowadays both `pip` and `uv` have a `--python` option\[1\] that would allow pipx to avoid the `.pth` file hack it uses at the moment _regardless_ of which installer it uses. * * * 1. and indeed, `pip` got it before `uv` even existed… ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [July 11, 2024, 1:37pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/6 "2024-07-11T13:37:47Z")
> [@pf\_moore](#): > > Also, some projects (still) expect pip to be installed in every environment. Actually, could you elaborate on this a bit? For example, any known workflow that would be broken? Is this a “we would need to do a deprecation cycle and teach users how to do it the new way” sort of thing, or… ? ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [July 11, 2024, 1:44pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/7 "2024-07-11T13:44:21Z")
The one I recall was IPython’s `%pip` magic, which runs `subprocess.run([sys.executable, "-m", "pip", ...])` in the background (see [here](https://github.com/ipython/ipython/blob/main/IPython/core/magics/packaging.py#L92)). It doesn’t include a dependency on pip, so this will fail in an environment without pip installed. Yes, it’s a “we need people to change their code/workflows to fix assumptions that may not be true in future” situation. But it’s not clear how we’d detect or deprecate this usage. There was [a Discourse thread](https://discuss.python.org/t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431) about this at the time we introduced the zipapp distribution of pip. ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [July 11, 2024, 4:23pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/8 "2024-07-11T16:23:10Z")
> [@pf\_moore](#): > > But it’s not clear how we’d detect or deprecate this usage. Hmm. I can’t really imagine detection. That seems like it would have required building something in to the recommended `subprocess` approach ahead of time. As far as deprecation goes, I similarly can’t see a way for Pip itself to say anything about it, and have it actual directed at the right people. So that just leaves the documentation, and praying the right people read it. Aside from that, there’s the reasoning that if people expect `python -m pip` in a new subprocess to work, then they’re expecting `pip` to exist in the current environment. Since Pip isn’t part of the standard library, these users therefore have `pip` as a dependency (even if they aren’t using it as a library), and should declare it. That would at least allow `pip install theirapp`, run from a different Pip, to notice that the destination environment lacks Pip and install it. (And when run from the environment’s Pip, everything works already and there’s no more to do.) That does defeat the purpose of the zipapp, but those developers (and their users) weren’t planning around the zipapp anyway. The situation hasn’t really gotten worse for them - there’s just a roadblock that prevents things from getting as much better for them as ought to happen. But. Notable from that discussion: > [@Pip plans to introduce an alternative (zipapp) deployment method](https://discuss.python.org/t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431/23): > > Probably the biggest thing that has come out of this discussion so far is that I want to explicitly add a note that pip does not support being quoted as a dependency of other tools (because it has no supported Python API). So maybe what’s needed is a tool that _actually does_ have that API - and is minimal and focused. Or maybe _just the library_ for it, never mind making a(nother) competitor to Pip. (Or maybe in the far future, Pip could use something like that internally.) Overwhelmingly, as far as I’m aware, people\[1\] do these tricks because they’re distributing an application with optional extras, and want the user to be able to obtain the extras at runtime (as opposed to only supporting `theirapp[feature]` at the initial installation, or expecting the user to understand and wrestle with Pip manually). One good example of this that I’ve run into is [Manim](https://www.manim.community/). So these are cases where sdist support is probably much less important on average: you wouldn’t be doing this in the first place if you had the kind of users who could deal with an sdist installation potentially failing\[2\]. Caching is probably also much less important: it’s relatively unlikely the user has the dependency in cache, and users who do would already have the skills to manage the environment manually. The main tasks are to resolve dependencies, grab wheels, figure out the right directory, and do the unpacking. Those could all be part of a developer toolchain. (And the last part is, to my understanding, already covered by `installer`.) * * * 1. That is, outside of cases like IPython/Jupyter/Spyder where the project is itself a Python development tool. 2. Although actually, the Manim stack _will_ try to pull in some huge AI-related stuff depending on exactly what you’re trying to do. I’ve been trying to build a much more lightweight approach to video rendering. But that’s another story… ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [July 11, 2024, 4:51pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/9 "2024-07-11T16:51:02Z")
Side note… > [@kknechtel](#): > > So maybe what’s needed is a tool that _actually does_ have that API There is [_`pip-api`_](https://pypi.org/project/pip-api/). I have not used it myself. I do not know if it would help/work in the use case(s) discussed here. And I assume that all warnings against using pip from your own Python code likely still apply. Just thought I would mention it. ---
**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\ **Post date:** [July 11, 2024, 5:13pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/10 "2024-07-11T17:13:38Z")
It’s neat that that exists, but I don’t think it’s what I’m looking for. It appears that fundamentally it’s just a wrapper for the subprocess logic, except that it uses `os.environ.get("PIPAPI_PYTHON_LOCATION", sys.executable)` as the Python to run (i.e. the default of using the current Python can be overridden). Although that probably does still solve problems for a fair number of users. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [July 11, 2024, 5:39pm UTC](https://discuss.python.org/t/whats-the-deal-with-the-pipx-shared-venv-exactly/57882/11 "2024-07-11T17:39:56Z")
> [@Pip plans to introduce an alternative (zipapp) deployment method](https://discuss.python.org/t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431/23): > > Notable from that discussion: > > > [@Pip plans to introduce an alternative (zipapp) deployment method](https://discuss.python.org/t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431/23): > > > > Probably the biggest thing that has come out of this discussion so far is that I want to explicitly add a note that pip does not support being quoted as a dependency of other tools (because it has no supported Python API). > > So maybe what’s needed is a tool that _actually does_ have that API - and is minimal and focused. Or maybe _just the library_ for it, never mind making a(nother) competitor to Pip. (Or maybe in the far future, Pip could use something like that internally.) I’m not actually sure I still feel that way, to be honest. IMO it’s legitimate for code that expects to be able to run `python -m pip` as a subprocess (using `sys.executable` as the Python interpreter) to declare pip as a runtime dependency. Pip _does_ have an API - it’s the CLI. What I said was it has no _Python_ API, i.e., there are no supported functions importable or callable from the user’s Python code. But that’s a pretty fine distinction, and I didn’t want to derail the discussion by expanding on that point. So, to put all this in context, nowadays, pipx could happily add a dependency on pip and/or uv, then use the command line interface for whichever tool the user prefers, including the `--python` flag, to install the requested application in its private venv. And I’m pretty sure there’s work going on in that direction.