Making venvs relocatable friendly

Ideas
1

Hi all,

In short: Have resolution of paths in pyvenv.cfg be relative to the pyvenv.cfg file.

Why? This allows a venv to be relocatable enough that the basic Python bootstrapping process gets far enough to hand over execution to application code that can handle the rest. (Spoiler: this is actually possible today! But is brittle and relies on some specific code paths being taken). By having a well-defined location (the pyvenv.cfg location) that paths are relative to, it allows constructing the surrounding environment to respect that.

The context here is build systems such as Bazel impose a couple of constraints when building (i.e. creating the files and directory structure a program will use at runtime) programs. In the context of Python, this means creating the equivalent of what one sees after running e.g. python -v venv myvevnv; e.g., a pyvenv.cfg file (to indicate a venv), a bin/python symlink, and lib/<version>/site-packages directory with third party packages.

(for context, I’m a maintainer of rules_python, which allows Bazel to build Python programs)

Systems like Bazel, however, impose two constraints:

  1. Absolute paths aren’t allowed. This is because the machine that generates files (performs the building) may differ from who runs it. For example, a cluster of remote build worker machines may collaborate and generate the pyvenv.cfg, bin/python, and site-packages etc files. Those other machines won’t have the same absolute path to files as your local machine.
  2. The build outputs, i.e. the venv directory tree, must be assumed to be read-only. This is because Bazel relies on the immutability to cache artifacts and share them to local/remote build processes or consuming users.

The first means the typical behavior of writing an absolute path to pyvenv.cfg’s home key isn’t an option. Writing a relative path doesn’t work because, currently, such paths get interpretered relative to the process’s CWD (which seems nonsensical and I can’t imagine intentional).

The second mean fixing up the pyvenv.cfg file in place isn’t possible, and thus requires e.g. creating a venv at runtime in a temporary directory (which has its own problems and is generally a headache best avoided).

Like I alluded to earlier, there is a workaround today, though: don’t write the home key at all in pyvenv.cfg. This happens to lead Python down a route where it recognizes it’s in a venv and sets up its home, sys.prefix, sys.executable, etc appropriately. Mostly, anyways. The key part is Python home is found and the venv-site packages is found, which allows application code to hook into the startup process (via site-packages pth files) and fix up one or two oddities (I forget exactly, but it’s just like a 1 line fixup to something in sys).

The net effect of this trick is the venv is relocatable (or more specifically, the directory tree Bazel generates is, which contains the venv and anything else necessary for the program). i.e. one can cp -r it to arbitrary other machines (within the confines of platform compatibility (OS, CPU, etc)).

Anyways, this is fairly brittle. It relies on bin/python3 being a symlink, which makes it tough to work on Windows, and symlinks-to-symlinks-to-symlinks can cause oddities.

So, back to my idea: Treat a relative path for the home key in pyvenv.cfg as relative to the pyvenv.cfg file. When one knows where the venv is relative to the python install (such as in the case of Bazel), one can then write a stable relative path for the home key. Now Python can directly find its home like normal without requiring symlinks.

Right now, relative paths in pyvenv.cfg are relative to the process’s CWD. But, that doesn’t seem to be intentional. Reading the getpath.py code, I get the impression it is assuming home is always absolute. It also doesn’t make sense – the CWD could be anywhere, which makes relative paths largely useless.

7 Likes
2

It’s not just pyvenv.cfg that contains absolute paths. All the scripts in the bin directory have hardcoded Python executable paths in their shebangs and the executables in Windows’s Scripts directory likewise have the full path to python.exe in them.

Changing that would be quite nice – you’d be able to move venvs around instead of recreating them (provided that the path in pyvenv.cfg stays absolute).

1 Like
3

The executables in the Scripts/bin directory are hard coded to the absolute venv location by design. There are real-world use cases where those files are copied to an alternate location while leaving the venv where it is - for that to work, an absolute path is essential.

I could imagine an optional way of generating script wrappers with relative paths, but it couldn’t be the only option - and for backward compatibility it could not be the default.

4 Likes
4

Acknowledging that moving scripts around means this can’t be the default, I still think this is worth pursuing.

I’ve only run into wanting to move venvs around in build systems (build or cache a venv, pass it to future stages). That’s a context in which the caveat around scripts seems acceptable.

A warning about “this makes the whole virtualenv relocatable but makes it unsafe to relocate scripts relative to the virtualenv itself” strikes me both as a good idea and likely to be ignored.

I therefore think it’s relevant how this mode should be enabled. I would consider a flag to venv a late addition, and I’d start with the runtime behavior and detection. What needs to happen to instruct the interpreter that the paths are relative?

3 Likes
5

runtime behavior and detection; what needs to happen to instruct the interpreter that the paths are relative

Identifying it is pretty easy. During Python startup, getpath.py knows the location of pyvenv.cfg and reads the raw value of the home key, so all it has to do is check if its absolute already or not. If it’s relative, then join(pyvenv_dir, value), and viola: the runtime has an absolute path for home like it desires, and the pyvenv.cfg gave a relative path. The rest of getpath.py continues on like normal.

So, IMHO, it really boils down to – what use is there in the existing behavior where a relative value for the home key means “relative to CWD” ? I’m hard pressed to think of one. Whether I do any of these:

cd /tmp
/whatever/myvenv/bin/python3

cd /whatever
myvenv/bin/python3

cd /whatever/myvenv/bin
./python3

I don’t expect that Python finds a different stdlib based on my CWD. I expect it to use what the venv configured (that’s kind of the point of a venv).

On the off chance that behavior is desirable, then I think there’s a variety of ways to express it. e.g. separate config key, special case explicit values starting with “.” or some other special char, envvar or interpreter -X arg, etc bikesheds.

2 Likes
6

Can these use cases be identified? Maybe they can be supported in a different way. Are they more important than the use cases where relocatable venvs are required?

7

Is it necessary to relocate with simple cp -r? A utility to clone a venv to different location exists already. According to its description it rewrites those absolute shebang lines etc.

Installing some packages involves compiling e.g. C language code. Are such packages relocatable? (I don’t know. If yes, no problem here.)

Could symbolic links help? The fixed configured location could be a symlink that can point to a different path in different installations. Something little bit like /etc/alternatives work in Linux.

1 Like
8

This may be all that matters to you, but as has been mentioned, there are a number of other places where absolute paths are embedded in a virtual environment, and it’s not clear that it’s OK to ignore those.

It’s worth saying that virtualenv had an option to make environments relocatable for quite a while, but it always had problems and ultimately that option was removed. I would strongly suggest that anyone interested in pursuing this investigate the history behind that flag, and make sure we’re not simply repeating the mistakes of the past.

Not offhand, no. One use case is pipx, I believe, which is a pretty important example. Another is my own personal use, but I don’t consider that to be an important consideration on its own (I have no idea if other people work the way I do).

They could, but that’s breaking backward compatibility, which is why I maintain that it’s relocatable venvs that should be “supported in a different way”.

I don’t think relocatable venvs get asked for that often, although there’s certainly an interest in them. More important? Someone would have to do the research. Otherwise, “status quo wins” and backward compatibility are likely to be the key factors.

This is a very good point. If there’s a utility already (I must admit I didn’t know about it, so maybe it needs a little more publicity?) then I’d certainly want to see good reasons why it’s not sufficient before adding extra (fragile) features to the stdlib.

I agree with this point, but for me that says that we should raise an error if the value isn’t absolute. If interpreting the value relative to the pyvenv.cfg file made everything work, I’d be more supportive of doing that. But if we’re going to get “I moved my virtual environment and now the pip command doesn’t work properly” bug reports, I’m not sure the advantage is as clear as you suggest.

Maybe a compromise would be to interpret relative paths as you suggest, but leave that behaviour undocumented and with no user interface. Then someone would have to go in and hand-edit pyvenv.cfg (and rely on undocumented behaviour) if they wanted to do this. IMO, that’s too user-hostile to be worth doing, but maybe it’s good enough for some people?

1 Like
9

This is the utility I found. I Did not use it till now.

virtualenv-clone utility on PyPi and on GitHub

10

It looks like there are some issues that need fixing (support for Windows exe wrappers, correctly updating the prompt in the activate scripts). But the project readme gives an extremely good summary of all of the things that need to be addressed when copying a virtual environment. And it’s obvious that “fix the path in pyvenv.cfg” is barely the start of what you need to do.

11

I think a new command line option for venv would be nice to be able to create venvs which are relocatable as a whole.

Note that this will never be perfect, though, since even if the venv installation is relocatable, installed packages will likely have added scripts with absolute paths in their shebangs to the bin/ dir.

2 Likes
12

A venv subcommand to relocate or clone a venv would be nice.

1 Like
13

I will share part of my workflow, and why I’d like to be able to relocate venvs.

I often start projects in ‘~/_Sync/2025/_Projects/Sandbox/{project name}’ (example project “generate-text2speech”). So I’ll add a ‘venv’ in there. Later, as the project gains more momentum, I might rename the project, add git support, drag the project folder out of Sandbox into its own dedicated project folder. Sometimes that’s stable, sometimes I’ll rename it/move again, depending on how the project evolves.

I find it frustrating that (1) 'venv’s don’t “survive” a move, (2) as a user, there aren’t errors explaining this - when I run ‘source venv/bin/activate’, in my terminal it looks like it’s working - but then I find out later the ‘python’ command doesn’t work.

In my mental model, python venv is like a project-specific toolbelt, and I ought to be able to take it anywhere.

Related, as you can see by my folder path above, I keep a major part of my user folder in ‘_Sync’ - it’s a SyncThing-managed folder, which syncs between my Macbook and a Debian Linux computer. This means I often create a ‘venv-mac’ and ‘venv-linux’ so that I can develop on a project from either computer, and the files synchronize & python works. I’m all ears if there is a better solution to achieve this. The net result is, when I rename/move a project folder, I have two venvs to delete/create/pip install again


I don’t mean to distract from the tech discussion of how to make venvs’s move-friendly, just wanted to share a users workflow to consider.

2 Likes
14

Hang on, am I missing something here? If you make home relative, that would make it less easy to move a virtual environment, as home points to the base Python interpreter, which is not going to be moved. So moving a venv with a relative path for home will break it.

15

The fact that you need these separate venvs shows why it doesn’t really make sense to share them. My suggestion would be to use git for synchronising code between the different machines and include a requirements.txt file for setting up a venv with needed dependencies on any machine.

5 Likes
16

Hang on, am I missing something here? If you make home relative, that would make it less easy to move a virtual environment, as home points to the base Python interpreter, which is not going to be moved. So moving a venv with a relative path for home will break it.

Have a read of the context and background in my original post.

The aspect of “relocatable” I’m interested in is avoiding absolute paths (they tend to be machine specific), which enables copying a minimal venv from one machine to another. Stated another way: I have a custom venv creation tool (rules_python in Bazel) that creates a directory with everything an application needs (which includes not only the venv itself, but can also include the python installation, among other things).

Some of the discussion has been about shebangs in the bin directory scripts – for me and my case, this sort of issue doesn’t come up (we don’t use python’s venv command, and it’s a hard requirement things be relocatable, and this particular piece isn’t implemented yet, so, however we eventually do it, it won’t be based on putting absolute paths to the venv interpreter in shebangs).

I hope that clarifies?

And it’s obvious that “fix the path in pyvenv.cfg” is barely the start of what you need to do.

Yes. Gotta start somewhere! And the ability for python to even find it’s home seems like the first part, if not absolute minimum.

1 Like
17

Another related issue is that venvs silently break if the project folder (or any other folder in its path) is renamed or moved. That’s a paper cut that I believe almost every single newcomer to Python encounters.

4 Likes
18

This brings to mind some of the exotic things which might happen, like a module which uses __file__ as some key part of its implementation. There will always be some potential for behavioral changes and therefore breakage involved in moving things around.

Setting aside “perfect” as unattainable, can we define reasonable and achievable behavior for a “relocatable” virtualenv?

I would posit this as a goal:

  • at startup, the interpreter discovers paths based on the virtualenv location
  • scripts in the bin/ or Scripts/ dir may be broken after the environment is moved
  • python -m venv.fix_scripts $VENV_DIR can be run to fix shebang lines (keeping them absolute) and regenerate the activate scripts

This is intentionally broader than OP’s suggestion, but I think it’s valuable to have a coherent story about “yes, some things may break, and there’s a supported path to fixing the ones which are under our control”.

If the above isn’t acceptable as an outcome, I don’t see a way of making this fly, since the alternative is to allow for a virtualenv to become strangely incoherent. You’d end up with scripts which work with venv/bin/python -m $utility but not with venv/bin/$utility when packages provide __main__.py.

19

For me, venvs are fundamentally disposable. The goal, especially now that we have standard lock files via PEP 751, should be to reproduce them on demand. If you find yourself in a spot where you can’t just rebuild your environment, I think that’s the real problem we should be looking at and trying to solve, rather than making the existing venv portable.

6 Likes
20

It feels like there’s two different concepts of “relocatable” being discussed?

One sounds roughly like what uv venv --relocatable provides. You can move the venv folder on a system and the venv will keep working - the python path in pyvenv.cfg is absolute in order for this to work. Scripts can’t be moved outside of the venv folder as a side effect (but they can be moved with the venv).

The other is what the original post here is asking for, where pyvenv.cfg also points towards a python runtime as a relative Path. So you can move the venv folder, but only if you also move the Python runtime folder (unless the relative path hasn’t changed).

I think this second kind should not be referred to as a ‘relocatable’ venv, because what most people probably have in mind when they hear that is the first one, and in this second case you can’t relocate the venv in isolation without possibly breaking it as the path to python may now be incorrect. I also don’t think this behaviour should be provided by the venv module for the same reason, I would expect to be able to move a ‘relocatable’ venv to a subfolder for example and have it still work.

The specific request “make it possible for venv python to resolve relative paths to ‘home’ in pyvenv.cfg relative to itself instead of relative to CWD” doesn’t seem unreasonable, but I’m trying to understand is the exact use case here. If the packaging is going to bundle the Python runtime itself does it even need a venv? Is this bundling a runtime with multiple venvs that need to be independent of each other?

If you’re including a Python runtime that you’ve built can you modify it to do this already?

4 Likes