Venv: `activate` script changes `$PATH` while executing the binary doesn't

Python Help
1

I spend a bunch of time to understand why one of my python command (west) didn’t work as expected. Finally, I found it was because I installed my package in a venv and west it executed commands provided by required packages. eg:

import subprocess
subprocess.run(["command-provided-by-a-required-package"])

So, the snippet below worked:

$ source /home/jerome/.local/pipx/venvs/west/bin/activate
(west) $ west
... <work as expected> ...

This one also worked:

$ export PATH=$PATH:/home/jerome/.local/pipx/venvs/west/bin
$ west
... <work as expected> ...

But running west without changing the $PATH didn’t work:

$ west
... <does not work as expected> ...

(note since I have no luck, the error message was unrelated to $PATH)

I thought that running a binary from a venv had exactly the same behavior than activating the venv and then running the binary. The documentation seems to say the same. Do you think this behavior is a bug or is expected? If it is expected, what change should I suggest to west to properly run with venv?

2

Your first two examples are equivalent; activating the virtual environment is done primarily to modify PATH in exactly the way you do so manually in your second example.

What happens when you (try to) execute west without modifying your PATH variable depends on whether a command named west is even found, or how the “other” west command compares to the one in your virtual environment.

1 Like
3

I forgot to mention:

$ which west
/home/jerome/.local/bin/west
$ ls -l /home/jerome/.local/bin/west
lrwxrwxrwx 1 jerome jerome 44 Feb 22  2024 /home/jerome/.local/bin/west -> /home/jerome/.local/pipx/venvs/west/bin/west*

The issue is not about finding the west command by itself, but finding the commands run by west.

4

Still, it boils down to: whatever command west executes is either not found outside your virtual environment, or a different version exists outside it. Path lookup is not exclusive to the shell; subprocess.run also uses path lookup to find command-provided-by-a-required-package.

5

But, the documentation says:

Furthermore, all scripts installed in the environment should be runnable without activating it.

My understanding is /home/jerome/.local/pipx/venvs/west/bin/west or sh -c 'source /home/jerome/.local/pipx/venvs/west/bin/activate; west' should give the same result.

6

It’s expected behaviour.

To be able to run a command line tool globally, it could be installed with pipx. Pipx’ll put it in its own venv, and add the tool’s executable to the global path.

7

That only applies to commands run directly from the shell.

8

hmm… so the idea is “a python package should never call a binary provided by another package”, that’s it? Yet, I believe it is a common pattern.

9

“Should” is a strong word. Why not simply import the package and call the entry point directly?

10

No, the idea is that you write the Python code with enough information to identify the external binary, given a suitable execution environment. Ideally, you are going to execute the command by name alone, not an absolute or relative path, and configure the (virtual) environment so that the command is found in a directory on your PATH. You seem to have done that in your first two examples, but not in the third.

11

Not quite, as the behaviour you’re describing is technically a bug (or at least a missing feature) in west rather than anything behaving unexpectedly in any of the packaging tools. Virtual environments only guarantee that sys.path will be adjusted - it is specifically venv activation that manipulates PATH.

For subcommand invocation like that to work reliably (without requiring an activated virtual environment), there are a few potential ways to do it:

  1. If a dependency offers a Python API, invoke it that way rather than via its CLI
  2. Invoke the dependency provided subcommands via [sys.executable, -m, relevant_module_name] rather than via their installer-generated CLI wrapper scripts. How viable this is depends on whether or not the dependencies involved have defined a -m compatible CLI invocation (there may not be a suitable module to execute that’s equivalent to running the wrapper script)
  3. Invoke the dependency subcommands via [sys.executable, -c, "from entry_point_module import entry_point_target; entry_point_target()] (these can be looked up in the project’s metadata files or its package build configuration). This is the workaround for cases where there’s no convenient -m compatible invocation published by the dependency.
  4. Attempt to locate the wrapper scripts relative to sys.path entries, invoke it via [sys.executable, path_to_wrapper_script] once found. Not commonly used, since one of the above options will usually be easier.
  5. Use sysconfig to manipulate PATH in the current process to ensure the environment’s wrapper script folder is present. This is less reliable than the above options, since it still involves making potentially invalid assumptions about the expected relationship between sys.path and PATH configuration. It’s fairly straightforward to make it work for the typical “everything in one venv” case, though.

The first three options are the most reliable, but the last two do turn up sometimes (especially in code old enough to predate the existence of the -m switch).

2 Likes
12

Thank you Alyssa for this detailed answer.

1 Like