Shutil.which is unable to utilize PATH [solved]

Python Help
1

Versions:

FreeBSD 13.1-RELEASE
Python 3.9.16

Paths

echo $PATH:
/home/user/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin

Code:

if shutil.which("rustc") is not None:
        try:
                rustc_output = subprocess.run(
                        ["rustc", "--version"],
                        capture_output=True,
                        timeout=0.5,
                        encoding="utf8",
                        check=True,
                ).stdout
                version = re.sub("^rustc ", "", rustc_output.strip())
                print(version)
                print("wow")
        except Exception as e:
            print("well OK")
            print(e)
else:
    print("OK NO")

Info and expectations

Running:
python3 -c "print(__import__('os').environ.get('PATH', None))"

returns:
/home/user/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin

When I use pdb to step through I notice that shutil.which() steps through various posix paths but never looks in the ~/.cargo/bin even though it is readily available in my shell cli.

Any insight available? Thank you.

2

Info and expectations

Running:
python3 -c "print(__import__('os').environ.get('PATH', None))"

returns:
/home/user/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin

When I use pdb to step through I notice that shutil.which() steps through various posix paths but never looks in the ~/.cargo/bin even though it is readily available in my shell cli.

Any insight available? Thank you.

Can you provide a small Python script which:

  • prints os.defpath
  • prints os.environ['PATH']
  • prints the results if shutil.which('rustc')

And a transcript of a shell session showing you running that script.

I’m just speculating that you’re not invoking this from your shell but
elsewhere (eg some desktop-started IDE) which has a different $PATH
environmnet variable. Also, shutil.which falls back to os.defpath if
there is no 'PATH' entry in os.environ.

Cheers,
Cameron Simpson cs@cskk.id.au

3

While making the script you asked for I realized shutil.which() was only failing to get the ~/.cargo/bin path inside the venv.

Outside of the venv I see the following paths, and shutil.which() finds it appropriately.:
/home/user/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin

But when inside the venv the cargo path is overwritten with a the the venv/bin path, eg:
/usr/home/user/pythontesting/env/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin.

So definitely not an issue with shutil, but why does the venv not pick up the path?

And no, I am not using any IDE, just venvs and vim.

Thanks

4

While making the script you asked for I realized shutil.which() was
only failing to get the ~/.cargo/bin path inside the venv.

Test scripts are great for this kind of thing.

Outside of the venv I see the following paths, and shutil.which() finds it appropriately.:
/home/user/.cargo/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin
But when inside the venv the cargo path is overwritten with a the the
venv/bin path, eg:
/usr/home/user/pythontesting/env/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin.

So definitely not an issue with shutil, but why does the venv not pick up the path?

Ah. So: how is your environment, particularly the .caro/bin path, get
set up? And how are you picking up the venv environment?

And no, I am not using any IDE, just venvs and vim.

Me too :slight_smile:

But I, atypically I suspect, never use the activate script.

A nice thing about the python executable in the venv/bin subdir is
that it does the correct hookups to use the venv packages all on its
own. There’s no need for activate. The activate script seems to me
to a) fiddle your shell prompt and b) fiddle your $PATH to put the
venv/bin at/near the front.

I hate both of those things. I want my environment to be unchanged. When
I want to use something from the venv I executable the python inside the
venv/bin.

This isn’t actually a painful thing to do. I’ve got an alias py3 which
runs dev python3, which sets my 'dev" environment then runs python3;
then python3 means the python3 from the venv/bin. Otherwise I
get the stock python with the stock modules (in my case this is the
python from my default ~/venv).

By not using the venv by default my working tools are not subject to
bugs in the dev environment. Particularly since sometimes the dev env
includes developing my working tools. Always bad to break your own dev
tools.

WRT to an IDE, I was mostly thinking that a desktop-gui-invoked app will
not have picked up your environment.

Cheers,
Cameron Simpson cs@cskk.id.au

5

Hey I figured it out and it is the kind of mistake that makes me want to leave the screen and become a farmer for the rest of my days. No, I don’t want to talk about it. Thanks for helping me debug – it got me there eventually!

6

I live on a farm. Not as idyllic as you might hope.

7

“All $PATH entries are equal, but some are more equal than others…”