Possible bug with py.exe launcher in 3.11

Python Help
1

I noticed a change with py.exe for Windows in Python 3.11.

Example test script - “myscript1.py”:

#!/usr/bin/env python3

import sys

print(f'Testing running using Python {sys.version}')

Note: The shebang syntax is officially supported for Python for Windows per:
4. Using Python on Windows — Python 3.11.0 documentation

Results running test script with Python 3.10 for Windows (64-bit version):

myscript1
Testing running using Python 3.10.8...

Results using Python 3.11 for Windows (64-bit version):

myscript1
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

Results using Python 3.11 using python binary:

python myscript1.py
Testing running using Python 3.11.0...

Results using Python 3.11, call py launcher directly:

py myscript1.py
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

Results using Python 3.10’s py launcher directly:

py310 myscript1.py
Testing running using Python 3.11.0...

Results using Python 3.11’s py launcher directly with any argument:

py -b myscript1.py
Testing running using Python 3.11.0...

If I edit myscript1.py and change the shebang line to this:

#!/usr/bin/env python

Then it works:

myscript1
Testing running using Python 3.11.0...

So, it appears the py.exe launcher included with Python 3.11 does not work if the shebang line is like this:

#!/usr/bin/env python3

Unless an argument is passed - then it works.

This is a change from Python 3.10 and since the behavior does not appear to be consistent it appears this could be a bug.

Before I open an issue on the GitHub CPython tracker, I wanted to check here.

Thoughts?
–Jim

1 Like
2

The old launcher special cases /usr/bin/env python3 to always run the preferred version of Python 3, based on the registered installations, “py.ini”, and PY_PYTHON3 environment variable. It does not search PATH for “python3.exe”. The new launcher tries to search PATH for “python3.exe” and falls back on the default version if “python3.exe” isn’t found. That’s generally a good change.

Unfortunately, Windows 10 and 11 include a “python3.exe” alias for a UWP app that launches the store to install Python. When executed with command-line arguments, this builtin app fails with instructions to either run without arguments or disable the alias. If you don’t want to install the app version of Python, then disable the builtin “python.exe” and “python3.exe” app execution aliases. If you install the app version of Python, then normal “python.exe” and “python3.exe” aliases will be enabled that actually run Python.

As you observed, with the new launcher, passing a command-line option for the interpreter prevents shebang parsing. Here’s the relevant code in the parseCommandLine() function:

    // Might have a script filename. If it looks like a filename, add
    // it to the SearchInfo struct for later reference.
    arg = search->restOfCmdLine;
    while(*arg && isspace(*arg)) { ++arg; }
    if (*arg && *arg != L'-') {
        search->scriptFile = arg;
        if (*arg == L'"') {
            ++search->scriptFile;
            while (*++arg && *arg != L'"') { }
        } else {
            while (*arg && !isspace(*arg)) { ++arg; }
        }
        search->scriptFileLength = (int)(arg - search->scriptFile);
    }

It skips looking for a script file in the command line if there’s a “-” command-line option. The old launcher, in contrast, uses the C argv array and looks for the first argument that doesn’t start with “-”. That approach has its own problems with reliability. For example, consider the command line py -X utf8 script.py. The old launcher will naively assume that “utf8” is the script file.

The new launcher’s choice to never guess here is a sound decision. If you need to use a shebang and also pass command-line options to the interpreter, pass them on the shebang line itself. For example, #!/usr/bin/env python3 -X utf8. That’s supported by both implementations of the launcher.

3

Hi Eryk,

Thank you for the detailed reply. So, it sounds like this was by design. One thing I wanted to point out is there are cases where Windows 10 or 11 don’t have the app execution alias. For example, one of the laptops I use for work. Is this case, with Python 3.11, Python scripts appear to just stop working - illustration example with myscript1.py:

#!/usr/bin/env python3

# Standard Library
import os
import sys

# 3rd Party:
import psutil

print(f'Testing running using Python {sys.version}')
invocation = psutil.Process(os.getppid()).cmdline()
myself = psutil.Process(os.getpid()).cmdline()
print(f'Script invoked from:  {" ".join(invocation)}')
print(f'      Script run as:  {" ".join(myself)}')

Behavior prior to Python 3.11:

PS> py310 myscript1.py
Testing running using Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)]
Script invoked from:  C:\working\github\sockduct\asa-api\py310.exe myscript1.py
      Script run as:  C:\Program Files\Python311\python.exe myscript1.py

Behavior with Python 3.11 - note there is no output or errors, thus this can be confusing:

PS> myscript1

I have tremendous respect for the people running the Python community, so I’m not here to argue about the decision. However, I’m wondering if this decision could at least be documented. What do you think about me contributing a PR to update the Python docs for Windows to document this behavior? Specifically, I’m thinking that this page (4. Using Python on Windows — Python 3.11.0 documentation) could be updated to explain the behavior change. From looking at your GitHub profile it appears you have some involvement with the community. Do you have any suggestions on how to document it or who should review? Or just follow the contribution guide and submit a PR?

Your thoughts?
–Jim

4

In PowerShell you can find where “python3” is located with (get-command python3).source. Or use where.exe python3. Also, if you set the environment variable PYLAUNCHER_DRYRUN (to any value, as long as it’s defined), the new launcher will print the command line that would be executed normally.

If “python3” resolves to “%LocalAppData%\Microsoft\WindowsApps\python3.exe”, then it’s probably an alias for an older version of the store launcher. The store launcher used to do nothing when run with command-line arguments. In later releases, it prints an error that tells the user to run without arguments in order to launch the store and how to disable the alias.

The behavior of the “/usr/bin/env” virtual command is documented as follows:

The /usr/bin/env form of shebang line has one further special property. Before looking for installed Python interpreters, this form will search the executable PATH for a Python executable. This corresponds to the behaviour of the Unix env program, which performs a PATH search. If an executable matching the first argument after the env command cannot be found, it will be handled as described below. Additionally, the environment variable PYLAUNCHER_NO_SEARCH_PATH may be set (to any value) to skip this additional search.

5

I am brand new to coding and I am having this issue. Even when I turn off the app execution aliases on Windows 10, I don’t get a solution. I just get this error:

python3 : The term ‘python3’ is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
python3 .\Project3\

+ CategoryInfo          : ObjectNotFound: (python3:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

Any help? :smiling_face_with_tear: My professors and TAs seem pretty stumped as well.

6

Hi Eryk,

Thank you for the suggestion with using PYLAUNCHER_DRYRUN. That let me see that my system does actually have the Windows App Execution Aliases on. I also see that’s documented, so shame on me for missing it and thank you for your patience walking me through it. Once I disable the App Execution Alias for Python3, then the py.exe launcher for 3.11 behaves the same as in 3.10.

From looking around I see you’ve answered many questions in this area. Do you think it would be helpful to submit a PR for the docs that amends the section you showed as follows?

The /usr/bin/env form of shebang line has one further special property. Before looking for installed Python interpreters, this form will search the executable PATH for a Python executable. This corresponds to the behaviour of the Unix env program, which performs a PATH search. If an executable matching the first argument after the env command cannot be found, it will be handled as described below. Additionally, the environment variable PYLAUNCHER_NO_SEARCH_PATH may be set (to any value) to skip this additional search.

Suggested Addition

New in version 3.11: The launcher will search the path for the python executable specified in the shebang line. If App execution aliases are enabled (they are by default in Windows 10 and 11) and in the path before installed Python interpreters, it could cause scripts using the shebang line to silently fail. See the Dry Run section below to check for this behavior.

If you think it would be helpful, I will look at submitting a PR. If you don’t, I’ll defer to your judgement.

Thanks again for all your help with this,
–Jim

7

Hi Elliot,

A few things:

By default, python in Windows is always “python.exe” regardless of whether it’s Python 2.x or Python 3.x. What happens if you just run “python”?

For example, here’s what I see on my system:

C:> python --version
Python 3.11.0

If you get the same error about “python” is not recognized, my guess is that Python isn’t in your PATH. When you installed Python, did you select the option to add it to your PATH?

From PowerShell (that looks like what you’re running from your example), use the following to inspect your PATH for Python:

$env:path -split ‘;’ | sls python

For example, here’s what I see on my system:

C:> $env:path -split ‘;’ | sls python
C:\Program Files\Python311\Scripts
C:\Program Files\Python311\

If you don’t see anything you need to fix your path. It’s hard to give you specifics because it depends on which version you installed and how you installed it. However, this is a good place to check out:

HTH,
–Jim