Subprocess returning 'incorrect' exit code for negative exit codes in Windows in 3.7

Python Help
1

Think I’ve got a bug in 3.7, but it’s a first time for me with this sort of thing, so I thought I’d post here first to make sure I’m not mising anything…

I’m updating some code so that it produces the same output with both Python 2.7 and Python 3.7 (in Windows)

The code runs an external program using ‘subprocess’ and reads the program’s exit code. If the external program runs for more than a specific time, it kills itself and returns a exit code of ‘-1’. (Some debate about this, but negative exit codes appear to be valid in Windows)

For 2.7, subprocess returns the correct return code of ‘-1’, but for 3.7, subprocess returns ‘4294967295’, so an integer overflow. This appears to be the case for all negative integers.

I’ve recreated this in simple Python by creating a small script, test_exit_rc_prog.py, that just exits with the exit code specified by a command line argument - i.e. : ‘python test_exit_rc_prog.py -1’ causes the script to exit with exit code ‘-1’:

import sys
print("Python Version: " + sys.version.split("(")[0])
exit_code_int = int(sys.argv[1])
print("Exiting with code:\t" + str(exit_code_int))
sys.exit(exit_code_int)

I then run this script with subprocess in another script for different values of return code. The second script looks like this:

import os, sys, subprocess
py_ver = sys.version.split("(")[0]

print("\nPython version: " + py_ver)

rc_list = ["0","1"," 999","-1","-999",]

for rc in rc_list:
    # Python version being used
    if py_ver[0] == "3":
        cmd = "python3 test_exit_rc_prog.py " + rc
    else:
        cmd = "C:\Python27\python.exe test_exit_rc_prog.py " + rc
        
    print("\n============================================================")

    # Test subprocess return code
    print("\nSUBPROCESS Output for Command: '" + cmd + "'")
    p_input = cmd.split()
    p = subprocess.Popen(p_input)
    p.communicate()
    p_rc = p.returncode
    print("SUBPROCESS return code:\t" + str(p_rc))

    # Test os.system return code for comparison only
    print("\nOS.SYSTEM Output for Command: '" + cmd + "'")
    os_rc = os.system(cmd)
    print("OS.SYSTEM return code:\t" + str(os_rc))

print("\n============================================================")

When run with python 2.7, I get the below output:

Python version: 2.7.16

============================================================

SUBPROCESS Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py 0'
Python Version: 2.7.16
Exiting with code:      0
SUBPROCESS return code: 0

OS.SYSTEM Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py 0'
Python Version: 2.7.16
Exiting with code:      0
OS.SYSTEM return code:  0

============================================================

SUBPROCESS Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py 1'
Python Version: 2.7.16
Exiting with code:      1
SUBPROCESS return code: 1

OS.SYSTEM Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py 1'
Python Version: 2.7.16
Exiting with code:      1
OS.SYSTEM return code:  1

============================================================

SUBPROCESS Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py  999'
Python Version: 2.7.16
Exiting with code:      999
SUBPROCESS return code: 999

OS.SYSTEM Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py  999'
Python Version: 2.7.16
Exiting with code:      999
OS.SYSTEM return code:  999

============================================================

SUBPROCESS Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py -1'
Python Version: 2.7.16
Exiting with code:      -1
SUBPROCESS return code: -1

OS.SYSTEM Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py -1'
Python Version: 2.7.16
Exiting with code:      -1
OS.SYSTEM return code:  -1

============================================================

SUBPROCESS Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py -999'
Python Version: 2.7.16
Exiting with code:      -999
SUBPROCESS return code: -999

OS.SYSTEM Output for Command: 'C:\Python27\python.exe test_exit_rc_prog.py -999'
Python Version: 2.7.16
Exiting with code:      -999
OS.SYSTEM return code:  -999

============================================================

But with Python 3.7 I get:

Python version: 3.7.9

============================================================

SUBPROCESS Output for Command: 'python3 test_exit_rc_prog.py 0'
Python Version: 3.7.9
Exiting with code:      0
SUBPROCESS return code: 0

OS.SYSTEM Output for Command: 'python3 test_exit_rc_prog.py 0'
Python Version: 3.7.9
Exiting with code:      0
OS.SYSTEM return code:  0

============================================================

SUBPROCESS Output for Command: 'python3 test_exit_rc_prog.py 1'
Python Version: 3.7.9
Exiting with code:      1
SUBPROCESS return code: 1

OS.SYSTEM Output for Command: 'python3 test_exit_rc_prog.py 1'
Python Version: 3.7.9
Exiting with code:      1
OS.SYSTEM return code:  1

============================================================

SUBPROCESS Output for Command: 'python3 test_exit_rc_prog.py  999'
Python Version: 3.7.9
Exiting with code:      999
SUBPROCESS return code: 999

OS.SYSTEM Output for Command: 'python3 test_exit_rc_prog.py  999'
Python Version: 3.7.9
Exiting with code:      999
OS.SYSTEM return code:  999

============================================================

SUBPROCESS Output for Command: 'python3 test_exit_rc_prog.py -1'
Python Version: 3.7.9
Exiting with code:      -1
SUBPROCESS return code: 4294967295

OS.SYSTEM Output for Command: 'python3 test_exit_rc_prog.py -1'
Python Version: 3.7.9
Exiting with code:      -1
OS.SYSTEM return code:  -1

============================================================

SUBPROCESS Output for Command: 'python3 test_exit_rc_prog.py -999'
Python Version: 3.7.9
Exiting with code:      -999
SUBPROCESS return code: 4294966297

OS.SYSTEM Output for Command: 'python3 test_exit_rc_prog.py -999'
Python Version: 3.7.9
Exiting with code:      -999
OS.SYSTEM return code:  -999

============================================================

Looks like a bug to me, especially as it used to work ‘OK’ for 2.7. Anyone think it isn’t a bug?

2

The exit code returned by WinAPI GetExitCodeProcess() is a DWORD (i.e. a 32-bit unsigned integer). Related functions also use an unsigned 32-bit integer, such as ExitProcess, GetExitCodeThread, ExitThread, GetLastError, and SetLastError. Starting with Python 3.3, subprocess.Popen returns the exit code as an unsigned integer, as the system returns it. os.system() still casts the exit code as a signed integer, as did subprocess.Popen prior to Python 3.3.

To convert an unsigned 32-bit integer to a signed value (two’s complement), if the value is greater than or equal to 2**31 (i.e. 2147483648), then subtract 2**32 (i.e. 4294967296). For example, (4294967295 - 4294967296) is -1, (4294966297 - 4294967296) is -999, and (2147483648 - 4294967296) is -2147483648.

2 Likes
3

Thanks for the reply and the advice on how to correct the value.

I think my problem was that I was thinking that it was returning an ‘incorrect’ value, whereas (I think) it’s returning the correct value, but as a 32-bit unsigned integer. Looks like this can be neatly translated to a 32-bit signed integer with:

import ctypes
rc_out = ctypes.c_int32(rc).value

I think this should be mentioned in the docs for subprocess.returncode, so I might raise that…

Cheers,
Nick

4

Certainly you could use the ctypes module that way, but the math isn’t really any harder:

def signed32(x):
    return (x - (1 << 32)) if (x >= (1 << 31)) else x

The struct module also does the job neatly:

import struct

def signed32(x):
    return struct.unpack('i', struct.pack('I', x))[0]

(Maybe it would be nice to have a .sign_extend(n) method - n being the number of bytes for the implied type - on integers?)

1 Like
5

Thanks for the other methods. Not sure which is the most efficient or ‘Pythonic’? I like the ctypes one as it’s immediately obvious to me what it’s doing from the function name.