How can I invoke an external program in an OS agnostic way?

Python Help
1

Hello all,

I’m writing a script that updates the firmware on OnePlus6T smartphones and am revising it to work under both Linux and Windows. The script has to invoke two different external programs during it’s execution, which I have implemented using subprocess.run(); this is fine if the script is to work under only one OS as the first argument of run() is (usually) a sequence of strings, each one being a single word of the command line. The problem however is that the syntax of a Linux shell (e.g BASH) and a Windows shell are obviously not the same.

I’ll not go into all the other considerations here, but is this a case of simply doing…

if sys.platform.startswith('win32'):
    cmd = ['windows', 'command', 'line', ...]
elif sys.platform.startswith('linux'):
    cmd = ['linux', 'command', 'line', ...]
else:
    print('platform not supported')

subprocess.run(cmd)

I’d love to know how other people handle this sort of case!

2

Can you clarify how the shells are involved here? Your call doesn’t use shell=True. If you do use shell=True, it’s generally wrong to pass an argument list instead of a command-line string. In POSIX it’s wrong because the arguments are consumed by the shell as parameters (e.g. $0, $1, etc).

3

Apologies for the late reply, I’ve been playing around with solutions all week and trying to understand the issue myself! Perhaps the language I’ve used is a bit sloppy or inaccurate, I’ll give the actual code I’m using as an example and try to explain the problem based off of that; I’ve also placed the solution I came up with below, but I invite any criticism of the method as I’m not sure it’s the best way to go about it myself! My original code is as follows…

fw_dir = <pathname of a directory to be used as the argument to the '-output' option of the external process> 
completed_process = run(['./payload-dumper-go', '-output', fw_dir, '-partitions', <partitions string>, './opflash/payload.bin'], stdout=PIPE, text=True)

and later in the script…

firmware_images = [<just a list of 20 strings, each one being the name of a firmware .img file>]
partitions = [<another list of 20 strings, each one being an element of firmware_images with the .img suffix removed>]

for i in range(len(firmware_images))
    completed_process = run(['fastboot', 'flash', '--slot=all', firmware_images[i], partitions[i], '2>&1'], stdout=PIPE, text=True)

as far as I can tell, the above code (written for a Linux OS) is not portable for two reasons:

  1. Linux pathname elements are separated by a forward slash, Windows a backwards slash. The first code snippet above is intended to invoke payload-dumper-go from the current working directory, however on Windows this would require .\payload-dumper-go. I now realise the constant os.sep contains the character used to separate pathname elements on the current platfrom and could be used as follows (note that this is made redundant by the solution at the bottom of this post):
completed_process = run([('.' + os.sep + 'payload-dumper-go'), '-output', fw_dir, '-partitions', <partitions string>, ('.' + os.sep + 'opflash' + os.sep + 'payload.bin')], stdout=PIPE, text=True)
  1. In relation to the second code snippet, on a Linux machine the program ‘fastboot’ will most likely be installed via the package manager and will be located in a directory that is included in the user’s $PATH (probably /usr/bin) so it will work. On a Windows machine the fastboot binary will most likely have been downloaded from a website into the users Downloads\ directory and will need to be moved into the Python interpreters current working directory OR a directory that is included in the Windows variable %PATH%, in order to be invoked using the program name alone. Alternatively I suppose I could have my script search for the fastboot binary and then invoke it using it’s full pathname.

In the end I went for the following solution of using shutil.which() to check if the binary is contained within a directory on the user’s PATH and, if not, using os.walk to recurse through the directory tree rooted at the users home directory and search for a a file of the same name as the respective binary for which the user has execute permission. If a file is found then it is assumed to be the binary/executable, and it’s containing directory is added to the users PATH:

### imports ###

from shutil import which
from os import access, X_OK
from os.path import expanduser, join, pathsep

found_binaries = {'payload-dumper-go':False,'fastboot':False}

for binary in ['payload-dumper-go','fastboot']:

    ### if the value returned by shutil.which() is truthy, binary is on PATH ###

    if which(binary):

        found_binaries[binary] = True

    else:

        for dirpath, _, filenames in walk(expanduser('~')):

            for filename in filenames:

              ### if a file with the correct name for which the user has execute permission is found, add it's containing diretory to PATH and break from the loop ###

                if filename == binary and access(join(dirpath, filename), X_OK):
                    found_binaries[binary] = True
                    environ['PATH'] += (pathsep + dirpath)
                    break

            ### if a matching file has been found, do not process the next directory/tuple returned by os.walk()

            if found_binaries[binary] == True:
                break

By making sure that the required binaries are on the user’s PATH, the call to subprocess.run() may refer to the programs using their name alone and is not required to include a relative or absolute pathname for them.

4

Sometimes Windows supports forward slash as a path separator. For example, much of the Windows file API supports forward slash as a path separator by internally rewriting the path to use backslashes. But the API doesn’t always support forward slash as a path separator, e.g. none of the path manipulation functions (i.e. Path[Cch]*) support forward slash as a path separator. Also, whether paths passed to an application on the command line support forward slash depends on the application.

In general, you’re right to assume that you need to use backslashes for paths in Windows. You can implement this by first normalizing paths with os.path.normpath().

Searching for binaries using os.walk() is excessive. Also, it could have unintended consequences if something unrelated happens to be named “fastboot”. (Note that the os.access check is not really implemented in Windows, but I presume you added it for POSIX. Anyway, while CreateProcess in Windows does check for execute access, generally a user has full control of all files in their profile directory, which includes execute access.)

Why not inform users that these binaries have to be located in a PATH directory? Most users should be able to figure out how to temporarily modify PATH in a command-line shell (e.g. in CMD, set "PATH=%PATH%;%USERPROFILE%\some\new\path"), or permanently modify PATH for the current user or system using the environment variable GUI editor.

5

Thanks! I wasn’t aware of this function. Interestingly the documentation for it states that it converts a forward slash to a backward slash on Windows, but doesn’t seem to suggest that the inverse is true.

I agree that the os.walk is excessive, I was trying to devise a solution that avoided the user having to manually modify their PATH, as although it’s a simple task, it’s only “relatively” simple and I have noticed a tendency for people to instantly give up when a computer related task presents itself (reading is hard) :roll_eyes: the access check was intended more to confirm that the file itself is an executable, because as you say, if a file can be executed and it’s in the user’s home directory, they will probably have execute permission and the check will succeed - a bit clumsy.

I didn’t realise os.access wasn’t fully implemented in Windows as the documentation didn’t mention it; is this due to a difference in the Windows permission system?

In terms of side effects in the event that something else is named ‘fastboot’, you are absolutely right of course, but I had concluded (using my very limited knowledge) that if all the script is doing is adding fastboot’s containing directory to PATH and later invoking it, the long series of arguments to the command would likely cause a fastboot program the script had selected by mistake to exit with an error (unless of course it was written to ignore arguments passed to it). I can’t see any difference between that and there being another program named ‘fastboot’ on the user’s PATH. However I must confess I’m no expert!

I had considered this. Despite what I wrote above, it’s probably not out of the way considering that the type of person that is looking to update the firmware on an android phone running a custom ROM (i.e what the script does) is probably comfortable enough with computers to do it.

6

On Windows, forward slash is disallowed in filenames, so normpath() can replace forward slash with backslash. On POSIX, backslash is allowed in filenames, so normpath() cannot replace backslash with forward slash.

On Windows, os.access() only checks for existence and the “readonly” file attribute, which on Unix would be like only checking the “immutable” file attribute. It doesn’t check file permissions, doesn’t support follow_symlinks, and doesn’t support effective_ids. It’s possible to implement access() on Windows, including support for follow_symlinks and effective_ids. It just hasn’t been implemented.

A user of your application that has a “fastboot” executable in PATH has probably configured this intentionally. But just rummaging through a user’s profile is open to unintended consequences.

1 Like