Implement `shutil.view` for cross-platform document opening (rework of `shutil.open` proposal from 2008)

Proposal

There’s considerable demand for a simple, cross-platform way to view documents from a running Python program. For example, this Stack Overflow Q&A has over 140k hits. It’s fairly easy to explain how to run a program via the subprocess standard library (or with os.system). But viewing (i.e. “opening” a document) isn’t really solved at all.

What I propose is to have a simple, cross-platform way to supply the name of a regular file (not a directory, although maybe following symlinks would be a good idea) and have the file opened for viewing in an appropriate program, assuming one could be determined. It should do so regardless of whether the system recognizes the file as “executable”: executable scripts should be treated as plain text, and compiled MZ/ELF/etc. binaries should be treated as generic binary blobs. The purpose is to provide a way to view the file contents that is safe, but optimized for the context (e.g. files that are supposed to represent images can be viewed as images).

By default, this would attempt to determine an appropriate viewer application, and then run it with subprocess.call. As a convenience, the choice of application could be overridden - this still simplifies the subprocess.call interface somewhat - but this code should very intentionally not offer anything else. Crucially, the path provided would never itself be executed.

Thus, something like:

# It might not be viable to separate this out;
# see notes in the next section
def viewer_for(path): 
    ... # implementation TBD

def view(path, viewer=None):
    _ensure_viewable(path) # raises exceptions for directories etc.

    # The actual code would include more error handling to determine
    # whether failures are due to a `viewer_for` failure (in turn
    # because of e.g. `xdg-open` being missing on Linux), a missing
    # `viewer` (whether explicitly provided or determined by
    # `viewer_for`), or a failure in `subprocess.call`.
    viewer = viewer or viewer_for(path)
    subprocess.call([viewer, path])

Prior work(arounds)

The standard recipe for “opening” a file is to detect the platform and use a corresponding OS-specific approach. For example, from the top answer on that Stack Overflow link:

import subprocess, os, platform
if platform.system() == 'Darwin':       # macOS
    subprocess.call(('open', filepath))
elif platform.system() == 'Windows':    # Windows
    os.startfile(filepath)
else:                                   # linux variants
    subprocess.call(('xdg-open', filepath))

However, this is substandard for several reasons:

  • It takes multiple standard library imports for a single task, and fairly lengthy. I would struggle to memorize something like this, especially since it doesn’t use subprocess.call on Windows.

  • It’s not clear what the correct way is to detect the platform (platform.system()? os.name? sys.platform?).

  • Any errors that come up aren’t translated in context, and it might not be clear whether they come from a successfully launched viewer or from the attempt to launch.

  • xdg-open isn’t guaranteed to be available on Linux (but it still makes sense to try to use it, and report an error otherwise).

  • It’s insecure: executables (including scripts with +x set on Linux) might execute rather than being handed to a viewer. Similarly, I would argue that opening Internet URLs should explicitly not be supported. (Besides, we already have webbrowser for that.)

It’s not clear to me whether open and xdg-open (and I guess start on Windows[1]) are actually fit for the task here. Ideally this would all use system functionality that only determines a path rather than actually spawning a process for the viewer. With appropriate checks up front, it seems like it could be made safe to end up calling something like subprocess.call(['xdg-open', path]). But 'xdg-open' would not be a very satisfactory result to return fro a hypothetical viewer_for. At any rate, the goal is explicitly not to provide an interface to these roughly-equivalent, platform-specific tools, but instead to try as much as possible to provide lowest-common-denominator, cross-platform functionality in a secure way.

Justifications (and history)

Adding functionality like this to the standard library was proposed in 2008, where it was mostly ignored until a flurry of activity in 2012 that still led nowhere (and some commentary in 2018). I think the lack of this functionality has been felt by many this whole time.

I think functionality like this belongs in the standard library because:

  • It’s hard to do by hand

  • It’s hard to find an implementation on PyPI (what would you search for?)[2]

  • It’s easy to mess up the semantics

  • shutil is a natural home for it

  • doing this might help discourage Windows devs from inappropriate use of os.startfile.

I argue that view is a suitable name for the functionality, because a properly scoped, secure version should not attempt to execute executables (a user who wants this functionality should be using subprocess instead), open URLs in a browser (the point is to work with the local filesystem), or support other actions the way that os.startfile does (that bloats the interface and exposes an additional, inferior way to do subprocess/os.system type things).

Other actions that could be taken with files, could be considered later as separate named functions - e.g. shutil.edit to require a program that is capable of editing the file, shutil.print to send it to a printer, shutil.explore to open a file explorer window for a directory, etc. These are out of scope for the current proposal, but the point is that they make more sense as separate functions. (The code in the original proposal tried to expose a bunch of os.startfile functionality, but then it would be silently ignored on non-Windows platforms - this seems hardly ideal.)


  1. Of note here: the standard library documentation describes os.startfile as an interface to start, but on Windows that program doesn’t (AFAICT) actually do the interesting work of looking up a file association - that’s done by the shell itself, and (again AFAICT, but I’m not set up to test it) something like subprocess.call("foo.txt", shell=True) would already work on Windows. On the flip side, I’m not certain that subprocess.call(["start", "foo.txt"]) works without shell=True. ↩︎

  2. I did find one attempt but it seems poorly maintained and not at all well known. It does vendor xdg-open which seems like a smart move, but on the other hand that probably has dependencies which haven’t been investigated). It also has most of the issues I described above; it’s still fundamentally trying to open a file rather than view it. ↩︎

2 Likes

This would be a fantastic library if it works, even just as an a library on pip. But possibly because it’s so ambitious, I see large potential for scope creep, and endless issues from users’ raised expections (“why doesn’t it open my file?..”).

As you’ve phrased it:

simple, cross-platform
have the file opened for viewing in an appropriate program, assuming one could be determined
The purpose is to provide a way to view the file contents that is safe

A lot of file types can be delegated to a web browser. For the rest, would it be necessary to audit a cross platform list of safe viewing applications for each file type? Would it recommend the user install something if an approved file viewing program is not available?

An example list of file extensions that it would support, would be helpful.

This project’s made a lot of progress, and supports a large number of file extensions:

While I agree that the this is an interesting suggestion, I can’t imagine a feasible way to implement this in a user friendly way.

Let’s take a very simple example, windows, two file kinds, .py, .txt. What is the proposed process to find the applications to use?

  • Use whatever the system registry would find? Doesn’t work, since it might execute .py files
  • Ignore user preference and default both of these to notepad? I would be very disappointed that notepad++, my preferred editor for both wouldn’t be chosen.
  • Filter the registry and decided which applications are ok? How would this work in the general case?

This is definitely something that should be tried out on pypi first before being added to the stdlib, to iron out UX problems.

2 Likes

I mean “safe” only in the sense that the specified file isn’t treated as an executable - it’s secured for example against foo.txt being a chmod+x shell script on Linux. Of course viewing anything entails starting another executable, and any user can install any insecure executable for any particular nominal purpose.

The default platform logic should be used to choose an application according to how it’s already configured. If it’s delegated to a browser, fine - it’s still viewing a local file, not a URL. Recommending installed software is way outside scope (I can’t even imagine an approach for that), as is auditing (that would require being highly opinionated, plus having access to a CVE database or something).

I intend that the platform-specific logic, and local file associations, would be used. I haven’t actually studied how e.g. xdg-open makes its choices; perhaps it’s not based entirely on the extension. But at any rate, file extensions (and headers) only diagnose what kind of file it is. It’s up to the local configuration to determine what the default text viewer, image viewer etc. are.

Good stuff. I think the feature I would like, is closer to what preview-generator has already done.

I.e. I’d prefer to avoid accidentally opening Visual Studio, simply to view a text or xml file with a particular extension, that a resource intensive app put itself in charge of when it installed.

As much as I am a big fan of “batteries included” and
use-only-stdlib-for-everything philosophy, I believe this exactly
is a project which at least should be developed as a standalone
PyPI module, which can be eventually later incorporated into
stdlib.

Best,

Matěj

1 Like

I can imagine two interpretations:

  1. It’s fine if .py files get executed, as long as this is documented as being within the spec - the user has configured python.exe as the “viewer” for Python files, and what happens from there isn’t shutil’s fault.

  2. Ideally, the system would have some existing generic association to open “plain text” files with Notepad++, and then we would determine the file type (perhaps by heuristics on the contents or via some other system utility) and then look up that association. But it seems that the OSes don’t actually work this way. If you override the native file-type detection scheme (which I think is purely by extension on Windows?) then you won’t end up with anything that can interoperate with native file-type association lookup. :frowning:

view and open are distinct actions. If I ask for a file to be viewed I do not expect it to be opened. At least in Windows the API has this distinction.

I’m not sure there is a view version of macOS open or unix xdg-open.
So… you seem to missing the API to implement shutil.view cross-platform.

So it isn’t “safe” in any meaningful way, you still can only do it with trusted files (since who knows what associations the end user has on their system), meaning you might as well use startfile on windows. Note that by this exact train of thought, I would expect the behavior on linux system to be to respect the shebang - Which is clearly not what you want I think.

Windows does have extensive File Associations, but these aren’t reliably maintained from my experience, open gets used as the generic choice, including execution, and there doesn’t appear to be an endorsed standard for a verb like view.

Does this API even reliably exists on windows? There appears to be a semi-standard edit, which however probably wouldn’t work for stuff like images or pdfs.

Thanks for the feedback. This looks completely not implementable, never mind its suitability for inclusion. But now I can at least do a proper writeup for Codidact about why that is (along with offering the “open” code with appropriate caveats). This also ties in to things I wanted to write generally about file formats and the nature of data, so.

Edit: As such, I’m going to take this out of Ideas . More interested in this point at looking at what semantics others are interested in and what APIs are available.

I basically agree with everything said so far in this thread, and I’d add a further comment - this sounds like it would be a really good source of CVEs, either for Python itself, or for applications that use the API. Beacuse by making the function “do what I mean”, you either commit to making sure it’s safe on the user’s behalf (a promise I doubt the core devs would be willing to make) or you make safety the responsibility of the code using the API (which, given the high level nature of the API and the complexity of determining safety, is unlikely to be something developers do correctly in practice).

I don’t write the sort of code that would need this myself, but IMO it would be a fantastic 3rd party module on PyPI. To address potential objections to that idea:

  1. You say it’s not discoverable, but shutil.view seems like a good name in the stdlib, so why wouldn’t fileview be a good name for a PyPI package?
  2. The faster release cadence of a PyPI package would make getting the right balance of “safe vs usable” much easier to achieve (through user feedback) than going straight to the stdlib.
  3. External development could be done by an individual or group with a strong interest in, and understanding of, the intended use cases and the relevant platform capabilities.
  4. Getting someone to put in the time and effort to implement this would be the same - there’s no reason to presume that the existing core devs have the time (or interest) to do this, so it would almost certainly be be a 3rd party PR anyway.
  5. A 3rd party package could leverage other PyPI libraries, if needed. That could make getting an initial implementation up and running easier.
1 Like

Wait, hold on. Does os.startfile actually recognize a 'view' operation? The documentation only tells me

When another operation is given, it must be a “command verb” that specifies what should be done with the file. Common verbs documented by Microsoft are 'open', 'print' and 'edit' (to be used on files) as well as 'explore' and 'find' (to be used on directories).

and I haven’t had much success finding said Microsoft documentation, either.

The trick is that “common verbs” is not a complete list. Arbitrary programs can define arbitrary verbs for arbitrary file extensions. That is what I meant above with “Does this API even reliably exists on windows?”. Programs can support view if they want to, and I am sure that some do, but it’s not exactly common.

1 Like

I don’t suppose you have a documentation link or guide for the full interface, then? I had imagined something where Windows was in control of the verb list. But I only ever knew as much about the Registry as I had to, so.

This I already linked above:

And these two that are linked from there:

From here there might be more interesting links, but I couldn’t really find anything.

1 Like

No, basically you (and by “you”, I mean the end user, not just the OS or a sysadmin or an installer program) can add entries to the registry that say, in effect “for file extension .xxx define the verb foo to run C:\MyProgs\bar.exe %1 -view” (where %1 is replaced by the filename)[1]. The rules are a little more complex than that, and are somewhat under-documented (there’s various %-substitutions) but that’s the gist.

But essentilally no, there’s no well-defined meaning for “view” on Windows, any more than there is on Linux.


  1. there’s also shell extensions, which are DLLs that the OS will load and call to implement certain verbs, again based on definitions in the registry ↩︎

I think I’m mis-remembering that there is open and print actions, so maybe not.

The Windows API function ShellExecuteW() is made available as os.startfile(). The latter is a bit of a misnomer since the ‘file’ can also be the name of any object in the “shell” namespace or protocol namespaces such as “http”. The function supports passing an operation (e.g. “Open”, “Edit”, “RunAs”), command-line arguments, the initial working directory, and whether a started application should automatically show a window (an app may ignore this).

Alternatively, WinAPI AssocQueryStringW() can be used to get the template command line that’s associated with a given file type or protocol (e.g. “.py” or “http”) for a given operation. After interpolating the parameters in the command, it can be executed via subprocess.Popen.

Usually the operation that gets executed is the default operation (e.g. double-clicking on a file in Explorer executes the default operation). This can be configured for a file type. Otherwise the shell uses “Open”. For example, “.py” files could be associated with a progid that configures the default operation as “Run”, which runs a script using the py launcher. This progid could also configure the operations “RunAs” (run as an administrator), “RunAsUser” (run as another user), “Open” (e.g. open in an editor), “Edit”, and “Print”.


On Windows, passing shell=True to subprocess.Popen uses the “ComSpec” shell, which is almost always CMD. In CMD, start is an internal command. The implementation first parses out the command from the command line and searches for it. If it’s either an internal command (e.g. pause) or a “.bat” or “.cmd” file, then start executes it in a new instance of CMD via the command line "%ComSpec% /K {command line}". If some other file type is found, start executes the given command line without modification.

If start isn’t able to resolve the command line to anything that it can execute by spawning a process, or if CreateProcessW() fails with one of a small set of error codes (e.g. ERROR_BAD_EXE_FORMAT or ERROR_ELEVATION_REQUIRED), then start falls back on the shell API function ShellExecuteExW(). This is an extended version of ShellExecuteW(). A key difference is that it can return the process handle (if the operation actually spawned a process) in order to be able to wait on the process, terminate it, and get its exit status.

1 Like

Why not a directory? I use open <dir> more often in my terminal than open <dir>. Opening the downloads folder for example seems like it could be a pretty useful thing to do.

1 Like