Handle not executable directories for os.listdir

EDIT: I don’t know why this was moved to “Python Help”, but I don’t need help with listing the files in these directories.

I have a directory with “rw-” permisions with a file inaccessible.txt:

>>> import os
>>> os.listdir("directory")
['inaccessible.txt']
>>> os.path.exists("directory/inaccessible.txt")
False

That’s inconsistent behaviour: os.listdir says directory/inaccessible.txt exists, while os.path.exists says it does not.
This can be solved simply be raising a PermissionError (or a subclass) for os.listdir when the directory is not executable.
I can’t think of any situation outside of a CTF where it would be useful to be able to list the files in such directories.

Note that this is different from a directory with “-wx” permisions with a file secret.txt:

>>> import os
>>> os.listdir("directory")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 13] Permission denied: 'directory'
>>> os.path.exists("directory/secret.txt")
True

In this case, files can still be accessed if you know the filename.

Just because you can’t think of a usecase doesn’t mean it doesn’t exists. Python shouldn’t hide information that exists (i.e. what a directory contains) unless it’s wrong.

I would guess that os.listdir and os.path.exists pretty directly map to underlying system calls. Unless there are very good reasons, we shouldn’t deviate from them. Having suprising behavior is something you can report to linux if you want to.

Note that similar “invariant” breakage can happen if a different process/thread deletes the file in the time between the os.listdir and os.path.exists/open call. You can’t actually rely on this if you want to write a resilient program.

2 Likes

To see exactly what python called you can run under strace.
Then try the os.listdir() and os.path.exists() to see what system calls are made.

strace --trace=%file python

I’d need to be more awake to research what exactly is going on.
But will note that normally the permissions on a directory are “rwx” or “rx”.
Is the “x” missing for a reason?

But will note that normally the permissions on a directory are “rwx” or “rx”.
Is the “x” missing for a reason?

I removed it with chmod: Execute vs Read bit. How do directory permissions in Linux work? - Unix & Linux Stack Exchange

If you use strace, as I suggested, you can find out the exact system calls python uses and look up their man pages and do other research to see why you get the results you do.

The results are expected and not a bug in linux or python.

It’s expected on POSIX that read and search (execute) access are independent. Python should not, for example, call access() or faccessat() on a directory in order to pretend that os.listdir() failed if the caller lacks search access. The os module is low-level, based directly on POSIX libc functions, such as opendir(), rewinddir(), readdir(), and closedir(). It’s not the place for implementing high-level policies that contravene the operating system.

On Windows it’s mostly a non-issue because users are almost always granted default-enabled “SeChangeNotifyPrivilege”. When this privileged is enabled, a filesystem must bypass checking permissions when traversing directories. It only checks the permissions of the opened file or directory. Even without this privilege, os.stat() on Windows falls back on getting stat information from the directory listing. WinAPI FindFirstFileW() is able to return data for a specific file, including the normalized name, file attributes, reparse tag (i.e. the file type if it’s a reparse point, such as a symlink or mountpoint), file size, and time stamps (creation/birth, modify, access).

1 Like

An alternative viewpoint here is that it’s the “False” that is wrong and that exists() should be raising the PermissionError. However, as noted in os.path — Common pathname manipulations — Python 3.12.2 documentation it is a documented possibility that you might get a False rather than an error.

2 Likes

Okay; and in doing so, is there a specific practical problem you hoped to solve?

The accepted answer there tells us:

So, you have a directory where the user is allowed to list the files, but not allowed to enter it to access the files. When you try using a function that claims it lists the files, you see the file listed. When you try using a function to test whether the file exists, you get a message indicating that there is no file to access.

Similarly, if you try using the shell:

$ mkdir nonexecutabledir
$ cd nonexecutabledir/
$ touch file.txt
$ cd ..
$ chmod -x nonexecutabledir/
$ ls -l nonexecutabledir/
ls: cannot access 'nonexecutabledir/file.txt': Permission denied
total 0
-????????? ? ? ? ?            ? file.txt
$ file nonexecutabledir/file.txt 
nonexecutabledir/file.txt: cannot open `nonexecutabledir/file.txt' (Permission denied)
$ file nonexecutabledir/file2.txt 
nonexecutabledir/file2.txt: cannot open `nonexecutabledir/file2.txt' (Permission denied)

You can’t get information about the file from ls, but you can know that it exists. You also can’t get information from other utilities (from outside the directory), and you get the same error message that you would with a file that actually isn’t there.

So I can’t really see a reason to expect anything else.

The information os.listdir() provides in this case is useless: you don’t have permission to this directory, so why do you need to know which files it contains?

I ran into this issue while creating my own file manager, but it applies to all folder processing:

It refreshes every time the directory is updated, but it still shows these files. Highlighting the selection with a create symbol, like when entering the name of a non-existent item.

Note: the program doesn’t use chdir (it lies), so it doesn’t get any errors.

With my wrapper function it does this:

And it does this for directories with -wx permission:

Ok, so you found a context where different behavior is useful, and you managed to implement it. Great! Why does that require a language change considering everything else also mentioned in this thread?

1 Like

I’ll just quote this as a response:

Yep. That.

2 Likes

I’m actually getting a permission error on macOS after listing the files with ls:

ls 666
secret		secret.txt
ls: fts_read: Permission denied

But I believe that’s because os.listdir() doesn’t use “fts_read”.

A quick look at fts shows its a high level library function. Higher level the os.listdir is. You could write a python fts library using the os functions, but not the other way around.

| Nice Zombies Nineteendo
April 1 |

  • | - |

I’m actually getting a permission error on macOS after listing the files with ls:

ls 666
secret		secret.txt
ls: fts_read: Permission denied

That’s odd, that doesn’t happen on my MacOS (10.12.6).

Back in the 70s and 80s, most Unix systems didn’t provide a library for reading directory entries. They documented a common file format for directories, and applications would read entries directly via the read() system call.

In 1983, 4.2BSD introduced the dirent functions, such as opendir() and readdir(), which provide an abstract interface to read directory entries. In 1988, System V Release 4.0 also supported these functions. POSIX standardized them in 1990. Nowadays many systems don’t even allow calling read() on an open directory, and readdir() relies instead on some other low-level system call. POSIX builds of Python use the standard dirent functions to implement os.listdir() and os.scandir().

In 1986, 4.3BSD introduced high-level functions for traversing a filesystem tree, such as fts_open() and fts_read(). In 1992, 4.4BSD revised the API. It was never standardized by POSIX. Note that fts_read() tries to fetch stat information for files via stat() or lstat() if the option FTS_NOSTAT isn’t specified. This will fail if the caller lacks search (execute) access to the directory.

4 Likes

The ls command is only showing you the literal file names, and can’t for example colour them to indicate file types, because of the folder permission. It’s just like what I showed in my post on Linux. A plain ls call there, too, will show a permission error while also listing the directory contents in a basic way.

@storchaka, why did you move this to “Python Help”? I would like to change the behaviour of os.listdir()

EDIT: I don’t know why this was moved to “Python Help”, but I don’t need help with listing the files in these directories.

Because its not a bug and the python implementation is not going to change.

  1. Its working how its designed to.
  2. If ever changed would break existing code.

That leave us explaining as “python help” why its implemented as it is.
Which comes down to os.listdir and os.path.exists exposing POSIX
file API behavior. Behavior that is 40+ years at this point in time I think.

5 Likes

I concur with @barry-scott.

It is not even particularly related to Python. It is how your OS works. If you think that this behavior is not correct, this is not the place to discuss it. We do not write OSes.

1 Like