Os.path.ishidden(), pathlib.Path.is_hidden() & os.DirEntry.is_hidden()

Currently when you use os.listdir(), Path.iterdir() / os.scandir() to list the items in a directory, this also includes hidden items.
You can check whether they start with ., but that doesn’t detect paths such as C:\ProgramData on Windows & os.path.expanduser(~/Library) on macOS.

My proposal is to add a function that handles these paths correctly:

import os
from os.path import basename
import stat

if os.name == "nt":  # ntpath.py
    if hasattr(os.stat_result, 'st_file_attributes'):  # Windows
        def ishidden(path):
            """Test whether a path is hidden."""
            try:
                st = os.stat(path)
            except (OSError, ValueError):
                return True

            return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
    else:
        def ishidden(path):
            """Test whether a path is hidden."""
            os.fspath(path)
            return False
else:  # posixpath.py
    if hasattr(os.stat_result, 'st_reparse_tag'):  # macOS
        def ishidden(path):
            """Test whether a path is hidden."""
            try:
                st = os.stat(path)
            except (OSError, ValueError):
                return True

            return bool(st.st_flags & stat.UF_HIDDEN)
    else:  # Unix
        def ishidden(path):
            """Test whether a path is hidden."""
            path = os.fspath(path)
            if isinstance(path, bytes):
                prefix = b"."
            else:
                prefix = "."

            return basename(path).startswith(prefix)

Usage:

from os import listdir
from os.path import join

directory = expanduser("~")
for item in listdir(directory):
    path = join(directory, item)
    if not ishidden(path):
        print(path)

Note: New files starting with . on macOS, as well as some system files on Windows aren’t hidden in the current implementation & the root on Windows is hidden.

It’s an interesting idea. I think it would make a bit more sense the other way around as ishidden() since “hidden” is a defined attribute. It’s telling that all three of your implementations return not :slight_smile:

I think there has been a proposal for this before; digging up and addressing any concerns from prior discussions should be the first step. Off the top of my head, one edge case to consider is a non-hidden file in a hidden directory: is it hidden or not?

1 Like

I am more in favor of adding a ‘hidden’ parameter to the os.listdir function instead. Also, it’s not clear where the proposed function should be added. However, without taking into consideration (and addressing) previous discussions, we would be repeating ourselves.

On unix systems hidden is a convention, filenames starts with “.” and is not “.” Or “…”. On Windows it is a attribute that requires an API call that is not reasonable for os.listdir to do us it is an extra call, as it will slow down os.listdir.

2 Likes

Actually, a hidden=True parameter for os.listdir() and os.scandir() would be nearly free to implement on Windows, though that in itself doesn’t make it a good idea. Directory entries on Windows include basic metadata, including the long filename, short filename (if any), file attributes, reparse tag (if any), file size, and timestamps (birth, modify, access).

4 Likes

There are also backup files ending with ~ , and files in a hidden directory which are inherently hidden.

A hidden folder (sometimes hidden directory) or hidden file is a folder or file which filesystem utilities do not display by default when showing a directory listing[1].

Using hidden=True doesn’t incur any additional cost since it reflects the current behavior. Introducing hidden=False as an option for os.listdir would bring it in line with common practices found in command-line tools like dir /a and ls -a


  1. Hidden file and hidden directory - Wikipedia ↩︎

I think it would make a bit more sense the other way around as ishidden() since “hidden” is a defined attribute. It’s telling that all three of your implementations return not :slight_smile:

True, I wanted to avoid needing to apply not to the result, but we can’t predict how the function will be used.

Off the top of my head, one edge case to consider is a non-hidden file in a hidden directory: is it hidden or not?

Windows treats the root directory as hidden, so we first need to decide whether we want to respect that fact.

I am more in favor of adding a ‘hidden’ parameter to the os.listdir function instead.

I was more thinking Path.iterdir() could filter out the hidden items using this function, as os.listdir() is a low-level function.

Also, it’s not clear where the proposed function should be added.

I’ve updated the description to clarify this.

I’m not sure this API is needed. I would guess that in most cases you can fetch your directory listing and use a list comprehension to filter out the visible or invisible files.

2 Likes

My counter proposal is about avoiding this block of code:

directory = "C:\\"
items = listdir(directory)
filtered_items = [item for item in items if not ishidden(join(directory, item))]
for item in filtered_items:
    print(item)

We can already use a platform-specific one-liner to check if a file is hidden or not, e.g., os.path.basename(os.fspath(path)).startswith(b'.' if isinstance(os.fspath(path), bytes) else '.'). Also, ‘hidden’ is just one of the file attributes in Windows.

For Windows, it’s best to use os.scandir() for this. It gets the basic stat result from the entries in the directory listing, which is read in a batch of entries per system call. For example:

import os
import stat

def hidden(e):
   attrs = e.stat(follow_symlinks=False).st_file_attributes
   return attrs & stat.FILE_ATTRIBUTE_HIDDEN

filtered_entries = [e for e in os.scandir('C:/Windows') if hidden(e)]
for e in filtered_entries:
    print(os.path.normpath(e.path))

I’m not. One can easily imagine wanting to know this for an arbitrary
filesystem path, not just “what I get from listdir”.

I have other objects to adding it to listdir:

os.listdir, like most os.* things, is a pretty direct mapping to the
OS API of that name, in this case the POSIX listdir; putting more
knobs on it breaks that idea

Once we start adding parameters to special things, where does one stop?
parameters for only-files, only-dirs, not-files, not-dirs,
only-particular special file types,
not-decodable-with-my-current-locale, etc etc. I can imagine uses for
all of these in some circumstances, and as such I do not want them in
listdir, but as some kind of filter applied later eg a list
comprehension.

6 Likes

And you’re off into policy land and the bike shed of many colours.

I’d be in favour of something in pathlib which could test “would my
current platform consider this hidden?” And/or in os.path to match.

I gather Windows has an actual “hidden” file attribute? Some UNIXy
systems have something similar.

The venerable UNIX “starts with a dot” convention should be honoured on
a UNIX platform. What should one do on a UNIX platform which also has a
“hidden” file attribute? I’d lean towards honouring that, too.

1 Like

Do you have a reference to docs on this?

Wouldn’t it be more consistent if we had os.path.ishidden() & os.DirEntry.is_hidden()
like os.path.isdir() & os.DirEntry.is_dir(), etc.
Then we have a function for arbitrary paths & an efficient one for directory entries.

I’d be in favour of something in pathlib which could test “would my
current platform consider this hidden?” And/or in os.path to match.

Yeah, adding pathlib.Path.is_hidden() would definitely be useful, which will need os.path.ishidden() to handle OS differences.

https://ss64.com/mac/chflags.html
https://man.freebsd.org/cgi/man.cgi?query=chflags&sektion=1

Also on OSX/MacOS. Basicly BSDs.

1 Like

The big real use case I can come up with is GUI file chooser dialog boxes. But those are implemented in high-level UI frameworks who have these things worked out already. Once again, I’m not sure these APIs make sense in the stdlib.

Can be CLI as well:

This is implemented using only standard libraries (and is cross platform).

Well, it is still a pretty narrow use case :wink:

4 Likes