# Ergonomics of new pathlib.Path.scandir() **URL:** \ **Category:** Core Development\ **Created:** [November 19, 2024, 11:58pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721 "2024-11-19T23:58:34Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [November 21, 2024, 9:23pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/21 "2024-11-21T21:23:32Z")
> [@steve.dower](#): > > Yeah, exactly. Which is why (unfortunately) the only good way to do it (now) is to fully implement the optimised functions on the subclasses, rather than trying to abstract the optimisations across different implementations. Fair enough. And just to be clear, you’re discounting the idea of a `PathBase.scandir()` method that returns `os.DirEntry`-like objects because, from a user perspective, its only effect is to add a `Path.scandir()` method that duplicates `os.scandir()`? ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [November 21, 2024, 9:26pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/22 "2024-11-21T21:26:02Z")
> [@barneygale](#): > > from a user perspective, its only effect is to add a `Path.scandir()` method that duplicates `os.scandir()`? Right. The only thing it can do is invert the call order, and/or make life harder for non-FS path implementations. Let it stay for filesystem paths only, and the easiest way to do that is to stick to calling the `os` version when needed. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [November 21, 2024, 9:38pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/23 "2024-11-21T21:38:31Z")
Thanks as ever for your input Steve, that’s very helpful. I think I’ll wait for [this PR](https://github.com/python/cpython/pull/126263) to land, and then put in another PR that renames `PathBase.scandir()` to `_scandir()` and removes the docs. I still believe I can motivate the addition of a `scandir()` method (see below), but probably only in conjunction with making `PathBase` public, which is still a ways away. > [@steve.dower](#): > > and/or make life harder for non-FS path implementations I think most non-toy implementations will want a `scandir()`-like method to make use of cached directory children info, otherwise they’re leaving performance on the table or setting themselves up for a heap of work re-implementing globbing. If there’s no information available other than child names, a user implementation of `scandir()` could `return contextlib.nullcontext(self.iterdir())` instead. Perhaps this should be the default implementation of `PathBase.scandir()`, with `PathBase.iterdir()` remaining abstract for ease of entry. ---
**Author:** ![Nineteendo](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nineteendo/32/19122_2.png) [@Nineteendo](https://discuss.python.org/u/Nineteendo)\ **Post date:** [November 21, 2024, 9:49pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/24 "2024-11-21T21:49:33Z")
Can’t we just define our own short-lived `DirEntry` with cached attributes, allow you to write an implementation for non filesystem paths and make `.path` a `Path` instance? Then we could return that for `Path.scandir()`. Sure, it doesn’t cover all usecases, but it doesn’t have to, right? ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [November 21, 2024, 9:59pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/25 "2024-11-21T21:59:00Z")
If the type of `os.DirEntry.path` were the only problem, I’d suggest we soft-deprecate `dir_entry.path` in favour of `os.fspath(dir_entry)`, which to my mind is more explicitly `str` / `bytes`. But IIUC, Steve’s argument is that any `scandir()` method isn’t compelling enough to add at the moment, even without that particular wart. ---
**Author:** ![Nineteendo](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nineteendo/32/19122_2.png) [@Nineteendo](https://discuss.python.org/u/Nineteendo)\ **Post date:** [November 21, 2024, 10:04pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/26 "2024-11-21T22:04:50Z")
I thought the main problem was that `os.scandir()` only returns file system paths: > [@steve.dower](#): > > Anything we attach to `Path` is an _abstract_ operation - we have to allow for it to be implemented by another kind of Path-based hierarchy, and its return values should return compatible Paths that remain within that hierarchy. For example, if you create a path referring to an S3 bucket/blob container, then `Path.scandir()` needs to return more paths within that container, rather than file system paths. So, if we allow you to overwrite the behaviour in your S3 implementation, that wouldn’t be an issue. But even so, this might be overkill and simply not worth the effort. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [November 21, 2024, 10:12pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/27 "2024-11-21T22:12:16Z")
FWIW, I do think there’s a bit of a divide between folks with a formal education in API design vs casual Python users in terms of how some `pathlib` functionality is viewed. Code like this might sends shivers up some folks’ backs (and I’m not saying they’re wrong): > But for a lot of beginner/intermediate Python users, being able to `path.rmdir()` rather than `os.rmdir(path)` is compelling, and this pattern might be _the_ selling point of the entire module for some folk. Whether `scandir()` deserves a spot in the pantheon is another matter, mind. ---
**Author:** ![dg-pb](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dg-pb/32/16248_2.png) [@dg-pb](https://discuss.python.org/u/dg-pb)\ **Post date:** [November 21, 2024, 10:28pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/28 "2024-11-21T22:28:20Z")
On a different (but possibly related) note. Are there some serious issues of not being able to inherit from `DirEntry` / initialise explicitly? E.g. I have my own `DirEntry` class defined to have consistent elements returned from some utilities. Maybe if it was a bit more flexible/inheritable object, dealing with this issue would be a bit easier? ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [November 22, 2024, 12:05am UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/29 "2024-11-22T00:05:21Z")
> [@Nineteendo](#): > > So, if we allow you to overwrite the behaviour in your S3 implementation, that wouldn’t be an issue. > > But even so, this might be overkill and simply not worth the effort. Yeah, I think it’s not worth the effort. Why should we make subclassers implement both `iterdir` and `scandir`? (FWIW, it’s probably _even more_ efficient to implement an online storage glob with a search query and not directory iteration, so implementing `scandir` isn’t always going to help anyway.) > [@barneygale](#): > > But for a lot of beginner/intermediate Python users, being able to `path.rmdir()` rather than `os.rmdir(path)` is compelling, and this pattern might be _the_ selling point of the entire module for some folk. Whether `scandir()` deserves a spot in the pantheon is another matter, mind. I mean, I much prefer having all the essential functionality on my `Path` directly, as it also means I can write algorithms around `Path`-like objects of any kind without having to assume that they’re file system paths. But `scandir()` doesn’t deserve that spot. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [December 6, 2024, 10:42pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/30 "2024-12-06T22:42:32Z")
I’ve now reverted the addition of `Path.scandir()`: > > > Remove documentation for \`pathlib.Path.scandir()\`, and rename the method to \`\_sc…andir()\`. In the private pathlib ABCs, make \`iterdir()\` abstract and call it from \`\_scandir()\`. > > It's not worthwhile to add this method at the moment - see discussion: https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721 > > > > \* Issue: gh-125413 > > > > > \---- > 📚 Documentation preview 📚: https://cpython-previews--127377.org.readthedocs.build/ There’s some discussion here about how we might expose the cached file type in pathlib: > > > Change the default value of \`PurePathBase.parser\` from \`ParserBase()\` to \`posixp…ath\`. As a result, user subclasses of \`PurePathBase\` and \`PathBase\` use POSIX path syntax by default, which is very often desirable. > > Move \`pathlib.\_abc.ParserBase\` to \`pathlib.\_types.Parser\`, and convert it to a runtime-checkable protocol. > > Add \`pathlib.\_types.DirEntry\`, which is a protocol for directory entries. Objects of this type provide a subset of the \`os.DirEntry\` API, specifically those methods and attributes needed to implement \`glob()\` and \`walk()\`. > > Add \`pathlib.\_types.StatResult\`, which is a protocol for stat() results. Objects of this type provide a small subset of the \`os.stat\_result\` API, specifically attributes for the file type, permissions and location/offset. > > No user-facing changes as the pathlib ABCs are still private. > > > \* Issue: gh-127456 IIUC the current proposal is as follows: we add a `Path.info` property, which exposes an object with `is_file()`, `is_dir()` and `is_symlink()` methods. Accessing the property doesn’t perform FS access. The property type is either `os.DirEntry` (for paths generated by `Path.iterdir()`) or a new pathlib type (other paths). In both cases, `stat()` results are cached. I propose that `p = Path(p)` is sufficient for clearing the `p.info` cache, and so we don’t need to provide further methods/property deleters/etc. If we add something like `Path.info`, then: - Users will be able to get at the `os.DirEntry` objects internally generated by `Path.iterdir()`, which are otherwise thrown away - Users writing performance-sensitive code can rely on `path.info` for cached file status _whether or not_ the path object was generated by `iterdir()` - In the pathlib ABCs, we can replace `PathBase.stat()` with `PathBase.info`. The former is too low-level and awkward to implement for most virtual filesystems. Here’s a rough example: ```python >>> from pathlib import Path >>> p = Path.cwd() >>> p.info >>> p.info.is_dir() True >>> q = next(p.iterdir()) >>> q PosixPath('Lib') >>> q.info ``` Thoughts? ---
**Author:** ![Ferdi265](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ferdi265/32/24060_2.png) [@Ferdi265](https://discuss.python.org/u/Ferdi265)\ **Post date:** [December 6, 2024, 11:47pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/31 "2024-12-06T23:47:01Z")
This sounds amazing! ---
**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\ **Post date:** [December 8, 2024, 6:01am UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/32 "2024-12-08T06:01:38Z")
@barneygale The `Path.info` approach sounds good to me. I think it would nicely avoid the conceptual problems that confounded the first attempt that relied solely on `DirEntry`. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [December 9, 2024, 9:22am UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/33 "2024-12-09T09:22:04Z")
Thinking about this further, I’m more and more convinced that there should be a `get_info()` _method_, which could have arguments: - `follow_symlinks` - `cached=True`. When you need fresh values, `p.get_info(cached=False)` would communicate the intent more clearly than `Path(p).info`. Consider not using `os.DirEntry` directly. A new type could _also_ have `st_*` attributes from `stat_result` (each only present if available for the filesystem), so you can use the same caching mechanism for the low-level values. A new class could also be extendable in other ways. Brainstorming: - `posix_permissions()` to get the standardized part of `st_mode` - `link_target()` (a path) - `get_key()` (for `samefile()`, defaulting to e.g. `(FSFileInfo, st_dev, st_ino)`) - `access_time()`, `mod_time()`, `change_time()`, `birth_time()` returning _datetime_ objects - other implementations could be encouraged to use a project-specific prefix for custom fields (`zip_compression`, `tar_offset`, `git_oid`, `s3_acl`). ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [December 9, 2024, 11:21am UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/34 "2024-12-09T11:21:55Z")
> [@encukou](#): > > Consider not using `os.DirEntry` directly. Yeah, or at least not _specifying_ that it’s used. I don’t see anything wrong with the object being defined as a protocol rather than a concrete object, and if it’s one that a `DirEntry` can fulfill for stdlib needs, well that’s just convenient 🙂 I would personally still like to see a model where the cached objects look enough like the `Path` instance that we can duck type the entire thing (IOW, a subclass that has pre-cached values, as I described on the issue). I’ve never hit my own case where the difference between the type of a path at the time I created/acquired\[1\] the `Path` and when I tested the value.\[2\] For whatever reason, every time I’ve held onto a path for a long time (i.e. between operations, where it’s reasonable for a user to have gone in and modified the files they told me to use) it’s been stored as a string. Maybe that’s just my scenarios, though? Storing a timestamp and invalidating the cache after ??? seconds would still be a win, perf-wise, if that’s a concern. The advantage is that _all_ users get performance improvements at very little risk, and users only have to learn a new concept if they’re doing the less common operation. I don’t like Python being known as a language where you have to learn lots of tricks to make it faster - it should just be faster, and the tricks are there to resolve complicated issues or mismatched expectations. * * * 1. I expect we’d only cache values when they’re known by the constructor. So `Path(x)` does _not_ cache, but `p.glob("*")` on a system with `readdir`/`FindNextFile` _would_ cache whatever it knows. 2. There’s still a TOCTOU risk that gets bigger with caching, but doesn’t go away by clearing the cache. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [December 9, 2024, 6:31pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/35 "2024-12-09T18:31:00Z")
> [@steve.dower](#): > > I don’t like Python being known as a language where you have to learn lots of tricks to make it faster - it should just be faster, and the tricks are there to resolve complicated issues or mismatched expectations. I wholly agree here, and I recall Guido previously saying something similar about adding new APIs vs fixing what’s already there. But IMO making `Path.is_dir()` and friends return cached results is a bridge too far. Users that rely on those methods retrieving up-to-date results would find their scripts break in hard-to-understand ways, and I can’t see a way to cushion the change with some kind of warning. Even if we restrict it to objects generated from `iterdir()`, the significant behaviour change remains. We could _maybe_ do it by introducing a new superclass (not subclass) of `Path` that’s allowed to cache results. That way we don’t break any `Path` guarantees. If there’s a route toward doing this then I’m open-minded, but right now it’s not clear to me. ---
**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\ **Post date:** [December 9, 2024, 6:42pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/36 "2024-12-09T18:42:57Z")
Maybe a keyword option for `iterdir` that enables caching on generated `Path` objects? That would at least make this an explicit opt in with very little hassle. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [December 9, 2024, 7:22pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/37 "2024-12-09T19:22:59Z")
Even that is problematic: ```python # 3rd party library code def delete(path): assert isinstance(path, pathlib.Path) os.unlink(path) assert not path.exists() # should be up-to-date # application code for p in Path().iterdir(cache_results=True): delete(p) ``` Suddenly `delete()` needs to deal with stale results, with no opt-out. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [December 9, 2024, 7:58pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/38 "2024-12-09T19:58:33Z")
> [@barneygale](#): > > Suddenly `delete()` needs to deal with stale results, with no opt-out. It always needed to deal with them. Unfortunately in this case, it goes from an occasional issue where the file is locked, isn’t a real file, or gets recreated quickly\[1\] into one that _always_ fails the assertion, but the possibility was already there. [LBYL](https://devblogs.microsoft.com/python/idiomatic-python-eafp-versus-lbyl/) simply can’t be relied on for filesystem stuff. The most obvious fix (for a caching change; not for the existing TOCTOU issue) would be to make it `assert not os.path.exists(path)`, since they’ve already chosen to use `os` methods. Or `path.unlink()` which could, conceivably, update/invalidate its cache. It makes the transition more complicated, but I’m not massively upset by mixing libraries leading to non-obvious behaviour. * * * 1. In relative terms. For a mounted network file system, there could be plenty of real time. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [December 9, 2024, 9:54pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/39 "2024-12-09T21:54:38Z")
I’m gathering some historical background on caching/not caching. Antoine’s initial version of PEP 428 posted to python-ideas included stat caching: [https://mail.python.org/archives/list/python-ideas@python.org/message/2FYSJJZ5XIVY5MAFDIE6G6K2AXGLOGPB/](https://mail.python.org/archives/list/python-ideas@python.org/message/2FYSJJZ5XIVY5MAFDIE6G6K2AXGLOGPB/) Guido warns about caching: [https://mail.python.org/archives/list/python-ideas@python.org/message/SSMN57AHKM6HBKDXQUDEGWMF2LNOGAUU/](https://mail.python.org/archives/list/python-ideas@python.org/message/SSMN57AHKM6HBKDXQUDEGWMF2LNOGAUU/) Alyssa explains why we might want different behavior in `os.DirEntry` and `pathlib.Path`: [https://mail.python.org/archives/list/python-ideas@python.org/message/KI2BQL64IAUVK5ZMR2ZRGXLCKQAHMXRO/](https://mail.python.org/archives/list/python-ideas@python.org/message/KI2BQL64IAUVK5ZMR2ZRGXLCKQAHMXRO/) Paul Moore laments the lack of caching in `pathlib.Path`: [https://mail.python.org/archives/list/python-ideas@python.org/thread/C7RDAJZN2DHNV5JQUKLHZZINVAFAHT3X/#C7RDAJZN2DHNV5JQUKLHZZINVAFAHT3X](https://mail.python.org/archives/list/python-ideas@python.org/thread/C7RDAJZN2DHNV5JQUKLHZZINVAFAHT3X/#C7RDAJZN2DHNV5JQUKLHZZINVAFAHT3X) In-depth discussion of pathlib caching behaviour kicked off by Ram: [https://mail.python.org/archives/list/python-ideas@python.org/thread/RFTCPWUR6BS6EDVYSV5YTY3IDK345HGA/#6XO2PWUQ44LLXXRD5MJZNEUZIKDHF6VE](https://mail.python.org/archives/list/python-ideas@python.org/thread/RFTCPWUR6BS6EDVYSV5YTY3IDK345HGA/#6XO2PWUQ44LLXXRD5MJZNEUZIKDHF6VE) Guido works on adding a stat cache based on Ram’s suggestion: > > > BPO | \[26031\](https://bugs.python.org/issue26031) > \--- | :--- > Nosy | @gvanrossum,… @pitrou > Files | \\[statcache.diff\](https://bugs.python.org/file41521/statcache.diff "Uploaded as text/plain at 2016-01-06.22:41:28 by @gvanrossum")\ > > \\*Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.\*\ > > \\Show more details\\ > > GitHub fields: > \`\`\`python > assignee = None > closed\_at = \ > created\_at = \ > labels = \['type-feature'\] > title = 'Add stat caching option to pathlib' > updated\_at = \ > user = 'https://github.com/gvanrossum' > \`\`\` > > bugs.python.org fields: > \`\`\`python > activity = \ > actor = 'gvanrossum' > assignee = 'none' > closed = True > closed\_date = \ > closer = 'gvanrossum' > components = \[\] > creation = \ > creator = 'gvanrossum' > dependencies = \[\] > files = \['41521'\] > hgrepos = \[\] > issue\_num = 26031 > keywords = \['patch'\] > message\_count = 4.0 > messages = \['257651', '257652', '257658', '380366'\] > nosy\_count = 2.0 > nosy\_names = \['gvanrossum', 'pitrou'\] > pr\_nums = \[\] > priority = 'normal' > resolution = 'out of date' > stage = 'resolved' > status = 'closed' > superseder = None > type = 'enhancement' > url = 'https://bugs.python.org/issue26031' > versions = \['Python 3.6'\] > \`\`\` > > \\ ---
**Author:** ![Ferdi265](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ferdi265/32/24060_2.png) [@Ferdi265](https://discuss.python.org/u/Ferdi265)\ **Post date:** [December 9, 2024, 11:01pm UTC](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/40 "2024-12-09T23:01:27Z")
Barney Gale’s suggestion is currently the design I’m most comfortable with: It allows caching in the new `.info` property, but doesn’t change any existing `Path` behaviour, while at the same time not being an overly convoluted change. The documentation for the info property could also prominently include that the information is potentially cached and that `Path.is_dir()` should be used instead of `Path.info.is_dir()` (or similar) if up-to-date info is required. [Previous page](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721.md?page=1) [Next page](https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721.md?page=3)