# Pathlib objects as glob template **URL:** \ **Category:** Ideas\ **Created:** [September 30, 2024, 10:12am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177 "2024-09-30T10:12:10Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [September 30, 2024, 10:12am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/1 "2024-09-30T10:12:10Z")
Sometimes, the user provide a glob template as a string, like `--files=s3://home/*/documents/*.txt`. Currently, there’s no obvious way to use glob using the pathlib API (we cannot use `shutil.glob` as it does not support Google cloud, s3 buckets): ```python path = 's3://home/*/documents/*.txt' path = s3_path.Path(path) files = list(path.glob()) # Not working ``` Currently it require some ugly hack, like: ```python Path(path.parts[0]).glob(os.fspath(Path(path.parts[1:]))) ``` It would be nice if the pathlib API had a standard way to using it as glob template ---
**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\ **Post date:** [September 30, 2024, 10:17am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/2 "2024-09-30T10:17:07Z")
I feel like the best solution is to add the ability for `Path.glob` to take in absolute patterns, locked behind a flag to prevent accidents. ---
**Author:** ![petercordia](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@petercordia](https://discuss.python.org/u/petercordia)\ **Post date:** [September 30, 2024, 10:17am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/3 "2024-09-30T10:17:32Z")
I’ve also had this situation, and ended up writing 3 different functions to process different versions of glob lookups all of which match this pattern. Added to that, In 80% of cases, I actually want one of ```python sorted(Path(path.parts[0]).glob(os.fspath(Path(path.parts[1:])))) or list(Path(path.parts[0]).glob(os.fspath(Path(path.parts[1:])))) ``` ---
**Author:** ![barry-scott](https://avatars.discourse-cdn.com/v4/letter/b/e9c0ed/32.png) [@barry-scott](https://discuss.python.org/u/barry-scott)\ **Post date:** [September 30, 2024, 11:58am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/4 "2024-09-30T11:58:25Z")
> [@Conchylicultor](#): > > Currently, there’s no obvious way to use glob using the pathlib API (we cannot use `shutil.glob` as it does not support Google cloud, s3 buckets): pathlib does not know about S3, why should it’s glob work for S3? ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [September 30, 2024, 12:16pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/5 "2024-09-30T12:16:44Z")
Pathlib defines an API which has many implementation. The official “standard” one is the one in the standard library, but many other implementations exists which add `s3`, `gcs`, `zipfile`, `git`,… Like: - [GitHub - fsspec/universal\_pathlib: pathlib api extended to use fsspec backends](https://github.com/fsspec/universal_pathlib) - [zipfile — Work with ZIP archives — Python 3.12.6 documentation](https://docs.python.org/3/library/zipfile.html#path-objects) - [Pathlib-like API — etils](https://etils.readthedocs.io/en/latest/epath.html) - … To maintain consistency across those libraries, it’s important pathlib API supports the use-cases of those implementations. Also see: [https://discuss.python.org/t/make-pathlib-extensible](https://discuss.python.org/t/make-pathlib-extensible) ---
**Author:** ![chepner](https://avatars.discourse-cdn.com/v4/letter/c/22d042/32.png) [@chepner](https://discuss.python.org/u/chepner)\ **Post date:** [September 30, 2024, 1:27pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/6 "2024-09-30T13:27:36Z")
> [@Conchylicultor](#): > > To maintain consistency across those libraries, it’s important pathlib API supports the use-cases of those implementations. And the interface is fine. `path = s3_path.Path(path)` doesn’t (I assume) raise a problem, only the call to `path.glob()`. Whatever library you are using needs to be responsible for querying what objects are actually available in a particular bucket. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [September 30, 2024, 6:44pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/7 "2024-09-30T18:44:29Z")
Could you retrieve a list of bucket keys (e.g. with [`ListObjectsV2`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)) and then filter them through [`PurePosixPath.full_match()`](https://docs.python.org/3.13/library/pathlib.html#pathlib.PurePath.full_match)? (That method is new in 3.13.) ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [October 1, 2024, 8:34am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/8 "2024-10-01T08:34:38Z")
I’m not sure I understand how this solve the issue. I simplified the use-case, but in practice, the implementation should not depend on any specific backend (as the same code should works for s3, gcs, local paths,…) I’m suggesting there should be an option to allow this: ```python path = epath.Path('gcs://home/*/documents/*.txt') # Works with arbitrary paths path.glob(is_template=True) ``` Currently, the way of implementing this is hacky and unatural, as pointed out in my first message ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [October 1, 2024, 9:49am UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/9 "2024-10-01T09:49:55Z")
> [@Conchylicultor](#): > > Currently, the way of implementing this is hacky and unatural, as pointed out in my first message I don’t think it’s “hacky and unnatural”. To execute a glob search, you need to start from some root. So `Path(path.parts[0]).glob(os.fspath(Path(path.parts[1:])))` does precisely that, it separates the root and the pattern, and uses the pattern to do a glob against the root. Certainly, helpers could be added which make the operation easier, but it’s not at all unnatural in my view. In actual fact, I find using a path object as a glob pattern in the first place to be the unnatural aspect - a path is a very different object than a glob pattern, and confusing the two feels like a bug. Also, if we did allow `Path.glob()`, we’d have to come up with a meaning for `Path("*://*/documents/*.txt")`. And that’s a whole other problem. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [October 1, 2024, 3:28pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/10 "2024-10-01T15:28:02Z")
Apologies, I misunderstood your message. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [October 1, 2024, 3:52pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/11 "2024-10-01T15:52:25Z")
> [@pf\_moore](#): > > Also, if we did allow `Path.glob()`, we’d have to come up with a meaning for `Path("*://*/documents/*.txt")`. And that’s a whole other problem. Does this argument apply also to `Path("~/.ssh")`? Arguably it’s another path-ish pattern that can be expanded into a “real” path with a specific method (`Path.expanduser()`). A few more thoughts: Supporting non-relative patterns means throwing away information from `self`. Folks already complain spiritedly when `os.path.join('foo', '/bar')` returns `/bar`, and we’d be doing the same thing with `Path('foo').glob('/bar')`. Perhaps we could allow non-relative patterns _only_ when `Path.parts` is empty? I’m not so keen on the flag suggested by @MegaIng as it complicates the interface a little. pathlib `PurePath` objects strip trailing slashes, which changes the meaning of patterns. I think this might be practically intractable. Expanding `self` as a pattern would change long-standing behaviour; albeit rarely, as glob wildcards don’t often appear in real paths Users would be able to supply the pattern in two places (`Path('*').glob()` and `Path().glob('*')`) rather than one. We could deprecate the method argument, but it would be another change for existing users. ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [October 1, 2024, 3:59pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/12 "2024-10-01T15:59:10Z")
> [@barneygale](#): > > Does this argument apply also to `Path("~/.ssh")`? Arguably it’s another path-ish pattern that can be expanded into a “real” path with a specific method (`Path.expanduser()`). I don’t think it applies there because `~` isn’t a wildcard, it’s a specific placeholder for the user directory. Whereas `*://` reads as something like “search all(?) possible URIs” which I’m not sure is even defined. ---
**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\ **Post date:** [October 1, 2024, 4:07pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/13 "2024-10-01T16:07:21Z")
Thanks. I don’t think my brain is working very well, as I’m still conflating different ideas in my previous post. Sorry for the noise. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [October 1, 2024, 4:24pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/14 "2024-10-01T16:24:57Z")
> [@jamestwebber](#): > > I don’t think it applies there because `~` isn’t a wildcard, it’s a specific placeholder for the user directory. Whereas `*://` reads as something like “search all(?) possible URIs” which I’m not sure is even defined. Yes, that was the point I was making. I think the point here is that we’re talking about an application that accepts user input of a “file pattern” and wants to turn that into a list of files. Traditionally, this is done with `glob.glob`, which takes a _string_ argument representing a pattern to match against the filesystem. The `pathlib.Path.glob` method is different - it takes a string as a pattern in a similar way, but the pattern is _required_ to be relative, and is matched against the contents of the path object in `self`. The OP’s use case is the first situation, but they want the ability to match against non-filesystem patterns, which `glob` doesn’t support. The extensibility of pathlib is attractive here, but only if we can construct a suitable base pathlib subclass. The problem is that if we follow the logic of `glob.glob`, the pattern is general, and _in theory_ the user could enter something that has a wildcard in the URI scheme part of the path. Clearly that makes no sense, and in particular it doesn’t allow us to know which pathlib subclass would apply. So in practice a fully general pattern string isn’t valid. _If_ we make the restriction that the “root” (however we choose to define that) of the pattern must be fixed (no wildcards) then it’s easy to implement the relevant search: ```python root, pattern = split_user_pattern(input_string) list_of_paths = Path(root).glob(pattern) ``` Implementing `split_user_pattern` is left as an exercise for the reader. It depends heavily on the application and valid path provider types, but something like ```python def split_user_pattern(input_string): path = Path(input_string) return path.parts[0], os.fspath(Path(path.parts[1:])) ``` isn’t an implausible implementation in the absence of any application-specific constraints. This is why I don’t see the one-line version of this as “hacky and unnatural”. The need to split and rejoin using `Path.parts` is annoying - a `Path.without_anchor` property would help a lot here - but that’s a very minor implementation detail, easily hidden in a helper function (as I did here). ---
**Author:** ![jamestwebber](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jamestwebber/32/12799_2.png) [@jamestwebber](https://discuss.python.org/u/jamestwebber)\ **Post date:** [October 1, 2024, 4:38pm UTC](https://discuss.python.org/t/pathlib-objects-as-glob-template/66177/15 "2024-10-01T16:38:58Z")
edit: deleted because I misunderstand @barneygale earlier. My brain is also not working.