# Make pathlib extensible

**URL:** <https://discuss.python.org/t/make-pathlib-extensible/3428>\
**Category:** Ideas\
**Created:** [February 24, 2020, 12:15am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428 "2020-02-24T00:15:07Z")\
**Posts on this page:** 20\
**Page:** 9

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [October 24, 2023, 5:56pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/161 "2023-10-24T17:56:16Z")

</div>

One thought - how does all of this affect `os.PathLike` and friends? The ` __fspath__ ` method is defined as returning `str` or `bytes`, and it should return “the file system path representation of the object”. A `Path` subclass that represents (say) a member of a zip archive can’t really do this, particularly as the user expectation is that `os.fspath(path)` will return something that could reasonably be passed to an external utility (such as the user’s editor).

Maybe I missed some discussion on this - can `_PathBase` subclasses opt out of being `PathLike`? I wonder if they should be “not `PathLike`” by default, and have to explicitly opt _into_ being convertible to a filesystem path.

Sorry if this is something you’ve already thought about - I’m not sure where I could look to check.

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [October 24, 2023, 6:16pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/162 "2023-10-24T18:16:55Z")

</div>

It’s a very good thought, and was briefly covered earlier (easy to miss in this long thread):

> [@Make pathlib extensible](https://discuss.python.org/t/make-pathlib-extensible/3428/135):
>
> For my fellow visual learners, here’s a venn diagram showing os.PathLike and pathlib.PathBase:

`pathlib._PathBase` won’t be `os.PathLike` by default for all the reasons you mentioned. It would be a catastrophe if `open(ZipPath('README.rst', zipfile=blah))` silently opened a file called `README.rst` on the local filesystem.

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [October 24, 2023, 6:29pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/163 "2023-10-24T18:29:48Z")

</div>

Cool, thanks. I’d imagined `ZipPath` returning something like `/.../archive.zip/README.rst` (the way zip files appear in the import system) so using it would be wrong, rather than disastrous, but what you’ve done is better 🙂

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [November 20, 2023, 10:16pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/164 "2023-11-20T22:16:02Z")

</div>

✨ November 2023 progress report ✨

I’ve merged [GH-106703](https://github.com/python/cpython/pull/106703), which adds a new [`glob.translate()`](https://docs.python.org/3.13/library/glob.html#glob.translate) function, and speeds up globbing in pathlib. Thank you @AA-Turner, @jaraco and @encukou for your help with this.

I’ve been considering backporting and publishing the latest pathlib as a standalone PyPI package (compatible with 3.8+), so that other PyPI packages like [dohq-artifactory](https://github.com/devopshq/artifactory) can make use of the new `_PathBase` ABC and validate its design. Previously I was planning to backport all of pathlib, but it’s rather more work than I was expecting, so I’m now looking at backporting `_PurePathBase` and `_PathBase` only.

I’ve played around with possible values for `_PurePathBase.pathmod` and how we support customization of low-level path syntax. The right answer still isn’t clear to me, but I think setting it to `posixpath` is a good first guess. I’ll make a PR for that once [GH-110670](https://github.com/python/cpython/pull/110670) lands, and then move on to the PyPI package.

Thanks to everyone following along and being encouraging, it means a lot

---

<div class="post-metadata">

**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\
**Post date:** [November 20, 2023, 10:39pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/165 "2023-11-20T22:39:48Z")

</div>

> [@barneygale](#):
>
> Previously I was planning to backport all of pathlib, but it’s rather more work than I was expecting

What’s the reason for this? Did you want the backported classes to be subclasses of the standard library classes (or `isinstance(pathlib_backport.Path(), pathlib.Path) == True`?

If you drop a link to the repo, I’d be happy to look at the code for answers.

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [November 20, 2023, 11:09pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/166 "2023-11-20T23:09:54Z")

</div>

Pathlib relies on new features elsewhere in the standard library - for example `glob.translate()`, `os.path.splitroot()` and `os.path.realpath(strict=True)`. It’s the `realpath()` implementations in `ntpath` and `posixpath` that scared me off - they might not be easy to backport.

I wouldn’t mind if someone else does a complete backport, but I reckon my PyPI package will be just the ABCs, at least initially.

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [December 18, 2023, 1:14am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/167 "2023-12-18T01:14:00Z")

</div>

✨ December 2023 progress report ✨

I’ve added a private `PurePathBase` class ([GH-110670](https://github.com/python/cpython/pull/110670)) and moved the base classes into a private `pathlib._abc` module ([GH-112881](https://github.com/python/cpython/pull/112881), [GH-112904](https://github.com/python/cpython/pull/112904)). Thank you @AlexWaygood and @tjreedy for helping solve a performance regresssion ([GH-112907](https://github.com/python/cpython/pull/112907)). The class hierarchy now looks like this:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/python1/original/3X/2/7/27f694e2f40d13cdf7d1e1c69926ff0325b6cc67.png)

I’ve copy-pasted the ABC code into a new repository:

> **[GitHub - barneygale/pathlib\_abc: Python base classes for rich path objects](https://github.com/barneygale/pathlib_abc)**
>
> Python base classes for rich path objects. Contribute to barneygale/pathlib\_abc development by creating an account on GitHub.

I intend to publish this as a PyPI package once some of the more [embarassing bugs](https://github.com/barneygale/pathlib_abc/issues) are fixed. The package will serve as a proving-ground for the ABCs, with a view towards adding them to the (public) standard library once they’ve matured. I’ll make a proper announcement once the first version of the PyPI package is published.

Happy holidays all 🙂

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [January 3, 2024, 4:41pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/168 "2024-01-03T16:41:33Z")

</div>

✨ January 2024 progress report ✨

I’ve published the pathlib ABCs as a PyPI package:

> **[pathlib-abc](https://pypi.org/project/pathlib-abc/)**
>
> Backport of pathlib ABCs

I’ve also written some basic docs, including an example `TarPath` implementation:

[https://pathlib-abc.readthedocs.io/en/latest/index.html](https://pathlib-abc.readthedocs.io/en/latest/index.html)

The pathlib ABCs are present in the Python 3.13+ standard library, in the private `pathlib._abc` module. They won’t be made a part of the _public_ standard library unless/until they mature as a PyPI package (and even then it’s not guaranteed!)

This will be my last monthly update in this thread. Folks interested in following the development of the ABCs may wish to watch the [pathlib\_abc](https://github.com/barneygale/pathlib_abc/) project on github - I’m using the issue tracker there as a todo list. Contributions are most welcome.

It’s been a pleasure to lead this work through its first phase, but I couldn’t have done it without the mentoring, support, insights and reviews from many others. I’m not sure how to thank all these folks - I’ll put together some sort of CREDITS file perhaps, and include it in the pathlib-abc 1.0 release announcement when it comes (some time this year?).

That’s all! Happy new year everyone 🎉

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [January 8, 2024, 4:49pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/169 "2024-01-08T16:49:57Z")

</div>

Thanks a lot Barney! Your work on `pathlib` has been excellent and it was fun watching the progress. You’ve definitely helped out the Python community a lot here.

---

<div class="post-metadata">

**Author:** ![moi90](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/moi90/32/8835_2.png) [@moi90](https://discuss.python.org/u/moi90)\
**Post date:** [January 10, 2024, 9:10am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/170 "2024-01-10T09:10:13Z")

</div>

Yes, thanks a lot @barneygale ! This will make it much easier to write custom implementations of the `pathlib` interface.

I will gladly used it in my [OmniArchive](https://github.com/moi90/omni_archive) package where I extracted the necessary parts of the `pathlib` interface for the time being. (The package is similar to your [`TarPath` example](https://pathlib-abc.readthedocs.io/en/latest/examples.html#tarpath), but also implements the same interface for zip files and aims to provide transparent access to any kind of archive.)

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [May 4, 2024, 8:04pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/171 "2024-05-04T20:04:46Z")

</div>

An irregular update:

I haven’t undertaken much pathlib work lately, because by the beginning of the year I’d already made quite a lot of changes in 3.13, and didn’t want to overload users. The [_What’s New in Python 3.13_](https://docs.python.org/3.13/whatsnew/3.13.html#pathlib) doc has all the public-facing stuff, and plenty has changed under the hood, too. Folks should find that pathlib is faster in 3.13 beta 1 than in earlier versions for a variety of common operations and scenarios. I’d love to see some independent confirmation of this (and only slightly less happy to see it refuted, if I get new optimisation targets).

With 3.13 beta freeze a few days away, I’m now thinking about what we could do for 3.14. My current plans are as follows:

**Bootstrapping the ecosystem** : I plan to release a few new PyPI packages that provide concrete implementations of various filesystems atop `pathlib-abc`. Not only will these be useful in and of themselves, they’ll also serve as templates that users can copy, adapt for other virtual filesystem backends, and perhaps release.

**Flat filesystems** : some virtual filesystems (tar and zip files, s3, git…) provide fast access to a list of all files (perhaps under a prefix), rather than direct children of a directory; in some cases, directories can only be inferred from file paths, and not directly recorded. I’m hoping to add a `pathlib_abc.FlatPathBase` class with efficient algorithms for these sorts of filesystems, and to steal all of @jaraco’s ideas from `zipfile.Path`! 😉

**File transfers** : for Python 3.14, I’m planning to add `copy()`, `copytree()`, `rmtree()` and `move()` methods to `PathBase` and `Path`, fulfilling [GH-73991](https://github.com/python/cpython/issues/73991). The implementations in `PathBase` will be generic, and so it will be possible to write something like:

```python
source = TarPath('images', archive=tarfile.open(...))
target = FTPPath('public_html', 'images', ftp=ftplib.FTP(...))
source.copytree(target)

```

**Backporting local paths** : once the `copy()` etc methods are in, users of `pathlib-abc` in earlier versions of Python will want to use them, and so I’ll need to backport 3.14’s `pathlib.PurePath`, `Path`, etc, for earlier versions. Maybe another new PyPI package.

**Collaborating with package maintainers** : with all the above in place, I’ll try to convince maintainers of pathlib-y packages of the merits of using `pathlib-abc`, and make PRs if they agree.

That’s all! My apologies to everyone who hates thread necromancy! 😅

---

<div class="post-metadata">

**Author:** ![eryksun](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/eryksun/32/697_2.png) [@eryksun](https://discuss.python.org/u/eryksun)\
**Post date:** [May 4, 2024, 8:58pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/172 "2024-05-04T20:58:46Z")

</div>

Barney, please consider moving the implementation out of “\_\_init\_\_.py” in 3.13+. I prefer to use it just for imports that define the API, like how asyncio is implemented, except with more explicit module names if the name is otherwise too generic, such as “\_pathlib\_abc.py” instead of “\_abc.py”. I’ll suffer in silence if you disagree, but I had to at least ask.

---

<div class="post-metadata">

**Author:** ![barneygale](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/barneygale/32/1882_2.png) [@barneygale](https://discuss.python.org/u/barneygale)\
**Post date:** [May 4, 2024, 9:00pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/173 "2024-05-04T21:00:43Z")

</div>

Hah, I was just considering doing that yesterday. PR coming up!

---

<div class="post-metadata">

**Author:** ![jaraco](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jaraco/32/57_2.png) [@jaraco](https://discuss.python.org/u/jaraco)\
**Post date:** [May 5, 2024, 3:46pm UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/174 "2024-05-05T15:46:28Z")

</div>

I’ve had a lot of experience with backports lately (`importlib_*`, `zipp` (`zipfile.Path`), `backports.tarfile`, `configparser`, `singledispatch`, `backports.functools_lru_cache`), all using slightly different approaches and tradeoffs. Maybe I’ll get around to documenting those, but in lieu of some useful documentation, feel free to reach out for advice if as you’re looking to backport.

tl;dr, I’d probably recommend using `backports.*` but keeping the CPython implementation as the canonical implementation.

---

<div class="post-metadata">

**Author:** ![blhsing](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/blhsing/32/25812_2.png) [@blhsing](https://discuss.python.org/u/blhsing)\
**Post date:** [May 11, 2024, 12:25am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/175 "2024-05-11T00:25:18Z")

</div>

Sorry to be so late to the party here, but just throwing my 2 cents that the easiest way to make `pathlib.Path` extensible is to make it inherit from `WindowsPath` or `PosixPath` at definition time.

Currently which of `WindowsPath` and `PosixPath` to use is decided at instantiation time, which is wholly unnecessary because `os.name` is never going to change for the life of the program.

So the new class hierarchy should look like this:

```python
class PathBase(PurePath):
    ... # I/O methods

class WindowsPath(PathBase, PureWindowsPath):
    ...

class PosixPath(PathBase, PurePosixPath):
    ...

class Path(WindowsPath if os.name == 'nt' else PosixPath):
    pass

```

This way, a user class inheriting from `Path` will never be missing a `_flavour` attribute.

I have a working demo in [Is there a `pathlib` equivalent of `os.scandir()`?](https://discuss.python.org/t/is-there-a-pathlib-equivalent-of-os-scandir/46626/25) if anyone’s interested.

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [May 11, 2024, 3:14am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/176 "2024-05-11T03:14:11Z")

</div>

> [@blhsing](#):
>
> ```python
> class Path(WindowsPath if os.name == 'nt' else PosixPath):
> pass
> 
> ```

That is broken, see: [hatch/src/hatch/utils/fs.py at hatch-v1.10.0 · pypa/hatch · GitHub](https://github.com/pypa/hatch/blob/hatch-v1.10.0/src/hatch/utils/fs.py#L15-L20)

---

<div class="post-metadata">

**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\
**Post date:** [May 11, 2024, 3:56am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/177 "2024-05-11T03:56:09Z")

</div>

Come to think of it, I would say there really shouldn’t be multiple sources of truth for that. Aside from which they aren’t even the same string:

```python
>>> sys.platform
'linux'
>>> os.name
'posix'

```

(Or `'win32'`/`'nt'`, respectively, for WIndows)

(Actually, for Linux systems, aren’t those two values exactly the wrong way around, if we grant that they should be different? I’ve never heard of installing a “posix distro” from “posix live media”; and if anything, the point of POSIX is to standardize interfaces, which sounds to me like the sort of thing a “platform” might do.)

---

<div class="post-metadata">

**Author:** ![blhsing](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/blhsing/32/25812_2.png) [@blhsing](https://discuss.python.org/u/blhsing)\
**Post date:** [May 11, 2024, 4:11am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/178 "2024-05-11T04:11:08Z")

</div>

> [@ofek](#):
>
> > [@blhsing](#):
> >
> > ```python
> > class Path(WindowsPath if os.name == 'nt' else PosixPath):
> > pass
> > 
> > ```
> 
> That is broken, see: [hatch/src/hatch/utils/fs.py at hatch-v1.10.0 · pypa/hatch · GitHub](https://github.com/pypa/hatch/blob/hatch-v1.10.0/src/hatch/utils/fs.py#L15-L20)

Good point, though my example was a simple reuse of the current code:

> <https://github.com/python/cpython/blob/35c436186b849f8f2f9fb866c59015c9d034d448/Lib/pathlib/_local.py#L114>

---

<div class="post-metadata">

**Author:** ![ofek](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ofek/32/1033_2.png) [@ofek](https://discuss.python.org/u/ofek)\
**Post date:** [May 11, 2024, 4:42am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/179 "2024-05-11T04:42:02Z")

</div>

Oh yes I understand thanks, I just wanted to avoid people copying that and having to dig deep to understand why type hinting did not work, as I had to do 🙂

---

<div class="post-metadata">

**Author:** ![blhsing](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/blhsing/32/25812_2.png) [@blhsing](https://discuss.python.org/u/blhsing)\
**Post date:** [May 11, 2024, 5:06am UTC](https://discuss.python.org/t/make-pathlib-extensible/3428/180 "2024-05-11T05:06:54Z")

</div>

Right, though come to think of it, it really should be mypy that needs to be fixed to recognize `os.name` as well.

[Previous page](https://discuss.python.org/t/make-pathlib-extensible/3428.md?page=8)

[Next page](https://discuss.python.org/t/make-pathlib-extensible/3428.md?page=10)