Make pathlib extensible

barneygale · July 16, 2023, 3:34pm

July 2023 progress report

Thank you to all those who have been providing feedback on naming, hierarchies, etc. It’s so useful to bounce ideas off such talented and experienced devs!

As I mentioned in a previous post, I’ve put up a PR that adds a private _PathBase class:

github.com/python/cpython

GH-89812: Add `pathlib._PathBase`

python:main ← barneygale:gh-89812-omgtarpath

opened 06:13PM - 02 Jul 23 UTC

barneygale

+608 -129

Add private `pathlib._PathBase` class. This will be used by an experimental PyPI… package to incubate a `tarfile.TarPath` class. ## Implementation notes In `pathlib.py`: - Split `Path` into `_PathBase` and `Path` - Implement `_PathBase.is_junction()` and `is_mount()` using `stat()`. In the latter case, restore the implementation deleted in #31458 - Implement `_PathBase.resolve()` using `absolute()` and `readlink()` - Implement `_PathBase._scandir()` using `iterdir()` In `test_pathlib.py`: - Add `PathBaseTest` to exercise the `_PathBase` class directly - Split `PathTest` into `DummyPathTest` and `PathTest`. The former exercises a representative subclass of `_PathBase`. - Add `DummyPathWithSymlinksTest` to exercise a similar subclass, this time with symlink support. ## Questionable bits In `pathlib.py`, the `_PathBase.is_junction()` and `_PathBase.resolve()` methods are new code. The latter is very performance-sensitive and may need more work. ## Future work - Make `pathlib._PathBase` public. There's still at least a couple of things to do first: - Make `PurePath._flavour` public - Figure out what to do with `_PathBase.__hash__()`, `__eq__()`, and other comparison methods. --- * Issue: gh-89812

That PR has been slimmed down: it originally added tarfile.TarPath too, but the expected behaviour of paths involving symlinks wasn’t clear, and so I’m going to work on TarPath in a PyPI package first.

When that PR lands, the remaining work is:

Add a public PurePath.pathmod class attribute (PR: GH-106533)
Figure out what to do with _PathBase.__hash__(), __eq__(), __lt__(), etc (any opinions?)
Make pathlib.PathBase public!

For the first time, I feel confident that this project will succeed. There are no architectural problems remaining in pathlib that would prevent it, nor any major decisions to be made (touch wood). It will be immediately useful upon release, and I think it could grow into one of Python’s best-loved features as third-party APIs begin to accept os.PathLike | pathlib.PathBase for path arguments. Eventually users should be able to do things like:

shutil.copytree(FTPPath(...), TarPath(...))
pandas.read_csv(S3Path(...))
image.save(TarPath(...))  # PIL

We’re doing for path objects what PEP 3116 and the io module did for file objects

That’s it for now. Thanks again to everyone who has helped with this!