Add cross-platform user directories to pathlib/standard library

The platformdirs package, and its predecessor appdirs, have been fulfilling a very useful purpose for a long time – they return the paths to the platform-specific data, cache, config, temp, etc. directories, which differ between OSes.

On Linux for example the module provides the output of the various XDG environment variables, but on macOS and Windows it provides the appropriate equivalents.

pathlib already provides Path.home(). And Path objects help a lot with platform-agnosticism. Given platformdirs’s similar mission to pathlib, its relatively small footprint, and its simple nature, would it be reasonable to think about incorporating it into the stdlib, perhaps as from pathlib import PlatformDirs?

8 Likes

OP also raised this on the platformsdirs issue tracker, where the maintainer at least wasn’t against it on principle Adding to stdlib/pathlib? · Issue #265 · platformdirs/platformdirs · GitHub

They mentioned their that their understanding was that the stdlib is trying to get smaller. I wouldn’t quite say that that is true, it just has to be a large enough benefit-to-maintance burden tradeoff. If platformsdirs is stable as a project and decently widely used, adding it to the stdlib I think could have a chance of being done.

But for that, the maintainers best mentioned here how stable the project/concept/ecosystem is. Is it foreseeable that at some point within the next few years a backwards incompatible change becomes necessary? I would doubt that because OSes generally try to be stable across similar timeframes.

Platformdirs is a great project!

The usual question: would end users benefit from it being in the stdlib, with only one feature release per year, rather than on PyPI, with releases as often as they like?

I see 20 releases in 2023 at platformdirs · PyPI. Ignoring patch releases, 14.

In the stdlib, that would be reduced to one per year, plus with no feature changes allowed for the last 5 months before release (first beta to official release).

10 Likes

I agree with Hugo. The problem might appear settled and stable, but we never know when a change in the operating systems or specifications will require adaptations in the lib, which would be very slow to come with Python or not come at all for security branches.

4 Likes

Reasonable concerns of course!

It looks to me like platformdirs uses some sort of automatic release process that results in many small releases, even days apart. Many of 2023’s releases are just for updates to the docs, changes to supported Python versions, linter changes, and so on.

So in reality I don’t think the pace of change is that fast. OS structure and file conventions don’t change much from year to year, after all. It would cause a lot of problems to a lot of software if they did.

The maintainers will have a better idea than me, an outsider, but it feels like once a year would be often enough for feature additions, if any are even necessary.

Sure. Currently though, without using platformdirs, a programmer has to hard code the paths. The consequences of changes to the way an OS does things results in similar things in each case (stdlib integration vs hard-coded):

Non-breaking changes to the recommendations for an OS: people’s code doesn’t break in either case. For the hard-coded case, the programmer would have to pay attention enough to find out about it themselves and actively change the code to meet the new recommendations, but more likely the program would just continue forever to follow the outdated convention. For the stdlib case, it would just be one of the things in the “list of changes in Python 3.1x” to be aware of when updating the Python version used for a project.

Changes to the way an OS does things that breaks the functionality (e.g. permissions changes): every program would be broken in both cases. If using the stdlib, only broken until a bugfix is released. With hard-coded paths, permanently broken until the programmer fixes it. Such a manual fix would also always be an option for anyone using the stdlib anyway. So the inconvenience of it being in the stdlib would be no greater, I would say.

3 Likes

But there’s a third option: use the third-party package. In that situation the fix can come much faster.

/cc @barneygale

Is the likelihood of breakage much higher than other OS-dependent modules in the stdlib like tempfile?

I wasn’t saying it is or it isn’t, just pointing out that your comparison left out the status quo. If there ever were any breaking changes, having a package is better than both of the options you described.

1 Like

I would think at this point that platformdirs is relatively stable with very few behavioral changes over time (As a user of it). There have continued to be convenience methods added over time that could otherwise be received by users more frequently, but the actual platform-specific behavior has not changed frequently (the only cases I’m aware of are in 2021 relating to the original fork from appdirs, fixing appdirs getting something wrong and another fix in 2023 which was a fix, but did change one of the paths returned for macos

Adding this to the standard library could be done in a very similar to the situation to dataclasses, where attrs and the standard library’s dataclasses are two separate namespaces, one updated more frequently than the other, but the standard library covers the base functionality that is the core need, with considering to roll things from platformdirs back into the standard library on a case by cases basis, but that’s probably something worth discussing with platformdirs.

At the same time, I’m not sure the opportunity cost is high here to leave it outside the standard library. This is a pure python dependency with a rather small and easily reviewable code base that does something specific and narrow in scope, and does it well. I would be surprised if there were issues for anyone including this as a dependency even in some of the strictest environments.

OK, then this could be something nice to have in the stdlib, mostly for scripts that want to do the right thing for config / cache / data dirs but have no dependency. I am not opposed to the proposal, but I think all new modules require a PEP now (the last one without was graphlib IIRC).

6 Likes

Would it make more sense to add a new module or to add its functionality (or a subset of it) to pathlib? And would the latter also require a PEP?

No, this is not related to object-oriented abstraction for path manipulation.
It would be its own module, like tempfile is.

5 Likes

I agree with Éric that this doesn’t belong in pathlib. To add an another reason: pathlib only distinguishes between Windows and POSIX, whereas platformdirs has specific algorithms for macOS IIUC.

I quite like the idea of including platformdirs somewhere in the stdlib – it’s very useful but slow-moving and self-contained, similar to tempfile or os.path.expanduser(). But I’m a newish core dev with a small area of responsibility, so take a handful of salt.

10 Likes

I don’t think this can be added. We discussed it in 2010 and never reached a consensus: Define a standard location and API for configuration files · Issue #51424 · python/cpython · GitHub

Well, there’s been nearly 15 years of standards evolution (FHS, XDG, systemd, etc) and platform convergence (many commercial *nix flavors giving way to Linux and BSD) since 2010, and the development and wide acceptance of appdirs and its successor platformdirs. So, I’m not sure how a (lack of) consensus reached way back then should preclude a fresh reconsideration of the topic now.

11 Likes

Yep, clear recommendations exist for all three of the main operating systems that Python targets. Once in a generation changes have historically and may yet happen, but the world’s software would move over to any new recommendations so slowly anyway that I don’t think the pace of Python releases would hinder anything.

I think it’s pretty well-accepted that the “correct” thing to do on Linux these days is to follow the XDG spec. In any case, the primary goal of the module on *nix would be to provide the values of those environment variables, and failing that, the default values provided by the spec. As platformdirs already does.

2 Likes

I really am completely inexperienced when it comes to this whole process, and I’ve seen different approaches in different parts of the forum, but I suspect here it would be easier to build consensus when a particular implementation is proposed? The likely maintenance requirements can be better assessed then, for example. I agreed to work with the primary maintainer in co-authoring a PEP, so I’ll look at preparing a draft – unless you think it’s better to discuss first.

The idea would be to mirror part or all of the current platformdirs in a similar way to other recently adopted modules like importlib.resources or tomllib. I wanted to discuss with him how the package could be slimmed down in scope to reduce maintenance, and to make the usage and syntax simpler/more Pythonic. I assume those are sensible goals?

Would any of the core developers who have chimed in here be prepared to sponsor a PEP?

2 Likes

For reference, platformdirs provides:

  • user data dir (user_data_dir)
  • user config dir (user_config_dir)
  • user cache dir (user_cache_dir)
  • site data dir (site_data_dir)
  • site config dir (site_config_dir)
  • user log dir (user_log_dir)
  • user documents dir (user_documents_dir)
  • user downloads dir (user_downloads_dir)
  • user pictures dir (user_pictures_dir)
  • user videos dir (user_videos_dir)
  • user music dir (user_music_dir)
  • user desktop dir (user_desktop_dir)
  • user runtime dir (user_runtime_dir)

Some of these sound too general for stdlib, e.g. user pictures dir won’t make much sense on a server.
Some offer opinion and roaming variants.
There’s a multipath option that gives you a list (I assume this is to better answer where should I look? rather than where should I write?)

Python already provides:

  • temporary dir (tempfile.gettempdir)
  • module installation/search paths (in sysconfig, in site, sys.path)
  • user home dir (pathlib.Path.home())
  • any user’s home directory (os.path.expanduser, pathlib.Path.expanduser)
  • current working directory

It might be worth it to include at least some of those?

Note that the tomllib PEP added a module that was already simplified, and widely used.
You might want to start with putting such a module on PyPI first. It would not only make discussions easier, but it would serve as a backport to current Python versions. And a PEP draft could serve well as docs. (IMO, documentation is platformdirs’ weakest point: e.g. what exactly is a runtime dir?)

And to answer your question:

In general, yes. But there’s a lot of details that need figuring out, and maybe there’s a blocker issue to be found.
That said, if you do take the effort to document the best version of this proposal, I can sponsor a PEP even if it ultimately ends up being rejected.

1 Like