Should there be a new standard for installing arbitrary data files?

[nhatkhai]

Thinking about it - these day we need to make a lot of automation stuff on many weird platforms.
Having pip suppprt egg allow it unzip, and run its setup.py install is the most simple way so solve corner cases that pip had not idea about.
Remove this feature would cause end less request and discussion about need non-arbitrary code execution from wheel package.

[pradyunsg (Pradyun Gedam)]

@aragilar

For examples of categories that a wheel might want to use, autotools provides a number of categories with descriptions . I personally would love to be able to look up man pages for scripts I install.

Indeed, this is a great list.

However, as noted earlier, it’s still unclear which of these we need to have and how we’d handle this on platforms where some of these concepts don’t make sense.

---

As a heads up, I’m gonna be unsubscribing from this discussion now, since I don’t see it going in a constructive direction in the short term.

[AWhetter (Ashley Whetter)]

@pradyunsg

Is there any prior art in language-specific package managers allowing for something like this?

Here’s what I can find, though I cannot speak to the language maintainers feelings on each solution:

• Ruby/rubygems (Specification Reference - RubyGems Guides): Gemspec files have a “files” array that take relative file paths and output those same files in the gem. Gem files are installed in their own namespace, meaning that you can put things anywhere, including the root directory, without fear of clashing with other gems (the exception to this is the lib/ directory which might get added to LOAD_PATH). Typically you would load a data file using a path relative to your current file, and it’s considered bad practise to load any files outside the lib directory of the gem. I’ve seen some examples of gems including man pages (see guard or ruby-irb), which repackagers can then easily move to a distro-specific location. There’s lots of examples of including licenses and readmes, but this might come from the fact that there are no sdists/wheels, only gems. You can’t install to anywhere outside the gem’s namespace.
• Lua/LuaRocks (Rockspec format · luarocks/luarocks Wiki · GitHub): Despite being geared towards being a build system, it has a copy_directories rule where specified directories are “copied to the rock installation prefix as-is”. They give typical use cases such as “installing documentation and other files such as samples and tests”. Similarly to Ruby, you can’t install to anywhere outside the rock’s prefix.
• Rust/cargo (The Manifest Format - The Cargo Book): Cargo is very geared towards statically compiled binaries, so it’s probably not a useful example. If it’s not documentation or source code, cargo gives you no help. You can only include or exclude files for packaging, similar to MANIFEST.in. There is a mechanism for running arbitrary commands as part of a build script, but again it’s geared towards compiling things. You can however do whatever you want and write things to wherever you want (For example see the build.rs file of zlib-sys), but the only output is binaries and therefore a repackager couldn’t make use of what was written out.
• Go: Very similar to Rust. The expectation seems to be to embed files in a binary, or use a tool other than the go package manager.
• Haskell/cabal (7. Package Description — Cabal 3.4.0.0 User's Guide): Arbitrary data files can be included. They get installed to /usr/share/<package_name>/ on Linux. Cabal even provides utilities for finding these data files so that packages can get installed to any location and still function.

@pradyunsg

What are the kinds of data that we want to allow users to place in distro-specific locations?

• Notably, why does this data need to be provided on a per-package basis?

I don’t think we need to support distro-specific locations, only OS specific ones. We could enable package maintainers to help out repackagers (if this is indeed one of the reasons that we want to continue supporting these sorts of files) by allowing the pip install step of repackaging to place more files in their correct locations for “Linux in general”, but I don’t think it’s unreasonable to expect repackagers to have to do a certain amount of work. Especially when it’s work that is specific to the distro they support.
I also don’t think it’s realistic to expect package maintainers to support more than even a couple of distros. Plus repackagers will be more in touch with where files are supposed to go for their distro than a package maintainer is.

More broadly, I think even having the package declare which of these extras files should get installed helps our repackagers a lot because currently they have to dig through repos to find what can be installed and there’s the opportunity to miss things.

Is there another use case involving docker images? If I’m installing packages into a docker image it would be nice to have things like documentation or .desktop files for those packages included in the image, but I wouldn’t want to have to repackage everything I’m installing to make that happen.
That’s assuming that the system Python is being used.

@pf_moore

• How do cross-platform issues impact these answers (for example man pages are useful on Linux, but not on Windows)? My specific concern is whether we’re trying to impose an inherently platform-specific concept (“distro-specific locations”) onto a platform-independent format (wheels).

I don’t think I understand this question. Was the purpose of asking this to get us to more carefully consider all operating systems when providing possible solutions?

@dholth

On the Ubuntu system I’m using most programs’ /usr/share/docs contain a README , a copyright and a changelog.Debian.gz which is not heavier than what we already put in .dist-info .

As mentioned above, I don’t think we should go so far as to consider distro specific files. But these are very good examples of files that we could be including in packages, and in fact these files are what I was seeing most often included in Ruby gems when I was researching the above answer.
Ruby gems aren’t in a good position to install things like .desktop files because they’re installed into a namespaced directory, but I did see man pages get installed by a gem and then repackagers taking that file and moving it into /usr/share/man. Python packages are in a position where they can skip the middleman (middleperson?) and go straight to a location under sys.prefix without the repackager needing to seek out those files and do anything extra with them.

[FFY00 (Filipe Laíns)]

@AWhetter

I don’t think we need to support distro-specific locations, only OS specific ones.

The distro is the operating system. Linux distros generally implement the Posix standard, which is what I assume would be the “OS” here based on the value of os.name, but there are a lot of degrees of freedom they can take.
Taking one of your examples, .desktop files, they are not even specified in the Posix standard, they are defined by the freedesktop.org spec. So, it wouldn’t be correct to install these files to the Posix OS location. Even getting past that, the freedesktop.org spec does not specify file location, it is usually used an extension to FHS (Filesystem Hierarchy Standard), which itself makes a lot of locations valid.
Taking another one, documenation. The location for documentation to be installed isn’t specified by Posix, most modern Linux distros implement the FHS standard, which is where that location comes from.

I don’t want to be pedantic here, but I think this does have a very significant impact here. In your proposal, how should a .desktop file be handled? What do Python packagers need to specify for its distribution, and what would installers do? How would documentation be handled? There are several standards distros might implement, and lots of ways they could implement them.

All this is then ignoring the fact that almost all modern distros assume they will be the only ones touching /usr, only letting components touch their own paths. An example of this is /usr/lib/python3.9/site-packages.
Having pip install to /usr/share/doc or /usr/share/man behind the distro will result in issues, causing a bad UX to users, and possibly unstable systems in other situations.

The proposal may sound good at first glance – let’s just let users package their own arbitrary data – but it fails to consider the impacts that it has on an OS/distro level. pip is not the system package manager, and it shouldn’t be used in that way. There are, however, valid use cases where someone might want to install extra data, so we need to come to a middle term. The solution should be to integrate Python installers in the OS, not making Python installers an OS-level installer.

[AWhetter (Ashley Whetter)]

@FFY00

The distro is the operating system. Linux distros generally implement the Posix standard, which is what I assume would be the “OS” here based on the value of os.name , but there are a lot of degrees of freedom they can take.

Ah this makes sense. Thanks for the clarification.

@FFY00

I don’t want to be pedantic here, but I think this does have a very significant impact here. In your proposal, how should a .desktop file be handled? What do Python packagers need to specify for its distribution, and what would installers do? How would documentation be handled? There are several standards distros might implement, and lots of ways they could implement them.

I don’t yet have a proposal really. I started this thread to collect the information that we would need to come up with a good solution for the problem. One of my goals for my initial post was to figure out use cases that could drive this, but so far it’s been easier to discount use cases than it has been to come up with valid ones.
I think the melding of @dholth’s suggestion and yours is the closest thing we have to a proposal at the moment. So that would be having categories of directories that an installer can install to a platform specific default location (if the platform even has a default) but also allowing overrides to these defaults at install time for distro packagers to make use of. It’ll be difficult to know if this is a decent solution to the problem when we don’t know how it’ll get used, but it will at least encourage users to do the right thing whilst still giving them the power to do what they want.
If this sounds like a decent starting place and we think we’re as far into this discussion that we can go without proposing a solution, then I can flesh this out into a more detailed proposal? I’m still unclear what categories we would want to support because there’s been yeas and nays for anything beyond a ${root}, as your questions point out.

To try and answer your questions though, I think it depends on what situations we want full support for them.
A .desktop file in a virtualenv isn’t really useful for example, but I don’t like the idea of installing a file or not depending on whether it’s in a virtualenv. So for virtualenvs, I don’t think that a default location for a “{desktop_files}" (or called whatever) category matters too much. If we're trying to help repackagers, then we can get away with providing a default value for the category "like {root}/share/applications” because the repackager is going to specify a default value for the location of the category anyway.
For the case of pip installing to the system when in a docker container, I think it’s a similar situation but the author of the Dockerfile acts like the repackager and it’s up to them to provide a real default for the value.
I think it’s a similar situation for other files. It probably doesn’t matter too much where they go, so long as they can be overridden at install time and potentially also so that the installed Python application has a way of querying them back at run time? Querying back wouldn’t be useful for every category, but presumably an application would want to query where the file that they put in ${datadir} ended up.

[nhatkhai]

Doing this with wheel force pip act like god tools ( it had to known everything ) due to a rules of not allows arbitrary code execution

Typical work for everyone would be a arbitrary code execution ( pip once supported this with eggs package < 20.* )

Here are more cases of package ask user do some form of addition step installation for run code that no longer could be run by pip

• GitHub - mhammond/pywin32: Python for Windows (pywin32) Extensions
• GitHub - tensorflow/tensorflow: An Open Source Machine Learning Framework for Everyone
• GitHub - pyusb/pyusb: USB access for Python

If I have projA , other person have projB and we both want automate everything. Hence package maker cannot implement arbitrary code, so now each project need to check it full dependency and write code to detect which version of Above package used to run arbitrary code for install the correct version in the correct way. This work is repeat as many as number of projects using above package.

This kind of issue been discussed many time over years and guess what we still been talk about similar thing here. It is simply a architecture yelling at us. Solution also already been there. Now it is none within pip and about to reinvent in some short.

[nhatkhai]

Pywin32 been having arbitrary code to solve issue I mention above. Here is the evident

• pywin32/setup.py at b212 · mhammond/pywin32 · GitHub

My own package also do similar thing as total automation solution. It now no longer useful.

[pf_moore (Paul Moore)]

@AWhetter

I think it’s a similar situation for other files. It probably doesn’t matter too much where they go, so long as they can be overridden at install time and potentially also so that the installed Python application has a way of querying them back at run time? Querying back wouldn’t be useful for every category, but presumably an application would want to query where the file that they put in ${datadir} ended up.

What is the point of including files in the wheel at all if you’re not going to install them into an arbitrary location that standard tools don’t recognise and the installed Python code isn’t going to use (files the Python code will use should probably be installed as package data files)? Expecting users to manually hunt in the Python installation directory for (for example) documentation files is a pretty bad UI (I recall old setuptools-based installs doing this, and honestly they might as well not have supplied documentation for all the use it was )

I’d like to see an actual use case where this was genuinely the UI that people wanted…

[nhatkhai]

@pf_moore

’d like to see an actual use case where this was genuinely the UI that people wanted

I’m certainly that I do not known what you looking for, and how many example would count enough of a full people. I had tried, but I’m failing you I guess.

[AWhetter (Ashley Whetter)]

My initial post was the collection of all related use cases requested by users over the years in various discussions about data_files that I could find. The only one we haven’t discussed so far is my own, very specific, use case. You could generalise that use case to mean “any integrations with any applications that have some sort of search path mechanism and understand virtualenvs”. But that’s not particularly useful as something to drive a new design of data_files.

The only other use case that I can find is Jupyter’s extension mechanism (Distributing Jupyter Extensions as Python Packages — Jupyter Notebook 5.7.6 documentation), and coincidentally it seems to fit the above description. Jupyter looks for both configuration files and data files in directories created using data_files (Jupyter Paths priority order - General - Jupyter Community Forum). Data files are usually javascript files that form the main content of an extension. Both configuration files and data files go into specific directories that Jupyter searches to find extensions.
However the shortcoming of data files means that packages also need to provide an additional installation step that is used when the data files don’t end up in the expected place. This additional installation step requires the data files to be duplicated as package data so that the package can locate them to report to jupyter. This also extends jupyter’s search paths with an additional directory per extension.
For the above reason, plus the fact that data_files are considered deprecated in setuptools and not supported by any other packaging backends yet, there has been a proposal to replace the current extension mechanism to not rely on data_files (RFP for successor to data_files-based extension discovery? · Issue #351 · jupyter-server/jupyter_server · GitHub). However there doesn’t seem to have been any ideas that address the same problems that the additional install command has, and there’s still a preference for data_files to stick around. Perhaps @minrk or someone else from the Jupyter team can speak more on this and possibly even on what they would want to see in a redesign of data_files.

[pf_moore (Paul Moore)]

@AWhetter

My initial post was the collection of all related use cases requested by users over the years in various discussions about data_files that I could find.

Sorry. I glossed over those because I thought the discussion had moved on, but they do deserve comment. See below.

@AWhetter

I think there’s two set of “arbitrary locations”. One is those files under sys.prefix , and the other is absolutely anywhere on the machine. Despite the fact that I don’t think that it makes sense to support installation outside of sys.prefix either, I was posing the question about those two sets of files separately because I wanted to discuss all potential use cases.

I think the consensus is pretty clear at this point that “absolutely anywhere on the machine” isn’t acceptable. In particular, I agree with @uranusjr’s comment:

@uranusjr

Python packaging is expected (at least these days, and increasingly so) to only modify the current Python installation (e.g. for reproducibility), so anything that has side effect outside of the Python prefix should not be supported.

I’m therefore going to limit myself to proposals that only allow “arbitrary data files” to be installed under sys.prefix.

Looking at your use cases (I’ll defer the first one for now, as it’s your specific case and you noted that hasn’t been discussed yet):

1. .desktop files. You said yourself they are platform specific and only useful if installed into system locations, i.e. not under sys.prefix. So the solutions we’re looking at won’t work for these.
2. Manual pages. Same as .desktop files, platform specific and need to be in a system location. I’ve already called these out specifically as a case where my experience is that placing them under sys.prefix where OS utilities can’t see them is pointless.
3. Files in /etc - again system locations, and not supported by solutions that install under sys.prefix.
4. Binaries - these work right now (there’s a “scripts” location that things get installed to, and console scripts go there so we know this works). I don’t understand your comment about not working in a virtualenv. Unless you mean “install to /usr/local/bin” or something like that, but we’re back to locations outside sys.prefix again.

So all of your use cases basically require “arbitrary locations” and I believe that we already have a general feeling that this isn’t acceptable. If you want to still argue for that behaviour, I suggest that you make a specific proposal that describes what you want. But be prepared for it to be rejected - I will definitely vote -1 on it as I’ve already said.

I’m not entirely sure about your VFX use case, but it seems like your solution using virtualenv with an application-specific plugin is a reasonably good approach, so I’m not sure anything more is needed. And if you want to install the files to a location that’s already on MAYA_SCRIPT_PATH, you’re back to installing outside of sys.prefix…

So to be more specific, I’m looking for examples of use cases where it would be necessary to be able to install “arbitrary data files” into particular locations under sys.prefix other than the ones that are already covered by the existing sysconfig locations.

[minrk]

Perhaps @minrk or someone else from the Jupyter team can speak more on this and possibly even on what they would want to see in a redesign of data_files.

Jupyter extension packages typically include files that should go in one or both of these:

• configuration files to enable extensions /specify config (in $prefix/etc)
• static resources in $prefix/share (javascript extension sources, html templates, kernel specification files for discovery, etc.)

We never expect to write outside sys.prefix as part of package installation. Ideally, these are staged into the right place using data_files (or whatever its replacement should be) at install time, but there are some exceptions where that can’t/won’t work, so we provide our own jupyter ... install commands which stage files into $prefix/etc or $prefix/share after installation time. This two-step install has led to lots of mixed-up installations, as removing or upgrading a package is no longer associated with removing or changing other files associated with it. data_files installs work great for this today.

Critically for Jupyter, none of Jupyter specs are Python-specific, and many of these things are not part of Python packages at all, so we standardize on evaluating paths relative to $PREFIX rather than something python specific (we generally use sys.prefix for this), and we don’t want to assume that everything comes from a Python package.

For install time, all we really want is a reliable way to write {sys.prefix}/share/ (or prefix/etc) and a way at runtime that returns the same {sys.prefix}/share|etc that should work for installs:

• in venv
• not in env
• --user

It’s the lack of symmetrical “where would you have put it?” API that’s been a challenge for us. We encourage data_files as the easiest way to do this which works almost all of the time (very reliably in venvs and conda envs), but we are aware of custom distutils install schemes like system Pythons where it can get weird, because sys.prefix is not actually where installed files end up.

[pf_moore (Paul Moore)]

@minrk

For install time, all we really want is a reliable way to write {sys.prefix}/share/ (or prefix/etc) and a way at runtime that returns the same {sys.prefix}/share|etc that should work for installs:

• in venv
• not in env
• --user

It’s the lack of symmetrical “where would you have put it?” API that’s been a challenge for us.

Cool, that makes a lot of sense to me. So if sysconfig added a new path for “share” (or “etc”), that would satisfy this use case? The wheel spec already covers the installation side of this, as the rule in the PEP applies to any scheme key that exists in sysconfig. So all you need is the lookup side, which sysconfig would provide.

[hroncok (Miro Hrončok)]

Fedora Python maintainer here. I need to pitch in as somebody who fundamentally disagrees that desktop files or manual pages "don’t work under sys.prefix" or that installing stuff to an arbitrary location under sys.prefix has an increased potential to create file-conflicts. It is matter of perspective. Let me describe a matrix of the following things I could think of:

Python modules in {sys.prefix}/.../site-packages

Python modules: sys.prefix is /usr

• The modules work naturally because they are by definition in sys.path.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and distro-package manager.

Python modules: sys.prefix is /usr/local or ~/.local

• The modules work naturally because they are by definition in sys.path.
• There is a potential of file-conflict between different Python packages.

Python modules: sys.prefix is within Python virtual environment

• The modules work naturally because they are by definition in sys.path.
• There is a potential of file-conflict between different Python packages, albeit the chances are very limited, as virtual environments tend to be one-purpose and the package set usually does not grow without bounds.

Python modules: sys.prefix is another arbitrary location including Windows

• The modules work naturally because they are by definition in sys.path.
• There is a potential of file-conflict with different Python packages.

Commands (scripts) in {sys.prefix}/bin

Commands: sys.prefix is /usr

• The commands work naturally because /usr/bin is (almost) always on $PATH.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and distro-package manager.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install scripts into this location, or manually created content.

Commands: sys.prefix is /usr/local

• The commands usually work because /usr/local/bin tends to be on $PATH.
• If needed, users can extend their $PATH easily to make it work.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install scripts into this location, or manually created content.

Commands: sys.prefix is ~/.local

• The commands usually work because ~/.local/bin tends to be on $PATH on modern distros.
• If needed, users can extend their $PATH easily to make it work.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install scripts into this location, or manually created content.

Commands: sys.prefix is within Python virtual environment

• The commands don’t work unless the virtual environment is activated: activate script adds the directory to $PATH.
• Users can add symbolic links to scripts in a virtual environment to directories on their $PATH.
• There is a potential of file-conflict between different Python packages, albeit the chances are very limited, as virtual environments tend to be one-purpose and the package set usually does not grow without bounds.

Commands: sys.prefix is another arbitrary location

• The commands don’t work unless user modifies their $PATH.
• Users can add symbolic links to scripts in arbitrary locations to directories on their $PATH.
• There is a potential of file-conflict witch anything else.

Commands: Windows

• This is handled differently on Windows and it seems to work.

Manual pages in {sys.prefix}/share/man

Manpages: sys.prefix is /usr

• The manual pages work naturally because /usr/share/man is (almost) always in manpath.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and distro-package manager.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install manual pages into this location, or manually created content.

Manpages: sys.prefix is /usr/local

• The commands usually work because /usr/local/share/man tends to be in manpath.
• If needed, users/distros can extend their config easily to make it work.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install manual pages into this location, or manually created content.

Manpages: sys.prefix is ~/.local

• The manual pages usually work because ~/.local/share/man tends to be on manpath on modern distros.
• If needed, users/distros can extend their config easily to make it work.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install manual pages into this location, or manually created content.

Manpages: sys.prefix is within Python virtual environment

• The manual pages don’t work out of the box.
• Users can make them work by setting/extending $MANPATH.
• If deemed useful, the activate script could be improved to set/extend $MANPATH.
• Users can add symbolic links to manual pages in a virtual environment to directories on their manpath.
• There is a potential of file-conflict between different Python packages, albeit the chances are very limited, as virtual environments tend to be one-purpose and the package set usually does not grow without bounds.

Manpages: sys.prefix is another arbitrary location

• The manual pages don’t work out of the box.
• Users can make them work by setting/extending $MANPATH.
• Users can add symbolic links to manual pages in arbitrary locations to directories on their manpath.
• There is a potential of file-conflict witch anything else.

Manpages: Windows

• The manual pages are not relevant there but they don’t hurt anything.

Desktop files in {sys.prefix}/share/applications

(This also applies to their icons in {sys.prefix}/share/icons or {sys.prefix}/share/pixmaps.)

Desktop files: sys.prefix is /usr

• The desktop files work naturally because /usr/share/applications is used by default.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and distro-package manager.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install desktop files into this location, or manually created content.

Desktop files: sys.prefix is /usr/local or ~/.local

• The desktop files work naturally because /usr/local/share/applications and ~/.local/share/applications is used by default.
• There is a potential of file-conflict between different Python packages.
• There is a potential of file-conflict between pip-installed packages and other language stack package managers that would also install desktop files into this location, or manually created content.

Desktop files: sys.prefix is within Python virtual environment

• The desktop files don’t work.
• Users might be able to make them work by some configuration (I have not explored this).
• Users can add symbolic links to desktop files in a virtual environment to the directories that work with desktop files.
• There is a potential of file-conflict between different Python packages, albeit the chances are very limited, as virtual environments tend to be one-purpose and the package set usually does not grow without bounds.
• If somebody wants to explore new ideas, we can have a concept of “activating a virtual environment for your desktop environment”, but I don’t think it would be that useful.

Desktop files: sys.prefix is another arbitrary location

• The desktop files don’t work.
• Users might be able to make them work by some configuration (I have not explored this).
• There is a potential of file-conflict witch anything else.

Desktop files: Windows

• The desktop files are not relevant there but they don’t hurt anything.

Static application data in {sys.prefix}/share/{app_name}

(E.g. Jupyter kernels.)

• They always work regardless of sys.prefix.
• There is a potential of file-conflict witch anything else that uses app_name.

---

tl;dr

• The stuff works quite fine for many values of sys.prefix; the degree of “works out of the box” varies, but there is some potential for improvement as well.
• The potential for file-conflicts is not worse than the existing potential (you can nuke a system by installing bash or sh script quite fine already even without data_files).
• Many files are useless on Windows but I consider that OK.

[bollwyvl (Nicholas Bollweg)]

Thanks, @AWhetter for looping Jupyter in, and @minrk for the thorough summary of our challenges!

To the various venv/not venv/--user cases, I’d add:

• uninstall leaving the file system “unharmed”
• a key “can’t/won’t” work case: pip install -e (and whatever flit and poetry do). I don’t know how this can be solved in a cross-platform way, but our experiments in going from 4 search paths to hundreds (via entry_points) were… not encouraging, and would require non-python Jupyter components to shell out to get such a list.

Semi-related: having at least some warning (which could ideally be elevated to an error with e.g. an environment variable) when two packages try to write to the same file in {sys.prefix}/etc or {sys.prefix}/share would be helpful… today we kinda wait for downstreams (or users) to find these issues. Our flagship first-party consumers of etc have well-known (well, at least documented) {sys.prefix}/etc/jupyter_*_config.d/ folders that have made this more robust, but one (well, usually two) badly-behaved packages can make a right mess of things depending on the order of installation.

[pf_moore (Paul Moore)]

In practice, this is somewhat of a side issue. The question of whether desktop files or manual pages "work under sys.prefix" is only significant if someone is arguing that wheels need to support installation of files to locations outside of sys.prefix - and I don’t think anyone is still trying to make that argument.

What we’re left with now, as far as I can see, is that the wheel spec supports custom install locations for any named path that is supported by sysconfig. Currently, sysconfig supports 8 paths (from the Python docs):

• stdlib : directory containing the standard Python library files that are not platform-specific.
• platstdlib : directory containing the standard Python library files that are platform-specific.
• platlib : directory for site-specific, platform-specific files.
• purelib : directory for site-specific, non-platform-specific files.
• include : directory for non-platform-specific header files.
• platinclude : directory for platform-specific header files.
• scripts : directory for script files.
• data : directory for data files.

Adding extra install locations is as simple as getting some extra locations added to the sysconfig module. It’s also exactly as hard as doing that - doing anything outside of the stdlib still requires getting agreement on locations for all platforms, supporting virtualenvs, handling ways of letting distros customise the locations, etc. I suspect people have a view that there’s an “easier” way than getting sysconfig changed, but honestly, I doubt that’s the case in practice.

So I think the “new standard” here might simply be a matter of requesting new install locations for the sysconfig module.

[dholth (Daniel Holth)]

We will also have to deal with getting KeyError when a wheel’s .data/ directory has a name not in sysconfig. I’ve wanted to just leave the data directory in site-packages in that case and possibly print() a warning.

[pf_moore (Paul Moore)]

That should be the case right now. The PEP doesn’t insist that wheels stick to a list of names, so tools have to be prepared for KeyError already.

I’d be fine with someone checking what existing installers do (pip, wheel, installer, poetry, …?) and if they all have a common behaviour, adding that to the PEP as a clarification. If they do different things, we’d have to say it’s currently implementation-defined and if we want to define a specific behaviour, that would be a proper PEP update.

[nhatkhai]

@AWhetter , @minrk, looking at the pySerial and dig depth into it. I realize the answer to our problem may fall into “Conda” tool - Conda for data scientists — conda 4.10.1.post2+b6d32c8d7 documentation

[nhatkhai]

https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/