Could you not just use
sysconfig.get_path("data")?
Yes, I think we can! As long as it’s documented that this is where data files go by default for pip install and pip install --user. I think that’s what’s been missing—a documented queryable api that e.g. returns /usr/local on debian. Then it could be considered a bug that debian debian sysconfig.get_path("data") returns /usr instead of /usr/local. As you said, every possible --target install is not critical for us to support. We have post-install configuration to deal with those cases, we mainly care about the really common ones getting us 90% there without that extra second step.
I’ll confess that I was surprised to learn that data_files is deprecated in the docs with the message:
It does not work with wheels, so it should be avoided.
Mainly because we use data_files in wheels all the time and it works great. So I don’t quite understand what the current situation is. We’ve always used ‘relative’ paths, e.g.
data_files=[("share/jupyter/kernels/python3", glob("kernelspec/*")),]
So what part of data_files is really deprecated? Is it only absolute path support? We’ve never needed that. Is the plan to have a new spec that can reasonably be supported and un-deprecate data_files with a more limited scope, or a new keyword arg with a new, very similar spec?
So from the Jupyter perspective, everything is working pretty well today except for an official consistent API that would return /usr/local on debian instead of /usr. If the fix is for debian to patch sysconfig in the same way they do distutils as proposed in that pip issue, that would probably work great for us. We’ve handled that well enough so far with a bit of special-casing for /usr/local anyway, so we have no pressing issues, unless data_files is actually going to be removed.
The Jupyter issue of locating the install location of data files at runtime, becomes a question of “what was the install scheme used to install this package?”
We actually want something that I think should be even simpler than that, since we don’t actually know what packages we are asking about (and critically, they might not be Python packages). Instead, the question we want to ask is:
- what is the default install scheme, and where would it put data files
- what is the
--userinstall scheme, and where would it put data files
so that we can put those on our default search path to make life easier for wheel installs. That’s really it.
To put it another way, data_files for us is distinct from package_data because it is for making files available to other tools or packages, which may even not be Python at all (e.g. xeus in c++ or nteract in javascript. Having a consistent $PREFIX/share|etc scheme is our approach to that.