I’m writing a script for automating installations of several packages, and it checks that the relevant command-line programs are where they’re expected to be after installation. Unfortunately, I cannot determine a foolproof way of always predicting exactly where pip will place console scripts. According to thesetwo StackOverflow questions, the rules are as follows:
For user installs: sysconfig.get_path('scripts', f'{os.name}_user') (which — always? — equals os.path.join(user_site, 'bin'), or os.path.join(user_site, 'Scripts') on Windows)
For non-user installs: sysconfig.get_path('scripts') (which — always? — equals os.path.join(sys.prefix, 'bin'), or os.path.join(sys.prefix, 'Scripts') on Windows), with a special exception that maps /System/Library/ paths on macOS to /usr/local/bin.
However, this is not the whole story, and I cannot find the rest of the story in the pip code. There are at least two situations where the above rules are incorrect:
When using Python installed through Homebrew on macOS, the predicted system install path is of the form /usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/bin, but console scripts are actually installed in /usr/local/bin.
On Ubuntu 20.04 (and possibly other versions), the predicted system install path is /usr/bin, but console scripts are actually installed in /usr/local/bin.
What are the exact rules to follow to take exceptions like these into account?
For historical reasons, pip uses distutils to get console script locations instead of sysconfig. The two should return the same values, but some systems (incorrectly) patch only one of them and not the other. That’s why you see this on Ubuntu (the behaviour is inherited from Debian). Not sure about Homebrew.
but some systems (incorrectly) patch only one of them and not the other
Sadly, there’s not much guidance on what exactly the sysconfig values should mean, or on how distros like Debian or Homebrew should correct them if they’re wrong.
I guess it’s historically due to lack of communication – both distros and CPython doing stuff their own way, while privately accusing the other party of doing stuff incorrectly.
Indeed. In the case of sysconfig specifically, the lack of communication is not only between CPython and Linux distributions, but also between PyPA and CPython core developers. Many packaging contributors were not aware the two modules are returning different things (or even there’s a second way to get the values at all!) until much too late, missing the opportunity to communicate the issue to relevant parties.
So, after some more digging through pip’s code, I’ve determined that the code block highlighted in this SO answer is only relevant when uninstalling scripts, not when installing them. When installing, the script directory equals pip._internal.locations.get_scheme("package-name", **kwargs).scripts for the appropriate **kwargs settings. This, in turn, determines the paths by calling distutils, which is where Ubuntu inserts its patch to make pip install in /usr/local/bin instead of /usr/bin. I can’t find where Homebrew does its patching, but it must be somewhere, as get_scheme("foo").scripts equals /usr/local/bin on Homebrew Python.
Now I have to decide whether to make use of this logic by calling a private function in pip or by using a standard library module that’s slated for removal …
It’s quite likely that pip will switch to sysconfig at some point, for what it’s worth (and when we do that, the internal function might be modified or even removed, so be careful if you take that route!) But the real problem here is that distutils and sysconfig don’t give the same answer, which is ultimately a bug (in the stdlib, or in the way distributions are patching the stdlib, I’m not entirely clear from what you said).
As a workaround for that bug, can’t you check that the console scripts at least end up in one of those locations? It’s not ideal, but it’s probably as good as anything else (after all, installers other than pip might not choose the same stdlib function as pip does).