I recently switched to pyproject.toml so that build time dependencies are not installed globally, but only in the temporary directory where pip installs packages needed for the build.
The issue is that the dependencies install headers and other goodies to that temp directory (e.g. E:\msys64\tmp\pip-build-env-707fj1u9\overlay\include), but I need to be able to pass that include/lib directory to the cython extensions. I cannot find a way to get that path from setup.py.
I concur that using the current working directory and constructing the path using it, is the way to go here.
My understanding is that pip will always invoke setup.py from the directory the setup.py file is in (through setuptools’ PEP 517 backend or directly in its legacy code path).
Hmm, I tried os.getcwd(), but although e.g. cython is installed to E:\msys64\tmp\pip-build-env-i26uu301\overlay\Lib\site-packages\Cython\__init__.py, os.getcwd() returns something like E:\msys64\tmp\pip-req-build-5sl60j8a. This happens whether I call it from subclassing build_ext. or if I call it in setup.py at the start of the file.
The first path, E:\msys64\tmp\pip-build-env-i26uu301\overlay, is what I’m after as that’s the prefix, the latter path is just a copy of the package being installed.
In some cases you can be able to get a list of installed files for dependencies (using e.g. importlib_metadata) and then “guess” which directory should be considered the root to add to your include path. There will end up being edge cases unless you have prior knowledge about a unique header file you expect to see for each dependency. This also won’t work if the packages were installed directly with setuptools or are managed by the OS-level package manager, which may not record the source information.
To be explicit, PEP 517 describes here the precise requirements on a build environment. It does not mention anything about “supplemental” files like headers being available - so it’s entirely valid for a build frontend to not even install the headers, and (to quote the PEP) “A build backend MUST be prepared to function in any environment which meets the above criteria”.
I can see the argument that supplementary files should be made available in the isolated build environment. But making that something that build scripts can rely on would require an extension to PEP 517. But in the absence of such an extension being proposed and agreed, adhoc approaches that involve educated guesses based on how build tools are likely to have set up the environment, are essentially all you have.
Pip could, as a non-standard extension to PEP 517, make guarantees about header files, etc. But we almost certainly won’t, as we’re working hard to ensure that tool behaviour is standardised as much as possible and does not rely on implementation-defined specifics.