CMake executed by a Python script can't find the file which exists

[ay0ks (Dimitriy)]

For some unclear reason the command executed by a Python script doesn’t work both in WSL and Windows, however works if I execute the command myself in a shell.

Script on GitHub: Project configurator script · GitHub

Linux

Command line (executed by python Configure.py -T Linux-x64):

cmake \
  -S /mnt/c/Users/ay0ks/Workspace/Red \
  -B /mnt/c/Users/ay0ks/Workspace/Red/Destination/Build/Linux-x64-Linux-x64/GNU-14.2.1-GNU-2.44/Debug \
  -DCMAKE_TOOLCHAIN_FILE="/mnt/c/Users/ay0ks/Workspace/Red/Auxillary/Toolchains/Linux-x64.cmake" \
  -DCMAKE_BUILD_TYPE="Debug" \
  -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" \
  -DCMAKE_MAKE_PROGRAM=ninja \
  -GNinja

Result:

❯ python Configure.py -T Linux-x64
-- Identifying the compilation environment
 > project("Identify")
 > message("${CMAKE_C_COMPILER_ID}~~~${CMAKE_C_COMPILER_VERSION}~~~${CMAKE_C_COMPILER_LINKER_ID}~~~${CMAKE_C_COMPILER_LINKER_VERSION}")
 > message("${CMAKE_CXX_COMPILER_ID}~~~${CMAKE_CXX_COMPILER_VERSION}~~~${CMAKE_CXX_COMPILER_LINKER_ID}~~~${CMAKE_CXX_COMPILER_LINKER_VERSION}")
 > cmake -Wno-dev -S /tmp/tmp7q12ok5j -B /tmp/tmp7q12ok5j
-- Identified environment:
 @ C/CXX compiler vendor: GNU
 @ C/CXX compiler version: 14.2.1
 @ Linker vendor: GNU
 @ Lompiler version: 2.44
-- Destination directory:
 "../../Destination/Build/Linux-x64-Linux-x64/GNU-14.2.1-GNU-2.44/Debug"
-- Bootstrapping the compilation configuration
 > cmake -S /mnt/c/Users/ay0ks/Workspace/Red -B /mnt/c/Users/ay0ks/Workspace/Red/Destination/Build/Linux-x64-Linux-x64/GNU-14.2.1-GNU-2.44/Debug -DCMAKE_TOOLCHAIN_FILE="/mnt/c/Users/ay0ks/Workspace/Red/Auxillary/Toolchains/Linux-x64.cmake" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" -DCMAKE_MAKE_PROGRAM=ninja -GNinja
CMake Error at /usr/share/cmake/Modules/CMakeDetermineSystem.cmake:152 (message):
  Could not find toolchain file:
  "/mnt/c/Users/ay0ks/Workspace/Red/Auxillary/Toolchains/Linux-x64.cmake"
Call Stack (most recent call first):
  CMakeLists.txt:9 (project)


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

~/Workspace/Red/Auxillary/Scripts ········································································ 3s
❯

Screenshot:

However, if I execute the command manually, it works like it should:

❯ cmake -S /mnt/c/Users/ay0ks/Workspace/Red -B /mnt/c/Users/ay0ks/Workspace/Red/Destination/Build/Linux-x64-Linux-x64/GNU-14.2.1-GNU-2.44/Debug -DCMAKE_TOOLCHAIN_FILE="/mnt/c/Users/ay0ks/Workspace/Red/Auxillary/Toolchains/Linux-x64.cmake" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" -DCMAKE_MAKE_PROGRAM=ninja -GNinja
-- The C compiler identification is Clang 19.1.7
-- The CXX compiler identification is Clang 19.1.7
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/sbin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/sbin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (4.6s)
-- Generating done (0.3s)
-- Build files have been written to: /home/ay0ks/Workspace/Red/Destination/Build/Linux-x64-Linux-x64/GNU-14.2.1-GNU-2.44/Debug

~/Workspace/Red/Auxillary/Scripts ···················································································································· 6s
❯

Screenshot:

Windows

Command line (executed by python Configure.py -T Windows-x64):

cmake \
  -S C:\Users\ay0ks\Workspace\Red \
  -B C:\Users\ay0ks\Workspace\Red\Destination\Build\Windows-x64-Windows-x64\MSVC-19.43.34808.0-MSVC-14.43.34808.0\Debug \
  -DCMAKE_TOOLCHAIN_FILE="C:\Users\ay0ks\Workspace\Red\Auxillary\Toolchains\Windows-x64.cmake" \
  -DCMAKE_BUILD_TYPE="Debug" \
  -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" \
  -DCMAKE_MAKE_PROGRAM=ninja \
  -GNinja

Result:

PS C:\Users\ay0ks\Workspace\Red\Auxillary\Scripts> python .\Configure.py -T Windows-x64
-- Identifying the compilation environment
 > project("Identify")
 > message("${CMAKE_C_COMPILER_ID}~~~${CMAKE_C_COMPILER_VERSION}~~~${CMAKE_C_COMPILER_LINKER_ID}~~~${CMAKE_C_COMPILER_LINKER_VERSION}")
 > message("${CMAKE_CXX_COMPILER_ID}~~~${CMAKE_CXX_COMPILER_VERSION}~~~${CMAKE_CXX_COMPILER_LINKER_ID}~~~${CMAKE_CXX_COMPILER_LINKER_VERSION}")
 > cmake -Wno-dev -S C:\Users\ay0ks\AppData\Local\Temp\tmp15a2x462 -B C:\Users\ay0ks\AppData\Local\Temp\tmp15a2x462
-- Identified environment:
 @ C/CXX compiler vendor: MSVC
 @ C/CXX compiler version: 19.43.34808.0
 @ Linker vendor: MSVC
 @ Lompiler version: 14.43.34808.0
-- Destination directory:
 "..\..\Destination\Build\Windows-x64-Windows-x64\MSVC-19.43.34808.0-MSVC-14.43.34808.0\Debug"
-- Bootstrapping the compilation configuration
 > cmake -S C:\Users\ay0ks\Workspace\Red -B C:\Users\ay0ks\Workspace\Red\Destination\Build\Windows-x64-Windows-x64\MSVC-19.43.34808.0-MSVC-14.43.34808.0\Debug -DCMAKE_TOOLCHAIN_FILE="C:\Users\ay0ks\Workspace\Red\Auxillary\Toolchains\Windows-x64.cmake" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" -DCMAKE_MAKE_PROGRAM=ninja -GNinja
CMake Error at C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.30/Modules/CMakeDetermineSystem.cmake:152 (message):
  Could not find toolchain file:
  "C:\Users\ay0ks\Workspace\Red\Auxillary\Toolchains\Windows-x64.cmake"
Call Stack (most recent call first):
  CMakeLists.txt:9 (project)


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
PS C:\Users\ay0ks\Workspace\Red\Auxillary\Scripts>

Screenshot:

However, same as on Linux, if I run the command myself, everything works like it should:

PS C:\Users\ay0ks\Workspace\Red\Auxillary\Scripts> cmake -S C:\Users\ay0ks\Workspace\Red -B C:\Users\ay0ks\Workspace\Red\Destination\Build\Windows-x64-Windows-x64\MSVC-19.43.34808.0-MSVC-14.43.34808.0\Debug -DCMAKE_TOOLCHAIN_FILE="C:\Users\ay0ks\Workspace\Red\Auxillary\Toolchains\Windows-x64.cmake" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always" -DCMAKE_MAKE_PROGRAM=ninja -GNinja
-- The C compiler identification is MSVC 19.43.34808.0
-- The CXX compiler identification is MSVC 19.43.34808.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.43.34808/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.43.34808/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (5.8s)
-- Generating done (0.3s)
-- Build files have been written to: C:/Users/ay0ks/Workspace/Red/Destination/Build/Windows-x64-Windows-x64/MSVC-19.43.34808.0-MSVC-14.43.34808.0/Debug
PS C:\Users\ay0ks\Workspace\Red\Auxillary\Scripts>

Screenshot:

I’ve mirrored this question to the CMake forum in case this is not exactly Python’s problem: https://discourse.cmake.org/t/cmake-executed-by-a-python-script-cant-find-the-file-which-exists/13798 (Their anti-spam filters have hidden the post temporarily, so it may not be available right now.)
Thanks in advance.

[JamesParrott]

The CMake config step needs a lot of environment variables, which you’ve not passed through from the parent process to the subprocess.run call. I did something like this in a hatch build hook that calls CMake:

        env = os.environ.copy()

        env.setdefault('VCPKG_INSTALLATION_ROOT','~/vcpkg')


        config_command_str = f"""
                cmake
                -G {config.generator}
                -B {build_dir}
                -S .
            """.lstrip().replace('\n','')


        subprocess.run(
            config_command_str,
            env=env,
        )

[ay0ks (Dimitriy)]

@JamesParrott

env = os.environ.copy()

I’ve added

environment = os.environ.copy()

to the beginning of the Program.main method
and added env=environment to every subprocess.run call but the problem still persists:

[JamesParrott]

The error’s clearly saying the env vars CMAKE_C(XX)?_COMPILER are not set. Figure out how they’re being set when you run CMake from the command line. I forgot that in my CMakeLists.txt I also added:

if (NOT MSVC)
    set(CMAKE_C_COMPILER "gcc")
    set(CMAKE_CXX_COMPILER "g++")
endif()

I also had to tell my subprocess the path to Ninja.

[ay0ks (Dimitriy)]

They are being set in the toolchain file that CMake can’t find for some reason (even though the absolute path to it is provided via -DCMAKE_TOOLCHAIN_FILE and that file exists on the machine:

It can find them if I execute the command manually in a shell.

I’m still waiting for CMake forum moderators to unhide my post so I can get additional info from them though.

[JamesParrott]

Oh yeah. Just few thoughts. I got it working using cmd in Windows, not Powershell. I’d double check which subshell, Windows’ Python launches, when run from WSL too.

Maybe make sure the paths are in CMake format too, with forward slashes even on Windows.

[ay0ks (Dimitriy)]

I think it’s just a problem with subprocess.run (even with specified environment), because I switched it with os.system and it works now both in WSL and Windows.

Edit: Solution CMake executed by a Python script can't find the file which exists. - #3 by ay0ks - Usage - CMake Discourse

[JamesParrott]

Thanks for the update. I’d previously been questioned on this forum, after mentioning that I needed to use shell=True on Linux (not on Windows though), with CMake in mind, so sorry I didn’t suggest that.