Move frozen modules outside the source tree to support immutable source directories

I’m looking to package Python for my own Linux distro and I’m building it with Zig. I’m looking to then build the OS inside of Nix for easy CI and caching. The problem I know I will run into is that I will have to be able to build Python inside the Nix cache and copying files will decrease build times and increase complications. The quicker solution is for Python to support having the frozen modules be located outside the source tree. I was looking to open a proposal and work on a PR but looking at the issue template, it says I should open a discussion first which I am doing so right now.

I’ve done some of the work by implementing the C source changes in my own fork which can be found at Partial implementation of immutable source support · MidstallSoftware/cpython@1006b68 · GitHub.

I’ve updated the title to directly describe the request.

Isn’t it already the case that frozen modules are in the build tree, not the source tree? When doing out-of-tree builds, I think I remember seeing the frozen modules generated there (that wasn’t the case for Windows, but I recently moved that too)

That’s not the case at least with Python 3.12.2. I tried building Python’s source code and adapting it to build in Zig and got an error in Modules/getpath.c that it is requiring stuff in frozen modules which are C files generated from Python files.

Can you share more details about what you tried and what kind of errors you got?

I just tested an out-of-tree build on Linux using the 3.12 branch, and it built successfully with all the frozen modules generated under the build tree.
The only things that ended up written to the source tree are __pycache__ directories.

Yeah so when I initially tried building Python, I got an error in cpython/Python/frozen.c at 6abddd9f6afdddc09031989e0deb25e301ecf315 · python/cpython · GitHub that the headers were missing. I looked at how to implement the build logic in Zig and discovered the frozen modules are written to the source tree so I made a fork and made any include which relied on the frozen modules be a macro. This fixed my build issues.

I don’t know anything about Zig, but in the regular Python build, these includes are picked up from the build tree via the -IPython flag that gets added to BASECPPFLAGS here. This is an example of where to CPython Makefile creates a frozen module header in the build tree (Python/frozen_modules/importlib._bootstrap.h is relative to the build tree).

Yeah, the problem is that I need to generate the frozen modules myself outside of the source tree. Autotools is the one thing that makes the source tree and build tree the same directory.