Build cpython from scratch

Python Help
1

I am currently working on building a minimal CPython interpreter using Meson to gain better control over static linking and potentially exchange implementation details. One idea is to create something akin to bun/deno but for CPython, where specific native code is implemented and bundled/ or solve the issue pyo3, many options seem interesting once building CPython was much easier.

I’ve shared the issue on the Meson WrapDB repo, and also on the buck2 issue tracker (re hermetic builds) hoping to nerd-snipe someone into this. I’m reaching out to your expertise to seek guidance in understanding the build process better.

Conceptually, building CPython from a few C files or generating some code should not be overly challenging. I’ve managed to put together a basic setup that gets me as far as running the following command:

meson compile -C builddir && ./builddir/Programs/python3

However, I’ve encountered a hurdle with the Python path configuration, resulting in the following error:

$ meson compile -C builddir && ./builddir/Programs/python3                                                                                                                                                         

INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /opt/homebrew/bin/ninja -C /Users/benediktmandelkow/Downloads/cpython/builddir
ninja: Entering directory `/Users/benediktmandelkow/Downloads/cpython/builddir'
ninja: no work to do.
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './builddir/Programs/python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/Users/benediktmandelkow/Downloads/cpython/Lib/python3.13'
  sys.path[0] = (not set)
  sys._base_executable = '/Users/benediktmandelkow/Downloads/cpython/builddir/Programs/python3'
  sys.base_prefix = '/Users/benediktmandelkow/Downloads/cpython/Lib'
  sys.base_exec_prefix = '/Users/benediktmandelkow/Downloads/cpython/Lib'
  sys.platlibdir = '/Users/benediktmandelkow/Downloads/cpython/Lib'
  sys.executable = '/Users/benediktmandelkow/Downloads/cpython/builddir/Programs/python3'
  sys.prefix = '/Users/benediktmandelkow/Downloads/cpython/Lib'
  sys.exec_prefix = '/Users/benediktmandelkow/Downloads/cpython/Lib'
  sys.path = [
    '/Users/benediktmandelkow/Downloads/cpython/Lib/python313.zip',
    '/Users/benediktmandelkow/Downloads/cpython/Lib/python3.13',
    '/Users/benediktmandelkow/Downloads/cpython/Lib/python3.13/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001dd501000 (most recent call first):
  <no Python frame>

If you have experience with the Python build process and could help me create a minimal example, I would greatly appreciate your support. I believe that with your expertise, we can put together a straightforward and understandable build system for a minimal working Python interpreter which is easy to hack on.

(As an aside, every complex build system is there for some reason, I don’t want to say that CPython should adapt meson or sth. like that, just that a minimal example of building CPython would really help adopting it in some areas/ creative ways)

Feel free to check the GitHub repo for the actual code meson_build

Kind regards

1 Like
2

Have you tried to build cpython using its build system?

I would assume when that works you have something to compare your meson build against.

3

Have you tried to build cpython using its build system?

Thats how I got pyconfig.h actually but other than that I’m not sure what to compare/ how to deduce the build steps the best way. For generating code for freezing the modules the comments were quite helpful but now I feel like to get further I’d need a much better understanding of the codebase or ask for help here.

I tried to trace the function calls of some initialisation routines but they were fairly deep so I did not manage to fully understand the procedures yet (not enough to remove/ deactivate them as much as possible).

A MVP would be a python interpreter which could just calculate 3 + 4 or print(“hello”), I’d be fine with excluding as much of the std as possible as I’d probably like to add needed functions using pyo3 and native implementations as one example.

Compiling to wasm-unknown-unknown (no emscripten) should also be trivial then which is currently not really that well documented as far as I can see.

5

the zig project has a working alternative build setup which at first glance only misses the invocations of freeze_module but is otherwise pretty self contained and might be a good starting point

6

Finally managed to port it to python build_cpython.py · GitHub and tested it on macos and alpine linux. This could quite feasibly ported to any other language (does not use host python for anything, could be written in c just as well), I will use this for further experiments regarding wasm/ import systems/ pyo3 in the future hopefully and if I come around to it create a proper repository for it. One convenient thing is that it already creates a compile_commands.json which works with clangd so for me it was already quite hackable/ configurable.

Maybe this is helpfull for others already who want to understand the cpython build process without diving through the configure system.

1 Like