Custom build python on mac11.6 can't load pybind11 bindings

I’m trying to create a python binding for a c++ project.

I’m planning to use pybind11 and started with its official example:

the code is very simple

#include <pybind11/pybind11.h>

int add(int i, int j) {
    return i + j;
}

PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring

    m.def("add", &add, "A function which adds two numbers");
}
c++ -g -Wall -shared -std=c++11 -fPIC -I/Users/xxx/work/masking_ai/extern/pybind11/include binding.cpp -I/Users/xxx/work/python_build/include/python3.9d -L/Users/xxx/work/python_build/lib -lpython3.9d -lintl.8 -o example$(/Users/xxx/work/python_build/bin/python3.9d-config --extension-suffix)

when I type import example in python, it hits a crash. I have tried different versions, 3.8.9, 3.9.1, 3.9.7 …

they are either built manually by me using

./configure
make

or installed by pyenv, which also builds python upon installation.

here is the call stack I get with debug symbol, it hit an assert assert(gilstate->autoInterpreterState);

it has something to do with the python version. I have tried 3.9.6 installed with brew, it worked well. But if the python is built on my machine, I will see these issues.

* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x00000001033fede5 example.cpython-39d-darwin.so`PyGILState_Ensure at pystate.c:1357:5
   1354     /* Ensure that _PyEval_InitThreads() and _PyGILState_Init() have been
   1355        called by Py_Initialize() */
   1356     assert(_PyEval_ThreadsInitialized(runtime));
-> 1357     assert(gilstate->autoInterpreterState);
   1358
   1359     PyThreadState *tcur = (PyThreadState *)PyThread_tss_get(&gilstate->autoTSSkey);
   1360     int current;
Target 0: (python3.9) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00007fff203b392e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff203e25bd libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007fff20337406 libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff203367d8 libsystem_c.dylib`__assert_rtn + 314
  * frame #4: 0x00000001033fede5 example.cpython-39d-darwin.so`PyGILState_Ensure at pystate.c:1357:5
    frame #5: 0x0000000103105419 example.cpython-39d-darwin.so`pybind11::detail::get_internals(this=0x00007ffeefbf59f0)::gil_scoped_acquire_local::gil_scoped_acquire_local() at internals.h:321:45
    frame #6: 0x0000000103104065 example.cpython-39d-darwin.so`pybind11::detail::get_internals(this=0x00007ffeefbf59f0)::gil_scoped_acquire_local::gil_scoped_acquire_local() at internals.h:321:66
    frame #7: 0x000000010310370d example.cpython-39d-darwin.so`pybind11::detail::get_internals() at internals.h:324:7
    frame #8: 0x00000001031034ff example.cpython-39d-darwin.so`::PyInit_example() at binding.cpp:7:1
    frame #9: 0x00000001002a8106 python3.9`_PyImport_LoadDynamicModuleWithSpec(spec=0x0000000100f34d20, fp=0x0000000000000000) at importdl.c:164:9
    frame #10: 0x00000001002a7a47 python3.9`_imp_create_dynamic_impl(module=0x0000000100a00890, spec=0x0000000100f34d20, file=0x0000000000000000) at import.c:2297:11
    frame #11: 0x00000001002a6eae python3.9`_imp_create_dynamic(module=0x0000000100a00890, args=0x0000000100cca8d8, nargs=1) at import.c.h:330:20
    frame #12: 0x0000000100125d96 python3.9`cfunction_vectorcall_FASTCALL(func=0x0000000100a09890, args=0x0000000100cca8d8, nargsf=1, kwnames=0x0000000000000000) at methodobject.c:426:24
    frame #13: 0x00000001000affb8 python3.9`PyVectorcall_Call(callable=0x0000000100a09890, tuple=0x0000000100cca8c0, kwargs=0x0000000100f39530) at call.c:231:16
    frame #14: 0x00000001000b01e9 python3.9`_PyObject_Call(tstate=0x0000000100906640, callable=0x0000000100a09890, args=0x0000000100cca8c0, kwargs=0x0000000100f39530) at call.c:266:16
    frame #15: 0x00000001000b02e2 python3.9`PyObject_Call(callable=0x0000000100a09890, args=0x0000000100cca8c0, kwargs=0x0000000100f39530) at call.c:293:12
    frame #16: 0x00000001002621f0 python3.9`do_call_core(tstate=0x0000000100906640, func=0x0000000100a09890, callargs=0x0000000100cca8c0, kwdict=0x0000000100f39530) at ceval.c:5092:9
    frame #17: 0x000000010025d29a python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100e76b30, throwflag=0) at ceval.c:3580:22
    frame #18: 0x000000010024d7df python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100e76b30, throwflag=0) at pycore_ceval.h:40:12
    frame #19: 0x0000000100263496 python3.9`_PyEval_EvalCode(tstate=0x0000000100906640, _co=0x00000001007ce5f0, globals=0x0000000100a00950, locals=0x0000000000000000, args=0x0000000100e10d20, argcount=2, kwnames=0x0000000000000000, kwargs=0x0000000100e10d30, kwcount=0, kwstep=1, defs=0x0000000000000000, defcount=0, kwdefs=0x0000000000000000, closure=0x0000000000000000, name=0x00000001007b6d60, qualname=0x00000001007b6d60) at ceval.c:4327:14
    frame #20: 0x00000001000b0a37 python3.9`_PyFunction_Vectorcall(func=0x0000000100a064b0, stack=0x0000000100e10d20, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:396:12
    frame #21: 0x000000010026540b python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a064b0, args=0x0000000100e10d20, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #22: 0x0000000100261d6a python3.9`PyObject_Vectorcall(callable=0x0000000100a064b0, args=0x0000000100e10d20, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:127:12
    frame #23: 0x0000000100261eac python3.9`call_function(tstate=0x0000000100906640, pp_stack=0x00007ffeefbf7680, oparg=2, kwnames=0x0000000000000000) at ceval.c:5072:13
    frame #24: 0x000000010025c86b python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100e10b90, throwflag=0) at ceval.c:3487:23
    frame #25: 0x00000001000b329f python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100e10b90, throwflag=0) at pycore_ceval.h:40:12
    frame #26: 0x00000001000b0bc9 python3.9`function_code_fastcall(tstate=0x0000000100906640, co=0x0000000100a23c70, args=0x0000000100e3d720, nargs=2, globals=0x00000001007c2b90) at call.c:330:24
    frame #27: 0x00000001000b05d8 python3.9`_PyFunction_Vectorcall(func=0x0000000100a580f0, stack=0x0000000100e3d710, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:367:20
    frame #28: 0x000000010026540b python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a580f0, args=0x0000000100e3d710, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #29: 0x0000000100261d6a python3.9`PyObject_Vectorcall(callable=0x0000000100a580f0, args=0x0000000100e3d710, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:127:12
    frame #30: 0x0000000100261eac python3.9`call_function(tstate=0x0000000100906640, pp_stack=0x00007ffeefbf8ab0, oparg=2, kwnames=0x0000000000000000) at ceval.c:5072:13
    frame #31: 0x000000010025c8ed python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100e3d590, throwflag=0) at ceval.c:3504:23
    frame #32: 0x00000001000b329f python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100e3d590, throwflag=0) at pycore_ceval.h:40:12
    frame #33: 0x00000001000b0bc9 python3.9`function_code_fastcall(tstate=0x0000000100906640, co=0x00000001007d4860, args=0x0000000100e623d8, nargs=1, globals=0x0000000100a00950) at call.c:330:24
    frame #34: 0x00000001000b05d8 python3.9`_PyFunction_Vectorcall(func=0x0000000100a06f50, stack=0x0000000100e623d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at call.c:367:20
    frame #35: 0x000000010026540b python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a06f50, args=0x0000000100e623d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #36: 0x0000000100261d6a python3.9`PyObject_Vectorcall(callable=0x0000000100a06f50, args=0x0000000100e623d0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:127:12
    frame #37: 0x0000000100261eac python3.9`call_function(tstate=0x0000000100906640, pp_stack=0x00007ffeefbf9ec0, oparg=1, kwnames=0x0000000000000000) at ceval.c:5072:13
    frame #38: 0x000000010025cab2 python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100e62250, throwflag=0) at ceval.c:3518:19
    frame #39: 0x00000001000b329f python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100e62250, throwflag=0) at pycore_ceval.h:40:12
    frame #40: 0x00000001000b0bc9 python3.9`function_code_fastcall(tstate=0x0000000100906640, co=0x00000001007d4ba0, args=0x0000000100817e30, nargs=1, globals=0x0000000100a00950) at call.c:330:24
    frame #41: 0x00000001000b05d8 python3.9`_PyFunction_Vectorcall(func=0x0000000100a07230, stack=0x0000000100817e28, nargsf=9223372036854775809, kwnames=0x0000000000000000) at call.c:367:20
    frame #42: 0x000000010026540b python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a07230, args=0x0000000100817e28, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #43: 0x0000000100261d6a python3.9`PyObject_Vectorcall(callable=0x0000000100a07230, args=0x0000000100817e28, nargsf=9223372036854775809, kwnames=0x0000000000000000) at abstract.h:127:12
    frame #44: 0x0000000100261eac python3.9`call_function(tstate=0x0000000100906640, pp_stack=0x00007ffeefbfb2f0, oparg=1, kwnames=0x0000000000000000) at ceval.c:5072:13
    frame #45: 0x000000010025cab2 python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100817c70, throwflag=0) at ceval.c:3518:19
    frame #46: 0x00000001000b329f python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100817c70, throwflag=0) at pycore_ceval.h:40:12
    frame #47: 0x00000001000b0bc9 python3.9`function_code_fastcall(tstate=0x0000000100906640, co=0x00000001007de6c0, args=0x0000000100e625f8, nargs=2, globals=0x0000000100a00950) at call.c:330:24
    frame #48: 0x00000001000b05d8 python3.9`_PyFunction_Vectorcall(func=0x0000000100a085f0, stack=0x0000000100e625e8, nargsf=9223372036854775810, kwnames=0x0000000000000000) at call.c:367:20
    frame #49: 0x000000010026540b python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a085f0, args=0x0000000100e625e8, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #50: 0x0000000100261d6a python3.9`PyObject_Vectorcall(callable=0x0000000100a085f0, args=0x0000000100e625e8, nargsf=9223372036854775810, kwnames=0x0000000000000000) at abstract.h:127:12
    frame #51: 0x0000000100261eac python3.9`call_function(tstate=0x0000000100906640, pp_stack=0x00007ffeefbfc720, oparg=2, kwnames=0x0000000000000000) at ceval.c:5072:13
    frame #52: 0x000000010025cab2 python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100e62450, throwflag=0) at ceval.c:3518:19
    frame #53: 0x00000001000b329f python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100e62450, throwflag=0) at pycore_ceval.h:40:12
    frame #54: 0x00000001000b0bc9 python3.9`function_code_fastcall(tstate=0x0000000100906640, co=0x00000001007de790, args=0x00007ffeefbfd880, nargs=2, globals=0x0000000100a00950) at call.c:330:24
    frame #55: 0x00000001000b05d8 python3.9`_PyFunction_Vectorcall(func=0x0000000100a08690, stack=0x00007ffeefbfd870, nargsf=2, kwnames=0x0000000000000000) at call.c:367:20
    frame #56: 0x00000001000b27eb python3.9`_PyObject_VectorcallTstate(tstate=0x0000000100906640, callable=0x0000000100a08690, args=0x00007ffeefbfd870, nargsf=2, kwnames=0x0000000000000000) at abstract.h:118:11
    frame #57: 0x00000001000b2ce1 python3.9`object_vacall(tstate=0x0000000100906640, base=0x0000000000000000, callable=0x0000000100a08690, vargs=0x00007ffeefbfda60) at call.c:792:14
    frame #58: 0x00000001000b2f80 python3.9`_PyObject_CallMethodIdObjArgs(obj=0x0000000000000000, name=0x000000010047b6a8) at call.c:883:24
    frame #59: 0x00000001002a5a5e python3.9`import_find_and_load(tstate=0x0000000100906640, abs_name=0x0000000100f34860) at import.c:1771:11
    frame #60: 0x00000001002a4bff python3.9`PyImport_ImportModuleLevelObject(name=0x0000000100f34860, globals=0x0000000100acc170, locals=0x0000000100acc170, fromlist=0x000000010046c5e0, level=0) at import.c:1872:15
    frame #61: 0x0000000100260d70 python3.9`import_name(tstate=0x0000000100906640, f=0x0000000100a6d960, name=0x0000000100f34860, fromlist=0x000000010046c5e0, level=0x000000010079c280) at ceval.c:5193:15
    frame #62: 0x0000000100259e12 python3.9`_PyEval_EvalFrameDefault(tstate=0x0000000100906640, f=0x0000000100a6d960, throwflag=0) at ceval.c:3097:19
    frame #63: 0x000000010024d7df python3.9`_PyEval_EvalFrame(tstate=0x0000000100906640, f=0x0000000100a6d960, throwflag=0) at pycore_ceval.h:40:12
    frame #64: 0x0000000100263496 python3.9`_PyEval_EvalCode(tstate=0x0000000100906640, _co=0x0000000100e026c0, globals=0x0000000100acc170, locals=0x0000000100acc170, args=0x0000000000000000, argcount=0, kwnames=0x0000000000000000, kwargs=0x0000000000000000, kwcount=0, kwstep=2, defs=0x0000000000000000, defcount=0, kwdefs=0x0000000000000000, closure=0x0000000000000000, name=0x0000000000000000, qualname=0x0000000000000000) at ceval.c:4327:14
    frame #65: 0x0000000100263f76 python3.9`_PyEval_EvalCodeWithName(_co=0x0000000100e026c0, globals=0x0000000100acc170, locals=0x0000000100acc170, args=0x0000000000000000, argcount=0, kwnames=0x0000000000000000, kwargs=0x0000000000000000, kwcount=0, kwstep=2, defs=0x0000000000000000, defcount=0, kwdefs=0x0000000000000000, closure=0x0000000000000000, name=0x0000000000000000, qualname=0x0000000000000000) at ceval.c:4359:12
    frame #66: 0x000000010024d76f python3.9`PyEval_EvalCodeEx(_co=0x0000000100e026c0, globals=0x0000000100acc170, locals=0x0000000100acc170, args=0x0000000000000000, argcount=0, kws=0x0000000000000000, kwcount=0, defs=0x0000000000000000, defcount=0, kwdefs=0x0000000000000000, closure=0x0000000000000000) at ceval.c:4375:12
    frame #67: 0x000000010024d66a python3.9`PyEval_EvalCode(co=0x0000000100e026c0, globals=0x0000000100acc170, locals=0x0000000100acc170) at ceval.c:826:12
    frame #68: 0x00000001002d15c5 python3.9`run_eval_code_obj(tstate=0x0000000100906640, co=0x0000000100e026c0, globals=0x0000000100acc170, locals=0x0000000100acc170) at pythonrun.c:1219:9
    frame #69: 0x00000001002cf92c python3.9`run_mod(mod=0x00000001038163a8, filename=0x0000000100ac6bd0, globals=0x0000000100acc170, locals=0x0000000100acc170, flags=0x00007ffeefbff3d8, arena=0x0000000100cbaf40) at pythonrun.c:1240:19
    frame #70: 0x00000001002ce4c5 python3.9`PyRun_InteractiveOneObjectEx(fp=0x00007fff806da190, filename=0x0000000100ac6bd0, flags=0x00007ffeefbff3d8) at pythonrun.c:273:9
    frame #71: 0x00000001002cdfab python3.9`PyRun_InteractiveLoopFlags(fp=0x00007fff806da190, filename_str="<stdin>", flags=0x00007ffeefbff3d8) at pythonrun.c:126:15
    frame #72: 0x00000001002cde54 python3.9`PyRun_AnyFileExFlags(fp=0x00007fff806da190, filename="<stdin>", closeit=0, flags=0x00007ffeefbff3d8) at pythonrun.c:85:19
    frame #73: 0x0000000100302493 python3.9`pymain_run_stdin(config=0x00000001018095e0, cf=0x00007ffeefbff3d8) at main.c:512:15
    frame #74: 0x0000000100301727 python3.9`pymain_run_python(exitcode=0x00007ffeefbff42c) at main.c:601:21
    frame #75: 0x00000001003014d8 python3.9`Py_RunMain at main.c:677:5
    frame #76: 0x0000000100301879 python3.9`pymain_main(args=0x00007ffeefbff4a0) at main.c:707:12
    frame #77: 0x00000001003018c7 python3.9`Py_BytesMain(argc=1, argv=0x00007ffeefbff518) at main.c:731:12
    frame #78: 0x0000000100001952 python3.9`main(argc=1, argv=0x00007ffeefbff518) at python.c:15:12
    frame #79: 0x00007fff203fdf3d libdyld.dylib`start + 1
    frame #80: 0x00007fff203fdf3d libdyld.dylib`start + 1

with a custom built 3.8, I got a different error:

>>> import example
Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

is there a build flag I need to turn on for this to work?

so one issue I noticed is that brew builds python into mac framework. whereas pyenv builds the normal static lib.