How to solve unresolved external symbol errors

Python Help
,
1

Hi folks. When building Python libraries that use C++, how do you understand the cause of and solve unresolved external errors for Python symbols?

  binding.obj : error LNK2001: unresolved external symbol __imp___PyArg_Parse_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyDict_SetDefault
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBool_FromLong
  binding.obj : error LNK2001: unresolved external symbol __imp___PyObject_CallFunction_SizeT

and many more along similar lines. While I know there’ll be a cause I’d like to understand how to approach this general issue to make this a more useful thread for future googlers.

(I know this looks like a dup of this topic, but I was asked to make a new thread.)

I am using Python 3.11 on Windows ARM64, and am installing tree-sitter 0.21. I have the MSVC 14 build tools installed.

Twist: I used to have Python 3.12 on this machine, but had to go back to 3.11 because tree-sitter 0.21 requires <3.12. It did not show any installed modules left over from 3.12 via pip list , nor do I see any references to 3.12 in the below compile messages, but I cleared to be sure via pip cache purge . So think this is a ‘clean’ version of Python 3.11 and there’s nothing from another version to confuse the linker.

Creating library build\temp.win32-cpython-311\Release\tree_sitter\_binding.cp311-win_arm64.lib and object build\temp.win32-cpython-311\Release\tree_sitter\_binding.cp311-win_arm64.exp
  binding.obj : error LNK2001: unresolved external symbol __imp___PyArg_Parse_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyDict_SetDefault
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBool_FromLong
  binding.obj : error LNK2001: unresolved external symbol __imp___PyObject_CallFunction_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyImport_ImportModule
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBytes_AsString
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBytes_FromObject
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_RuntimeError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyLong_FromLong
  binding.obj : error LNK2001: unresolved external symbol __imp__PyErr_SetNone
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_MemoryError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBytes_FromString
  binding.obj : error LNK2001: unresolved external symbol __imp___Py_NoneStruct
  binding.obj : error LNK2001: unresolved external symbol __imp__PyMemoryView_FromObject
  binding.obj : error LNK2001: unresolved external symbol __imp__PyList_Size
  binding.obj : error LNK2001: unresolved external symbol __imp__PyErr_SetString
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_ValueError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyLong_FromUnsignedLong
  binding.obj : error LNK2001: unresolved external symbol __imp___PyArg_ParseTupleAndKeywords_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyErr_Format
  binding.obj : error LNK2001: unresolved external symbol __imp__PyTuple_Type
  binding.obj : error LNK2001: unresolved external symbol __imp___Py_FalseStruct
  binding.obj : error LNK2001: unresolved external symbol __imp___Py_Dealloc
  binding.obj : error LNK2001: unresolved external symbol __imp__PyTuple_GetItem
  binding.obj : error LNK2001: unresolved external symbol __imp__PyModule_AddObjectRef
  binding.obj : error LNK2001: unresolved external symbol __imp___PyArg_ParseTuple_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyUnicode_FromFormat
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_GetBuffer
  binding.obj : error LNK2001: unresolved external symbol __imp__PyList_New
  binding.obj : error LNK2001: unresolved external symbol __imp__PyModule_Create2
  binding.obj : error LNK2001: unresolved external symbol __imp__PySlice_New
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_SyntaxError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_GetAttrString
  binding.obj : error LNK2001: unresolved external symbol __imp___PyObject_CallMethod_SizeT
  binding.obj : error LNK2001: unresolved external symbol __imp__PyErr_Clear
  binding.obj : error LNK2001: unresolved external symbol __imp__PyList_Append
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_RichCompareBool
  binding.obj : error LNK2001: unresolved external symbol __imp__PyTuple_Size
  binding.obj : error LNK2001: unresolved external symbol __imp__PyType_GetModuleState
  binding.obj : error LNK2001: unresolved external symbol __imp___Py_HashPointer
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBytes_Size
  binding.obj : error LNK2001: unresolved external symbol __imp__PyDict_SetItem
  binding.obj : error LNK2001: unresolved external symbol __imp__PyDict_New
  binding.obj : error LNK2001: unresolved external symbol __imp__PyList_GetItem
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_IsInstance
  binding.obj : error LNK2001: unresolved external symbol __imp__PyLong_FromVoidPtr
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_StopIteration
  binding.obj : error LNK2001: unresolved external symbol __imp__PyLong_AsVoidPtr
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_GetItem
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_TypeError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyType_FromModuleAndSpec
  binding.obj : error LNK2001: unresolved external symbol __imp__PyExc_NameError
  binding.obj : error LNK2001: unresolved external symbol __imp__PyTuple_Pack
  binding.obj : error LNK2001: unresolved external symbol __imp__PyCallable_Check
  binding.obj : error LNK2001: unresolved external symbol __imp__PyModule_GetState
  binding.obj : error LNK2001: unresolved external symbol __imp__PyList_SetItem
  binding.obj : error LNK2001: unresolved external symbol __imp__PyUnicode_FromString
  binding.obj : error LNK2001: unresolved external symbol __imp__PyLong_FromSize_t
  binding.obj : error LNK2001: unresolved external symbol __imp__PyBuffer_Release
  binding.obj : error LNK2001: unresolved external symbol __imp__PyObject_Call
  binding.obj : error LNK2001: unresolved external symbol __imp__PyUnicode_FromStringAndSize
  build\lib.win32-cpython-311\tree_sitter\_binding.cp311-win_arm64.pyd : fatal error LNK1120: 61 unresolved externals
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.41.34120\\bin\\HostX86\\x86\\link.exe' failed with exit code 1120
2

What version of python was tree_sitter compiled against?
You are using python 3.11 to run the module.

3

As far as I can tell, tree-sitter is compiling as part of installation, and all references seem to be 3.11 (eg temp.win32-cpython-311\Release\tree_sitter\_binding.cp311-win_arm64.exp). I have only Python 3.11 installed.

4

I would assume it is not compiled as there are binary wheels on pipy for tree-sitter.

Maybe ask the authors what is wrong?

5

Tree-sitter doesn’t include a wheel for Windows ARM64. I filed that issue, and they may add the platform, but meanwhile I’d like to solve why it isn’t linking – and learn how to understand these sort of issues for Python libraries in general.

6

In short the python library experts symbols for the C API.
Only the exported symbols can be used by the extension module.

Usually when you see errors looking up symbols you would assume that the module was linked against the API of a different version of python.

7

Seems that that tree-sitter build issue was closed 2 hours ago - so, I’d simply try again.

To learn to understand these kind of issues yourself, the best way, imo, is to checkout the source code of such a library and try to build it yourself, from scratch. You’ll quickly notice whether your compiler install is ok or not. Assuming that that is ok, inspect any command line traces (similar to the traces that a build in pip will give if you activate all pip debug options). Then, when you know there is probably an issue with running on a different platform, make sure the compiler and linker options (as specified in the error trace, or as specified in the setup.py or elsewhere) are all suitable for that platform.

Even if you know almost nothing about the particular compiler you’re using, you can make a quick stab at that by - first making a calculated guess (after trying to look up the options) and tinkering with those options (turn them on/off, check any compiler/linker paths they refer too) and running only the specific command lines.

8

I know this is an old thread, but I came across it when trying to resolve the same issue myself when building a Python extension on Windows on ARM so I feel it’s worth adding the solution I found to a similar kind of issue.

I believe the cause of the issue is when a version of the build tools that targets x86 is mistakenly used to build a Python extension for ARM64. For example, in a failing case I see as output:

creating C:\Users\gmlocal\numbadev\numba\build\lib.win32-cpython-314\numba
"C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.44.35207\bin\HostARM64\x86\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\gmlocal\numbadev\numbaenv\libs /LIBPATH:C:\Users\gmlocal\AppData\Local\Python\pythoncore-3.14-arm64\libs /LIBPATH:C:\Users\gmlocal\AppData\Local\Python\pythoncore-3.14-arm64 /LIBPATH:C:\Users\gmlocal\numbadev\numbaenv\PCbuild\win32 "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.44.35207\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.44.35207\ATLMFC\lib\x86" "/LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.44.35207\lib\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x86" /EXPORT:PyInit__dynfunc build\temp.win32-cpython-314\Release\numba\_dynfuncmod.obj /OUT:build\lib.win32-cpython-314\numba\_dynfunc.cp314-win_arm64.pyd /IMPLIB:build\temp.win32-cpython-314\Release\numba\_dynfunc.cp314-win_arm64.lib
   Creating library build\temp.win32-cpython-314\Release\numba\_dynfunc.cp314-win_arm64.lib and object build\temp.win32-cpython-314\Release\numba\_dynfunc.cp314-win_arm64.exp
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_Malloc
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyType_GenericNew
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_SelfIter
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyErr_Occurred
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyExc_RuntimeError
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_GC_UnTrack
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__Py_BuildValue
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyDict_SetItemString
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyErr_SetString
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyDict_Type
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp___Py_Dealloc
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_Free
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyModule_AddObject
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_ClearWeakRefs
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyUnicode_AsUTF8
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyList_New
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyModule_Create2
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyType_Ready
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyObject_GetAttrString
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyDict_New
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyLong_FromVoidPtr
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyType_GenericAlloc
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyCMethod_New
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyErr_NoMemory
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__Py_IsFinalizing
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyLong_AsVoidPtr
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyArg_ParseTupleAndKeywords
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyTuple_Pack
_dynfuncmod.obj : error LNK2001: unresolved external symbol __imp__PyArg_ParseTuple
build\lib.win32-cpython-314\numba\_dynfunc.cp314-win_arm64.pyd : fatal error LNK1120: 29 unresolved externals
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.44.35207\\bin\\HostARM64\\x86\\link.exe' failed with exit code 1120

(note the “x86” in many paths used for the tools)

I’m not familiar with the nuance of how different build tools are selected, so it’s hard for me to give full guidance - however, for me it was sufficient to open an ARM64 Native Tools Command Prompt", activate my Python venv in it, and build the extension there.