Building debug version of extensions on Windows without debug binaries

[Andrej730 (Andrej)]

@steve.dower

@Andrej730

Are there plans to include debug binaries (pythond, python311_d.dll) to PythonTest or to some separate tag?

No, there are too many external dependencies needed (specifically, you have to have installed all the dependencies you’d need to build it yourself, so we suggest just building yourself - then you’ll also have sources accessible for debugging CPython, which is the only reason you’d want debug binaries).

Branching off from Python Installation Manager conversation - I thought debug binaries were required to build debug version of Python extension, since they are used for linking automatically if _DEBUG is defined and it’s defined by default in Debug configurations (at least in cmake).

pyconfig.h code:

But turned out if I just temporarily undefine _DEBUG before including Python header, then it will work fine - debug build will compile and link without issues and I will be able to use debug build with the normal Python.

#if defined(_DEBUG)
# undef _DEBUG
# include <Python.h>
# define _DEBUG 1
#else
# include <Python.h>
#endif

Is it indended way to do so? It seems there could be some kind of flag USE_NON_DEBUG_PYTHON that can avoid using python312_d.lib for linking even if _DEBUG is set and would proceed to python312.lib.

Then it would be possible to streamline this flag to the build systems used to build Python extensions, as indeed it seems debug binaries are needed only if you want to debug Python itself and not if you need debug version of your extension.

E.g. cmake’s FindPython module would be able to set this flag for Debug configs automatically when users are adding their libraries with Python_add_library, as in 99% cases they probably just want Debug version of their extension, not to debug Python. And then they also may never meet _d suffix appearing in their extensions, since it’s unlikely that they actually need it.

I’ve also found that swig already defined such option called SWIG_PYTHON_INTERPRETER_NO_DEBUG (code) to allow to compile debug version of extensions that would work with Release Python.

Maybe some flag like this can be added to the CPython itself?

—

Searching more on the topic, I’ve found a similar issue with someone using undef / define hack (github) - it was back in Python 3.4 and at the time, it seems, just undefining _DEBUG could have led to some other issue later on and python_d indeed was required and this could be the reason why debug binaries started to be included since 3.5.

But later in 3.8 (release notes) debug and release builds ABI became compatible and, maybe this is what allowed building undef _DEBUG more safely and use otherwise debug extension with release Python.

[steve.dower]

Typically when you want a debuggable build, you want four things:

• optimizations disabled
• debug info generated
• internal C runtime library assertions
• own code assertions

Only the last of these is connected to the preprocessor (I don’t remember if it’s connected to the _DEBUG variable or the NDEBUG variable, but it’s one of those - I think the latter). The rest are controlled by other command line options.

Optimizations can be disabled by passing /Od instead of any other /O option.

Debug info can be enabled with /Zi to the compiler and some linker option (either /DEBUG or /PDB, again, I don’t recall off the top of my head).

The C runtime selection comes from building with /MTd//MDd (debug) vs. /MT//MD. For static linking, these options become more complicated, but generally for CPython extensions you’ll want /MD[d] (dynamic linking), even if you then do tricks to statically link the C++ libraries.

I’m pretty sure that specifing /M?d is what implicitly defines _DEBUG, so that the C headers can tell whether they’re targeting the debug C runtime or the release one. Since the difference between a debug CPython build and a release one on Windows is the choice of C runtime, we match it. But we also mostly avoid bleeding C runtime objects out of CPython, and avoid using most of them directly within it, so you can link to two different C runtime libraries successfully in a lot of cases. If you don’t use any file IO, locale-aware, or thread-aware functions in your extension, then mixing runtimes likely won’t hit any issues.

But in general, it should be sufficient to use a “release” build with optimizations disabled and debug info enabled. Possibly undefining NDEBUG will let your own assert() statements break, but you won’t get assertions from inside CPython or the C runtime^[1].

The main reason you need a CPython debug build is to get these assertions, or to resolve issues due to shared state between your own C runtime and CPython’s (in the case where you’ve forcibly linked a debug C runtime build to a release CPython build, such as with #undef _DEBUG).

---

1. Most places that would assert in the C runtime via CPython are caught and turned into a Python exception. So you’re disabling a popup dialog, but I don’t think any asserted failures will pass silently unless you let them. ↩︎

[Andrej730 (Andrej)]

@steve.dower

I’m pretty sure that specifing /M?d is what implicitly defines _DEBUG, so that the C headers can tell whether they’re targeting the debug C runtime or the release one

Yeah, was just testing this, /M?d is indeed the option that’s producing it.

@steve.dower

But in general, it should be sufficient to use a “release” build with optimizations disabled and debug info enabled. Possibly undefining NDEBUG will let your own assert() statements break, but you won’t get assertions from inside CPython or the C runtime .

Would it make sense to standardize the use of this with something like below?
As that’s what the majority of users probably would expect when making Debug build for an extension and it would allow to toggle this behaviour by defining one flag.

// pyconfig.h
/* Allows debug builds of extensions to link against release Python safely in most cases. */
#if defined(_DEBUG) and defined(Py_WIN_NO_DEBUG_LIBS)
#   define Py_WIN_NO_DEBUG_LIBS_RESTORE
#   undef _DEBUG
#endif


// End of pyconfig.h
#if defined(Py_WIN_NO_DEBUG_LIBS_RESTORE)
#   define _DEBUG 1
#   undef Py_WIN_NO_DEBUG_LIBS_RESTORE
#endif

[steve.dower]

@Andrej730

Would it make sense to standardize the use of this with something like below?

No, but there might be a way to change the logic around the pragma comment(lib, ...) parts so that you either can always reference the release lib, or that you must explicitly choose to reference the debug lib. Hard to say which is the more disruptive change.

There should already be a flag to skip that logic entirely and make you reference it explicitly (though not in your screenshot, but I think it landed after 3.12), which is also an option.

We shouldn’t be undefining built-in compiler definitions anywhere.

[CEXT-Dan (Dan)]

I do this with my package, since AutoCAD’s SDK is also release only
It’s a pseudo debug, optimizations are turned off, string pooling is off, generate debug info is on

[Andrej730 (Andrej)]

@steve.dower

There should already be a flag to skip that logic entirely and make you reference it explicitly (though not in your screenshot, but I think it landed after 3.12), which is also an option.

Yeah, there is Py_NO_LINK_LIB (code), which actually is enough to build with _DEBUG and Release Python libraries. But if there’s Py_INCREF or Py_DECREF used in the code, then they would refer to symbols from debug libs. Those symbols are used if Py_REF_DEBUG defined, which comes from Py_DEBUG, which comes from _DEBUG.

extension.cpp.obj : error LNK2019: unresolved external symbol __imp__Py_NegativeRefcount referenced in function Py_DECREF
extension.cpp.obj : error LNK2019: unresolved external symbol __imp__Py_INCREF_IncRefTotal referenced in function Py_INCREF
extension.cpp.obj : error LNK2019: unresolved external symbol __imp__Py_DECREF_DecRefTotal referenced in function Py_DECREF

Missing symbols is actually the same issue users would have meet if they compile with Py_DEBUG on Unix, but on Unix Py_DEBUG has to be defined explicitly and doesn’t come from general _DEBUG. Any ideas why on Windows it is? Or it’s mostly now historical? So maybe introducing Py_WIN_NO_PY_DEBUG that with combination Py_NO_LINK_LIBwould resolve this?

Found when it was added to get more context, it’s been awhile ago

@CEXT-Dan

I do this with my package, since AutoCAD’s SDK is also release only
It’s a pseudo debug, optimizations are turned off, string pooling is off, generate debug info is on

You mean the build doesn’t use debug C runtime, so _DEBUG is not defined and you don’t run into this issue in debug buiilds?

Btw, is there a need for # comment( lib , “python312.lib” ) in the code below for some case? Since Pyhon.h would include the same statement automatically from pyconfig.h. Or it’s just to make it more explicit?

PyRxCore/StdAfx.h

[CEXT-Dan (Dan)]

@Andrej730

comment( lib , “python312.lib” )

comment( lib , “python312.lib” )
Ouch, didn’t know that was in pyconfig.h, bad habit

you mean the build doesn’t use debug C runtime, so _DEBUG is not defined and you don’t run into this issue in debug buiilds?

Right, my debug configuration is a pseudo debug, _DEBUG is not defined, and /MD, no issues, except I can’t step outside my module, unless there’s release .PDB I’ve seen some setups where there’s three build configurations, Release, Debug, FullDebug, where FullDebug is MDd and _DEBUG is defined

[steve.dower]

@Andrej730

But if there’s Py_INCREF or Py_DECREF used in the code, then they would refer to symbols from debug libs. Those symbols are used if Py_REF_DEBUG defined, which comes from Py_DEBUG, which comes from _DEBUG.

Okay, so there’s one more check needed to make it work. This was likely part of Victor’s change to make non-debug builds ABI-compatible with debug runtimes (but apparently not the other way around).

@Andrej730

on Unix Py_DEBUG has to be defined explicitly and doesn’t come from general _DEBUG. Any ideas why on Windows it is?

I think just because building debug on Windows is a more integrated part of the normal workflow, while it isn’t on Unix?

Being able to somehow undefine Py_REF_DEBUG seems like the right path here. Unless there’s a wild scattering of non-obvious features that would justify a broad scenario-specific flag, we’d rather stick to feature-specific flags for special cases like these.

The lack of documentation on building extensions, which would include these options, is known. There’s an open issue somewhere.

[Andrej730 (Andrej)]

@steve.dower

Being able to somehow undefine Py_REF_DEBUG seems like the right path here. Unless there’s a wild scattering of non-obvious features that would justify a broad scenario-specific flag, we’d rather stick to feature-specific flags for special cases like these.

Yeah, makes sense. Tested now with just disabling Py_REF_DEBUG and there seems to be no other issues. Was mainly thinking about flat to disable Py_DEBUGentirely to avoid weird state you mentioned when Python is in debug mode, but not entirely - I thought Py_REF_DEBUG is more like a flag to document features related to debug mode and references, not an independant thing. And option to disable Py_DEBUG entirely would make debug extension would allow to achieve consistent behaviour between Windows and Unix platforms.

Anyways, existing Py_NO_LINK_LIBflag and just disabling Py_REF_DEBUG resolves the main problem with the need of undef _DEBUG or using debug libraries, glad we narrowed it down.