Debugging cythonized using visual studio 2022

I am trying debug my cythonized python code in a visual studio 2022 ide. My setup file is as follows:

from setuptools import setup, Extension
from Cython.Build import cythonize

python setup.py build_ext --build-lib ./build

extensions = [
Extension(“ticker”, [“ticker.py”], define_macros=[(‘CYTHON_TRACE’, ‘1’)]),
Extension(“ticker_dep”, [“ticker_dep.py”], define_macros=[(‘CYTHON_TRACE’, ‘1’)]),
]

setup(
ext_modules=cythonize(extensions, annotate=True, gdb_debug=True, compiler_directives={‘linetrace’: True})
)

Cython is creating the pyd files for the ticker.py and ticker_dep.py files and also a cython_debug folder with the debugging files for the above pyd. I placed a breakpoint in both these files but the debugger is not landing on the breakpoints even though the code is being executed (I know this because of the print statements in my code).

I also have Enable Native Debugging enabled in visual studio.

What can be wrong about this?

Thanks for any assistance.

There are a few more options you’ll need on Windows. I wrote them up at Debugging Cython with WinDBG | Steve Dower a while back (it says for WinDBG, but VS uses essentially the same debug engine so it’ll help there, too).

Thanks for your quick response Steve. I am now able to step through my code but I am running into problems with an external library which is also built mostly with Cython.

I am in touch with the 3rd party and will post additional questions if not successful.

Thanks.

Hi Steve,

On the off-chance that you may be familiar with this library (Spacy), I’m posting below the stack trace of the error I’m getting when importing

when stepping in the debugged code, I get the following error from the import statement below:

from spacy.tokens import Token

Message=ForwardRef._evaluate() missing 1 required keyword-only argument: ‘recursive_guard’
Source=C:\RTFilingsNLPC\businesslib\build\fragment_search.py
StackTrace:
File “C:\RTFilingsNLPC\businesslib\build\fragment_search.py”, line 7, in (Current frame)
from spacy.tokens import Token
File “C:\RTFilingsNLPC\businesslib\build\sc.py”, line 6, in
from fragment_search import FragmentSearch
File “C:\RTFilingsNLPC\businesslib\build\rtf_bootstrap.py”, line 10, in
from sc import ScParser
TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: ‘recursive_guard’

Running this code as pure python works correctly.

Have you run into a similar problem?

Thanks again.

Never seen this before.

ForwardRef._evaluate() missing 1 required keyword-only argument: ‘recursive_guard’

These are not part of Python or Cython as far as I know, so your best bet will be to find who defines and uses them. It may be something specific to spacy.

They are part of python, specifically typing

However, _evaluate is an internal method that changes regularly. You should figure out who is calling it and why.

1 Like

It looks like this issue TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard' · Issue #9609 · pydantic/pydantic · GitHub. So very little to do with “debugging Cython code”. Or “Core development” which is really a topic for developing Python itself.

1 Like

If it’s part of the standard library, then Core Development is a fine place to discuss it. Especially if it’s broken and affecting users who are doing something other than running a type checker.

Assuming it is that issue then it looks like pydactic may have their own copy of the typing module. It’s possible that I’m wrong though.

Or at least a partial copy of some features of it

I strongly doubt this is the stdlib, I suspect some other library is accessing typing internals (maybe cython itself?) for some reason, and that additionally there is a mismatch in python version and project version. The stacktraces OP posted are so messed up that I am not sure what library it is we are talking about however.

Cython doesn’t access the typing library. It understands a few of the type hints from typing (but that’s just to generate static types - it doesn’t generate any runtime code). Older versions of Cython used to have issues when users tried to inherit from typing types (but that doesn’t look like what’s going on here).

So I don’t think it’s something that Cython itself is doing.

1 Like

Ok, thanks for all your replies. I will follow up with the library manufacturer.

@steve.dower I was able to build and run the python/cython application using your setup configuration and the app correctly stopped at the breakpoint in the python file (rtf_bootstrap.py) but did not break at the breakpoint in the cythonized file (see the application flow below). There was an error in the command window which I have pasted to the bottom of this message. Any assistance would be greatly appreciated.

setup.py:

extensions = [
Extension(“ticker_dep”, [“ticker_dep.py”],
extra_compile_args=[“-Ox”, “-Zi”],
extra_link_args=[“-debug:full”])
]

setup(
ext_modules=cythonize(extensions, emit_linenums=True)
)

build command:

python setup.py build_ext --build-lib ./build

application start up:

python businesslib\build\rtf_bootstrap.py:

from ticker_dep import TickerParserDEP

businesslib\build\ticker_dep.cp312-win_amd64.pyd (cythonized version of businesslib\ticker_dep.py):

debug breakpoint placed in businesslib\ticker_dep.py:

def __init__(self):
    try:
		# breakpoint placed on the line below
        print('top of ticker_dep.init')

get the following error in the command window:

Traceback (most recent call last):
File “C:\Program Files\Python\Lib\runpy.py”, line 198, in _run_module_as_main
return _run_code(code, main_globals, None,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Program Files\Python\Lib\runpy.py”, line 88, in run_code
exec(code, run_globals)
File "c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy_main
.py", line 39, in
cli.main()
File “c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy/…\debugpy\server\cli.py”, line 430, in main
run()
File “c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy/…\debugpy\server\cli.py”, line 284, in run_file
runpy.run_path(target, run_name=“main”)
File “c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py”, line 321, in run_path
return _run_module_code(code, init_globals, run_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py”, line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File “c:\program files\microsoft visual studio\2022\professional\common7\ide\extensions\microsoft\python\core\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py”, line 124, in _run_code
exec(code, run_globals)
File “C:\RTFilingsNLPC\businesslib\build\rtf_bootstrap.py”, line 13, in
from ticker_dep import TickerParserDEP
File “ticker_dep.py”, line 2, in init ticker_dep
TypeError: ‘NoneType’ object is not callable

You’re going to have to share more code than that, I’m afraid. I can’t tell how Python execution got from an import statement to File “ticker_dep.py”, line 2, in init ticker_dep in order to produce the TypeError.

If your code works without the breakpoint, you are probably best to report this to the Visual Studio team (probably at GitHub - microsoft/debugpy: An implementation of the Debug Adapter Protocol for Python). If your code doesn’t work even outside of the debugger, you might need the Cython team (at GitHub - cython/cython: The most widely used Python to C compiler).

The line from ticker_dep import TickerParserDEP automatically loads the ticker_dep module and executes the init function (this is what happens in a pure Python project). Below are the lines that get executed before the TickerParserDEP import:

import datetime
import win32evtlog
import inspect
import logging
import ctypes
from typing import List, Set, Dict, Tuple, cast, Optional
from util import EventLog, TraceLog
from rtf_server import RtfServer
from sc_13d import Sc13dParser
from ticker_dep import TickerParserDEP

they all execute correctly but the ticker_dep import lines fails with the error.

It’s not “the ticker_dep import” that’s failing, or you’d have an ImportError.

Can you post either the full contents of ticker_dep.py or a link to it? I can’t guess what you’re doing from vague descriptions.

Alternatively, try getting support from one of the projects that you are using, as I listed above.

@steve.dower Thanks for your help. Unfortunately I am unable to share the code of ticker_dep.py as it is proprietary. I will open an incident with Microsoft tech support.

Hi Steve,

I’ve been trying to debug a very simple Cythonized project in Visual Studio 2022 Professional (zip file containing the scripts and the project file is attached), and the IDE correctly stops at the breakpoints but I am unable to inspect any variables.

In this case, I’ve placed breakpoints in the init method of the TickerParserDEP class in ticker_dep.py. My startup script is rtf_bootstrap.py which is placed in the build folder which is a subfolder of the businesslib folder. structure is as follows:

C:\CythonDebug\businesslib\build

The businesslib folder contains all the .py scripts and the build folder contains the Cythonized binaries. I’ve included the setup.py script which builds the binaries from the 2 script files (ticker.py and ticker_dep.py). You need to run the command python setup.py build_ext --build-lib ./build from the businesslib folder.

As rtf_bootstrap.py is in the build folder, Visual Studio correctly debugs the binaries. Everything runs fine but when I try to inspect the i or j variable in the TickerParserDEP.init method, I get the following error in Visual Studio:

Document

Name | Value | Type |

  • | - | - | - |

j | identifier “j” is undefined | |

Do you have experience debugging Cythonized binaries using Visual Studio? Or do you prefer some other tools for debugging? My project has Enable native code debugging option checked and the Search Paths property set to …\businesslib in the Debug tab of the project properties. Also, it is using Python 3.12 as the Interpreter (see the General tab).

Any information would be much appreciated.

Ronny

setup.py:

from setuptools import setup, Extension
from Cython.Build import cythonize

python setup.py build_ext --build-lib ./build

extensions = [
Extension(“ticker”, [“ticker.py”],
# extra_compile_args=[“-Ox”, “-Zi”],
extra_compile_args=[“-Od”, “-Zi”],
extra_link_args=[“-debug:full”]),
Extension(“ticker_dep”, [“ticker_dep.py”],
# extra_compile_args=[“-Ox”, “-Zi”],
extra_compile_args=[“-Od”, “-Zi”],
extra_link_args=[“-debug:full”])
]

setup(
ext_modules=cythonize(extensions, emit_linenums=True)
)

ticker_dep.py:

from ticker import Ticker

import cython

class TickerParserDEP(Ticker):
# @cython.locals(j=cython.int, i=cython.int)
def init(self):
print(‘top of ticker_dep.init’)
j = 99
Ticker.init(self)
i = 100
print('bottom of ticker_dep.init: ’ + str(i) + ‘/’ + str(j))

def start(self) -> None:
    print('top of ticker_dep.start')

ticker.py:

import abc

class Ticker(metaclass=abc.ABCMeta):
def init(self):
print(‘top of ticker.init’)

rtf_bootstrap.py:

from ticker_dep import TickerParserDEP

class RtfBootstrap:
def init(self, print_to_console: bool=False, print_stats: bool=False):
print(‘top of bootstrap.init’)
tickerDep = TickerParserDEP()
tickerDep.start()

def start(self) -> None:
    print('top of rtf_bootstrap.start')

__rtf_bootstrap = RtfBootstrap(print_to_console=False, print_stats=False)
__rtf_bootstrap.start()

Cython replaces all the “nice” names with “mangled” ones. You’ll need to browse the Locals window for the actual name of your variable - it should be fairly obvious, they normally just add a prefix.

Unfortunately, there’s no nice way to map variable names in the debugger. I looked into it some years ago, and even asked the debugger team if they’d add it, but they didn’t go for the idea.