Interception of methods from Python modules compiled into the interpreter

Some background: Generally speaking, one could easily intercept function calls & returns by using the sys module and setting sys.setprofile(custom_trace_function) whereby in this custom_trace_function we could execute anything depending whether the event was a new scope entered, return from scope, a bytecode is about to be executed, a C function is about to be entered, a C function is about to be returned form, etc…

My Question: I am not able to intercept some methods whose implementation is compiled into the interpreter; so how can I intercept them dynamically?

Multithreading with Python:
The Python threading module, a higher-level module for _thread, is compiled into the interpreter itself. It includes some synchronization methods such as acquire() and release() order to administer a mutex that lets us avoid having race conditions on some global variable, just like this:

lock=threading.Lock()
lock.acquire()
# do something
lock.release()

The problem with these methods is that their implementation exists in the C code, and thus I cannot intercept them with the sys module if let’s say I administer the lock with context management:

lock = threading.Lock()
with lock:
    # do something
    pass

The lock is acquired at the beginning of the with statement as if I called lock.acquire() during the __enter__() function. And at the end of the with scope, it’s as if lock.release() is called by __exit()__ function. So this is already taken care of by the module.

Usually, I would see the dunder functions being called [at least by intercepting the C functions being called] when the with statement is executed on some object. However in the case when the object is a threading.Lock() primitive instance,that is compiled into the interpreter, I cannot see that. Thus I have no way of knowing when a lock is acquired and when it is released during runtime.

I’m aware that I could do static code analysis; however, I need this to be detected at runtime.
So is there a way to intercept such calls in Python [and know when they occur] and possibly manipulate the lock object being called on?

Hi Mohamad,

Your question seems to be very advanced. You might get more responses if
you start by giving an example of how you would intercept a function
that works the way you want, and a second example of trying to intercept
something that doesn’t work as you expect.

1 Like

A more concrete example is given here:

import threading
import sys

def tracer(frame, event, arg):
    if event == 'call':
        print(f"Entering: {frame.f_code.co_name}")
        return tracer
    elif event == 'c_call':
        print(f"C --- Entering: {arg.__name__}")
    elif event == 'return':
        print(f"Returning: {arg!r}")
    elif event == 'c_return':
        print(f"C --- Returning from: {arg.__name__}")

lock = threading.Lock()

sys.setprofile(tracer)
threading.setprofile(tracer)

print("Begin...")

# In the following case, we would see the C functions being called: acquire and release
lock.acquire()
print("Do some routine with explicit locking")
lock.release()

print("In the middle...")

# In the following case, we cannot see that any locking is happening, despite it working
with lock:
    print("Do some routine with locking using context management")

print("End...")

Hi, thanks for your feedback; just added an example in the comments