Interception of methods from Python modules compiled into the interpreter

Python Help
,
1

Some background: Generally speaking, one could easily intercept function calls & returns by using the sys module and setting sys.setprofile(custom_trace_function) whereby in this custom_trace_function we could execute anything depending whether the event was a new scope entered, return from scope, a bytecode is about to be executed, a C function is about to be entered, a C function is about to be returned form, etc…

My Question: I am not able to intercept some methods whose implementation is compiled into the interpreter; so how can I intercept them dynamically?

Multithreading with Python:
The Python threading module, a higher-level module for _thread, is compiled into the interpreter itself. It includes some synchronization methods such as acquire() and release() order to administer a mutex that lets us avoid having race conditions on some global variable, just like this:

lock=threading.Lock()
lock.acquire()
# do something
lock.release()

The problem with these methods is that their implementation exists in the C code, and thus I cannot intercept them with the sys module if let’s say I administer the lock with context management:

lock = threading.Lock()
with lock:
    # do something
    pass

The lock is acquired at the beginning of the with statement as if I called lock.acquire() during the __enter__() function. And at the end of the with scope, it’s as if lock.release() is called by __exit()__ function. So this is already taken care of by the module.

Usually, I would see the dunder functions being called [at least by intercepting the C functions being called] when the with statement is executed on some object. However in the case when the object is a threading.Lock() primitive instance,that is compiled into the interpreter, I cannot see that. Thus I have no way of knowing when a lock is acquired and when it is released during runtime.

I’m aware that I could do static code analysis; however, I need this to be detected at runtime.
So is there a way to intercept such calls in Python [and know when they occur] and possibly manipulate the lock object being called on?

2

Hi Mohamad,

Your question seems to be very advanced. You might get more responses if
you start by giving an example of how you would intercept a function
that works the way you want, and a second example of trying to intercept
something that doesn’t work as you expect.

1 Like
3

A more concrete example is given here:

import threading
import sys

def tracer(frame, event, arg):
    if event == 'call':
        print(f"Entering: {frame.f_code.co_name}")
        return tracer
    elif event == 'c_call':
        print(f"C --- Entering: {arg.__name__}")
    elif event == 'return':
        print(f"Returning: {arg!r}")
    elif event == 'c_return':
        print(f"C --- Returning from: {arg.__name__}")

lock = threading.Lock()

sys.setprofile(tracer)
threading.setprofile(tracer)

print("Begin...")

# In the following case, we would see the C functions being called: acquire and release
lock.acquire()
print("Do some routine with explicit locking")
lock.release()

print("In the middle...")

# In the following case, we cannot see that any locking is happening, despite it working
with lock:
    print("Do some routine with locking using context management")

print("End...")
4

Hi, thanks for your feedback; just added an example in the comments