How to efficiently know a PyObject liveness

Python Help
1

Hi, I am doing some PyObject tracing profiling within CPython, and I have an additional customized field in PyObject struct called hotness. I need to frequently access this hotness field for the traced PyObjects, inside a separate thread spawned by PyThread_start_new_thread.

The problem I’m facing is, my tracing thread has no knowledge whenever a PyObject is freed (no matter which memory context in obmalloc.c). So it get segfaults when trying to access op->hotness for those logically “freed” objects, before the same pointer is assigned to a new one.

Specifically, in _Py_dealloc():

(*dealloc)(op);

is basically populating the value pointed by op, to the lastfree pointer address. After this statement, and before the same op is assigned to a new object, I basically cannot do any read/write to PyObject field (e…g., reading ob_refcnt field), since it won’t even make:

Segmentation fault (core dumped)
make: *** [Makefile:1329: Python/frozen_modules/zipimport.h] Error 139

Thus, as a walk around, I maintain a set that bookkeep “live” objects, and erase from the set inside _Py_dealloc(). However, this adds up runtime overhead. Is there a more elegant and thinner way to identify the PyObject liveness? Thanks.

2

(Even if you are working on the C internals of a custom build, to my understanding the Core Development section is intended for discussion of development of the actual main CPython project. So I relabelled the thread.)

Why not have the tracing thread explicitly hold a reference to whichever objects it’s tracing (i.e. in the PY_INCREF sense), and release the reference, say, when it notices that the traced object doesn’t have any other references? Because what you’re currently doing with the set sounds a lot to me like what a garbage collector does already…

3

Reference cycles won’t get cleared up any more, unless the OP’s code is fully reproducing a garbage collector?

4

What if you set your own dealloc function on each object you are tracing?
Save the original dealloc first.
When your dealloc runs update your book keeping and then call the original dealloc function.