Exposing the source location of a global instance causing an exception

I’m using ProcessPoolExecutor, and in Windows that means ‘spawn context’, which caused this exception in my code:

  File "stringsource", line 2, in zmq.backend.cython.socket.Socket.__reduce_cython__     
  TypeError: no default __reduce__ due to non-trivial __cinit__

From the stack I could see it was caused here when process_obj is dumped, makes sense.

What wasn’t in the stack or the exception message is the source location of the instantiation of the offending Socket instance. In my case I was eventually able to find it (buried in a log handler), but only after a lot of digging around.

I know it’s probably a long shot, but is there any way (in principal) Python could know at run time where the offending instance was instantiated (and so give that to the user)?

I think Python can now what socket failed, but your error message contains “cython” twice, which the error may have occured in natively compiled Cython code (if I understand cython correctly). But you might be correct.

In principle, yes in theory.

However:

  • CPython (and likely other Pythons) deliberate try to keep the basic
    form of an object as small as possible, and this would increase that
    size
  • the place an object is instantiated is often not useful without the
    call stack leading to it, and that is quite a lot of data

Consider an object which itself allocates a bunch of other objects, and
one of these other objects causes your trouble. They’ll each be recorded
as being made in the outer main object’s __init__, which tells you
nearly nothing.

As a counter point, I’ve recently built a general purpose task thing for
work and it records the call stack leading to the Task creation. In a
database entry. That has a space and time cost to it, but making these
tasks is comparitively infrequent and I want to know after the fact what
lead to their creation. But I would not do it for every object, just
strategic objects. Even then I’m only recording the call stack code
locations, not eg the function parameters etc.

IMO, the easy thing (just the code point where made) is nearly useless
and still too expensive, and the useful thing is far too expensive in
both space and time.

But there can be value in maybe making some utility class or mixin to
annotate some specific strategic classes with this info for debugging
purposes.

Cheers,
Cameron Simpson cs@cskk.id.au

3 Likes