I’m using ProcessPoolExecutor, and in Windows that means ‘spawn context’, which caused this exception in my code:
File "stringsource", line 2, in zmq.backend.cython.socket.Socket.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
From the stack I could see it was caused here when process_obj is dumped, makes sense.
What wasn’t in the stack or the exception message is the source location of the instantiation of the offending Socket instance. In my case I was eventually able to find it (buried in a log handler), but only after a lot of digging around.
I know it’s probably a long shot, but is there any way (in principal) Python could know at run time where the offending instance was instantiated (and so give that to the user)?
I think Python can now what socket failed, but your error message contains “cython” twice, which the error may have occured in natively compiled Cython code (if I understand cython correctly). But you might be correct.
CPython (and likely other Pythons) deliberate try to keep the basic
form of an object as small as possible, and this would increase that
size
the place an object is instantiated is often not useful without the
call stack leading to it, and that is quite a lot of data
Consider an object which itself allocates a bunch of other objects, and
one of these other objects causes your trouble. They’ll each be recorded
as being made in the outer main object’s __init__, which tells you
nearly nothing.
As a counter point, I’ve recently built a general purpose task thing for
work and it records the call stack leading to the Task creation. In a
database entry. That has a space and time cost to it, but making these
tasks is comparitively infrequent and I want to know after the fact what
lead to their creation. But I would not do it for every object, just
strategic objects. Even then I’m only recording the call stack code
locations, not eg the function parameters etc.
IMO, the easy thing (just the code point where made) is nearly useless
and still too expensive, and the useful thing is far too expensive in
both space and time.
But there can be value in maybe making some utility class or mixin to
annotate some specific strategic classes with this info for debugging
purposes.