Pickle exception handling could state object path

Ideas
1

Feature or enhancement

Proposal:

Consider the case that you get some exception during unpickling. This could be anything, in maybe your custom object __setstate__ or whatever else. For example, we got this crash:

...
  File "/u/dorian.koch/setups/2024-10-11--denoising-lm/recipe/returnn/returnn/util/multi_proc_non_daemonic_spawn.py", line 156, in NonDaemonicSpawnProcess._reconstruct_with_pre_init_func
    line: reconstruct_func, reconstruct_args, reconstruct_state = pickle.load(buffer)
    locals:
      reconstruct_func = <not found>
      reconstruct_args = <not found>
      reconstruct_state = <not found>
      pickle = <global> <module 'pickle' from '/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/pickle.py'>
      pickle.load = <global> <built-in function load>
      buffer = <local> <_io.BytesIO object at 0x74bbaa61e610>
  File "/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/synchronize.py", line 110, in SemLock.__setstate__
    line: self._semlock = _multiprocessing.SemLock._rebuild(*state)
    locals:
      self = <local> <Lock(owner=unknown)>
      self._semlock = <local> !AttributeError: 'Lock' object has no attribute '_semlock'
      _multiprocessing = <global> <module '_multiprocessing' from '/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/lib-dynload/_multiprocessing.cpython-311-x86_64-linux-gnu.so'>
      _multiprocessing.SemLock = <global> <class '_multiprocessing.SemLock'>
      _multiprocessing.SemLock._rebuild = <global> <built-in method _rebuild of type object at 0x74bbb60322c0>
      state = <local> (132092164476928, 1, 1, '/mp-2wkdacg_')
FileNotFoundError: [Errno 2] No such file or directory

(The exception traceback was printed using better_exchook for an extended traceback which adds the involved local variables and some more.I thought the additional info might help to better understand where this exception comes from.)
So, SemLock.__setstate__ fails here for some reason. Maybe some race condition. But when I saw this crash, my first thought was, where actually do we have a SemLock inside the pickled object?

So, this is what I would like: In case of an exception during unpickling, it can show me the object path during the construction which lead to this object. (In case there are multiple references to the object, just show me the first.)

I’m not sure exactly how this should be done. It means some overhead. For every single object that pickle creates, we would need to store the creating parent object + name/index/whatever. So maybe this is a feature which should be optional. It would be fine for me if I run unpickling first without, and if I get some exception, I run unpickling again with this debug flag enabled. Maybe it’s also fine if this is only in the pure Python implementation.

Maybe I can already do sth like this by checking the local self in the stack frame where the exception occured and then using gc.get_referrers to get back to the root?

I also don’t care how it would show this information, or how I can get this information on the object path, or how it would represent this information, as long as I have any way of getting this information. I leave this open for suggestions. E.g.:

  • My first idea was that it could simply extend the exception.msg, but maybe that’s too hacky. Sth like:
try:
  _load(...)
except Exception as exc:
  exc.msg += f"\n\nGot exception during unpickling of object {self._recent_obj_info()}"
  raise
  • Or it could wrap any exception thrown during unpickling, and do sth like this:
try:
  _load(...)
except Exception as exc:
  raise UnpicklingException(f"Got exception during unpickling of object {self._recent_obj_info()}") from exc

(But this would change the current behavior, so this would definitely need some flag to enable this behavior.)

  • Or it could expose such Unpickler.recent_obj_info, and then I can get that info in my own exception handling code and do what I want with it.

(Note, I already posted this to GitHub #130621, as I thought this is a minor feature, but it was suggested to post this here as well.)

1 Like
2

The correct feature to use for this is Exception notes: Built-in Exceptions — Python 3.13.2 documentation

This seems pretty reasonable, if it’s implementable without a huge performance cost I think it should be implemented.

1 Like
3

I haven’t looked through the pickle code but I would have thought that if an exception occurs while pickle is trying to create self then any parent/root object that would have had a reference to self won’t exist yet.

4

Did you try the future Python 3.14?

5

Ah, right. Well, they do exist, because it calls sth like object.__new__(cls) first, so that it can already refer to this, but then it creates all its state, and only then it assigns this (via __setstate__ or so). So, when the exception occurs, those parent objects exist but they don’t refer to the child object yet, so this idea with gc.get_referrers would not work.

6

No. Why? Is this already implemented there?

7

As I said I haven’t looked through the pickle code but it is more complicated than this because there is __getstate__ but also __getnewargs__ and __reduce__ and more. At best the object that you want to get is not fully initialised if it does exist so there is a good change that it does not function in a usable way e.g. you can expect that str(obj) would not work.

8

Yes, that as well. But that is all only about the idea of using gc.get_referrers. So yes, that would not work. This means, to get what I want, we would need to extend the unpickling code. The unpickler knows basically everything that I want to know, specifically the trace of why it wants to construct the most recent object where the exception occured. In most standard cases (e.g. some nested dict/list/object/etc), it then can also show me key/index/attrib chain, and even for cases where there is no clear key/index/attrib, it still always knows the parent which triggered the creation.

9

Yes, it is already implemented. Please test any 3.14 alpha version, your feedback is important for us.

Similar notes are also added for JSON serialization.

3 Likes
10

I just tried now. No, this is not implemented yet. Currently, it catches exceptions during pickling, and then adds notes to the exception in a very similar manner that I described. However, I want basically the same feature but for unpickling.

I wrote a small demo here:

class MyClass:
    def __init__(self, value):
        self.value = value

    def __getstate__(self):
        if args.exc_in_getstate:
            raise Exception("getstate")
        return self.value

    def __setstate__(self, state):
        if args.exc_in_setstate:
            raise Exception("setstate")
        self.value = state


obj = MyClass(42)
d = {"a": {"b": {"c": obj}, "d": obj}}

s = pickle.dumps(d)
d2 = pickle.loads(s)

I tested with Python: 3.14.0a5 (tags/v3.14.0a5:3ae9101482, Mar 1 2025, 21:50:29) [Clang 15.0.0 (clang-1500.3.9.4)].

With exception in __getstate__, this gives:

Traceback (most recent call last):
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 36, in <module>
    s = pickle.dumps(d)
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 24, in __getstate__
    raise Exception("getstate")
Exception: getstate
when serializing dict item 'c'
when serializing dict item 'b'
when serializing dict item 'a'

With exception in __setstate__, this gives:

Traceback (most recent call last):
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 37, in <module>
    d2 = pickle.loads(s)
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 29, in __setstate__
    raise Exception("setstate")
Exception: setstate
11

Pickling is top-down recursive. At any point you know the path from the top object. But unpickling is bottom-up state machine – it starts from primitives on the lowest level and build objects at upper levels to use them to build objects in even upper levels. You don’t know the path to the current object, because the upper objects have not yet be built. You can extract some information for few standard containers (dict, list, tuple, set), but you need to duplicate the size of the unpickler code to support walking up without building an object. And this would not work for self-referencing collections. And this would not work for non-trivial types, the case when you need detailed error report the most.

So I think that this is not practical. If you need to debug the pickle file, use pickletools.dis().

12

Oh right, I didn’t really thought about how the pickle bytecode works. But wouldn’t it still be possible to analyze the remaining bytecode to see where the current constructed object would be used for? I don’t care if it really ends up as an attrib of some object or if is just used as temporary state object for some __setstate__ call. Both would be equally helpful for me to know.

Or phrased in a different way: via pickletools.dis(), I can also see exactly what I want to know, i.e. where exactly the object would end up. So the information is there, and can be shown.

Instead of pickletools.dis(), maybe more readable is sth like the pickle debug tool from Sage: A tool for inspecting Python pickles: explain_pickle

13

I can also see exactly where the object would end up in the output of pickletools.dis() because you are human and you tried relatively small and simple pickles. But this is not so simply for computer:

  • The size of the unpickler code will be increased at least twofold. Every operation should be implemented for the normal mode and for the “analyzing” mode.
  • It needs to analyze all pickle data to the end. If exception occurs at the beginning of a multimegabyte file, all these megabytes should be read and parsed before reporting an error. Most users expect fast responce on error.
  • You can get report like:
    when unpicling item 3 of tuple
when unpickling item "abc" of dict
when unpickling some value of dict
when unpickling argument 2 of function foo.bar
when unpickling argument 2 of some callable
    Some dict keys and some callables can be dynamic, i.e. they cannot be just read from the pickle file, but are results of other calls encoded in the pickle file. Error report for such cases will be not very helpful.
  • If the original data contains recursive references, the pickled data can contain the POP opcodes. I.e. the chain leads to the object which is dropped. You need to backtrack and check all intermediate results which can be used to create the final object.
  • If the pickle data is broken, all this is futile.

You can try to implement this in your project. Just clone the standard pickle module and add your features. If you find that module useful in your other work, publish it on PyPI. If it is used by other people, and is not too complex, we can consider adding such feature in the stdlib.

1 Like
14

Yes that makes sense.

Note, to be pragmatic: Maybe when an exception occured, it would be helpful that I can at least get the information about the current byte offset in the pickled data, so that I can see in pickletools.dis() at what place the exception occured.

Well, I could get that information maybe by checking file.tell() afterwards?

Or another idea: Maybe in case of an exception, I could use explain_pickle and execute that generated code. Then, when I get the same exception again, I would also know exactly where it occurred, and the surrounding code gives me the relevant information I want to know.

15

Adding a note that includes the offset or the position in the file may work (although there may be small technical issues).