# Pickle exception handling could state object path

**URL:** <https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395>\
**Category:** Ideas\
**Created:** [February 27, 2025, 3:50pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395 "2025-02-27T15:50:35Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [February 27, 2025, 3:50pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/1 "2025-02-27T15:50:36Z")

</div>

# Feature or enhancement

### Proposal:

Consider the case that you get some exception during unpickling. This could be anything, in maybe your custom object ` __setstate__ ` or whatever else. For example, we got this crash:

```python
...
  File "/u/dorian.koch/setups/2024-10-11--denoising-lm/recipe/returnn/returnn/util/multi_proc_non_daemonic_spawn.py", line 156, in NonDaemonicSpawnProcess._reconstruct_with_pre_init_func
    line: reconstruct_func, reconstruct_args, reconstruct_state = pickle.load(buffer)
    locals:
      reconstruct_func = <not found>
      reconstruct_args = <not found>
      reconstruct_state = <not found>
      pickle = <global> <module 'pickle' from '/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/pickle.py'>
      pickle.load = <global> <built-in function load>
      buffer = <local> <_io.BytesIO object at 0x74bbaa61e610>
  File "/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/multiprocessing/synchronize.py", line 110, in SemLock. __setstate__
    line: self._semlock = _multiprocessing.SemLock._rebuild(*state)
    locals:
      self = <local> <Lock(owner=unknown)>
      self._semlock = <local> !AttributeError: 'Lock' object has no attribute '_semlock'
      _multiprocessing = <global> <module '_multiprocessing' from '/work/tools/users/zeyer/linuxbrew/opt/python@3.11/lib/python3.11/lib-dynload/_multiprocessing.cpython-311-x86_64-linux-gnu.so'>
      _multiprocessing.SemLock = <global> <class '_multiprocessing.SemLock'>
      _multiprocessing.SemLock._rebuild = <global> <built-in method _rebuild of type object at 0x74bbb60322c0>
      state = <local> (132092164476928, 1, 1, '/mp-2wkdacg_')
FileNotFoundError: [Errno 2] No such file or directory

```

(The exception traceback was printed using [better\_exchook](https://github.com/albertz/py_better_exchook) for an extended traceback which adds the involved local variables and some more.I thought the additional info might help to better understand where this exception comes from.)  
So, `SemLock. __setstate__ ` fails here for some reason. Maybe some race condition. But when I saw this crash, my first thought was, _where_ actually do we have a `SemLock` inside the pickled object?

So, this is what I would like: In case of an exception during unpickling, it can show me the object path during the construction which lead to this object. (In case there are multiple references to the object, just show me the first.)

I’m not sure exactly how this should be done. It means some overhead. For every single object that pickle creates, we would need to store the creating parent object + name/index/whatever. So maybe this is a feature which should be optional. It would be fine for me if I run unpickling first without, and if I get some exception, I run unpickling again with this debug flag enabled. Maybe it’s also fine if this is only in the pure Python implementation.

Maybe I can already do sth like this by checking the local `self` in the stack frame where the exception occured and then using `gc.get_referrers` to get back to the root?

I also don’t care how it would show this information, or how I can get this information on the object path, or how it would represent this information, as long as I have _any_ way of getting this information. I leave this open for suggestions. E.g.:

- My first idea was that it could simply extend the `exception.msg`, but maybe that’s too hacky. Sth like:

```python
try:
  _load(...)
except Exception as exc:
  exc.msg += f"\n\nGot exception during unpickling of object {self._recent_obj_info()}"
  raise

```

- Or it could wrap any exception thrown during unpickling, and do sth like this:

```python
try:
  _load(...)
except Exception as exc:
  raise UnpicklingException(f"Got exception during unpickling of object {self._recent_obj_info()}") from exc

```

(But this would change the current behavior, so this would definitely need some flag to enable this behavior.)

- Or it could expose such `Unpickler.recent_obj_info`, and then I can get that info in my own exception handling code and do what I want with it.

(Note, I already posted this to GitHub [#130621](https://github.com/python/cpython/issues/130621), as I thought this is a minor feature, but it was suggested to post this here as well.)

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [February 27, 2025, 4:06pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/2 "2025-02-27T16:06:29Z")

</div>

The correct feature to use for this is Exception notes: [Built-in Exceptions — Python 3.13.2 documentation](https://docs.python.org/3/library/exceptions.html#BaseException.add_note)

This seems pretty reasonable, if it’s implementable without a huge performance cost I think it should be implemented.

---

<div class="post-metadata">

**Author:** ![oscarbenjamin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/oscarbenjamin/32/1209_2.png) [@oscarbenjamin](https://discuss.python.org/u/oscarbenjamin)\
**Post date:** [February 27, 2025, 4:34pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/3 "2025-02-27T16:34:27Z")

</div>

> [@albertzeyer](#):
>
> Maybe I can already do sth like this by checking the local `self` in the stack frame where the exception occured and then using `gc.get_referrers` to get back to the root?

I haven’t looked through the pickle code but I would have thought that if an exception occurs while pickle is trying to create `self` then any parent/root object that would have had a reference to `self` won’t exist yet.

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [February 27, 2025, 4:49pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/4 "2025-02-27T16:49:06Z")

</div>

Did you try the future Python 3.14?

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [February 27, 2025, 8:39pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/5 "2025-02-27T20:39:45Z")

</div>

Ah, right. Well, they do exist, because it calls sth like `object. __new__ (cls)` first, so that it can already refer to this, but then it creates all its state, and only then it assigns this (via ` __setstate__ ` or so). So, when the exception occurs, those parent objects exist but they don’t refer to the child object yet, so this idea with `gc.get_referrers` would not work.

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [February 27, 2025, 8:40pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/6 "2025-02-27T20:40:32Z")

</div>

No. Why? Is this already implemented there?

---

<div class="post-metadata">

**Author:** ![oscarbenjamin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/oscarbenjamin/32/1209_2.png) [@oscarbenjamin](https://discuss.python.org/u/oscarbenjamin)\
**Post date:** [February 27, 2025, 11:39pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/7 "2025-02-27T23:39:53Z")

</div>

> [@albertzeyer](#):
>
> it calls sth like `object. __new__ (cls)` first, so that it can already refer to this, but then it creates all its state, and only then it assigns this (via ` __setstate__ ` or so).

As I said I haven’t looked through the pickle code but it is more complicated than this because there is ` __getstate__ ` but also ` __getnewargs__ ` and ` __reduce__ ` and more. At best the object that you want to get is not fully initialised if it does exist so there is a good change that it does not function in a usable way e.g. you can expect that `str(obj)` would not work.

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [February 28, 2025, 12:22am UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/8 "2025-02-28T00:22:33Z")

</div>

Yes, that as well. But that is all only about the idea of using `gc.get_referrers`. So yes, that would not work. This means, to get what I want, we would need to extend the unpickling code. The unpickler knows basically everything that I want to know, specifically the trace of why it wants to construct the most recent object where the exception occured. In most standard cases (e.g. some nested dict/list/object/etc), it then can also show me key/index/attrib chain, and even for cases where there is no clear key/index/attrib, it still always knows the parent which triggered the creation.

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [February 28, 2025, 7:28am UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/9 "2025-02-28T07:28:46Z")

</div>

Yes, it is already implemented. Please test any 3.14 alpha version, your feedback is important for us.

Similar notes are also added for JSON serialization.

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [March 1, 2025, 9:03pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/10 "2025-03-01T21:03:28Z")

</div>

I just tried now. No, this is not implemented yet. Currently, it catches exceptions during _pickling_, and then adds notes to the exception in a very similar manner that I described. However, I want basically the same feature but for _unpickling_.

I wrote a small demo [here](https://github.com/albertz/playground/blob/master/py-pickle-test-exception-gh130621.py):

```python
class MyClass:
    def __init__ (self, value):
        self.value = value

    def __getstate__ (self):
        if args.exc_in_getstate:
            raise Exception("getstate")
        return self.value

    def __setstate__ (self, state):
        if args.exc_in_setstate:
            raise Exception("setstate")
        self.value = state

obj = MyClass(42)
d = {"a": {"b": {"c": obj}, "d": obj}}

s = pickle.dumps(d)
d2 = pickle.loads(s)

```

I tested with `Python: 3.14.0a5 (tags/v3.14.0a5:3ae9101482, Mar 1 2025, 21:50:29) [Clang 15.0.0 (clang-1500.3.9.4)]`.

With exception in ` __getstate__ `, this gives:

```python
Traceback (most recent call last):
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 36, in <module>
    s = pickle.dumps(d)
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 24, in __getstate__
    raise Exception("getstate")
Exception: getstate
when serializing dict item 'c'
when serializing dict item 'b'
when serializing dict item 'a'

```

With exception in ` __setstate__ `, this gives:

```python
Traceback (most recent call last):
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 37, in <module>
    d2 = pickle.loads(s)
  File "/Users/az/Programmierung/playground/py-pickle-test-exception-gh130621.py", line 29, in __setstate__
    raise Exception("setstate")
Exception: setstate

```

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [March 2, 2025, 10:35am UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/11 "2025-03-02T10:35:34Z")

</div>

Pickling is top-down recursive. At any point you know the path from the top object. But unpickling is bottom-up state machine – it starts from primitives on the lowest level and build objects at upper levels to use them to build objects in even upper levels. You don’t know the path to the current object, because the upper objects have not yet be built. You can extract some information for few standard containers (dict, list, tuple, set), but you need to duplicate the size of the unpickler code to support walking up without building an object. And this would not work for self-referencing collections. And this would not work for non-trivial types, the case when you need detailed error report the most.

So I think that this is not practical. If you need to debug the pickle file, use `pickletools.dis()`.

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [March 3, 2025, 8:34pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/12 "2025-03-03T20:34:38Z")

</div>

Oh right, I didn’t really thought about how the pickle bytecode works. But wouldn’t it still be possible to analyze the remaining bytecode to see where the current constructed object would be used for? I don’t care if it really ends up as an attrib of some object or if is just used as temporary state object for some ` __setstate__ ` call. Both would be equally helpful for me to know.

Or phrased in a different way: via `pickletools.dis()`, I can also see exactly what I want to know, i.e. where exactly the object would end up. So the information is there, and can be shown.

Instead of `pickletools.dis()`, maybe more readable is sth like the pickle debug tool from Sage: [A tool for inspecting Python pickles: explain\_pickle](https://doc.sagemath.org/html/en/reference/misc/sage/misc/explain_pickle.html#sage.misc.explain_pickle.explain_pickle)

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [March 3, 2025, 9:40pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/13 "2025-03-03T21:40:02Z")

</div>

I can also see exactly where the object would end up in the output of `pickletools.dis()` because you are human and you tried relatively small and simple pickles. But this is not so simply for computer:

- The size of the unpickler code will be increased at least twofold. Every operation should be implemented for the normal mode and for the “analyzing” mode.
- It needs to analyze all pickle data to the end. If exception occurs at the beginning of a multimegabyte file, all these megabytes should be read and parsed before reporting an error. Most users expect fast responce on error.
- You can get report like:

```python
when unpicling item 3 of tuple
when unpickling item "abc" of dict
when unpickling some value of dict
when unpickling argument 2 of function foo.bar
when unpickling argument 2 of some callable

```

Some dict keys and some callables can be dynamic, i.e. they cannot be just read from the pickle file, but are results of other calls encoded in the pickle file. Error report for such cases will be not very helpful.
- If the original data contains recursive references, the pickled data can contain the POP opcodes. I.e. the chain leads to the object which is dropped. You need to backtrack and check all intermediate results which can be used to create the final object.
- If the pickle data is broken, all this is futile.

You can try to implement this in your project. Just clone the standard `pickle` module and add your features. If you find that module useful in your other work, publish it on PyPI. If it is used by other people, and is not too complex, we can consider adding such feature in the stdlib.

---

<div class="post-metadata">

**Author:** ![albertzeyer](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/albertzeyer/32/26285_2.png) [@albertzeyer](https://discuss.python.org/u/albertzeyer)\
**Post date:** [March 3, 2025, 10:22pm UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/14 "2025-03-03T22:22:18Z")

</div>

Yes that makes sense.

Note, to be pragmatic: Maybe when an exception occured, it would be helpful that I can at least get the information about the current byte offset in the pickled data, so that I can see in `pickletools.dis()` at what place the exception occured.

Well, I could get that information maybe by checking `file.tell()` afterwards?

Or another idea: Maybe in case of an exception, I could use [`explain_pickle`](https://doc.sagemath.org/html/en/reference/misc/sage/misc/explain_pickle.html#sage.misc.explain_pickle.explain_pickle) and execute that generated code. Then, when I get the same exception again, I would also know exactly where it occurred, and the surrounding code gives me the relevant information I want to know.

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [March 4, 2025, 9:01am UTC](https://discuss.python.org/t/pickle-exception-handling-could-state-object-path/82395/15 "2025-03-04T09:01:27Z")

</div>

Adding a note that includes the offset or the position in the file may work (although there may be small technical issues).
