Sixth element of tuple from __reduce__(), inconsistency between pickle and copy

Core Development
1

Both the pickle and copy modules of the standard library make use of a class’s __reduce__() method for customizing their pickle/copy process. They seem to have a consistent view of the first 5 elements of the returned tuple:
(func, args, state, listiter, dictiter) but the 6th element seems different. For pickle it’s state_setter , a callable with signature state_setter(obj, state)->None , but for copy it’s deepcopy with signature deepcopy(arg: T, memo) -> T .

This seems to be unintentional, since the pickle documentation states:

As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the __reduce__() special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects. 4

It seems like in order to make a class definition for __reduce__() returning all 6 elements, then the __reduce__() would have to do something very awkward like examining its call stack in order to determine if it is being called in pickle or copy context in order to return an appropriate callable? (Naively providing the same callable in both contexts would cause errors for one or the other).

Is the situation that the 6th element is unsafe to return for a class that supports both pickle and copy (in which case the documentation of pickle and copy should be updated with this caveat) or is there some way to fix the APIs eg maybe providing an in_pickle_context_q() helper to package up the fragile stack inspection?

2

This should probably just be a bug in bugs.python.org.

Do you have a small example file demonstrating this?

3

This defines two classes making use of a __reduce__() returning a 6 element tuple. One class Pickleable can be duplicated via pickling, but not deepcopied. The converse is true for the Copyable class.

Other than the 6th element of the tuple returned from __reduce__() the classes are identical.

from copy import deepcopy
from pickle import dumps, loads
from datetime import datetime


def state_setter(obj, state):
    print(f"setting state with: {state}")
    obj._state = state["_state"]


class Pickleable:
    def __init__(self):
        self._state = datetime.now()

    def __str__(self):
        return str(self._state)

    def __reduce__(self):
        return (
            object.__new__,
            (type(self),),
            {"_state": self._state},
            None,
            None,
            state_setter,
        )


def deep_copier(arg, memo=None):
    print(f"deepcopying with: arg={arg}, memo={memo}")
    return deepcopy(arg, memo)


class Copyable:
    def __init__(self):
        self._state = datetime.now()

    def __str__(self):
        return str(self._state)

    def __reduce__(self):
        return (
            object.__new__,
            (type(self),),
            {"_state": self._state},
            None,
            None,
            deep_copier,
        )


p1 = Pickleable()
c1 = Copyable()

# these work:
p1_p = loads(dumps(p1))
c1_c = deepcopy(c1)

# these fail:
p1_c = deepcopy(p1)
c1_p = loads(dumps(c1))
4

I strongly recommend that you create a new issue on bugs.python.org and write up your findings there.

FWIW it looks like these are independent developments:

I’m guessing the folks doing the latter weren’t aware that deepcopy already uses the 6th arg. Sorting this out will be painful.

5

Thanks, I did that as Issue 46336: Sixth element of tuple from __reduce__(), inconsistency between pickle and copy - Python tracker