Custom unpickler and pickler for the shelve module

Ideas
1

Hi,

As you know we can’t pickle the lambda functions in the standard pickle module. So, for that reason the shelve library also can’t do that.

What if we can define the custom pickler and unpickler for the shelve module?

We couldn’t do this before;

import shelve
with shelve.open("test_file") as sh:
    squared = lambda x: x ** 2
    sh['test_key'] = squared

Now, we can easily do it; (Just a basic modification for support custom pickler and unpickler for shelve module · GitHub)

import dill
import shelve

with shelve.open("test_file_2", pickler=dill.Pickler, unpickler=dill.Unpickler) as sh:
    squared = lambda x: x ** 2
    sh['test_key'] = squared

I could easily solve this problem I had while using the Shelve module. I believe that shelve module will be a more useful module by adding support for custom unpickler and pickler.

Kindest regards,
Furkan Onder

2

What format were you proposing we use for pickling functions?

3

The way I read the proposal is not for pickle to support lambdas, but rather for shelf to support other picklers, such as dill or cloudpickle, which have solved function pickling one way or another.

4

You’re right and I should have seen that.

The feature sounds sensible then, the question is do enough people need this to warrant adding it?

Shelve is very old and perhaps should be deprecated.

1 Like
5

Especially for distributed computing communities, they seem to have been leveraging pluggable pickling endpoints (changing the default pickle with any module that provides a loads(data: bytes) -> object and dumps(obj: object) -> bytes interface; like cloudpickle or dill) quite a lot. This even includes multiprocessing where you can change the intermediary format (using the ConnectionWrapper). For shelve, I agree that it is a sensible change that makes it much more flexible without any maintainance cost.

I’ve searched a bit about this and saw a couple of stackoverflow posts ([1]) that suggest monkeypatching shelve directly (which is unnecessarily convoluted than simply offering an option) and there are more than a handful examples in real world projects: shelve.Pickler = on GitHub search

I’d be interested in having it in shelve as a new argument.

2 Likes
6

Go for it then!

1 Like
7

I created a pull request for this improvement. @isidentical @guido