# Custom unpickler and pickler for the shelve module

**URL:** <https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178>\
**Category:** Ideas\
**Created:** [November 17, 2022, 8:19pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178 "2022-11-17T20:19:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![furkanonder](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/furkanonder/32/27074_2.png) [@furkanonder](https://discuss.python.org/u/furkanonder)\
**Post date:** [November 17, 2022, 8:19pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/1 "2022-11-17T20:19:25Z")

</div>

Hi,

As you know we can’t pickle the lambda functions in the standard pickle module. So, for that reason the shelve library also can’t do that.

What if we can define the custom pickler and unpickler for the shelve module?

We couldn’t do this before;

```python
import shelve
with shelve.open("test_file") as sh:
    squared = lambda x: x ** 2
    sh['test_key'] = squared

```

Now, we can easily do it; ([Just a basic modification for support custom pickler and unpickler for shelve module · GitHub](https://gist.github.com/furkanonder/436d1b23eede54650107924e747b5d77))

```python
import dill
import shelve

with shelve.open("test_file_2", pickler=dill.Pickler, unpickler=dill.Unpickler) as sh:
    squared = lambda x: x ** 2
    sh['test_key'] = squared

```

I could easily solve this problem I had while using the Shelve module. I believe that shelve module will be a more useful module by adding support for custom unpickler and pickler.

Kindest regards,  
Furkan Onder

---

<div class="post-metadata">

**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\
**Post date:** [November 17, 2022, 9:19pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/2 "2022-11-17T21:19:31Z")

</div>

What format were you proposing we use for pickling functions?

---

<div class="post-metadata">

**Author:** ![psarka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/psarka/32/9277_2.png) [@psarka](https://discuss.python.org/u/psarka)\
**Post date:** [November 18, 2022, 2:02pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/3 "2022-11-18T14:02:59Z")

</div>

The way I read the proposal is _not_ for pickle to support lambdas, but rather for shelf to support other picklers, such as dill or cloudpickle, which have solved function pickling one way or another.

---

<div class="post-metadata">

**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\
**Post date:** [November 18, 2022, 3:10pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/4 "2022-11-18T15:10:44Z")

</div>

You’re right and I should have seen that.

The feature sounds sensible then, the question is do enough people need this to warrant adding it?

Shelve is very old and perhaps should be deprecated.

---

<div class="post-metadata">

**Author:** ![isidentical](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/isidentical/32/4201_2.png) [@isidentical](https://discuss.python.org/u/isidentical)\
**Post date:** [November 20, 2022, 3:57pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/5 "2022-11-20T15:57:09Z")

</div>

Especially for distributed computing communities, they seem to have been leveraging pluggable pickling endpoints (changing the default `pickle` with any module that provides a `loads(data: bytes) -> object` and `dumps(obj: object) -> bytes` interface; like `cloudpickle` or `dill`) quite a lot. This even includes multiprocessing where you can change the intermediary format (using the `ConnectionWrapper`). For shelve, I agree that it is a sensible change that makes it much more flexible without any maintainance cost.

> [@guido](#):
>
> The feature sounds sensible then, the question is do enough people need this to warrant adding it?

I’ve searched a bit about this and saw a couple of stackoverflow posts ([[1]](https://stackoverflow.com/questions/52927236/how-to-use-dill-library-for-object-serialization-with-shelve-library)) that suggest monkeypatching shelve directly (which is unnecessarily convoluted than simply offering an option) and there are more than a handful examples in real world projects: [`shelve.Pickler = ` on GitHub search](https://github.com/search?q=shelve.Pickler%20%3D%20&type=code)

I’d be interested in having it in shelve as a new argument.

---

<div class="post-metadata">

**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\
**Post date:** [November 20, 2022, 5:28pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/6 "2022-11-20T17:28:49Z")

</div>

Go for it then!

---

<div class="post-metadata">

**Author:** ![furkanonder](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/furkanonder/32/27074_2.png) [@furkanonder](https://discuss.python.org/u/furkanonder)\
**Post date:** [December 1, 2022, 6:17pm UTC](https://discuss.python.org/t/custom-unpickler-and-pickler-for-the-shelve-module/21178/7 "2022-12-01T18:17:16Z")

</div>

I created a [pull request](https://github.com/python/cpython/pull/99632) for this improvement. @isidentical @guido