AttributeError behaviour in multiprocessing - missing name attribute

Python Help
1

Hi there,

I am facing a problem during exception handling in multiprocessing.
A raised AttributeError looks different in multiprocessing environment.

Given the following minimal example:

import operator
from dataclasses import dataclass
import multiprocessing


@dataclass
class ClassWithAttributes:
    name: str

def do_something_with_attribute(instance: ClassWithAttributes):

    attributename_name:str = "name"
    attrgetter_name = operator.attrgetter(attributename_name)

    attributename_no_attribute:str = "no_attribute"
    attrgetter_no_attribute = operator.attrgetter(attributename_no_attribute)

    value_name = attrgetter_name(instance)
    print(value_name)

    # raises an attribute error
    value_no_attribute = attrgetter_no_attribute(instance)
    print(value_no_attribute)

instance1 = ClassWithAttributes("instance1_name")
instances_list = [instance1]

print("-----------------------")
print("do with for loop")
for instance in instances_list:
    try:
        do_something_with_attribute(instance)
    except AttributeError as e:
        print("Error in For Loop")
        print(e.name) # here name is set

print("-----------------------")
print("do with multiprosessing")
with multiprocessing.Pool() as pool:
    try:
        entities_result = pool.map(do_something_with_attribute, instances_list)

    except AttributeError as e:
        print("Error in Multiprocessing")
        print(e.name) # here name is not set

Result:

-----------------------
do with for loop
instance1_name
Error in For Loop
no_attribute
-----------------------
do with multiprosessing
instance1_name
Error in Multiprocessing
None

The object of the AttributeError in for loop looks different than in multiprocessing.
The name of the error is not set within multiprocessing.

I am not sure, if this is an expected behaviour?

Thanks and best,
Christoph

2

This looks like a bug in how the AttributeError is pickled:

C:\Users\csm10495\Desktop>python
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> a = AttributeError("test text", name="test name", obj="test obj")
>>> a.name
'test name'
>>> pickle.loads(pickle.dumps(a)).name
>>> pickle.loads(pickle.dumps(a))
AttributeError('test text')

multiprocessing communicates back and forth via pickle under the hood, so since its losing attributes (like .name) its losing it in your example too.

Filed: Pickling is losing some fields on exceptions · Issue #103333 · python/cpython · GitHub

2 Likes
3

Thanks.
I understand, that this issue is a pickle issue and not a multiprocessing issue.

Also thanks for your effort solving the issue in the python core.