queue.Empty() and queue.Full() not available in multiprocessing namespace

Python Help
1

Hello all,

I’m a bit confused about how namespaces work when importing multiple modules. Till now my understanding has been that the interpreter maintains a namespace for each module, and when a callable object invoked from __main__ references a global variable, it is the namespace of the module in which the callable is defined that is searched for the global, NOT that of __main__ (unless of course it is actually defined in __main__).

I thought it is for this reason that callables designed to be imported by another module cannot rely on global variables defined in that module; they may rely on constants defined in their own module, but failing that any objects they require must be passed in to them.

I still think that’s all true but I can’t work out how it relates to this quote from the multiprocessing documentation.

multiprocessing uses the usual queue.Empty and queue.Full exceptions to signal a timeout. They are not available in the multiprocessing namespace so you need to import them from queue.

According to my current understanding, a method of multiprocessing.Queue that tried to raise queue.Empty or queue.Full would need them to be defined in, or imported by multiprocessing itself.

I hope that makes sense, and if it does I’d appreciate some guidance!

2

Yes, multiprocessing must somewhere import the exceptions from queue in order to raise them. But multiprocessing.Empty and multiprocessing.Full are intentionally name errors.

You should import them directly from queue, and not try to find them in one of the multiprocessing submodules, if you want to catch them in an exception clause.

3

Thanks. Just to clarify, is it that methods of multiprocessing.Queue can raise the exceptions, and to do this they must be imported by multiprocessing. But for them to be caught in __main__, it also needs to import them?

4

Yes. multiprocessing.queues imports Empty and Full and defines classes Queue, JoinableQueue(Queue), and SimpleQueue. The first two raise exceptions in various places. All three have to be initialized with an OS-dependent context. Multiprocessing.context imports these classes (but not the exceptions, which is does not need) wraps them, and ‘combines’ them with the OS default context. Multiprocessing.init imports context and adds the bound wrappers to its globals. The structural result:

>>> import multiprocessing
>>> multiprocessing.Queue
<bound method BaseContext.Queue of <multiprocessing.context.DefaultContext object at 0x000002610976A660>>

Sort of baroque, but the behavioral result is that when you call the bound wrapper, it automatically initializes the class with the appropriate default context.

Any user code, including the main script, must independently import the exceptions to catch them.

1 Like