Differences between 3.8 and 3.9 in importing module?

Python Help
1

I encoundered similar error to https://bugs.python.org/issue41567.

Not only multiprocessing.Pool, I saw various ImportError in multiprocessing in 3.9.
For example, the sample code below raises the following error(on macOS 10.15.5).

from concurrent.futures import ThreadPoolExecutor

def f3():
    pass

def f2(arg):
    queue: Any = multiprocessing.Queue()
    p = multiprocessing.Process(target=f3)
    p.start()
    p.join()

def f1():
    with ThreadPoolExecutor(max_workers=20) as e:
        ret = e.map(f2, range(10))
        for x in ret:
            print(x)

if __name__ == '__main__':
    f1()

Traceback (most recent call last):
  File "/Users/ishimoto/importerror_multiprocessing.py", line 20, in <module>
    f1()
  File "/Users/ishimoto/importerror_multiprocessing.py", line 16, in f1
    for x in ret:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 600, in result_iterator
    yield fs.pop().result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/ishimoto/importerror_multiprocessing.py", line 8, in f2
    queue: Any = multiprocessing.Queue()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 102, in Queue
    from .queues import Queue
ImportError: cannot import name 'Queue' from partially initialized module 'multiprocessing.queues' (most likely due to a circular import) (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py)

The multiprocessing has a lot of local imports, so multiprocessing is not thread-safe.

But I wonder why these codes start failing in Python 3.9. They worked fine in 3.7 and 3.8 for years and I can not find the change that caused these differences between 3.8 and 3.9 in multithreading module.

2

It looks the issue is not in the multiprocessing module itself, but Python’s import machinery somewhat changed.

Following files reproduce similar error.

main.py

from concurrent.futures import ThreadPoolExecutor
import threading

def f2(arg):
    print("enter f2", threading.get_ident())
    try:
        from sub.sub1 import func
        func()
        print("leave f2", threading.get_ident())
    except:
        print("err f2", threading.get_ident())
        raise

with ThreadPoolExecutor(max_workers=2) as e:
    ret = e.map(f2, range(2))
    for x in ret:
        print(x)

sub/init.py

# blank

sub/sub1.py

def func():
    from .sub2 import VAR

sub/sub2.py

import time
import threading

print("begin sub2", threading.get_ident())
time.sleep(1)
VAR  = 100

In Python 3.8, the second thread blocks until the first thread completes importing sub.sub2 module. But in Python 3.9, the second thread fails soon.

3

Ok, the difference came from bpo-35943

4

For the record, this is a regression in 3.9: Issue 43517: Fix false positives in circular import detection with from-imports - Python tracker