Differences between 3.8 and 3.9 in importing module?

I encoundered similar error to https://bugs.python.org/issue41567.

Not only multiprocessing.Pool, I saw various ImportError in multiprocessing in 3.9.
For example, the sample code below raises the following error(on macOS 10.15.5).

from concurrent.futures import ThreadPoolExecutor

def f3():
    pass

def f2(arg):
    queue: Any = multiprocessing.Queue()
    p = multiprocessing.Process(target=f3)
    p.start()
    p.join()

def f1():
    with ThreadPoolExecutor(max_workers=20) as e:
        ret = e.map(f2, range(10))
        for x in ret:
            print(x)

if __name__ == '__main__':
    f1()
Traceback (most recent call last):
  File "/Users/ishimoto/importerror_multiprocessing.py", line 20, in <module>
    f1()
  File "/Users/ishimoto/importerror_multiprocessing.py", line 16, in f1
    for x in ret:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 600, in result_iterator
    yield fs.pop().result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/ishimoto/importerror_multiprocessing.py", line 8, in f2
    queue: Any = multiprocessing.Queue()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 102, in Queue
    from .queues import Queue
ImportError: cannot import name 'Queue' from partially initialized module 'multiprocessing.queues' (most likely due to a circular import) (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py)

The multiprocessing has a lot of local imports, so multiprocessing is not thread-safe.

But I wonder why these codes start failing in Python 3.9. They worked fine in 3.7 and 3.8 for years and I can not find the change that caused these differences between 3.8 and 3.9 in multithreading module.

It looks the issue is not in the multiprocessing module itself, but Python’s import machinery somewhat changed.

Following files reproduce similar error.

main.py

from concurrent.futures import ThreadPoolExecutor
import threading

def f2(arg):
    print("enter f2", threading.get_ident())
    try:
        from sub.sub1 import func
        func()
        print("leave f2", threading.get_ident())
    except:
        print("err f2", threading.get_ident())
        raise

with ThreadPoolExecutor(max_workers=2) as e:
    ret = e.map(f2, range(2))
    for x in ret:
        print(x)

sub/init.py

# blank

sub/sub1.py

def func():
    from .sub2 import VAR

sub/sub2.py

import time
import threading

print("begin sub2", threading.get_ident())
time.sleep(1)
VAR  = 100

In Python 3.8, the second thread blocks until the first thread completes importing sub.sub2 module. But in Python 3.9, the second thread fails soon.

Ok, the difference came from bpo-35943