Venv not function properly in macOS within an exFAT disk

Python Help
1

TLDR

macOS creates ._* files for every file in certain file systems like exFAT. As a result, when creating a virtual environment in exFAT, a ._*.pth file is generated, which contains binary-encoded content that prevents Python from functioning properly.

Computer information

macOS version: Ventura 13.2.1
Mac model: Mac mini 2023 with m2
Python version: 3.11.3

Issue

After attempting to create a virtual environment by running python3 -m venv venv on an external exFAT-formatted disk, I executed the command pip install -r requirements.txt. This resulted in the following error:

Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

At this point, the python command also produced the same error. However, python -S executed successfully.

Investigation

I ran the python -vvv command and extracted the relevant log information below:

Processing global site-packages
Adding directory: '/path/to/project/venv/lib/python3.11/site-packages'
Processing .pth file: '/path/to/project/venv/lib/python3.11/site-packages/._distutils-precedence.pth'
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte
# destroy site

then I inspected the ._distutils-precedence.pth file:

(venv) ➜  site-packages git:(master) ✗ cat ._distutils-precedence.pth
Mac OS X        	2��ATTR��
                                 �
                                  com.apple.provenance$u�@�This resource fork intentionally left blank   ��%
(venv) ➜  site-packages git:(master) ✗ file ._distutils-precedence.pth
._distutils-precedence.pth: AppleDouble encoded Macintosh file
(venv) ➜  site-packages git:(master) ✗ file -I ._distutils-precedence.pth
._distutils-precedence.pth: application/octet-stream; charset=binary

It is clear that this file was generated by macOS. After researching the issue, I found this explanation: macos - Why are dot underscore ._ files created, and how can I avoid them? - Ask Different (stackexchange.com).
I deleted the ._distutils-precedence.pth file, and everything is working correctly now.

Conclusion

I’m not sure how to permanently resolve this issue. Deleting all ._* files every time after creating a venv seems to be the best workaround for now. Is there anything I can do or Python can do to prevent from this?

Perhaps I should consider switching to Linux in the future.

3 Likes
2

Just adding a ‘me too!’. I bought a new external drive and formatted it as ExFAT in case I need to use it with Windows & Linux too. I want to store my ML projects there and keep the models in the same place.

This also causes PyCharm problems when creating a virtual environment which is how I encountered it. I’ve logged an issue with JetBrains but expect they’ll say it’s MacOS’s fault.

In the meantime, I’ve used dot_clean to remove the files. Hopefully that won’t delete anything used by git?

3

You could watch for folder changes using the built-in Automator.app (Look for ‘Folder Changes’ then ‘Run Shell Script’) or the newer Shortcuts app or other suggestions here -

Shortcuts has options for scripting via js & applescript and also a ‘run shell script’ module

4

+1 on having this problem on an exFAT drive on macos.
+1 on running dot_clean . to solve these problems. It’s a bit iterative since I didn’t setup the Automator as suggested, but it works.

Thanks guys