Venv not function properly in macOS within an exFAT disk

TLDR

macOS creates ._* files for every file in certain file systems like exFAT. As a result, when creating a virtual environment in exFAT, a ._*.pth file is generated, which contains binary-encoded content that prevents Python from functioning properly.

Computer information

macOS version: Ventura 13.2.1
Mac model: Mac mini 2023 with m2
Python version: 3.11.3

Issue

After attempting to create a virtual environment by running python3 -m venv venv on an external exFAT-formatted disk, I executed the command pip install -r requirements.txt. This resulted in the following error:

Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

At this point, the python command also produced the same error. However, python -S executed successfully.

Investigation

I ran the python -vvv command and extracted the relevant log information below:

Processing global site-packages
Adding directory: '/path/to/project/venv/lib/python3.11/site-packages'
Processing .pth file: '/path/to/project/venv/lib/python3.11/site-packages/._distutils-precedence.pth'
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte
# destroy site

then I inspected the ._distutils-precedence.pth file:

(venv) ➜  site-packages git:(master) ✗ cat ._distutils-precedence.pth
Mac OS X        	2��ATTR��
                                 �
                                  com.apple.provenance$u�@�This resource fork intentionally left blank   ��%
(venv) ➜  site-packages git:(master) ✗ file ._distutils-precedence.pth
._distutils-precedence.pth: AppleDouble encoded Macintosh file
(venv) ➜  site-packages git:(master) ✗ file -I ._distutils-precedence.pth
._distutils-precedence.pth: application/octet-stream; charset=binary

It is clear that this file was generated by macOS. After researching the issue, I found this explanation: macos - Why are dot underscore ._ files created, and how can I avoid them? - Ask Different (stackexchange.com).
I deleted the ._distutils-precedence.pth file, and everything is working correctly now.

Conclusion

I’m not sure how to permanently resolve this issue. Deleting all ._* files every time after creating a venv seems to be the best workaround for now. Is there anything I can do or Python can do to prevent from this?

Perhaps I should consider switching to Linux in the future.

3 Likes

Just adding a ‘me too!’. I bought a new external drive and formatted it as ExFAT in case I need to use it with Windows & Linux too. I want to store my ML projects there and keep the models in the same place.

This also causes PyCharm problems when creating a virtual environment which is how I encountered it. I’ve logged an issue with JetBrains but expect they’ll say it’s MacOS’s fault.

In the meantime, I’ve used dot_clean to remove the files. Hopefully that won’t delete anything used by git?

You could watch for folder changes using the built-in Automator.app (Look for ‘Folder Changes’ then ‘Run Shell Script’) or the newer Shortcuts app or other suggestions here -

Shortcuts has options for scripting via js & applescript and also a ‘run shell script’ module

+1 on having this problem on an exFAT drive on macos.
+1 on running dot_clean . to solve these problems. It’s a bit iterative since I didn’t setup the Automator as suggested, but it works.

Thanks guys

Another workaround is to simply initialize the new Venv on your local machine and then move it onto your exFAT disk before using it.

@lincw, could you file an issue about this in the CPython repository?

I’ve reproduced the issue, but this required some manual tweaking on my end:

  1. Create a disk image using the exFAT format
  2. Create a venv on that
  3. Install setup tools
  4. Open the “Get Info” panel in the finder for the distutils_hack.pth file and add a note

The last step forces the creation off metadata that exFAT cannot store natively, and therefore results in the creation of an “._” file.

With that file I get the same error as you.

I’ve also filed an issue about this: Crash when there a "._xxx.pth" on macOS · Issue #113356 · python/cpython · GitHub

I have the impression that it is the MacOS Finder which makes ._*
files. Does your venv have any ._* files?

Not just the Finder. These “._” files appear when the filesystem doesn’t support metadata that the system wants to write.

One way to reproduce this in the command-line on an exFAT filesystem:

$ touch hello.txt
$ xattr -w answer 42 hello.txt

There’s now a “._hello.txt” file as well to contain the extended attribute. You won’t see this file with APFS and HFS+ filesystems because those natively support these.

1 Like