Venv not function properly in macOS within an exFAT disk

Python Help
1

TLDR

macOS creates ._* files for every file in certain file systems like exFAT. As a result, when creating a virtual environment in exFAT, a ._*.pth file is generated, which contains binary-encoded content that prevents Python from functioning properly.

Computer information

macOS version: Ventura 13.2.1
Mac model: Mac mini 2023 with m2
Python version: 3.11.3

Issue

After attempting to create a virtual environment by running python3 -m venv venv on an external exFAT-formatted disk, I executed the command pip install -r requirements.txt. This resulted in the following error:

Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

At this point, the python command also produced the same error. However, python -S executed successfully.

Investigation

I ran the python -vvv command and extracted the relevant log information below:

Processing global site-packages
Adding directory: '/path/to/project/venv/lib/python3.11/site-packages'
Processing .pth file: '/path/to/project/venv/lib/python3.11/site-packages/._distutils-precedence.pth'
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 982, in exec_module
  File "<frozen site>", line 616, in <module>
  File "<frozen site>", line 599, in main
  File "<frozen site>", line 531, in venv
  File "<frozen site>", line 384, in addsitepackages
  File "<frozen site>", line 226, in addsitedir
  File "<frozen site>", line 179, in addpackage
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte
# destroy site

then I inspected the ._distutils-precedence.pth file:

(venv) āžœ  site-packages git:(master) āœ— cat ._distutils-precedence.pth
Mac OS X        	2ļæ½ļæ½ATTRļæ½ļæ½
                                 ļæ½
                                  com.apple.provenance$uļæ½@ļæ½This resource fork intentionally left blank   ļæ½ļæ½%
(venv) āžœ  site-packages git:(master) āœ— file ._distutils-precedence.pth
._distutils-precedence.pth: AppleDouble encoded Macintosh file
(venv) āžœ  site-packages git:(master) āœ— file -I ._distutils-precedence.pth
._distutils-precedence.pth: application/octet-stream; charset=binary

It is clear that this file was generated by macOS. After researching the issue, I found this explanation: macos - Why are dot underscore ._ files created, and how can I avoid them? - Ask Different (stackexchange.com).
I deleted the ._distutils-precedence.pth file, and everything is working correctly now.

Conclusion

Iā€™m not sure how to permanently resolve this issue. Deleting all ._* files every time after creating a venv seems to be the best workaround for now. Is there anything I can do or Python can do to prevent from this?

Perhaps I should consider switching to Linux in the future.

3 Likes
2

Just adding a ā€˜me too!ā€™. I bought a new external drive and formatted it as ExFAT in case I need to use it with Windows & Linux too. I want to store my ML projects there and keep the models in the same place.

This also causes PyCharm problems when creating a virtual environment which is how I encountered it. Iā€™ve logged an issue with JetBrains but expect theyā€™ll say itā€™s MacOSā€™s fault.

In the meantime, Iā€™ve used dot_clean to remove the files. Hopefully that wonā€™t delete anything used by git?

3

You could watch for folder changes using the built-in Automator.app (Look for ā€˜Folder Changesā€™ then ā€˜Run Shell Scriptā€™) or the newer Shortcuts app or other suggestions here -

Shortcuts has options for scripting via js & applescript and also a ā€˜run shell scriptā€™ module

4

+1 on having this problem on an exFAT drive on macos.
+1 on running dot_clean . to solve these problems. Itā€™s a bit iterative since I didnā€™t setup the Automator as suggested, but it works.

Thanks guys

5

Another workaround is to simply initialize the new Venv on your local machine and then move it onto your exFAT disk before using it.

6

@lincw, could you file an issue about this in the CPython repository?

7

Iā€™ve reproduced the issue, but this required some manual tweaking on my end:

  1. Create a disk image using the exFAT format
  2. Create a venv on that
  3. Install setup tools
  4. Open the ā€œGet Infoā€ panel in the finder for the distutils_hack.pth file and add a note

The last step forces the creation off metadata that exFAT cannot store natively, and therefore results in the creation of an ā€œ._ā€ file.

With that file I get the same error as you.

Iā€™ve also filed an issue about this: Crash when there a "._xxx.pth" on macOS Ā· Issue #113356 Ā· python/cpython Ā· GitHub

8

I have the impression that it is the MacOS Finder which makes ._*
files. Does your venv have any ._* files?

9

Not just the Finder. These ā€œ._ā€ files appear when the filesystem doesnā€™t support metadata that the system wants to write.

One way to reproduce this in the command-line on an exFAT filesystem:

$ touch hello.txt
$ xattr -w answer 42 hello.txt

Thereā€™s now a ā€œ._hello.txtā€ file as well to contain the extended attribute. You wonā€™t see this file with APFS and HFS+ filesystems because those natively support these.

1 Like