I have run into similar things. I’ve also resorted to a venv with pip install for distribution, with the same concerns.
This discussion does get rather technical and down in weeds, but it looks like there is an appetite for improving the Python distribution story.