I started this project as a way of avoiding the dependency issues that I’ve been having while building FreeCAD from source so that I can get IME support to work in FreeCAD on Linux Mint 22.3, which is an LTS distro. FreeCAD AppImages and Flatpacks don’t work with IMEs. It’s a build issue and they’ve not figured it out yet.
I cloned the current main repository of FreeCAD, which works when building with pixi. However FreeCAD also checks the user site for dependencies in various locations for various reasons, most of which involve working around PEP 668 and distro locked user sites and the various versions of dependencies in those various locations. However as I thought about all of the locations that could be hiding code that touches this issue, Addon Manager, workbenches, etc. The scope grew very large very quickly. So I thought I should focus more on the underlying issue. Clearly that is python’s dependency issues. So I thought about how to fix THAT issue instead. Sounds insane, right?
Actually it wasn’t very hard. If you consider why things are why they are, not wanting to have multiple versions of the same package on the same machine to save disk space (historically and maybe still today), then Linux distros staking claim on system python and the user site dependencies as well, which leads to pep 668 and appimages, flatpacks, snaps, etc all trying to address the same issue but actually making it worse by requiring people to keep multiple copies of the same library on their computer instead of a single copy of multiple versions, it becomes clear that the answer is to simply allow for the retention of historical copies of the dependencies.
So I cloned cpython and implemented a method of resolving library imports in a particular order. I based this fork on PEP 621 and 751 to know which dependencies are needed. In the case of FreeCAD I pulled the python dependencies from the pixi dependency list and put them into pyproject.toml and pylock.toml, which I added to the repo root. From here, my cpython implementation will check the user site location for available dependencies, if it does not find the version it needs there, it goes to my historical package site and checks my hierarchical list for the available dependency there. If it finds it, it resolves to that directory for the library import. If it does not, it throws an error. Then I can use a prototype pip that I modified to download dependencies into my historical package site.
Then when I build FreeCAD using my version of Python 3.11.14, which is the version included in FreeCAD, pixi takes what it can from the user site, and what it cannot find, it takes from my historical site. This resolves all dependency contamination and FreeCAD can build and load correctly without disabling FreeCADs access to the user site.
So what does this mean?
- venvs are mostly unnecessary. They’re mostly used for dependency conflicts and contamination prevention which is not an issue for me anymore. You can still use them, but most people won’t need to.
- PEP-668 stops frustrating people. The OS can still lock down the user site, and really who cares? The real user site is now much more robust and less prone to contamination.
- Its backwards compatible. This requires no changes to any python code. Everything that was written in the past still functions as expected and things that don’t function as expected get rerouted to the correct dependency so that they function as expected.
- Distributions are smaller in size. Native builds, appimages, flatpacks, snaps, etc no longer need to include python dependencies. You save bandwidth on every version and on every download.
- Pip is expanded to check both the user site packages and historical site packages and downloads to the historical site packages for use by other scripts. Pip can list the user site list and the historical site list.
- Native builds of applications can now exist on LTS OSes. Since you can download the needed version to the historical site, there is no reason why python has to block current application versions on LTS OSes.
What burden does this create?
- You have to use pyproject.toml and pylock.toml to use packages outside of the OS protected user site.
- There could be edge cases where you don’t save much disk space if every AppImage you use has a different set of dependencies, or if you never use many containers in the first place. Not much of a burden, but not a benefit I guess.
I started with 3.16, but moved to 3.11.14 when I realized that’s what FreeCAD uses. I have since made a port of 3.12 so I can replace my system Python with my prototype and continue testing.
Now I’m still testing, but so far I haven’t run into many issues. I fixed one problem with dependencies that only include one .py file, and I’m still testing things to see what other fixes need doing, but FreeCAD just runs and I didn’t change any existing FreeCAD code. I just point at my new python build.
The diff is about +450 / -2 (in 3.11). If you include tests, which it has passed all applicable tests (I don’t have the tooling for 2, and 1 was set to be offline but it tried to connect online anyway. Those are the only 3 tests it failed out of the many many found in the Cpython repo), then the diff is closer to +600 / -2.
I built this prototype using Codex 5.6 Sol High Thinking. I’m not suggesting anyone merge my prototype, but the method seems sound. I for one would rather keep 1 copy of the various versions of dependencies I actually need than multiple copies of the same versions spread across every venv, appimage, flatpack, snap, etc. The disk space and bandwidth that can be saved are not insignificant.
Look at how many packages PyPa serves daily. It’s in the BILLIONS. How many of those could be prevented if every computer had access to the files they needed without having to download them again every time their cache is cleared or goes stale? How about through new releases of AppImages and FlatPacks? Release builds will all reduce in size.
There’s are many other benefits I’m neglecting I’m sure, and I haven’t finished testing. I still need to try replacing my system python, but if you want to try it out for yourself or just look at the code. My github account only has the 3 forks I mentioned here: FreeCAD with the dependency .toml files, my prototype cpython builds, and my prototype pip build.
There are many things I don’t know about python, so I cannot say these prototypes are perfect and ready to merge. However the idea is looking to be sound. I invite you to consider this method seriously and let’s stop all the wasted effort people are building and maintaining every day to work around python dependency issues when the real fix could be quite simple and small.