# Compileall option to hardlink duplicate optimization levels bytecode cache files **URL:** \ **Category:** Ideas\ **Created:** [January 16, 2020, 3:51pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014 "2020-01-16T15:51:02Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [January 16, 2020, 3:51pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/1 "2020-01-16T15:51:02Z")
Hello. In Fedora, we are looking at [ways how to minimize the filesystem footprint of Python](https://github.com/hroncok/python-minimization/blob/master/document.md). One of the ideas is to deduplicate **identical** `.pyc` files across optimization levels by hardlinking them between each other. In the 3.8 standard library, this could save 10+ % of what we ship by default. If we do it with all our Python packages, we could save a lot of space. As a single data point: On my workstation I have 360 MiB of various Python 3.7 bytecode files in `/usr` and I can save 108 MiB by doing this. We utilize the [compilall](https://docs.python.org/3.9/library/compileall.html) module to bytecompile the files. We have [recently added the ability to compile for various optimization levels at a time](https://github.com/python/cpython/pull/16012) (for example, `compileall -o0 -o1 -o2`). The idea here is to add an optional flag (called for example `--hardlink-dupes/-H`) that would compare different optimization level bytecode caches after compiling them and if they are identical in content, `compileall` would make them hardlinks instead of copies. Example: ```auto $ python -m compileall -o0 -o1 -o2 --hardlink-dupes ... ``` This would hardlink `module.cpython-3?.pyc` with `module.cpython-3?.opt-1.pyc` with `module.cpython-3?.opt-1.pyc` if identical. Or just the two identical files. Or nothing if different. Given the nature of the bytecode caches, the non-optimized, optimized level 1 and optimized level 2 `.pyc` files may or may not be identical. Consider the following Python module: ```python 1 ``` All three bytecode cache files would by identical. While with: ```python assert 1 ``` Only the two optimized cache files would be identical with each other. And this: ```python """Dummy module docstring""" 1 ``` Would produce two identical bytecode cache files but the opt-2 file would differ. Only modules like this would produce 3 different files: ```python """Dummy module docstring""" assert 1 ``` Is that idea worth doing? We’d rather have the support written in `compileall` than doing it by some external tool. We are willing to maintain that code. ---
**Author:** ![nad](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nad/32/51_2.png) [@nad](https://discuss.python.org/u/nad)\ **Post date:** [January 17, 2020, 3:38am UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/2 "2020-01-17T03:38:47Z")
That sounds like a clever way to save some disk space with no obvious downsides, assuming that the overall wall clock time to do a `make install` is not significantly increased. If someone does a naive copy of the files afterwards, the worst that should happen is that you end up with separate files again, the same as today. The code would need to handle the case where hard links aren’t supported on the file system in use. Unless someone thinks of some other downside, I’d say let’s do it. ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [January 17, 2020, 8:49am UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/3 "2020-01-17T08:49:38Z")
The idea is to make the option nondefault at least for now, so make install won’t be affected yet. We would use it in Fedora and later we can eventually have another conversation about making it the default on POSIX like systems. For the same reason, we might want to have the option available on all OSes – users of Windows might try it, but they won’t be affected by hardlink quirks until they opt-in. If we realize it may never work there, we can eventually make the option only defined on POSIX. ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [January 18, 2020, 2:35pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/4 "2020-01-18T14:35:16Z")
There are separate general utilities which deduplicate files. Why not use them in your build scripts? And it is easy to write a Python scripts which does this for this particular case. What happen if you update `.py` file and run `compileall` or just import the Python file? Would not the compiler overwrite the content of linked `.pyc` files for different optimization levels? To make the optimization having an effect, you need a Python file which do not contain docstrings (very unlikely) nor `assert` statements or uses of ` __debug__ ` constant. ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [January 18, 2020, 3:01pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/5 "2020-01-18T15:01:30Z")
> [@storchaka](#): > > There are separate general utilities which deduplicate files. Why not use them in your build scripts? And it is easy to write a Python scripts which does this for this particular case. We use `compileall` in various places: when we build Python, when we build RPM packages form Python packages and when we build custom RPM packages that happen to contain some importable Python modules. If we add this functionality to the module, we just need to add the new flag to the various places and others can benefit form this as well. If we just adapt our build scripts to add additional code, we would need to maintain this somewhere else or copy it to various places, no 3rd party benefits from this. Generally, my motivation was that doing it in `compileall` makes sense – it’s easier for us and others can benefit as well. > [@](#): > > What happen if you update `.py` file and run `compileall` or just import the Python file? Would not the compiler overwrite the content of linked `.pyc` files for different optimization levels? The compiler replaces the files and effectively “unlinks” them. We might want to add some tests to make sure this is always the case. When you update the source and run Python in different optimization levels, the files are no longer linked even if they are the same. Worst case scenario is that you end up with the same amount of bytes as before using the new deduplicate switch. > [@](#): > > To make the optimization having an effect, you need a Python file which do not contain docstrings (very unlikely) nor `assert` statements or uses of ` __debug__ ` constant. The standard library (Python 3.8.1, various tests modules excluded): - 607 modules have bytecode files - 454 identical optimization 0 and 1 pairs - 68 identical optimization 1 and 2 pairs - 62 identical optimization 0, 1 and 2 triads (already counted in both of the above) We anticipate the files without docsrings are most likely empty, OTOH most of the non-empty files have no asserts or ` __debug__ ` conditionals. From [python-minimization/document.md at master · hroncok/python-minimization · GitHub](https://github.com/hroncok/python-minimization/blob/master/document.md#solution-9-deduplicate-bytecode-cache) counted in [python-minimization/python-minimization.ipynb at master · hroncok/python-minimization · GitHub](https://github.com/hroncok/python-minimization/blob/master/python-minimization.ipynb) ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [February 3, 2020, 10:29am UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/6 "2020-02-03T10:29:01Z")
We have a working draft implementation ready at [https://github.com/fedora-python/compileall2/pull/19](https://github.com/fedora-python/compileall2/pull/19) ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [February 5, 2020, 12:48pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/7 "2020-02-05T12:48:53Z")
How should we proceed? We can either start using our implementation in Fedora and come back with some data, or we can propose an upstream Pull Request. The discussion here seem stalled. Should we bring this up on the python-dev mailing list instead? ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [February 5, 2020, 1:07pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/8 "2020-02-05T13:07:45Z")
I think the discussion is over, rather than stalled. Ned said “let’s do it” and Sehryi’s points are addressed. (Sehryi, please shout if you’re -1 on this; from here it doesn’t like you are.) I’d go for opening a pull request after some testing in Fedora. ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [February 10, 2020, 10:43pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/9 "2020-02-10T22:43:33Z")
The feature is now part of the [compileall2 0.7 release](https://pypi.org/project/compileall2/0.7.0/). ---
**Author:** ![hroncok](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hroncok/32/18696_2.png) [@hroncok](https://discuss.python.org/u/hroncok)\ **Post date:** [February 11, 2020, 2:56pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/10 "2020-02-11T14:56:07Z")
The installation size of Python 3.9 gets from 125 MiB to 103 MiB. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [May 14, 2020, 2:19pm UTC](https://discuss.python.org/t/compileall-option-to-hardlink-duplicate-optimization-levels-bytecode-cache-files/3014/11 "2020-05-14T14:19:23Z")
I just merged the PR: [https://github.com/python/cpython/commit/e77d428856fbd339faee44ff47214eda5fb51d57](https://github.com/python/cpython/commit/e77d428856fbd339faee44ff47214eda5fb51d57) will be part of the incoming Python 3.9.0 beta1!