packaging module conflict
As we all know, we use a package name to download a distribution from pypi.org. However, we import the module with its module name.
So the problem is that, a package name is unique in the pypi.org while module names are not. What happens if I download two packages and they have the same module (same module name and different content)? I do some experiments and here is the result.
Here are the two packages I constructed. The first package has two modules, mod1 and mod2, while they have their own add and sub submodules. The second package also has two modules, mod1 and mod2, but they have their own div and multi submodules. I install the first package first and then the second package. We can see that both packages are installed in the site-packages directory at the same time, and their structure is shown in the figure.
It is interesting that, except for the metadata files of the two packages, which are stored in their own folders, the rest of the modules are jumbled together due to a module conflict between the two packages. What’s worse, later installed packages will overwrite modules that previously installed(e.g. mod1/__init__.py and mod2/__init__.py). And the other modules are installed in the same folder.
This seems to be the convention, and there has been a lot of work noticing module conflicts, but such an overwrite setting seems to break the local environment. For example, if there is a module foo.py in the local environment, and a user accidentally overwrites it when installing an open source package subsequently, then the user will use the wrong module when importing foo module.
A special case is represented in the dependency graph. If pip installs an open source package with conflicting modules in its dependencies, pip will have an overwrite problem when installing these dependencies into the same path. This may cause the functionality of some packages to fail, or an attacker may use this situation to compromise common modules in the local environment to achieve an attack. However, pip does not have any warning messages for this situation.
In addition I did a large-scale analysis of module conflicts in the dependency graph and found that nearly 4.71% of the packages on pypi have module conflicts in the dependency graph.