# Python module conflict discussion **URL:** \ **Category:** Packaging\ **Created:** [February 8, 2023, 1:57pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659 "2023-02-08T13:57:35Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![unsatisfying](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/unsatisfying/32/11030_2.png) [@unsatisfying](https://discuss.python.org/u/unsatisfying)\ **Post date:** [February 8, 2023, 1:57pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/1 "2023-02-08T13:57:35Z")
# packaging module conflict ## phenomenon As we all know, we use a package name to download a distribution from [pypi.org](http://pypi.org/). However, we import the module with its module name. ## some experiment So the problem is that, a package name is unique in the [pypi.org](http://pypi.org/) while module names are not. What happens if I download two packages and they have the same module (same module name and different content)? I do some experiments and here is the result. Here are the two packages I constructed. The first package has two modules, mod1 and mod2, while they have their own add and sub submodules. The second package also has two modules, mod1 and mod2, but they have their own div and multi submodules. I install the first package first and then the second package. We can see that both packages are installed in the site-packages directory at the same time, and their structure is shown in the figure. ![1](https://us1.discourse-cdn.com/flex002/uploads/python1/original/2X/7/74d580767a0bbaaf3add8a7c7de2e2696f5c6e8c.png) It is interesting that, except for the metadata files of the two packages, which are stored in their own folders, the rest of the modules are jumbled together due to a module conflict between the two packages. What’s worse, later installed packages will overwrite modules that previously installed(e.g. mod1/\_\_init\_\_.py and mod2/\_\_init\_\_.py). And the other modules are installed in the same folder. ## analysis This seems to be the convention, and there has been a lot of work noticing module conflicts, but such an overwrite setting seems to break the local environment. For example, if there is a module foo.py in the local environment, and a user accidentally overwrites it when installing an open source package subsequently, then the user will use the wrong module when importing foo module. A special case is represented in the dependency graph. If pip installs an open source package with conflicting modules in its dependencies, pip will have an overwrite problem when installing these dependencies into the same path. This may cause the functionality of some packages to fail, or an attacker may use this situation to compromise common modules in the local environment to achieve an attack. However, pip does not have any warning messages for this situation. In addition I did a large-scale analysis of module conflicts in the dependency graph and found that nearly 4.71% of the packages on pypi have module conflicts in the dependency graph. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [February 8, 2023, 2:15pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/2 "2023-02-08T14:15:53Z")
I suspect this discussion will keep coming up until pip refuses to extract a file that already exists. That itself will break a range of packages,\[1\] but as we move toward more predictable build backends it should be easier to ensure those packages release without conflicting files. * * * 1. e.g. those who want to act like namespace packages but want the safety of having an ` __init__.py` and so all possible children will include it - _yes_ I know this is wrong and there are better ways to get that safety, but even the people who ask me for advice on this keep deciding to ignore it 😉 ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [February 8, 2023, 2:30pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/3 "2023-02-08T14:30:31Z")
I can’t quote footnotes, but you’re right, legacy-style namespace packages with ` __init__.py` are a big reason pip can’t do this. That, and the broader “there is no bad practice so awful that there isn’t a massive closed-source project that relies on it” issue. See [xkcd: Workflow](https://xkcd.com/1172/). ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [February 8, 2023, 2:33pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/4 "2023-02-08T14:33:43Z")
Possibly a “pip fails on file conflict unless the contents is identical” is sufficient to get past that case? ---
**Author:** ![fungi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/fungi/32/5963_2.png) [@fungi](https://discuss.python.org/u/fungi)\ **Post date:** [February 8, 2023, 2:49pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/5 "2023-02-08T14:49:13Z")
Another place this is relied on is when you have two packages providing the same API but with alternative implementations behind it. There is a desire to be able to use one (often a successor) as a drop-in replacement for the other with no changes to existing code that imports it. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [February 8, 2023, 2:54pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/6 "2023-02-08T14:54:27Z")
> [@steve.dower](#): > > pip fails on file conflict unless the contents is identical The problem will creep up again on uninstallation. This obviously doesn’t work now either, but if you “fix” installation people would reasonably expect uninstallation to also work. ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [February 8, 2023, 2:57pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/7 "2023-02-08T14:57:07Z")
> [@uranusjr](#): > > The problem will creep up again on uninstallation. It’s always been there, and it’s why when we set up the Azure SDK (which uses namespace packages _extensively_) we put the package ` __init__.py`’s in their own package, so they get installed exactly once and uninstalled exactly once (or more likely, never 😉 ). Reference counting conflicting files is the only real way around it, but that still doesn’t answer how to handle the conflict in the first place - fail (because it’s different), or increase the count? ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [February 8, 2023, 3:17pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/8 "2023-02-08T15:17:46Z")
> [@steve.dower](#): > > we put the package ` __init__.py`’s in their own package This is a better idea the more I think of it. With pkgutil-style namespace packages mostly in the rear mirror, a reasonable way out like this (fork the projects, remove the ` __init__.py`, and put it in a separate package) is good enough IMO. (We need a deprecation period for this to be implemented by potentially impacted users, of course.) With the workaround available, I think erroring out on any file conflict is the right thing to do. ---
**Author:** ![srittau](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/srittau/32/237_2.png) [@srittau](https://discuss.python.org/u/srittau)\ **Post date:** [February 8, 2023, 11:04pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/9 "2023-02-08T23:04:24Z")
> [@pf\_moore](#): > > I can’t quote footnotes, but you’re right, legacy-style namespace packages with ` __init__.py` are a big reason pip can’t do this. That, and the broader “there is no bad practice so awful that there isn’t a massive closed-source project that relies on it” issue. Maybe one way around this and towards a safer world would be to prevent overwriting files by default, but allowing it by using an `--allow-overwrite` flag. At least the user needs to consciously opt-in and the annoyance of that flag might nudge package maintainers towards a safer alternative, like the “metadata” alternative, described [here](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/). At some final stage, pip could completely disallow infringing on another package’s module space, unless an explicit flag is specified. ---
**Author:** ![cnspary](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cnspary/32/11045_2.png) [@cnspary](https://discuss.python.org/u/cnspary)\ **Post date:** [February 9, 2023, 1:56am UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/10 "2023-02-09T01:56:05Z")
The pkgname is unique on PyPI, so is it possible to use pkgame as its namespace by default when installing packages (similar to metadata folders, creating separate folders for each package and declaring them when using them, such as _ **from pkgname import mod\_name** _ or _ **from pkgname.mod\_name import function/class** _). This lets pip perform correctly during installation and uninstallation, and allows developers to be aware of which package they are using, thus avoiding the use of the wrong package or even a malicious package. ---
**Author:** ![pradyunsg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pradyunsg/32/206_2.png) [@pradyunsg](https://discuss.python.org/u/pradyunsg)\ **Post date:** [February 9, 2023, 4:20am UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/11 "2023-02-09T04:20:23Z")
I don’t enjoy doing this but we’re repeating a discussion that has largely taken place on pip’s issue tracker already. I suggest we move the rest of this discussion there and, uhm… please read the existing discussion before posting further there! > > > \* Pip version: any (tested on 1.5.4 and 9.0.1) > > \* Python version: any (tested …on 2.7.6 and 3.4.3) > > \* Operating system: Should be OS independent (tested on Ubuntu 14.04.5 and Mint 17.3) > > \### Description: > > I'm installing packages with with \`pip\`, and it happens that some of these packages (with different names) have files with name clashes. Unfortunately, \`pip\` silently ignores the issue and the results depend on the order in which the installs happens. In an ideal world, I would expect this to be impossible, i.e. to force the package maintainer to use a unique namespace. As a (far worse, but better than current behavior) second choice, I would expect \`pip\` to warn the user before silently overwriting existing files, \*\*especially\*\* if \`pip\` itself installed those files. > > \### What I've run: > > Create a clean slate to work on: > > \`\`\` > /tmp $ virtualenv NAMECLASH > New python executable in NAMECLASH/bin/python > Installing setuptools, pip...done. > /tmp $ cd NAMECLASH > /tmp/NAMECLASH $ source bin/activate > (NAMECLASH)/tmp/NAMECLASH $ pip show -f pyjwt > (NAMECLASH)/tmp/NAMECLASH $ pip show -f jwt > (NAMECLASH)/tmp/NAMECLASH $ > \`\`\` > > Install packages: > > \`\`\` > (NAMECLASH)/tmp/NAMECLASH $ pip install jwt > Downloading/unpacking jwt > Downloading jwt-0.5.2.tar.gz > Running setup.py (path:/tmp/NAMECLASH/build/jwt/setup.py) egg\_info for package jwt > > Downloading/unpacking cryptography==1.7.2 (from jwt) > Downloading cryptography-1.7.2.tar.gz (420kB): 420kB downloaded > Running setup.py (path:/tmp/NAMECLASH/build/cryptography/setup.py) egg\_info for package cryptography > ... > Successfully installed jwt cryptography typing idna pyasn1 six setuptools enum34 ipaddress cffi pycparser > Cleaning up... > (NAMECLASH)/tmp/NAMECLASH $ > > \`\`\` > Lots of irrelevant stuff removed from log above (for the sake of making this issue easier to read). Analyzing what has been installed: > > \`\`\` > (NAMECLASH)/tmp/NAMECLASH $ pip show -f jwt > \--- > Name: jwt > Version: 0.5.2 > Location: /tmp/NAMECLASH/lib/python2.7/site-packages > Requires: cryptography, typing > Files: > ../jwt/jws.py > ../jwt/jwk.py > ../jwt/exceptions.py > ../jwt/jwkset.py > ../jwt/utils.py > ../jwt/jwa.py > ../jwt/jwt.py > ../jwt/\_\_init\_\_.py > ../jwt/jws.pyc > ../jwt/jwk.pyc > ../jwt/exceptions.pyc > ../jwt/jwkset.pyc > ../jwt/utils.pyc > ../jwt/jwa.pyc > ../jwt/jwt.pyc > ../jwt/\_\_init\_\_.pyc > ./ > PKG-INFO > requires.txt > SOURCES.txt > dependency\_links.txt > top\_level.txt > > (NAMECLASH)/tmp/NAMECLASH $ grep Invalid /tmp/NAMECLASH/lib/python2.7/site-packages/jwt/exceptions.py > class InvalidKeyTypeError(JWTException): > \`\`\` > > Now install \*silently conflicting\* package, and analyzing what happened: > > \`\`\` > (NAMECLASH)/tmp/NAMECLASH $ pip install pyjwt > Downloading/unpacking pyjwt > Downloading PyJWT-1.5.2-py2.py3-none-any.whl > Installing collected packages: pyjwt > Successfully installed pyjwt > Cleaning up... > (NAMECLASH)/tmp/NAMECLASH $ pip show -f pyjwt > \--- > Name: PyJWT > Version: 1.5.2 > Location: /tmp/NAMECLASH/lib/python2.7/site-packages > Requires: > Files: > Cannot locate installed-files.txt > (NAMECLASH)/tmp/NAMECLASH $ grep Invalid /tmp/NAMECLASH/lib/python2.7/site-packages/jwt/exceptions.py > class InvalidTokenError(Exception): > class DecodeError(InvalidTokenError): > class ExpiredSignatureError(InvalidTokenError): > class InvalidAudienceError(InvalidTokenError): > class InvalidIssuerError(InvalidTokenError): > class InvalidIssuedAtError(InvalidTokenError): > class ImmatureSignatureError(InvalidTokenError): > class InvalidKeyError(Exception): > class InvalidAlgorithmError(InvalidTokenError): > class MissingRequiredClaimError(InvalidTokenError): > InvalidAudience = InvalidAudienceError > InvalidIssuer = InvalidIssuerError > \`\`\` > > As you can see, \`pip\` silently allow the two packages to overwrite each other's \`jwt/exceptions.py\` file (and in particular the \`InvalidKeyTypeError\` is now gone, creating a whole lot of problem, depending on the order in which pip install ran) ---
**Author:** ![PythonCHB](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pythonchb/32/2551_2.png) [@PythonCHB](https://discuss.python.org/u/PythonCHB)\ **Post date:** [February 9, 2023, 4:43am UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/12 "2023-02-09T04:43:09Z")
> [@srittau](#): > > prevent overwriting files by default, but allowing it by using an `--allow-overwrite` flag. +1 – certainly as a start. To go a bit further, I think there are two different potential use cases here: 1. Two different distributions that have nothing to do with each-other use the same package name – no idea what can be done, but at least the user should know something’s up! 2. Cooperating distributions, such as namespace packages – then maybe pip could know that it’s OK to overwrite certain files. OK – just saw the link to the issue tracker – I’ll go there now. ---
**Author:** ![davidism](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/davidism/32/3295_2.png) [@davidism](https://discuss.python.org/u/davidism)\ **Post date:** [February 9, 2023, 2:57pm UTC](https://discuss.python.org/t/python-module-conflict-discussion/23659/13 "2023-02-09T14:57:29Z")
Closed on February 9, 2023, 2:57pm UTC Maintainer requested following the original discussion on GitHub, reading it fully before posting. [pip overwrites existing files unconditionally during installation · Issue #4625 · pypa/pip · GitHub](https://github.com/pypa/pip/issues/4625)