I found https://github.com/jawah/urllib3.future library that is unpacking as urllib3 (overshadowing the original urllib3 library). This new library is starting to get some usage and I happened to have a package that used both urllib3.future and original urllib3. I stumbled upon bug in the new library that was not supposed to be used in the function I used but because of overshadowing it replaced the original.
If the PyPi is allowing for overshadowing of packages, shouldn’t at least pip issue warning when it detects that installation of virtualenvironment has conflicts? For now I created constraint that I am using to guard pip from installing this library at all but I would like to only do so if conflict is detected. This would need some meta-package or wrapper around pip and honestly it is not clean solution to me.
In this case I don’t think it’s overwriting. The purpose of this package is to be a drop-in replacement for urllib3. Using the same name is on purpose, so that no one has to refactor their code. Given that, it seems like a bad idea to try to use both in the same project–they aren’t meant to be compatible.
A warning might be useful but I’m not sure pip can detect this–it doesn’t necessarily know the import names of the packages its installing, because they don’t have to match the name of the package.
7 years? Wow I am lucky I didn’t stumble onto this issue up until then
It wasn’t my choice. This dependency was 4th in the chain, not even direct dependency of what I installed. But given that world evolves around internet and HTTP is a barebones of the internet, there could be a time when conflicts like this will be in every second package. Will there be a fight for the package name? Should I also fork the urllib3 and create an competitor library?
I don’t get the idea of not having to refactor the package being beneficial. ruamel.yaml · PyPI shows already how to do a good fork. It is one line change because interfaces are the same.
Yeah, I think there would be significant caveats implementing this at the installer level:
I don’t see how .pth files could be taken into account
Any dynamic runtime behaviour wouldn’t be known by the installer
The installer would have no idea who is “overshadowing” who
Given Python’s setup I think problems like this must ultimately be solved at a community level, understanding why packages have decided to use the same namespace, and being clear and specific, without being contentious, about what problems it’s causing you. Hopefully such projects are aiming to be good neighbours and weighed up the problems they faced before deciding to take such an action.
I think for short term technical solution available are:
Implement allow list or deny lists if they manage their own local index
Use some kind of lock-like file using pip-compile, poetry, pdm, uv, etc. and review any package before it gets installed into their environment
Use a “constraint” to make sure some packages don’t get installed (supported by pip, pdm, and uv)
Wheels built by setuptools ship a top_level.txt with the top level package or module names in it, which pip can(but do not) leverage it to detect the conflicts. But it is not standardized and not generated by other build backends.
Core metadata also used to allow fields Provides and Obsoletes that seem to work similarly but they are marked as deprecated now.
Unfortunately it’s not the case for this project. One of contributors of urrlib3 asked to not use their naming but ofc was refused. In this spirit I shall create a fork of both niquists and urllib3.future and just keep it in-sync with upstream trying to convince package maintainers to use these instead of the originals to unpollute the namespace .
Just to make clear, urllib3.future is not shadowing the standard library’s urllib, but installs a package with the same name as the extensively used, third party library urllib3 (of which it is a fork as they make perfectly clear).
They claim to have added much value, and I see no reason to doubt that. They regularly sync with urllib3 for patches and security fixes.
urllib3.future can be installed without shadowing urllib3, by installing from an sdist with the env variable SHOULD_PREVENT_FORK_OVERRIDE set.
The request I saw on GitHub was not clear what the specific problem was, nor did it seem a good faith understanding of why the library had made that choice, nor did it seek any other solution available.
Being a good neighbor is a two way process, you can’t solve community problems by starting off assuming your position is the only possible way to solve something.
I can only imagine this would be taken as an actively hostile move given the motivation.
You can extract the same information from RECORD fairly easily, but rather than detecting conflicts I’d like to see this presented on PyPI. Firstly, it’d be real helpful to search by “what module will this let me import”, and it’d also make it very obvious if the package name doesn’t match the module name.
A check/audit tool might also inspect RECORD to determine and warn if two dists are installing to the same directories, though in many cases this is intentional and safe (especially when the same publisher has released both dists), but these tools are the better place for heuristics than trying to have pip detect conflicts.
There have been some discussions about this recently[1]. I thought there was some reason this wouldn’t be trivial to implement, but I’d have to find the thread to remember.
probably you were involved in them, I don’t remember ↩︎
Looks like it’s just waiting on a PEP to decide where to put the list of top-level names in metadata, but the general consensus was that RECORD is fine for 99.9% of cases and the 0.1% should feel a little bit bad about whatever shenanigans they’re up to.
even if he added value, he also introduced bugs and I stumbled over one of them. The bug that was easy oversight of bytes → str conversion missing in one place. It is new code, new code always introduces new bugs. I do not need any new feature, I need old features to work as intended.
In any case, the issue of urllib3 and urllib3.futures is something to discuss with the maintainers of those projects, who I don’t think are lurking in this forum.
If the metadata issues were solved, I think a feature where installers warned you about conflicting names would be handy, but likewise that’s an issue for pip / uv etc to figure out for themselves, once it is reasonably possible.
)