Nice to meet you guys here,
Package version is a long-history problem of python since there is so many versions of a package.
Is there any way to allow different versions of a package work non-interferencely ？
I have an idea, that is Python recognize Python packages by this way in scripts:
if a package should be imported, use this(use matplotlib as an example):
import matplotlib if you want to import the newest(or default) version.
or
import matplotlib(1.1.0) or import matplotlib(1.1.0-2.0) if you want to import the specific version.
So that different packages can work together without creating one and another virtual environments.
I think all the way down the stack, imports would have to be versioned to even consider something like this.
If matplotlib1 uses numpy3 but matplotlib2 uses numpy4: you’d have to have matplotlib’s import of numpyX be versioned as well so that it imports the correct version.
Even then I don’t think all imports should be pinned, maybe there would be wildcards allowed for patch fixes? What about other variants?
How would you detect an issue where an internal import isn’t versioned but needed to be?
I think this type of change would be really tough to make with the ecosystem we have today. If starting over, this would be more possible to me.
Virtualenvs are really nice for this type of thing based off the current import flow.
1 Like
It’s really nice to read your opinion.
Yep, however we can change it on later Python versions. For example, when we make Python 4, we can use this new way to import. Just as what you mentioned in the last. 
Actually it’s a scary problem. However, we can also choose to retain some important versions.
A problem confuzing me now is that some packages must be installed by conda. I think Python version control is one of the main reasons.
I like simple and purity, and I think a large part of the Python coders would like them, too.
(It shouldn’t be taken for granted that Python 4 will literally ever happen. AFAIK, the current plan is for it not to ever happen.)
Virtualenvs are great until the packages you want for a single project have unsolvable dependencies. However, adding to the import syntax seems really noisy and hard to work with for such a rare case.
Other languages have an ecosystem where you expect to be able to install multiple versions of the same library side by side, and your code can easily use the appropriate one. It works in Javascript for example because the importing syntax directly specifies paths. That’s rather unpleasant to work with, IMO, and it can waste tons of space when the separate versions aren’t actually necessary.
The design I’m thinking of is something where a package in site-packages could have sub-folders for versions, perhaps with some kind of metadata in the package root folder to indicate this situation; and then packages that depend on that library would have their own metadata folder indicating the expected version when importing specific names. Then, when the package installer encounters a broken solve, it could just grab two versions to satisfy the requirements more easily (or add one to the existing one), and write the corresponding metadata in the appropriate places. The importers on sys.meta_path would check for those metadata files and re-route their folder searches correspondingly. Ideally, it would be possible to intervene in the package-solving process and manually specify versions, too. Of course, that’s all still pretty nebulous, but.
It’s not just a package X that decides which version of a library Y it needs. It has to be completely consistent. If some package Z imports X and Y, and X imports Y, then the version of Y that’s chosen has to be consistent with both Z and X.
If you really want something like this, I think it would mean that for any user script, you would need to resolve dependencies and create a virtual environment for that script. I think it’s totally doable to write a tool that does this; maybe build it as an extension to a package manager like poetry:
poetry script some_script.py
The tricky part is figuring out the dependencies (without a pyproject.toml), and just by looking at the script.
Similarly, for every installed package with an entropy point, you would have to create a virtual environment for it, and then bind the entry point’s shell command to a script that runs the command from the created virtual environment. Also, seems totally doable today if someone wants to go to the trouble.
This is such an important feature in my opinion. The amount of problems I encountered because of version conflicts is enormous. However, it can cause some unintuitive behavior when it comes to which version will take priority when a library can work with both versions of a dependency.
This feature will help alot with infrastructures to run arbitrary code. You can be assured the users will be able to run their code without worrying you broke their behavior for needing some dependency you both happen to have in common.
Hmm, good point. The code could well depend at runtime on Y being the same module object (to share data), so it can’t just import two different versions of Y (even if the module cache design were adapted to allow for it).
In my opinion, this feature is completely undesirable. If a package need to work with a specific version of a dependency, they should just vendor that dependency or fork them with a different library name. That will cause it to have a different import path.
By allowing multiple versions to co-exist at runtime, that means that you can pass in objects returned by libone that internally uses mydep==1.7 to libtwo that takes that object and passes it to code from mydep==1.0. That’s going to cause endless amount of problems, an exponential explosion of testing matrix, and it’ll become much, much harder to deprecate anything because new versions of the library would continue to be expected to be able to handle objects created from older versions indefinitely.