Proposal - enable pip to install and manage multiple versions of libraries side by side

Packaging
1

Hi, I’ve had this idea come across my mind recently about supporting multiple versions of libraries in pip. Sorry in advance if this sounds unprofessional or vague, or if any PEPs or discussions already exists with the same idea, I’m new to the community, and wanted to raise what I think of would be a revolutionary idea.

So the proposal:

In simple terms, it’s just enabling pip to install and manage multiple versions side by side. Here’s a small snippet in action.
pip install mylib@1.2.3

What I am trying to raise with this change is the ability to install and use multiple versions of the same library in python scripts.

What’s the use for this new behaviour?

One potential use case might be reducing usage of venv and it’s siblings by installing the specific versions of the dependencies needed globally, while staying out of the forced lock in, enabling multiple projects

For example, say I use a “lib” package in two projects x and y, and i use the same computer for both
On x, the version of lib used is 1.2 but on y, 1.3 is used. This is where venv came into to the picture.

The change I’m proposing is that we could pull this off with a simple manifest file instead (I’ve heard there’s something called .toml files now, which could be used), which could massively simplify the user side setup with just a single script. This approach can even enable “use git to control dependency versions” type of workflows, which appears to ve be common in CI/CD environments.

And now what about older versions?

We could add a --keep-old-versions flag on pip install/upgrade, which will keep the older versions and install the newer ones seperately, and keep track of versions properly.

Resolving behaviour with import xyz

A version resolution order can be implemented which goes like

  1. Version manifest files
  2. Latest version installed (considering more than 1 version is installed)
  3. The default version (legacy behaviour)

This is what I have for now. I also thought about a phase 2 which touches the import statement’s semantics, will bring it up and write a seperate proposal if there’s interest from the community.

(Sneek peek - import lib@1.2)

I’m also open to rethink any changes which I might have missed.

Thanks for reading, and I’m hoping for favourable replies from the core devs. Appreciate the work you guys put to maintain the world’s easiest computer language.

2

Also, I’ll be honest, I don’t know the inner working of pip. I had this idea crop up in my mind, and I wrote this up, with help from my FriendGPT, but the ideas are mine, and I thought about it.

3

The previous discussions on this:

The issue isn’t to do with pip, Python does not support this.

You can read there, but one classic problem is, say you had two versions of numpy installed, and they implemented ndarray differently in memory, when you (or libraries you installed) pass an ndarray around, and it ends up being created by one version of numpy and used by a different version so of numpy, it could crash or read the data wrong.

7 Likes
4

Installing with pip is not actually the interesting question.

The problem is having multiple versions of a library installed at the same time and how to import them. That is that part you need to find a solution for.

5 Likes
5

@MegaIng

For that, I propose a minor tweak to the import statement of the core language.

For eg - from numpy@1.21 import ndarray

This will import from numpy 1.21, assuming another version, say 1.19 is installed on the parallel. This is a part of my original proposal, just split it up into 2 phases to see the community’s response.

6

@notatallshaw

Now that’s an interesting view point. I thought about this for a minute, but didn’t get anything. Skimmed through your links, and I can get where you are coming from.

C ABI is something which is already messy, even without the python interpreter and the deps getting involved. I have worked with C code in a few projects, and the issue you say is real in that domain too, and by extension, it’s creeping up in C compiled python libraries.

I might need to take more time to think about this. But as far as I know, this is not an issue with pure python libraries.

Thanks for the response.

7

It can affect pure Python libraries too. If an instance of a class from one copy is passed to a function or method from the other copy, then the latter may try to access (potentially private/unstable) fields that doesn’t exist in the former.

It also breaks isinstance().

4 Likes
8

Also look at issues and pitfalls related to importlib.reload

9

Why not use a venv for each version you want to use?

10

@bwoodsend

Yes, your concern is a hole in the design I had in my mind. Though this runs contradictory to my idea, the solution for this specific problem (and one which damian mentioned) is to treat function calls between different versions of the same library as error.

For eg:

import numpy@1.19 as np_old
import numpy@1.21 as np_new

arr_119 = np_old.ndarray() # throw warning if anything from 1.21 tries to use this array

np_new.foo(arr_119) # error

I’ll be honest, I’m sensing the discussion is going off topic, but we need to discuss this issue first, there’s chances of many accidental bugs here.

11

I am going to be honest: AFAIK what you want is impossible to do in a sensible way for a wide variety of reasons.

It makes 0 sense to discussion the packaging implications (let alone targeting specifically only pip) before the runtime implications are clear - there is no point in having a thread about what pip should be doing in this regard.

A few more sticks to throw your way:

What if you have multiple packages, some of them depending on numpy<2.0, some of them depending on numpy>=2.0? They are all using the import numpy style - how could this ever work correctly?

If you use the import numpy@version syntax, are you marrying yourself to that specific version? That is conceptually incompatible with the way packing is currently supposed to work.

Who is responsible for tracking this? The extension? The Interpreter? What about pure-python libraries? Does this apply to all objects? If the Interpreter, how does it know about different libraries and their inter-version compatibilities? If the library, how much boilerplate is it, how would that ever work?

And I am sure there are hundreds of further edge cases that need to be considered. None of them concern pip in any meaningful way.

3 Likes
12

If version numbers are hard corded into the import statement then you can almost never update your dependencies, nor can two libraries share objects between each other (without endless warnings under your proposal) unless they exactly agree on the versions of all the dependencies and transative dependencies.

But more importantly, changing the import syntax requires changing Python, so this stops being a packaging issue, and you must propose exactly how this works to the wider Python community (i.e. the ideas category).

2 Likes
13

Welcome, and thanks for contributing!

As it happens, this is the old way packages used to be installed, until we found it wasn’t manageable/safe for users. You need to go back to distutils/setuptools and easy_install (and the --single-version-externally-managed flag, which eventually became the default) to find it, so perhaps add those keywords to your research.

The problems with multiple version installs have largely been brought up already, and stem from “Python only uses the module name as the runtime identifier”, and so two different modules with the same name can’t both be loaded (normally).

The typical workaround these days is vendoring, where you copy and rename the package you want to be independent from all the others, and include it in your own package. Renaming the root module is sufficient to satisfy the runtime, and well-written libraries don’t rely on their own name (using relative imports) to make this easier to support.

8 Likes
14

So apparently, this thing was discussed in the past, and all your arguments seem valid too. I was like, why didn’t no one think of this, and now, I’ve learnt so many new things from these discussions, about the past workings and stuff.

My insights about this problem comes from a very different background. To be more specific, microsoft’s vcpkg package manager, which does the versioned side by side installs (atleast the pip part). And the language supporting multiple versions via pinning, it’s a “shower thought” which seemed legit to me.

I saw the tip of the iceberg and judged too early. But I don’t feel bad either. Atleast I learnt something. I’ll come back when I get another idea, on this topic, or something different.

9 Likes
15

Yes that is not a new idea in Python community. Armin also wrote about this: Multiversion Python Thoughts | Armin Ronacher's Thoughts and Writings

16

Did any other language implement such a thing satisfactorily by the way?

3 Likes
17

Depends on what you mean by “satisfactorily”. The JavaScript ecosystem has a couple ways to do this that I know of.

One option is to install using explicit aliases, so that you install the package foo multiple times as, say, foo-old, foo-new, etc. When doing this you have to refer to each version using the correct alias when writing out import statements.

The other option is to rely on the way the module search works; each package in node_modules can recursively have its own copy of its dependencies in its own package-local node_modules directory, which will be found first.

So, it works something like this:

  • node_modules/
    • package-a/
      • node_modules/
        • foo/
    • package-b/
      • node_modules/
        • foo/

Where package-a and package-b can each require a different version of foo and each will only see its own version. In theory PEP 582 could have introduced a similar mechanism to Python, but as I understand it, it would require more invasive changes to the Python import process to pull that off.

2 Likes
18

Rust also supports multiple versions at the same time, but doing so requires explicit language considerations: users can’t import from crates that aren’t explicitly provided via their own top-level dependencies, even if provided transitively by other dependencies. This leads to a funny (and relatively rare) pathological situation where a project’s dependency trees have too many copies of nearly-identical versions due to over-aggressive locking.

(It’s also not fully general: any crate that provides non-Rust linkages can’t be mangled, so cargo won’t let you install conflicting copies of tree-sitter, for example. This is analogous to the native linkages in Python extensions problem :slightly_smiling_face:.)

2 Likes