Allowing Multiple Versions of Same Python Package in PYTHONPATH

TLDR; I wanted to get feedback on a potential feature that may be added to nixpkgs that allows multiple versions of the same python package to be installed in the same PYTHONPATH. This is a general approach that is not specific to nixpkgs and could be used in other package managers. The only nix specific part is the tooling to allow for the building of these specialized packages. All of the materials/demo is in this repo https://github.com/costrouc/python-multiple-versions. Sorry discourse prevented sharing all links… to only 2 so go to the repo for actual html links. I want to clarify I definitely don’t think this is a feature that should be regularly used nor depended on. But for a package manager to provide a consistent place where all packages are “nearly” compatible a trick like this is needed for nixpkgs and possibly others.

Demo of Multiple Python Versions

This is a self contained demo of having multiple versions of a python
package in the same PYTHONPATH. In this demo bizbaz requires flask=0.12.4 and foobar requires flask>=1.0. It requires nix (sorry no windows support in nix). This
idea is not nix specific but would rely on package managers/builds to
allow for multiple versions.

$ nix-shell
...
[nix-shell:~/p/python-multiple-versions]$ python
Python 3.7.4 (default, Jul  8 2019, 18:31:06) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import foobar; foobar.foobar()
I am using flask version 1.0.3
>>> import bizbaz; bizbaz.bizbaz()
I am using flask version 0.12.4
>>> quit()
$ echo $PYTHONPATH
...:/nix/store/f3j11lk2m8ddw2j2axvcdfc2al2bk98c-flask-0.12.4/lib/python3.7/site-packages:.../nix/store/wv42si07c8wd64ravd4va4kh4j7prwlk-python3.7-Flask-1.0.3/lib/python3.7/site-packages:...

Motivation

In nixpkgs we like to have a
single version of each package (preferably latest) with all packages
compatible with one another. Often times it is true that two packages
may be incompatible with one another but if it is a compiled
library/binary we have luxury of rewriting the shared library path
allowing two packages that use different versions of a package to
coexist. In python this philosophy breaks down because all packages
are specified in the global PYTHONPATH. This means that if a package
requires import flask it searches the path for flask and uses the
one that it finds.

For nixpkgs this is troublesome because it prevents all packages from
being compatible with one another.

Examples of Issue

  1. jsonschema. jupyterlab_server
    requires jsonschema >= 3.0.1 and cfn-python-lint did not
    support jsonschema 3 until about a month
    ago. 3.0 was released in February!

  2. Some packages fix the version of a package such that other packages
    in the same PYTHONPATH cannot depend on the latest version. For
    example apache-airflow fixes pendulum ==
    1.4.4. That
    pendulum release is over 1.5 years old and libraries .io reports that 400+ packages depend on pendulum. We cannot let a single package restrict the version of other packages.

How does this work?

I wrote a tool python-rewrite-imports that helps to make multiple versions possible. Lets say that package bizbaz wants an old version of flask==0.12.4 but we have another package foobar that requires the latest version of flask>=1.0. Normally these two packages would be incompatible. In order to do this we:

  1. Create a build of flask for 0.12.4 and install
  2. Use Rope to rewrite all the imports of flask of itself to flask_0_12_4_1pamldmw2y7g and rename the package to flask_0_12_4_1pamldmw2y7g
  3. Rename the dist in site-packages and move the package to flask_0_12_4_1pamldmw2y7g
  4. Rewrite all imports of flask in bizbaz to flask_0_12_4_1pamldmw2y7g

Rewriting all imports is done with Rope a robust python refactoring tool.

Current Limitations

  • Wanting several versions of a package that builds c-extensions
    looks a little hard than rewriting the imports?
  • Suppose package A requires C==1.0.0 and B requires
    C>=1.1. Let’s say that package B calls a method in A with a
    structure built from C>=1.1 and then A proceeds to call its
    package C with that data. This will probably not happen often.
  • Rope does not handle all rewrites currently in
    python 3. Expressions within fstrings are the only example that I
    know of.
  • It is impossible for Rope to handle all import rewrites. For
    example. import flask; globals()[chr(102) + 'lask'].__version__

I believe for the vast majority of packages that require multiple
versions these issues will be rare.

1 Like

Links:

  • nixpkgs a general package manager with 45k+ packages and 5k+ python packages.
  • Rope python refactoring tool used in editors

Crosslink to discussion at Nixpkgs Discourse https://discourse.nixos.org/t/allowing-multiple-versions-of-python-package-in-pythonpath-nixpkgs/3849

Situations that will need to be covered as well:

  1. In a development environment where one works with plain Python as source, one needs to be able to perform an import flask instead of import flask_0_12_4_1pamldmw2y7g. I suppose shims are going to be needed that map to a certain hashed import.
  2. Similarly, dynamic imports will resolve to something like import flask (as already mentioned in “Current limitations”).

My worry is that this will cause tons of bugs to be filed on random packages because of subtle breakage you’re introducing, and package maintainers will have no idea what’s going on. All those “current limitations” are things that real packages do.

If you do this, please print some kind of banner at startup and in exception tracebacks noting that this is a nix-modified python setup and that any bugs should be reported to nix only.

1 Like

Note that setuptools did something like this (but not nearly as invasive, as far as I know, as it didn’t have multiple versions of a package active at the same time), and even that has since been mostly abandoned as a bad idea (as I understand it).

I agree with @njs, this should be considered extremely non-standard, and very definitely “use at your own risk”. Package maintainers should not be expected to support this usage.