Custom importer for overriding a buggy vendored transitive dependency

Python Help
1

I maintain Library A, which intends to be usable with Python 3.9 and later, notably including Python 3.12.

Library A depends on Library B. Library B officially still intends to be usable with Python 2, and bundles a copy of six to facilitate this. However, its copy of six is old and incompatible with Python 3.12. Library B is large and complicated, so I don’t want to bundle it myself, and its maintainer is currently unresponsive.

I therefore need to monkey-patch B somehow so that when it tries to load its vendored copy of six it gets a current version that’s compatible with 3.12. I think this ought to be possible using a custom importer, but I do not understand how to write custom importers, even after reading the documentation carefully, and I especially don’t understand how to write a custom importer that overrides the module selected for a particular import. To make matters more complicated, Library B refers to its bundled six using both relative and absolute imports.

(Since Library A is not compatible with Python 2, it doesn’t matter if B’s usability with Python 2 is lost when B is loaded as a dependency of A. It is also fine if a package that loads both A and B needs to load A first in order to work with 3.12.)

Where can I find advice on how to write this kind of custom importer?

1 Like
2

As long as there is only one copy of six, probably in a path like libB.vendor.six or something like that, you can just overwrite it in sys.modules before importing libB:

import six # The up-to-date version
import sys
sys.modules["libB.vendor.six"] = six
import libB

No custom importer required. If you get weird results (i.e. if libB is doing something really weird like monkey patching six), you might need to make a copy of the six module object before assigning it or reimporting it using one of the recipes for explicitly importing a module file.

2 Likes
3

Thanks! I’ll give that a try. I don’t think it’s doing anything weird but we’ll see.

4

The simple thing you suggest doesn’t work because six itself does an unusual thing: it adds its own custom importer to sys.meta_path that takes responsibility for the six.moves namespace, mounting that namespace at (whatever its module’s __name__ is) + ‘.moves’. Thus, importing six as six and then poking it into sys.modules as libB.vendor.six fails to make libB.vendor.six.moves available.

However, I was able to get around that using importlib instead of an ordinary import:

spec_six_new = importlib.util.spec_from_file_location(
    "libB.vendor.six",
    six.__file__,
)
six_new = importlib.util.module_from_spec(spec_six_new)
sys.modules["libB.vendor.six"] = six_new
spec_six_new.loader.exec_module(six_new)

This loads a second copy of the original six module with its __name__ adjusted, so its custom importer supplies the libB.vendor.six.moves namespace.

Thanks again for the help. I would never have thought of this approach.

2 Likes
5

Aha yep, wasn’t aware of six.moves. But the solution you showed is one of “recipes for explicitly importing a module file”.

Glad to help.