Suppose that there is a main package that dynamically loads and imports 3rd-party plugin packages. The standard way is to use package entrypoints to enumerate and import the 3rd-party packages inside the same virtualenv (just like how pytest does).
Currently, there is no way to isolate the dependency trees of the main program and the plugin packages as they reside in the same virtualenv “namespace”.
I’d like to design a plugin subsystem that uses an explicitly defined communication protocol instead of direct module imports, where the update cycles and dependency trees of the main program and plugin packages can be completely decoupled.
Some prior discussions about multi-version package support, possibly related:
I think it would be nice to revisit the topic as now we have subinterpreters, which would simplify many rough edges in the design and implementation.
Proposed (rough) design:
Allow mapping a separate virtualenv when spawning a new subinterpreter.
Each virtualenv and subinterpreter just works like before. (No multi-version stuff from prior discussions!)
The hypothetical plugin subsystem should use an explicit communication protocol based on object interchange schemes of the Python subinterpreter.
It also should manage the plugin packages and their virtualenvs by its own.
(optional) It would be much nicer if we could have a more native-feeling direct function/module calls to the fellow subinterpreters.
Desired effects:
Decouple the update cycle of the main program and its plugins.
Allow using incompatible dependencies in plugin packages.
Keep the lifecycle of the main program and plugins in sync.
Use (hopefully) simpler, faster communication means.
When using scie and Pantsbuild, the self-contained self-executable bootstrapper implicitly creates an isolated PEX virtualenv. Currently it is difficult to “load-and-import” external wheel files into it. With subinterpreter-specific venvs, I could just create a new venv to auto-install the plugin wheels and fire up a new subinterpreter running the plugin.
Potential technical issues:
Loading multiple versions of native dependencies in a single Python process may cause link issues, even though the importing subinterpreters are different.
Would there be any way to avoid this issue by “namespacing” symbols on the fly or using some thread-specifics?
Possible objection:
Why not just running two separate Python server instances, installed in separate virtualenvs?
It’s to reduce the deployment and operation burden (see ‘Desired effects’). The plugins are passively invoked by the main program only and may not have any other public service interfaces.
It would be more intuitive to write communication protocols once we have a simpler, faster way to interchange (serialized) Python objects between subinterpreters.
I don’t think you need virtual environments for this. All you need is to manipulate sys.path to put the appropriate set of packages onto the import path for each subinterpreter (as part of subinterpreter startup).
It should be possible to do this entirely in user code, so I suggest writing a package to manage subinterpreters the way you describe. Even a prototype would be useful in determining the viability of the approach.
This approach assumes that the plugin wheels have a fully compatible dependency set with the main program, as it just “overrides” the sys.path with the discovered wheel files.
I want to have a better way to handle the dependencies of the target wheel, to decouple the dependency trees.
For instance, the plugin may no longer work if the main program’s dependency gets upgraded or vice versa.
For the library symbol conflict issue, could the auditwheel tool help as it transforms the shared library symbols into a statically linked ones, since it’s pretty much a standard procedure to build production wheels?
In addition to my prior reply, I haven’t tried prototyping with actual subinterpreters (_interpreters module in Python 3.13+ and _xxsubinterpreters in Python 3.12) and manual sys.path manipulation.
It would be also interesting to combine with PEX to automate the venv bootstrapping of the plugin dependency set.
It should be possible, but there are some tricky details with native extensions, as we discussed on the linked issue. Though, those same issues already exist, they would just be exacerbated by having completely different environments on the same process.
Statically linking dynamic library symbols into extensions (are you sure this is what auditwheel does?) does not work when you are working with arbitrary wheels, I think, as you don’t know if multiple extensions rely on shared (global) data, so you have to rewrite them to make them unique.
On top of rewriting the symbol names, you probably also need to rewrite DT_SONAME (ELF) and LC_ID_DYLIB (Mach-O), to ensure two environments trying to load a dynamic library with the same name will not get the same handle.
That said, rewriting the symbols will break code that loads them dynamically at runtime (dlopen/ctypes). So… it’s tricky. If you can limit your supported platforms to ones whose linker property supports RTLD_LOCAL, then it’s easier, you only need to deal with the special global symbols, mitigating the issue in most cases, but still not fixing it completely.
Perhaps you could write a LD_PRELOAD that routes symbols at runtime based on the context… eh?
But well, I don’t know, my brain is frying already, so take all this with a grain of salt…
I’m afraid if I continue I will end up writing a custom linker
This is a very minimal PoC without interfacing of the plugin via interpreter queues.
pex.pex_bootstrapper.bootstrap_pex_env() automatically extracts and populates a hermetic virtualenv for the packaged PEX archive in a subinterpreter, without affecting the main interpreter.
$ uv run hello.py
msgpack is available in the plugin as expected
msgpack is not available in the main program as expected
Yeah, it seems to be difficult to resolve the conflicting symbol issue completely.
Then, could there be a way to verify potential conflicts for a given set of dependency trees?
At least, we could emit warnings or errors in such cases.
