Best practices for managing dynamic imports in plugin-based architecture?

Python Help
1

Hi everyone,

I’m working on a plugin-based CLI tool where new plugins can be added at runtime. Right now, I’m using importlib to dynamically load modules based on folder structure.

I’d love to hear how others are managing dynamic imports—especially around:

  • Safety and validation
  • Managing dependencies per plugin
  • Keeping plugin discovery clean

Are there any preferred community patterns or maybe lesser-known libraries that help with this?

Thanks in advance!

2

I think the best plugin system I’ve ever worked with is pytest’s although not really for any of the factors you’re asking about.

  • Its plugin API is generalised enough that you can usually do what you intend to do (a lot of plugin systems feel like they were written with a couple of first party plugins in mind and never quite work out whenever you want to do something the author hadn’t already thought of)
  • It does a good job of only exposing objects that you’re allowed to safely manipulate
  • It has first class support for testing the plugin
  • It has a dedicated exception type which can be used by plugins to send user errors to the user without distracting tracebacks
  • It picked a naming convention early on to aid user’s discovery
  • It’s then able to use that naming scheme to generate a discovery page

Regarding detecting installed plugins, some use entry points, some do import globbing (using pkgutil.iter_modules()), some don’t do any discovery and instead require that the user list the import names of the plugins they want loaded in some configuration file (e.g. coverage.py’s .coveragerc). Avoiding entry points means that users can write (or at least proof of concept) plugins as ad-hoc modules without fully packaging/installing them. Requiring the explicit list of plugins in a configuration file can make debugging/triaging bug reports easier since a user can’t be unaware of what plugins they have loaded. But I’ve not seen anything to give me too strong an opinion on the matter.

I think these are both issues that you can’t really solve. I suggest wrapping the loading of each plugin in some defensive try/except/ the following error came from x plugin so that the user at least has good diagnostics (or you can maybe downgrade errors in plugins to warnings) if either goes wrong. Anything beyond that is out of your control.

3 Likes
3

The plugin system used by pytest is called pluggy and it’s very well designed and easy to use… but it is not designed for plugins to registered/added at runtime. In my experience it relies on setuptools hook registrations for plugin discovery.

2 Likes
4

Thanks Brénainn and Kevin for the thoughtful breakdowns, that’s super helpful. I hadn’t considered looking deeper into pytest/pluggy as a model, but it sounds like there’s a lot of solid design lessons there. The points about discovery methods and defensive loading especially give me a clearer direction for handling plugins in my setup. Really appreciate the guidance :folded_hands: