How to implement namespace packages (as in PEP 420) via import hooks?

Python Help
1

In a previous post in the context of PEP 660 (editable installs), @pf_moore and I were discussing that currently it is difficult to handle namespace packages using a MetaPathFinder, the main challenge being dynamic path computation.
After testing some PoCs during the weekend, I think I manage to find 2 possible implementations that would work for this scenario:

  • (A) Consider _NamespacePath to be an implementation detail and re-implement another “path class” that emulates the behaviour described in 5. The import system — Python 3.12.1 documentation :

    Namespace packages do not use an ordinary list for their path attribute. They instead use a custom iterable type which will automatically perform a new search for package portions on the next import attempt within that package if the path of their parent package (or sys.path for a top level package) changes.

    Comments:

    • This approach requires re-implementing a lot of stuff. Even when relying on pkgutil.extend_path (as I did in my PoC), a lot of code is still needed to support path entry hooks.
    • It also requires the MetaPathFinder in the beginning of sys.meta_path, which is something I would avoid if possible.

  • (B) Instead of using a MetaPathFinder, create a custom PathEntryFinder and add a “bogus” entry to sys.path just to trigger it. This way the existing importlib.machinery.PathFinder will take care of creating the _NamespacePath under the hood.
    Comments:

    • This approach seems more straightforward than (A), but it still requires a “bogus” entry to sys.path. It would be nice to be able to implement finders for namespaces without this correspondence, to align with the definition in Python docs:

      Namespace packages may or may not correspond directly to objects on the file system; they may be virtual modules that have no concrete representation.

The code for both PoCs is available at How to implement Namespace packages via `MetaPathFinder`? · GitHub.

I would like to use this discussion thread to collect some feedback and ask the following questions:

  1. The assumption in (A) is that the expression “a custom iterable type” to refer to any iterable type that fulfils the requirements, not a specific class of the stdlib. Is this a fair assumption or is there any risks that internally Python checks for _NamespacePath?
  2. Is pkgutil (and particularly pkgutil.extend_path) still considered first class citizen of the standard library or is it better to avoid it when writing new code?
  3. Is the behaviour observed in (B), by any chance, accidental? For a PEP 660 implementation it would be nice if this behaviour could be considered stable.
  4. Is there any conceptual problems with approaches (A) and (B)?
  5. Is there any recommended/other way of implementing this feature? (The main use case I am interested is PEP 660).
2

I opened a new issue in pypa/packaging-problems: Uncertainty on how to implement namespace packages (as in PEP 420) via import hooks · Issue #588 · pypa/packaging-problems · GitHub.

3

I think you probably need to open an issue against CPython asking for clarification on how to support namespace packages using importlib hooks. I don’t think the original PEP for implicit namespace packages, nor the Python documentation, addresses this question, and while it’s possible that the CPython core devs might say “sorry, this is a question of how do I use Python, we don’t provide help in writing your own code”, I think it’s worth asking.

To be clear, I don’t personally (either as the author of editables or as a core python developer) have any idea how to implement this, short of somehow reimplementing the whole of the namespace package support (and even that might not support having a namespace path supplied partly from a hook and partly from a filesystem directory).

If it isn’t possible to support implicit namespace packages using an import hook, I consider this a flaw in the way implicit namespace packages are defined/implemented, and therefore something that needs to be addressed in core python. However, I think it’s possible that it might be considered as something that won’t get fixed. I’ve never used namespace packages, explicit or implicit, so I have no feel for the importance of this “dynamic path search” feature - clearly there was some need for it when the original PEP was written (as otherwise why add the complexity?) but I don’t know any details.

1 Like
4

Thank you very much Paul. I opened Clarification about how to implement namespace packages (as in PEP 420) via import hooks · Issue #92054 · python/cpython · GitHub.