Module: implementation and representation

Core Development
1

It is not so uncommon that functions and classes are implemented in one module, and then exposed to user in other module. This happens with extension modules (e.g. _locale and _pickle), but there are also examples when the pure Python implementation is moved to other module (e.g. _collections_abc and _pydecimal) or when they are defined in submodules but exposed for public in the main pockage namespace (e.g. multiprocessing and asyncio).

There are two issues with this.

  1. Pickling. Where the function or the class are implemented is an implementation detail. They are pickled by name, and we want to use the public name, independent from implementation.
  2. Pydoc. If __all__ is not defined, it only shows non-underscored names defined (not imported) in the specified module. Except for builtins – I made a mistake of applying this rule for builtins – and many names defined in extension modules disappeared from the pydoc output, but this was unnoticed until I fixed this error many months later.

To solve the pydoc issue, we can define __all__, but there are public names which should not be included in __all__ (should not be imported by the star-import) because they conflict with builtins or module names, or just not so useful fro the start-import, but which should be shown by pydoc. And thsi would not help the first issue.

Common solution of the first issue – use the fake module name in the implementation module. Set __name__ = "fakename" in the module code. This has a bad side effect – inspect.getsourcelines() can no longer find the source of the class. In worst case it can return incorrect result. To work around this in the stdlib I used ugly hacks (for example, by completely replacing the interface module with the implementation module), but these hacks should be applied for each particular case, details differ from case to case, and this in reliable, it would not work in more complex cases.

So, I have an idea which can help to solve both issues. Since one implementation module only provides content for one interface module, but different implementation module can content for the same interface module (e.g. asyncio), define some module global, for example __implements__, specifying the name of the interface module. For example, _pydecimal and _decimal should set it to "decimal". Then pickle and pydoc will look in sys.modules[obj.__module__].__implements__, not in obj.__module__, if it is defined.

The question is about the name of this attribute. I have no preferences. Is __implements__ good? Would not it conflict with the name used in third-party code? Maybe other name is used for similar purpose in third-party code?

4 Likes
2

Is this not a place where __qualname__ has subtly different semantics from __name__ that are useful here?

You can’t use __qualname__ to look up a type, but you can use it to look up sources (factoring in that you may need to ignore parts of the name).

Pickle can and should override serialization to point to the public name, and then support unpickling back to the native type - Yury did this recently for UUID (maybe still in PR?).

By definition, all __dunder__ names are ours, so they can’t conflict with third-party code. But I don’t think we need a new marker for this just for inspect.

1 Like
3

Perhaps some decorator @implements(<source>) would both be easier to use in some API, and not need to add some dunder attribute to an already growing list (sure they are reserved, but if you want to filter for them or such, it’s usually best to make a list of them).

The decorator would take any n > 1 number of strings, which are equal to the __qualname__ of the object they expose. Those could then be saved in some registry, where tools like pydoc and inspect can find it. Therefore we would be able to do @implements("_module1.object_name", "_module2.object_name2").

This approach sadly makes either some import, or a new builtin (which I suppose is very unlikely) necessary, which is not too much of an issue though (IMO). Perhaps such a decorator could be added to typing, which would enable type checkers to trace the objects root (although this is not only typing related).

But yeah, IMO an API for this is definitely needed, I wished for something like this (mainly for documentation and typing purposes) for some while now.