PEP 713: Callable Modules

PEPs
1

PEP 713: Callable Modules

This PEP proposes support for making modules directly callable by defining a __call__ object in the module’s global namespace, either as a standard function, or an arbitrary callable object.

14 Likes
2

In reference to the second example of how to use this (defining a class and then subsequently assigning the class to __call__), would the following be supported as well?

class __call__:
    pass

This style might be useful if an author of a new library wants to export a single class and is opinionated enough to not want to also give that class a separate, importable name (from hello import __call__ would of course work but any style guide would frown upon it in a way that from hello import Hello might not be).

3

I honestly find calling a module object confusing from the intuitiveness and readability perspective.

My current impression is that this merely allows making import mod a shorthand for from mod import mod, the latter of which is more explicit. The PEP would be better served if it gave examples of what modules it expects this feature can be used in, and what it brings compared to the current approach (if more than a slight shortening of the import line.)

Module-level __getattr__ and __setattr__ have interesting use cases when a module needs to do fancy things as part of its implementation while keeping it as an internal detail. For example: lazy loading of module items. It is an extension of “basic” module objects that still makes the module look and feel like a normal module (something you can look up attributes on). With __call__, I don’t immediately see what new capabilities it brings, and it sounds very different from what I would want to do with a module.

Are there existing modules that are already trying to do this? (To give a point of comparison, Pygments did dynamic lookup in modules long before PEP 562, using a types.ModuleType subclass.)

10 Likes
4

I believe that would work, because it’s in the global namespace as __call__, but for the reasons you mention, I would probably expect/encourage module authors to do something more like this:

class Foo:
    pass

__call__ = Foo
1 Like
5

My current impression is that this merely allows making import mod a shorthand for from mod import mod, the latter of which is more explicit.

More explicit, but also more redundant. Yes, this is mostly for shorthand, specifically for the purpose of DRY when packages/modules have one primary/default purpose.

But also, as mentioned in the PEP, type checkers do not yet support this behavior, making it difficult/impossible to use the shorthand method without triggering errors when running mypy.

With __call__, I don’t immediately see what new capabilities it brings

The primary feature of the PEP is providing an official mechanism (with less boilerplate) for behavior that was already possible, and providing a standard that type checkers can use to accurately support this behavior.

Are there existing modules that are already trying to do this?

I have made multiple “single serving modules” like this already, usually for standalone features that don’t fit well in other existing libraries. I have encountered packages on PyPI in the past that used ModuleType hacks to enable this behavior, but the prevalence of type checkers has pushed these use cases towards requiring the explicit “double mention” just to make the type checkers happy.

The shorthand is less verbose and “ugly” IMO, especially when the primary feature is a single decorator or context manager that matches the name of the library, like the first example given in the Motivation section of the PEP.

For many modules, this could allow users of the “default” behavior to use a friendlier import:

6

Has there been any prior discussion of this feature? It seems rather pointless to me. In particular, the statement in the motivation section:

Many modules have only a single primary interface to their functionality

doesn’t match at all with my experience. I find it pretty rare that a module only exposes a single callable as its interface - and when it does, I see no real problem with the syntax from mymodule import interface.

Has there been any previous support for this proposal? Are there any examples of real-world cases where it would make a significant difference to the usability of a module (note, that’s going to be a hard sell, because as I say, I don’t see from mod import mod as being perceptibly worse than import mod…)

14 Likes
7

+1. I would like to be able to pprint(obj) rather than having to worry about whether it was import pprint or from pprint import pprint. It’s a small feature but a useful one.

11 Likes
8

Has there been any prior discussion of this feature?

I presented this idea as a lightning talk at the Language Summit and got generally positive responses and a suggestion to write the PEP by multiple participants. The general consensus seemed to be that this is a small ergonomic improvement with no obvious downside.

I find it pretty rare that a module only exposes a single callable as its interface

It might not be the majority of cases, but I’ve found it frequently enough over the years. Just in the stdlib, there are a couple of modules with only a single documented callable:

  • array.array
  • fractions.Fraction

But there are several more where I would personally consider it useful to expose the “main” API from each module as a potential “default” callable:

  • configparser.ConfigParser
  • dis.dis
  • enum.Enum
  • fnmatch.fnmatch
  • getopt.getopt
  • glob.glob
  • mmap.mmap
  • queue.Queue

I can also imagine cases where stdlib packages could have had more ergonomic usage with callable modules, and perhaps some better naming. Consider:

import dataclass

@dataclass
class Foo:
    something: str
    many_things: list[str] = dataclass.field(...)

import sqlite

with sqlite(":memory:"):
    ...

Beyond stdlib, looking at the top packages on PyPI, a few also stand out to me as potentially gaining a cleaner import for their primary UX:

  • decorator.decorator
  • filelock.FileLock
  • frozenlist.FrozenList
  • iniconfig.IniConfig
  • tqdm.tqdm
  • wrapt.decorator

As I mentioned earlier in the thread, there are multiple cases where I’ve personally written (or maintained) libraries with a single class or function that serves as the entrypoint to the library, including aiosqlite.connect, diff_match_patch.diff_match_patch, and trailrunner.Trailrunner.

that’s going to be a hard sell, because as I say, I don’t see from mod import mod as being perceptibly worse

Maybe it’s not a huge QoL improvement in a single instance, but is there really a downside to reducing the amount of text, typing, and redundancy in imports? Small papercuts add up over time, and IMO from mod import mod is a papercut that shouldn’t exist if we can solve it with a dozen lines of C code.

Other than “I’m fine with the status quo”, is there an actual concern with the feature? I’m happy to add more of these example use cases to the PEP if the only concern is that the PEP is light on real world examples.

14 Likes
9

In your dataclass example, what happens if I write

help(dataclass)
1 Like
10

dataclass would still be a standard module object, except with an additional __call__ function listed, similar to calling help() on a class that defines its own __call__ method.

# fancy.py

def __call__(value):
    print(value)

Help on module fancy:

NAME
    fancy

FUNCTIONS
    __call__(value)

FILE
    /Users/amethyst/workspace/cpython/fancy.py
11

I see, that makes sense.
I still find the dataclass example (or any other example where a module exports multiple classes/ functions) potentially confusing for someone who doesn’t know about the default __call__ and wants to find out what is being used.
For modules with a single export this should be fine, though.

12

One more example that could benefit from this PEP is tqdm.

2 Likes
13

I’m sure that help() could be extended to understand callable modules and present the default parameters similar to how it presents the constructor for classes.

14

For the sake of this example, suppose I am writing a module that provides itertools.chain. Initially, I only implement it as a generator and all is fine. People can do import chain; chain(...).

Now, I want to add the static method chain.from_iterable. I turn the generator into a class and implement the method. But this does not work anymore for my callers. They could be confused seeing that they are using chain fine but they cannot use chain.from_iterable, without realizing that this comes from the import site (a place you wouldn’t usually scan for problems).

15

I’m not sure how this would be a problem. Here’s how the module would look, I think:

# chain.py
def __call__(*iters):
    ...
    ...
    yield stuff
    ...

def from_iterable(iterable):
    return __call__(*iterable)

Or alternatively:

# chain.py
class Chainer:
    def __init__(self, *iters):
        ... as above ...

def from_iterable(iterable):
    return Chainer(*iterable)
def __call__(*iters):
    return Chainer(*iters)

or various others. Static methods off your default import are just methods on your module.

16

I’m not sure how that’s any different from any other problem involving API stability as implementations evolve. This PEP is not suggesting to make all modules callable — only to provide the option for modules to be callable where it makes sense for the use case.

3 Likes
17

Thank you for the PEP! It would be great to have this. Unfortunately, there’s a rather nasty devil hiding in the details: __call__ is a special method, which is looked up on the class rather than the instance.

Here are some tricky questions for the PEP to answer:

  • What will ModuleType.__call__ be? (This is what Python will call, unless the PEP changes the calling protocol itself.)
  • What will mod.__call__ be if mod’s __call__ is not set? (Edit: the PEP does specify that!)
  • What will callable(mod) return?
6 Likes
18

How was this handled for __getattr__?

19

How was this handled for __getattr__?

ModuleType.__getattr__ itself always exists. PEP 562 changed it so that where it would raise AttributeError, it tries to get mod.__dict__['__getattr__'] and calls that if it’s found. (Going through __dict__ ignores the class’s __getattr__ that would be found by normal attribute lookup.)

If you know C, follow along with the implementation:

  • try to get the attribute normally (PyObject_GenericGetAttr)
  • if not found, get mod.__dict__['__getattr__'] (the PyDict_GetItemWithError)
    • if found, call the result (PyObject_CallOneArg)
  • if still not found, spend a lot of lines constructing a nice AttributeError
2 Likes
20

Ahh, fair enough. So __call__ isn’t strictly parallel, and adding this functionality would effectively require that ModuleType grow a __call__ method that checks the module’s dictionary for something callable.

1 Like