Performance implications of this PEP are minimal, as it defines a new interface. Calling a module would trigger a lookup for the name
__call__on a module object.
Feature emulation:
import module
function = module.__call__ # 1 lookup
function()
Suggested feature:
import module
module() # 1 lookup/call
How to avoid 1 lookup/call?
Please guide me if I have missed something.
In case if this PEP gets passed, would any change be foreseen for stdlib callables that shadow their module names, like copy.copy, pprint.pprint or datetime.datetime? Would it be possible to just import datetime and have its functionality to be identical to that of from datetime import datetime?
As mentioned in the PEP, this is the right way to make a callable object. A module is essentially an object.
class FancyModule(types.ModuleType):
def __call__(self):
return
sys.modules[__name__].__class__ = FancyModule
Everything works:
import fancy
print(fancy()) # None
print(isinstance(fancy, types.ModuleType)) # True
print(callable(fancy)) # True
It boils down to type checkers not supporting callable objects, because they don’t treat a module as an object?
That will have to be discussed on a case-by-case basis. For datetime, we could make datetime.__call__ return a datetime instance, but then datetime.datetime’s classmethods still wouldn’t work, and isinstance(dt, datetime) also wouldn’t pass, so it might actually make things more confusing.
From a Steering Council perspective, it doesn’t seem like there is actual consensus on this issue.
I’m seeing what I’ll call “obvious” desire to alleviate past library/module/package naming mistakes that just make some things unnecessarily awkward to use (pprint.pprint, etc.). But this PEP isn’t suggesting we change those to be callable. Which could confusing people even more when they switch back and forth between Python versions if we did, so it isn’t clear that we should. Given a time machine we’d likely have just chosen more appropriate namespace layout 20-30 years ago there without the repetition.
But the concerns about this one such as what does isinstance(when, callable_module_whos_call_returns_a_new_instance) do vs people’s confusion of what they expect when they simply have a class (aka type) instead of a module and how to make callable() work properly along with existing (somewhat hacky but well defined) way to make modules callable today via the __class__ assignment… lead us to conclude:
We aren’t seeing a strong argument in favor here, even though there are all times when we wish we could’ve had this as it appears simple at first glance. In reality it doesn’t seem so simple.
What next? As is, we’re not ready to accept this. If no seemingly strong consensus can be reached here, that’ll wind up as a decision to reject from the SC.
part if the problem with the datetime example is that datetime.datetime is a class, with its own special behavior, and not just a function
you can define the __isinstance__ method of datetime
We aren’t seeing a strong argument in favor here
Considering the questions:
In terms of teaching, I think this might be easier to teach the concept of “calling” something if modules - like virtually every other object - could be callable.
In terms of maintenance, I don’t think this is a big lift considering this is just syntactic sugar for something that already is possible. It might be an easier heuristic for mypy to use in type checking than the current way it must be implemented if modules want this feature. In fact this also mitigate the isinstance concern, because if mypy implemented this heuristic it could detect incorrectly using modules as classes.
So, how much does this benefit people who write Python? Well, I would have liked to use it in: Allow the Timerit class to be used without being explicitly imported by kalekundert · Pull Request #29 · Erotemic/timerit · GitHub instead of the current boilerplate. To me it makes so much sense how __call__ would work, and I wish I could use that instead of the boilerplate that is currently necessary.
As a 3rd party module developer, I would argue in favor of this PEP. I think it’s a good idea to avoid adding it to the stdlib (for now). I also think it would be useful especially if you are teaching the more advanced concept of module dunder methods. More correspondence between what you can do with modules and what you can do with a class feels right and more “complete”, because a module is a class.
While every object could be callable (including modules, already), the everyday objects that beginners are using (number, strings, dictionaries, lists, sets) are not. I think this makes teaching harder, not easier:
That seems more confusing to me, not less.
It seems like the primary motivation is “I want to import a function from a module that only has one”. I think it’d be more useful (but more of a lift) if that had its own syntax (like def import time, but I’m not seriously suggesting that), so that the interpreter could enforce a rule like “there’s only one public function” and it’s clear what the author intended.
the everyday objects that beginners are using (number, strings, dictionaries, lists, sets) are not. I think this makes teaching harder, not easier:
Like __getattr__ and other dunder methods, __call__ is not a basic topic. Most modules will not be callable. The way you teach the basics doesn’t change. But this does make teaching the dunder methods easier.
It seems like the primary motivation is “I want to import a function from a module that only has one”. I think it’d be more useful (but more of a lift) if that had its own syntax (like def import time, but I’m not seriously suggesting that), so that the interpreter could enforce a rule like “there’s only one public function” and it’s clear what the author intended.
The motivation I have is interactive use. There is some discussion of this in the linked timerit PR. A callable module gives maintainers like me the option of establishing more concise patterns of use. Of course it is our responsibility to document this. The stdlib is not going to make its modules callable, so the only concern is that new module developers could use this to make their code more concise.
I would not use the word “boilerplate” here. Boilerplate code refers to a situation where you repeat the same (or very similar) code many times in order to do things that are done frequently. Making a module callable is not exactly something that would ever be done very frequently though: it can happen at most once per module but actually most modules will not do this at all. Needing to write a few lines of code to do something unusual is not “boilerplate”: it’s just code.
If this PEP is accepted then I will still consider that making a module callable is an anti-pattern that should be avoided/rejected. It will just confuse people and mean another variation on how a module exposes its API that would need to be remembered. Just the existence of some packages that have this sort of design will add to confusion for users of all packages including those that do not have callable modules (why can’t I call this module?). If this PEP is ever approved then I can guarantee that I will end up arguing in a PR about why it is still a bad idea to make modules callable regardless of the idea being seemingly blessed by an approved PEP. The PEP should be rejected simply because accepting it gives the impression that having callable modules is considered reasonable in the first place.
I can think of only one situation where it might make sense to make a module callable which is if there is a redesign that introduces a submodule with the same name as a prior function so e.g. you had module mod with function func that downstream code uses like
from mod import func
Now if you decide that this design was a mistake and actually you want a submodule mod.func then you might consider making the module object callable like:
class DeprecatedCall(sys.modules[__name__].__class__):
def __call__(self, *args, **kwargs):
warn("deprecated: Use from mod.func import new_func instead")
return new_func(*args, **kwargs)
sys.modules[__name__].__class__ = DeprecatedCall
This is not a common thing that should be done and so it does not need to be made any easier (unless perhaps by a library that helps with making deprecation warnings).
I would not use the word “boilerplate” here.
I stand by my use of the word “boilerplate”. This stack overflow post demonstrates that this non-varying pattern has been used multiple times. It doesn’t have to be a daily activity to be boilerplate.
The idea that you shouldn’t do this in most circumstances (which I agree with) is not a strong argument against the PEP. I assert that you shouldn’t use __getattr__ in most cases as well. But it’s nice that it’s there when you want it.
accepting it gives the impression that having callable modules is considered reasonable in the first place.
This assertion is not supported. You might not like the pattern - just like I don’t like decorators - but that doesn’t mean it’s an unreasonable pattern.
There are several reasonable use cases. I have an RSI, and using callable modules can allow for more concise usage patterns. Reducing my typing is health consideration. Now, you might counter: should we allow for all syntactic sugar under the sun? No, because then they all have to be documented. However, __call__ is already an established pattern in Python, and it’s a special case that it doesn’t work for modules like it does for classes, so it shouldn’t be considered something new, wild, or unreasonable.
mean another variation on how a module exposes its API that would need to be remembered.
It’s no different than adding any other complexity to an API. It’s the API deveper’s responsibility to ensure all paths are documented.
Just the existence of some packages that have this sort of design will add to confusion for users of all packages including those that do not have callable modules (why can’t I call this module?).
The same question currently exists for objects. Why can I call this object but not that one? The answer is: “well it’s time to learn about __call__”.
If I understand correctly then what you are referring to here is the difference between needing to type
from mod import func
func()
as compared to
import mod
mod()
and specifically you mean when typing in an interactive context. Personally I use ipython for interactive work and it has many ways to reduce typing so e.g. if I regularly type from mod import func then I can do f, <up>, <enter> to repeat the previously typed command that begins with f. Alternatively I don’t use this but presumably there are ways to configure ipython automatically to import things that you want to use regularly. I actually usually use isympy rather than ipython which is just a wrapper that imports and defines a few extra things that are useful when using SymPy (to reduce typing for those of us who use it regularly).
Basically my point is that there are many ways to reduce typing in an interactive context that do not require bending the basic semantics of imports, modules, functions etc.
I disagree with this strongly. The usual expected semantics are that there are modules, classes, functions and objects. Users of Python do need to understand the distinction between these concepts clearly in order to use them correctly. You need to be clear about which is which not just in any import statement but also in any lines of code that follow on from that import statement. It is already problematic that some Python packages confuse what is a function and what is a class (e.g. numpy.array vs numpy.ndarray) but blurring the lines between modules and functions is much worse because their usage is starkly different in terms of import.
Yes, on some level everything is an object that may or may not have a __call__ method but no actually the vast majority of users of Python never need to know that in order to use Python correctly. What they do need to know is that you either do this:
from mod import func
func()
or otherwise you do this:
import mod
mod.func()
The fact that this is ambiguous is unfortunate:
from a.b.c import d
Is d a module or a function? It is an unfortunate ambiguity but at least attempting to call a module should be a TypeError so that we are clear about the distinction: modules are not callable.
A lot of your arguments are great for why you probably don’t want to make your module callable from a design persepctive, but only one of them is a valid argument against the PEP itself.
modules are not callable.
They could be, even in the current version of Python.
The usual expected semantics are that there are modules, classes, functions and objects.
And that doesn’t change, and every one of these is an object. From the datamodel: “All data in a Python program is represented by objects or by relations between objects.”
The fact that this has been true doesn’t change how you introduce the major flavors of objects to people. You introduce them as separate things, and later as they learn more Python they learn that these things are all unified.
You need to be clear about which is which not just in any
importstatement but also in any lines of code that follow on from that import statement
You should, but you don’t need to. Python is a dynamically typed language. For better or worse from mod import * is a thing.
If I understand correctly…
Correct. I use IPython too. It’s great. I also have lots of custom macros and utilties to help with my RSI.
I also want the ability to make my modules callable, which is good because I can do that right now - with or without this PEP. We don’t need to debate if I should or not. A lot of the discussion here is losing sight of that, and I think it makes it seem like there are more arguments against this PEP than there really are.
What should be debated is: should modules get syntactic sugar to allow them to access the dunder methods that are already available to them in a roundabout way? Should modules have an conceptually ellegant way to access their internal objectness? I think the answer to this is a resounding yes. Modules should be able to define __call__ and __iter__ with the same ease that you can do with any other object, but only __call__ is in the scope of this PEP, so let’s focus on that.
The strongest argument I’ve heard against this PEP is that this will “bless” the practice of defining __call__ and cause module developers to over-use the capability. I think that will happen to some extent, but I also think it won’t happen often enough to cause real problems. There are only a few cases where it is a good idea to have a callable module. As you said: “most modules will not do this at all”.
I think this argument is entirely countered by the fact that it removes the disconnect of why a(b) doesn’t find __call__ when a is a module, but it does when a is a class-instance, even though they share the same data model.
Why do modules need a second way to do what they can already do? Is it a good idea to blur the distinction between different concepts (module and function)? What does it gain us to do so?
Your assertion that this is an “elegant way to access a module’s internal objectness” is entirely subjective, and somewhat arbitrary. We don’t have an “elegant way” to access an integer’s “internal objectness”. What makes that different from this case?
There’s the much simpler argument that there’s no need for this feature, and far more important things for the people working on Python to focus their energies on.
And you say yourself “I also think it won’t happen often enough to cause real problems”. So you’re arguing for a feature that you don’t think should be used often, where there’s already a way of doing it for people who care enough to want to. It seems to me that your argument for the feature is also extremely weak.
Ultimately, as with any proposal, the “status quo wins” rule applies. It’s not particularly important that you counter the arguments against the feature. Or even that people offer arguments against it. What you need to do is present compelling arguments for the feature. So far it seems you’re arguing
Unless I’ve missed something, that’s not very compelling…
I think that this is the major weakness of the PEP. Early on it was conflated with a separate proposal (that I would support) to allow __setattr__ for modules. The reason I would like __setattr__ is to help users avoid bugs like e.g.:
There are probably other examples where defining methods on modules could be useful and perhaps a general mechanism for defining relevant methods on modules could be valuable to satisfy a variety of use cases. I might support a PEP that proposed something like that.
PEP 713 though focuses only on __call__ which to me is a method that I don’t really want anyone to define on a module.
FWIW the proposal does not entice me, and makes me worried about two kind of things:
we would be able to do import datetime then dt = datetime(y, M, d, h, m, s) but not datetime.now() or isinstance(dt, datetime): this feels incomplete and confusing
«callable modules» are already a concept: python3 -m module (usually named «calling modules as scripts»); this new capability would be something different with a smilar name
We don’t have an “elegant way” to access an integer’s “internal objectness”. What makes that different from this case?
There is an ellegant way to access an integer’s internal objectness.
In [1]: class foo(int):
...: def __call__(self):
...: return "you called me"
...:
In [3]: foo(1)()
Out[3]: 'you called me'
There’s the much simpler argument that there’s no need for this feature , and far more important things for the people working on Python to focus their energies on.
I can mostly buy this. But I’ll add that supporting this is probably an easier feature to add.
Unless I’ve missed something
Your bullets are all correct points (although I would say that I’m not the only person who can think of compelling use cases where they would want it, and I want it enough to where I’m willing to spend my time and engery arguing for it), but it doesn’t contain what I think the strongest argument is one of corresponence with object-y things:
The data model when defining a class should be as easilly available when defining a module through dunder method of the same name.
I think @oscarbenjamin alludes to an important point when discussing __setattr__. A flaw I see in this proposal its scope is too narrow. I’m only in favor of __call__ because it moves closer to what I see as the desirable end state: All of the dunder methods that are available for objects and make syntatic sense (i.e. not __init__ because how do you even invoke it?) in terms of operators should be respected by modules. Adding all of them at once, rather than incrementally would allieviate the concern of people thinking that certain operators are now “blessed”. The only thing that is “blessed” is that dunder method are a good idea and giving users the option to hook into Python’s duck typing system concicesly is a power and responsibility that they are in control of.
@merwok Making stdlib modules callable is not part of this proposal. But “callable modules” already being an existing concept is a real issue. But if this PEP was superceded by something that unified the dunder methods when defining classes and the dunder methods that define modules, then that would be less of an issue.
I’m personally not convinced by the “unification of classes and modules” angle, despite the language data model ultimately using objects for everything. In a Smalltalk-like language, as I understand it, everything being an object is a core part of the language from top to bottom and integral to a good mental model of it. In Python, everything being an object feels more like an implementation detail. Modules being represented by objects at runtime is not an important part of their conceptual model. That is of course subjective, I admit it’s not a concrete reason - but I think Python already has a well-established separation of concepts, with different syntax and use cases for each, so it’s not compelling to me to increase overlap between them.
Besides, what’s that quote, “make simple things easy and complex things possible”? Based on that stack overflow link earlier subclassing ModuleType is already straightforward - it’s already possible to have a class masquerade as a module just as a class can masquerade as a function (or both).
That makes an int subclass callable. We already have the same mechanism for making module subclasses callable. What we don’t have is a “literal” form for those subclasses - just like we don’t have a literal form for subclasses of int, or for that matter any other user defined type.
So the point becomes - we don’t have user defined literals for any other subtype of a built in type, why should the module type be any different in this regard?