I’m not sure how that could be made to work in general:
# foo.py
import types
import sys
class _Foo(types.ModuleType):
pass
sys.modules[__name__].__class__ = _Foo
Then:
>>> import foo
>>> type(foo).__getattribute__ = lambda self, attr: 42
>>> foo.stuff
42
I imagine that the fast path depends on the fact that it isn’t possible to mess about with the builtin module type like this.
I was only seeing a 2.5% difference in attribute access times using a dynamic ModuleType subclass, is this the level of slowdown you are worried about, or are these specific cases where it would be more severe? I think I tested Python modules and native extensions.
The OP of the other thread has timings (maybe it would be better to discuss this there).
Repeating those timings with CPython 3.11 on amd64 Linux I get:
$ cat b.py
x = 1
$ python -m timeit -s 'import b' 'b.x'
20000000 loops, best of 5: 17.2 nsec per loop
$ cat c.py
import sys, types
x = 1
class _Foo(types.ModuleType): pass
sys.modules[__name__].__class__ = _Foo
$ python -m timeit -s 'import c' 'c.x'
5000000 loops, best of 5: 53.3 nsec per loop
(Using CPython 3.11.3 on Linux amd64)
That’s a 3x slowdown for a plain attribute access. Of course if you call a function or something then the overhead would be reduced in relative terms. Here is what it looks like when calling an almost trivial function:
$ cat b.py
def f(x):
return 2*x
$ python -m timeit -s 'import b' 'b.f(2)'
5000000 loops, best of 5: 68.3 nsec per loop
$ cat c.py
import sys, types
def f(x):
return 2*x
class _Foo(types.ModuleType): pass
sys.modules[__name__].__class__ = _Foo
$ python -m timeit -s 'import c' 'c.f(2)'
2000000 loops, best of 5: 116 nsec per loop
Either way I think that changing the class of the module is too much messing around to be something that I would want to do except in a momentary monkey-patch type situation so my motivation for __setattr__ is not primarily about performance. Having a clear way to make module attributes effectively read-only seems like a basic not unreasonable feature to me (much more so than changing the class of a module or making modules callable or adding other dunder methods). Of course every feature may or may not be worth its cost.
Ok, I think I see what’s going on. The attribute lookup specialisation is already disabled if there is a __getattr__() function in the module. With the specialisation, I get a slowdown of about 75% for CPython 3.11 on Intel macOS.
To keep the faster attribute access with module subclasses: Does anything break if PyModule_CheckExact() in specialize.c gets an additional subclass check?
On the other hand, one could also just say no specialisation for “magic” modules with __getattr__() and beyond.
Hmm, I see no difference (or even worse):
$ git diff
diff --git a/Python/specialize.c b/Python/specialize.c
index 33a3c4561c..c989a558c3 100644
--- a/Python/specialize.c
+++ b/Python/specialize.c
@@ -752,7 +752,7 @@ _Py_Specialize_LoadAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name)
SPECIALIZATION_FAIL(LOAD_ATTR, SPEC_FAIL_OTHER);
goto fail;
}
- if (PyModule_CheckExact(owner)) {
+ if (PyModule_Check(owner)) {
if (specialize_module_load_attr(owner, instr, name))
{
goto fail;
$ cat b.py
x = 1
$ cat c.py
import sys, types
x = 1
class _Foo(types.ModuleType): pass
sys.modules[__name__].__class__ = _Foo
$ ./python -m timeit -r11 -s 'import b' 'b.x'
10000000 loops, best of 11: 34.7 nsec per loop
$ ./python -m timeit -r11 -s 'import c' 'c.x'
2000000 loops, best of 11: 121 nsec per loop
There is a further check in specialized.c, plus two others in bytecodes.c and generated_cases.c.h.
Since these seem to be the only uses of PyModule_CheckExact() in cpython, you can for a test replace the definition of PyModule_CheckExact() by PyModule_Check() in moduleobject.h, and then you do get essentially the same attribute lookup performance from module subclasses.
I don’t know if this has ill effects elsewhere, but it seems to work.
% python3 -m timeit -s 'import b' 'b.x'
20000000 loops, best of 5: 14.4 nsec per loop
% python3 -m timeit -s 'import c' 'c.x'
10000000 loops, best of 5: 11.9 nsec per loop
I have played with this a bit more, and I would propose to change the PEP to add more broadly “magic modules” instead of merely callable ones.
If you want to give it a try, here’s a simple proof of concept against current cpython that implements generic call, getattr, and setattr. One could think of other useful methods. It’s debatable where one would put the patching of the magic methods.
diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
index e4fcaa6..62f0f32 100644
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -814,6 +814,19 @@ def _load_backward_compatible(spec):
pass
return module
+
+def _add_magic(module):
+ """Add magic methods to an existing module by changing its type."""
+ magic = {"__call__", "__getattr__", "__setattr__"}
+ module_dict = module.__dict__
+ subclass_dict = {}
+ for name in module_dict.keys() & magic:
+ subclass_dict[name] = staticmethod(module_dict[name])
+ if subclass_dict:
+ cls = module.__class__
+ module.__class__ = type("magic_module", (cls,), subclass_dict)
+
+
def _load_unlocked(spec):
# A helper for direct use by the import system.
if spec.loader is not None:
@@ -845,6 +858,8 @@ def _load_unlocked(spec):
except KeyError:
pass
raise
+ # If module has magic methods, make them magic
+ _add_magic(module)
# Move the module to the end of sys.modules.
# We don't ensure that the import-related module attributes get
# set in the sys.modules replacement case. Such modules are on
diff --git a/Python/bytecodes.c b/Python/bytecodes.c
index 9de0d92..ab948f5 100644
--- a/Python/bytecodes.c
+++ b/Python/bytecodes.c
@@ -1676,7 +1676,7 @@ dummy_func(
}
inst(LOAD_ATTR_MODULE, (unused/1, type_version/2, index/1, unused/5, owner -- res2 if (oparg & 1), res)) {
- DEOPT_IF(!PyModule_CheckExact(owner), LOAD_ATTR);
+ DEOPT_IF(!PyModule_Check(owner), LOAD_ATTR);
PyDictObject *dict = (PyDictObject *)((PyModuleObject *)owner)->md_dict;
assert(dict != NULL);
DEOPT_IF(dict->ma_keys->dk_version != type_version, LOAD_ATTR);
diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h
index 864a4f7..96c764f 100644
--- a/Python/generated_cases.c.h
+++ b/Python/generated_cases.c.h
@@ -2338,7 +2338,7 @@
uint32_t type_version = read_u32(&next_instr[1].cache);
uint16_t index = read_u16(&next_instr[3].cache);
#line 1679 "Python/bytecodes.c"
- DEOPT_IF(!PyModule_CheckExact(owner), LOAD_ATTR);
+ DEOPT_IF(!PyModule_Check(owner), LOAD_ATTR);
PyDictObject *dict = (PyDictObject *)((PyModuleObject *)owner)->md_dict;
assert(dict != NULL);
DEOPT_IF(dict->ma_keys->dk_version != type_version, LOAD_ATTR);
diff --git a/Python/specialize.c b/Python/specialize.c
index 33a3c45..527b454 100644
--- a/Python/specialize.c
+++ b/Python/specialize.c
@@ -752,7 +752,7 @@ _Py_Specialize_LoadAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name)
SPECIALIZATION_FAIL(LOAD_ATTR, SPEC_FAIL_OTHER);
goto fail;
}
- if (PyModule_CheckExact(owner)) {
+ if (PyModule_Check(owner)) {
if (specialize_module_load_attr(owner, instr, name))
{
goto fail;
@@ -925,7 +925,7 @@ _Py_Specialize_StoreAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name)
SPECIALIZATION_FAIL(STORE_ATTR, SPEC_FAIL_OTHER);
goto fail;
}
- if (PyModule_CheckExact(owner)) {
+ if (PyModule_Check(owner)) {
SPECIALIZATION_FAIL(STORE_ATTR, SPEC_FAIL_OVERRIDDEN);
goto fail;
}
With this, the magic works as expected:
# echo.py
def __getattr__(key): return key
def __setattr__(key, value): raise RuntimeError("not allowed")
def __call__(word): print(word)
>>> import echo
>>> echo.hello
'hello'
>>> echo.hello = 'hi'
RuntimeError: not allowed
>>> echo('hello world')
hello world
Indeed, with patching also generated_cases.c.h I got comparable timings.
$ git diff
diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h
index 864a4f7bca..96c764f396 100644
--- a/Python/generated_cases.c.h
+++ b/Python/generated_cases.c.h
@@ -2338,7 +2338,7 @@
uint32_t type_version = read_u32(&next_instr[1].cache);
uint16_t index = read_u16(&next_instr[3].cache);
#line 1679 "Python/bytecodes.c"
- DEOPT_IF(!PyModule_CheckExact(owner), LOAD_ATTR);
+ DEOPT_IF(!PyModule_Check(owner), LOAD_ATTR);
PyDictObject *dict = (PyDictObject *)((PyModuleObject *)owner)->md_dict;
assert(dict != NULL);
DEOPT_IF(dict->ma_keys->dk_version != type_version, LOAD_ATTR);
diff --git a/Python/specialize.c b/Python/specialize.c
index 33a3c4561c..c989a558c3 100644
--- a/Python/specialize.c
+++ b/Python/specialize.c
@@ -752,7 +752,7 @@ _Py_Specialize_LoadAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name)
SPECIALIZATION_FAIL(LOAD_ATTR, SPEC_FAIL_OTHER);
goto fail;
}
- if (PyModule_CheckExact(owner)) {
+ if (PyModule_Check(owner)) {
if (specialize_module_load_attr(owner, instr, name))
{
goto fail;
$ ./python -m timeit -r11 -s 'import b' 'b.x'
10000000 loops, best of 11: 37.2 nsec per loop
$ ./python -m timeit -r11 -s 'import c' 'c.x'
5000000 loops, best of 11: 41.1 nsec per loop
PyModule_Check() is much more expensive call than PyModule_CheckExact().
-1. This seems like a minor convenience in some cases, and a major annoyance in many others.
I expect that the (admittedly rather satisfying) half a second I save by typing import tabulate instead of from tabulate import tabulate will quickly be lost once I start having to explain to code reviewing colleagues that modules normally are modules can sometimes double as functions and at other times look like classes except that you can’t do isinstance(x, callable_module) or subclass from a module pretending to be a class.
I’d also be very wary of just how similar but also dissimilar this is to Java imports. import some_module.SomeClassSubmodule is exactly the same syntax as Java but is very different in functionality.
Opinions differ. I would say that it messes with the mental model of Python that just defining a module level __call__ function would not work. I think I understand reasons that it doesn’t, but they are not immediately obvious.
I like the proposal and hope it’s accepted!
Thanks to @brandtbucher, Python 3.12 will ship with this nice usability improvement:
>>> import dis
>>> dis("what is this")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable. Did you mean: 'dis.dis(...)'?
>>> dis.dis("what is this")
0 0 RESUME 0
1 2 LOAD_NAME 0 (what)
4 LOAD_NAME 1 (this)
6 IS_OP 0
8 RETURN_VALUE
>>>
This should provide an easier way for users to discover their mistakes if they get confused between a module and a module member of the same name.
I think it would be useful if there was a decorator which, when used, would make any magic/dunder method work. To demonstrate what I mean, I will use the (imaginary) sys.method decorator. (sys.method does not exist and is purely for demostration)
Example:
@sys.method
def __call__(...):
... # function as the __call__ method
#without the decorator it would not
def __repr__(...):
... # does not act as a method because it does not have the decorator
I think this decorator should apply to all magic/dunder methods. For example, it would be useful if you could define the __str__ and __repr__ methods to customize the appearance of your module.
I think that’s doable. Here’s a possible implementation:
import sys
def modulemethod(func):
module = sys.modules[func.__module__]
if type(module).__name__ != "PatchedModule":
class PatchedModule(type(module)): pass
module.__class__ = PatchedModule
setattr(module, func.__name__, func)
return func
Making this work in packages is left as an exercise for the reader.
The problem with that implementation is that setting an attribute to a function does not make that function into a working method. If you defined it inside a class it would be of the type MethodType but because you used setattr the type is FunctionType instead of MethodType.
I think that this would actually require a reworking of the way methods work.
Did you try it?
$ python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Example:
... pass
...
>>> def method(self, x):
... return f'method of {self}, called with {x}'
...
>>> setattr(Example, 'method', method)
>>>
>>> Example().method(1)
'method of <__main__.Example object at 0x...>, called with 1'
>>> type(Example.method)
<class 'function'>
>>> type(Example().method)
<class 'method'>
setattr called on the class sets an attribute of the class itself, exactly the same as if that function had been defined within the class block. It is a function, not a method, either way (in 3.x). Methods are created on the fly when the name is looked up from an instance.
so it does define a method properly (I should have tested that) but it still does not work as a special method. for example, if I use that on a __call__ function in a module, it does not make the module callable
You can make a module callable by setting its class to a subclass of types.ModuleType and implement the call in that subclass. I have an implementation of a function that makes a module callable and allows to set any function as the called function. You can even make stdlib modules callable (though I do not recommend doing that).
Here’s the file callmod.py:
import types
def make_module_callable(module, func=None):
if func is None:
func = getattr(module, '__call__', None)
if not callable(func):
raise ValueError("func should be a callable")
class CallableModule(types.ModuleType):
def __call__(clas, *args, **kargs):
func(*args, **kargs)
module.__class__ = CallableModule
and a module calltest.py to test it on:
def __call__(*args, **kargs):
print(f"Module {__name__} was called with {args=} and {kargs=}")
And here is the result:
>>> from callmod import make_module_callable
>>> import calltest
>>> make_module_callable(calltest)
>>> calltest(6, 7, k='me')
Module calltest was called with args=(6, 7) and kargs={'k': 'me'}
>>> import os
>>> make_module_callable(os, print)
>>> os("Obscure", "print", "function")
Obscure print function
This thread seems to focus a lot on the consequences for a serious code base, but I’d like to suggest that callable modules would have benefits in the context of an interactive session or small scripts. It’s an important use case for Python that should not be neglected despite the fact the language is improving its industrialization story these days.
If you are in a python shell, a notebook, or in pdb, it is a noticeable quality of life improvement to have callable modules, especially while debugging where you don’t always have a history of past session, and you restart often from scratch. Most people don’t have a .pdbrc after all.
In fact, before we had breakpoint(), I could see beginners using basic editors, struggling just to remember what to type for import pdb; pdb.set_trace(). I believe import pdb; pdb() would have helped.
And I do see regularly new comers attempt to call modules, it’s only counter intuitive to us because we are used to it.
I believe this has been mentioned, in particular in the case of pprint, and I agree this would make things more comfortable. But I also feel like this will make things inconsistent and harder to explain in the end, (see this post for example).
I also think of cases like import json. Should we make it so that json() loads JSON or dumps JSON?