PEP 726: Module __setattr__ and __delattr__

PEPs
21

I haven’t yet read this PEP but it sounds like my idea for using the module object as the global namespace would give the desired behaviour. I had prototype that was passing nearly all tests so I think the backwards compatibility issues are not too horrible. A high level summary of my idea is that global lookups do getattr() on a module object, rather than __getitem__() on a dictionary-like object. The import machinery would pass around module objects rather than the module globals dict, like it currently does. Maybe my idea is not very good or not explained very well. As Guido suggested, I should write a PEP for it.

3 Likes
22

The sys module API is designed around the idea of setting its attribute, but there are other stdlib modules with some attributes which can be overriden (and it’s explained in their doc, it’s the way to go).

Some more examples:

  • threading.excepthook doc: can be overridden to control how uncaught exceptions raised by Thread.run() are handled.

  • warnings

    • warnings.showwarning() doc: You may replace this function with any callable by assigning to warnings.showwarning
    • warnings.filters is not documented (!), but it’s commonly modified. Its own warnings.catch_warnings() functions does self._module.filters = self._filters to restore the attribute to its previous value.

There are some constants which may be interesting to implement as constants: either deny modifying them, or at least disallow removing them. Examples:

I’m not declaring a war to monkey patching (even if I not really like that). If the PEP is approved, each of these attributes should be discussed on a case by case basis.

On the other side, it seems like io.DEFAULT_BUFFER_SIZE can be modified on purpose. Just don’t assign it to a string, right? :wink:

23

@skirpichev @AA-Turner: Would you mind to update the PEP to mention that the API is limited on purpose and list limitations? In short, summarize what has been discussed here. You might also add a few more examples in the PEP, look at examples that I gave.

24

I was hoping the major limitation is mentioned in the Specification section:

Do you suggest to rephrase this or to extend with an explanation for this limitation (which is very clear, I believe)?

Maybe it’s better to replace an artificial example (inspired by the mpmath) in the Motivation section? It was here just to illustrate all use cases for the proposed interface (read-only attributes, input validation, monitoring attribute change) in one shot.

Or we can add a new section (Discussion?) and add real-world examples from the discussion thread. BTW maybe this is a suitable place also to mention reasons for limitations of the API.

25

No, I think that it should be elaborated. See previous messages.

26

FYI: PEP was updated to address intentional limitations of the API and to include few more examples from the discussion thread.

1 Like
27

So, what’s the next step? Ask the Steering Council to take a decision on that PEP?

Or are there still open questions or points to discuss?

28

So, what’s the next step? Ask the Steering Council to take a decision on that PEP?

I think it’s ready. I would appreciate @AA-Turner opinion on the PEP state, so lets wait few days. Then I’ll open an issue.

Edit:
JFR, the issue thread has additional example of using module’s __setattr__ to handle deprecation warning (actually, something like this was mentioned before in the discussion).

29

Hi.
I’m following up on the comment by Sergey.

There is an open issue regarding the warnings module not being thread safe, namely the bit regarding catch_warnings() and monkeypatching of the showwarning function. I have commented on that issue and came up with a very rought draft branch. [1]

This is unfortunatelly an anti-pattern with some bits of the Python stdlib, where specific behavior is controlled by monkeypatching global symbols. This is unsafe.

Compare for instance sys.excepthook vs atexit.register(callback). The latter is a good example of how to setup hooks or callbacks. The first one not so much.

Back to warnings, fixing that module would require forbidding all monkeypatching (and many other implementation details with a stack of catch_warnings objects, not relevant for this discussion).

However, that module is as you well know quite a central part of the python stdlib, and any change in its behavior needs 1. to go through a deprecation cycle, plus, such changes should not introduce any 2. performance penalties.

Therefore, with these concerns, 1. requires a __setattr__ hook to warn that monkeypatching that function will no longer be supported, then in a latter version, raise an Exception. The performance part 2. kind of negates the choice of subclassing the warnings module with a class with a setter, because of the perormance penalty when accessing attributes.

To conclude, the proposal of a __setattr__ in modules is something that would not have that wide use, but there are at least some niche use cases, including these problems with warnings that I mentioned which do not have any other good options. Hopefully, people should not be designing APIs that require patching global functions, but that ship has sailed long ago.
Moreover, I personally like when APIs are complete, symmetric and consistent, so it’s only natural that if a module object has a __getattr__ that it would too have a __setattr__.

Thank you.

PS: I think that there should be a PEP about removing all monkeypatching from stdlib and change those APIs to use register callback functions. It would not be much work implementation wise, but it would be a bit bureocratic.

[1] warnings.catch_warnings is async-unsafe · Issue #91505 · python/cpython · GitHub

1 Like
30

First, a minor question about this comment referenced in the PEP:

That should be “dict” right?

>>> type(sys.modules)
<class 'dict'>

If so, that should be fixed in the PEP.

But my bigger concern with that statement in the PEP:

Reading more of the discussion, it looks like I could do this?

import sys

sys.modules['sys'].__dict__['modules'] = 123

If I can bypass the __setattr__, doesn’t that mean we still can’t “make sure that sys.modules is always a list” (or dict)?

That would make the motivation for this a lot weaker.
When I read this:

That sounded good to me, to get the performance improvement from the C code.
But if there’s a way to bypass it, then we can’t avoid the error checking in the C code.

And it seems kind of misleading for the PEP to say that we can “ensure” when we can’t ensure it.

It makes accidental change less likely, but it doesn’t ensure it.

1 Like
31

Yep, good catch. Thanks!

Yes, this is correct.

True. But we can’t catch all cases when someone want to shot own leg.

For example (maybe not a better one, but), how can we be sure that all instances of some class have its methods? 
>>> class Foo:
...     def spam(self):
...         pass
... 
>>> del Foo.spam
>>> bar = Foo()
>>> # del Foo.spam  # or here...
>>> isinstance(bar, Foo)
True
>>> bar.spam()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'spam'
In a similar way we could easily break other invariants that might be assumed by some code, e.g. in the stdlib... 
>>> import fractions
>>> from fractions import Fraction as F
>>> half = F(1, 2)
>>> half._denominator = 'spam'
>>> half + half
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/fractions.py", line 615, in forward
    return monomorphic_operator(a, b)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/fractions.py", line 714, in _add
    g = math.gcd(da, db)
        ^^^^^^^^^^^^^^^^
TypeError: 'str' object cannot be interpreted as an integer

Maybe correct wording is: “It makes accidental change impossible.”?

32

The Python Steering Council has discussed PEP-726 – Module __setattr__ and __delattr__ and decided to Reject it.

We did not see clear consensus in the community about use cases such as read-only module attributes, and we are concerned that going further down the route of modules behaving semi-class-like may lead to future performance optimization challenges and confusing semantics. We do not find there to be a strong need for having read-only module attributes, and even if there is one, this is not the ideal solution to that problem. Preventing one possible mis-use of modules did not seem compelling enough.

There are already workarounds that people can use to achieve similar results. That some people consider those workarounds such as replacing a module with another object to be gross is not considered a problem as it is an uncommon need.

Thanks for the PEP and thorough discussion,
– The 2024 PSC

15 Likes
33

Thanks to all for discussions, including previous one, and help. Especially to @oscarbenjamin and to @vstinner for a lot of advocacy (and pr review). Sorry that my efforts led to the death of the idea.

Very well, instead we will live with existing performance penalties. Another attempt to fix this (issue #103951) seems to be fallen as well.

6 Likes
34

As a supporter of the idea, it makes me sad, but it’s great to have a full PEP which writes down the rationale and limitations of this design. Thanks a lot Sergey for your great work on this topic!

9 Likes
35

Oh, too bad. Hopefully, the performance hit when using a custom class for a module will be fixed then.

36

Best way to ensure it’s fixed is to research it and submit a patch :wink:

6 Likes