Use frozendict as type and module namespace (dictionary)

Ideas
1

Hi,

I would like to propose adding a new __frozendict__ = True marker to type definition or module namespace to make them use a frozendict for their namespace. The type/module is created as usual, but once the initialization is complete, the namespace becomes a frozendict and no longer possible to modify the namespace. Only the namespace is read-only. If a type/module attribute is mutable (list, dict, whatever), it is still possible to modify the attribute value.

What do you think of this idea? Would it be useful for your use case?

Example on a class defined in Python. Trying to override an attribute or setting a new attributes raises an exception:

class A:
    __frozendict__ = True
    attr = 1

assert type(A.__dict__) == frozendict
A.attr = 2  # error
A.xyz = 3  # also an error

Note: A.__dict__ is not a types.MappingProxyType of a frozendict, but directly the frozendict.

Module example:

__frozendict__ = True
def func():
    pass
attr = 1

It’s no longer possible to override a module attribute or define a new module attribute:

import mod
assert type(mod.__dict__) == frozendict
mod.func = str  # error
mod.attr = 2 # error
mod.new_attr = 3 # error

For a class, it’s different than the Java final keyword, since it remains possible to inherit from the class and override methods and attributes in the child class. It’s also different than typing.Final which only applies to a single attribute (and not the whole class namespace).

I implemented my idea in my fork: pull request. To test my idea, I modified a bunch of stdlib modules and some stdlib classes to use __frozendict__ = True.

Example on the modified math extension:

>>> import math
>>> type(math.__dict__)
<class 'frozendict'>

>>> math.pi
3.141592653589793
>>> math.pi = 4
TypeError: frozendict object does not support item assignment

>>> math.floor
<built-in function floor>
>>> math.floor = math.ceil
TypeError: frozendict object does not support item assignment

>>> math.new_attr = 123
TypeError: frozendict object does not support item assignment

Multiple stdlib modules cannot be modified to use __frozendict__ = True, because their test modifies module attributes (for example, using unittest.mock). Tests is the major blocker point to use __frozendict__ = True more widely. Well, and backward compatibility obviously.

See also the rejected PEP 726 – Module __setattr__ and __delattr__.

Victor

11 Likes
2

If I’m understanding you correctly, this would allow us to have the typing benefits of @dataclass(frozen=True) without the runtime cost of the class becoming slower to initialise? But this benefit wouldn’t extend to @dataclass(frozen=True, slots=True)?

I don’t think I’d ever use this any other way. Blocking monkeypatching makes testing harder, as well as various kinds of debugging and experimentation. And I trust my colleagues not to add code that modifies module attributes.

5 Likes
3

I like it particularly for pieces which do a lot of constants / config. Opt-in way way to encouraging readable coding practices.

For the testing cases is it possible to “wrap” the frozendict module in a non-frozen when mocking? How efficiently can an additional attribute be added to a frozen module?

4

Oh, when preparing a reply, I found a flaw in my implementation of frozen module namespace: functions keep a reference to the mutable dict namespace!

Module mod.py:

__frozendict__ = True
global_var = 1

def func():
    global global_var
    global_var += 1
    return global_var

Example:

>>> import mod
>>> mod.func()
2
>>> mod.global_var
1

>>> type(mod.func.__globals__)
<class 'dict'>
>>> type(mod.__dict__)
<class 'frozendict'>

The func() is supposed to raise an error when trying to override the global namespace, since the module namespace is a frozendict. But it works, because the function (__globals__) keeps a reference to the original mutable dict namespace!

mod.global_var is frozen and is still 1, but in mod.func() mutable namespace, global_var can be increased!

Maybe it would be possible to patch module functions to override their __globals__ attribute. But it’s complicated, because all functions are not directly in the module namespace. For example, class methods also keep a reference to the module namespace, and are not in the module namespace, but in the class namespace.

Module mod.py with a class:

__frozendict__ = True
global_var = 1

class A:
    @classmethod
    def meth(cls):
        global global_var
        global_var += 1
        return global_var

Example:

>>> import mod
>>> mod.A.meth()
2
>>> mod.A.meth()
3
>>> mod.global_var
1
5

@dataclass(frozen=True) doesn’t create a frozen/immutable type, only instances have frozen attributes. In the example below, it’s still possible to override an existing attribute or set a new attribute on the frozen dataclass A:

import dataclasses

@dataclasses.dataclass(frozen=True)
class A:
    x: int
    y: int

A.x = 2  # override attribute
A.z = 3  # set new attribute

One option would be to create a module copy with a mutable namespace and store it temporarily in sys.modules[name]. It would not work on modules which already imported the mocked module, since these ones will continue to use their reference to the old module (don’t use the new mocked module).

Another option would be to override temporarily the module __dict__. The problem is that module functions keep a reference to the module namespace in their __globals__ attribute. So the function would use the old namespace, not the mocked one.

It would be simpler if there would be a switch to make a dict read-only or mutable: dict.set_readonly() :slight_smile: But currently, frozendict is a type separated from dict.

There is no magic solution working in all cases.

2 Likes
6

I don’t understand why we keep coming back to this idea of forced immutability. Python’s consenting adults philosophy works. I have never even seen or heard of anyone accidentally writing to something they weren’t supposed to without knowing what they’re doing. What problem are we solving? Are there some hidden concurrency benefits to make free threading more effective perhaps?

Being able to monkeypatch anything for testing/debugging purposes is such an underrated strength of Python. Untestable code pretty much just doesn’t exist because of it. Recently I was able to properly verify some defensive handling of antivirus filesystem interference[1] by monkeypatching the os functions I used to randomly throw OSError. It should have been near impossible but it was easy.

  1. antivirus software will often lock or effectively lock by scanning recently written files so that any subsequent IO on that file fails until the scan is finished ↩︎

13 Likes
7

What would be the benefit vs slots?

8

Slots only affected type instances, not the type itself.

class A:
    __slots__ = ()
    attr = 1

A.attr = 2      # accepted
A.new_attr = 3  # also accepted
3 Likes
9

Ok - you’ve shown it could work.

What do you think of this idea?

I wonder “but works for what”? Depythonize Python?
Your o.p. doesn’t mention a single advantage of that.

The message conveyed is that “making things more static and more imutable is better”, I suppose?

We could retrieve the 2 or 3 threads on frozen data structures - I can see value on that. But everytime I needed to override a value in a module, and it is less than a handful times, I was very grate that I was in a language were that was possible: either for patching something before testing, or workaround a bug in a 3rd party library (which I also usually report upstream).

I mean - this would even shutdown unittest.mock.patch, wouldn’t it?

Do we get any speed gains? Via the JIT maybe? Or is this just part of a slow and steady a de-dynamization of the whole language just to, I don’t know, look better to ‘serious people’?

Would it be useful for your use case?

Which use case? And this is a serious question.

Any speed? Security?
Projects which policy is agreed to “not change things that should not be changed” or even to "prevent acidental modification’ already have `typing.Final`.

And I guess, if at some point, freezing a module in this way is really that important, it could be done manually, I believe . (/pauses → REPL

3 Likes
10

Ok - I object to this idea, as I can only perceive this as a trend to de-dynamize Python.

But apart from that, some bike-shedding here: changing how a class is constructed based on an attribute on that class was Python 2’s way - with the `_metaclass_` attribute.

Maybe this frozendict mark for a class could be a parameter for the metaclass - so, it should be passed as a named argument at the `class` statement declaration. (And, yes, I favor using the dunders even there to avoid clashing with possibly existing named parameters in the wild). What would be your take there? (actually, no strong opinions here, but it sounds less Python2ish)

So,

class A(__frozendict__=True):
   attr = 1

rather than having another attribute inside the class change the way it is built (I know `_slots_` do that - but it is also a decision taken a long time ago, and dataclasses, for example, preferred the “named argument on declaration” way, even for slots)

(Now derailing the bikeshed colors;
this could be a parameter passing a mapping class to `type` algogether - maybe defaulting to `frozendict` if it is not a callable - but otherwise, it could just be used as a factory for whatever mapping would be the `cls._dict_` on the class instances. After all, this is not customizable today. If it would be a mutable mapping, then, that is on the class creator for eventual malfunctions when changing dunder-attributes via the mapping only)
```

class A(__dict__ = MyWeirdMapping): 
    attr = 1
3 Likes
11

There’s plenty of reasons to want this. If there weren’t, we wouldn’t have frozendicts in the first place. Or frozen dataclasses. Immutability is a nice property in many cases, especially those involving concurrency, and it prevents misuse in larger code bases.

What looks safe to modify may not be, and while it may be obvious when first designed, that doesn’t mean people working with the same code later are going to be as aware. Designing it to not be able to be modified in such cases is preventative care.

To turn the over-used “consenting adults” bit on its head, “Hey, this namespace ['s author] is explicitly denying consent to others making runtime modifications to this namespace. If you don’t like them doing that, you don’t have to use their code.”

7 Likes
12

I think this is too likely to break things on modules, especially if used on the stdlib.

It may be that the stdlib tests don’t rely on monkey patching certain modules but other peoples’ tests may. Anything that uses plain import <stdlibmodule> and needs to patch for a test will stop working if the module dict is now frozen.

There are also packages that offer the ability to patch stdlib modules like gevent that this will almost certainly break.

This also doesn’t mix well with lazy imports, which are supposed to replace themselves in the module dictionary on being accessed. The function still currently works presumably due to the bug you mentioned earlier about keeping a reference to the original dict.

lazy_bug.py

__frozendict__ = True
lazy import typing as typing # intended as a re-export
def f():
    return typing.Any

>>> import lazy_bug
>>> lazy_bug.typing
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    lazy_bug.typing
TypeError: frozendict object does not support item assignment
>>> lazy_bug.f()
typing.Any
>>> lazy_bug.__dict__["typing"]
<lazy_import 'typing'>

On classes as you’ve already noticed this changes the behaviour of __annotations__ which is normally created as a cache if the annotations on a class are resolvable. This also doesn’t work with things like dataclasses that operate on the class after it has been constructed[1].

In both cases I think this is going to cause more problems than it solves.

  1. With my proposal for lazy method generation on dataclasses to improve construction time and hence import time of modules with heavy dataclass use, it really couldn’t work. ↩︎

13

Agreed. But to be more specific, I think that having the ability to use a frozen __dict__ is safe, but applying that ability to pretty much any existing module, and certainly any module in the stdlib, is likely to cause significant breakage (in people’s test suites, even if not in production code).

For new libraries, maybe being able to say “the modules in this library are immutable” would be helpful[1], but is that benefit sufficient to justify the huge attractive nuisance of having people think it’s something that can be retrofitted to existing libraries?

  1. Although I’m not entirely sure how ↩︎

1 Like
14

In the linked PR, you can eat both cakes

from abc import ABCMeta, abstractmethod


class Iterable(metaclass=ABCMeta):

    __frozendict__ = True
    __slots__ = ()

    @abstractmethod
    def __iter__(self):
        while False:
            yield None


class Iterator(Iterable):

    __frozendict__ = True
    __slots__ = ()

    @abstractmethod
    def __next__(self):
        raise StopIteration
1 Like
15

That’s what things like the single leading underscore say. “Hey, the author is explicitly denying consent to you modifying this attribute. If you don’t like that, well, it’s your code, and if it breaks, now you have two pieces.”

Python’s philosophy is not “consenting adults, but some adults have more power over you than others do”. It’s “you are the programmer, and this is your code”. We have ways to express concepts like “touching this will void your warranty” but we don’t have ways to say “you’re a Java programmer now”. [1]

  1. Technically, Jython… ↩︎

10 Likes
16

I’m not sure it is helpful though? Even in new libraries this would break them for anyone that wanted to use -X lazy_imports=all.

This may also cause failures in cases where it’s expected that module.__dict__ is an actual dict and not a different mapping. There’s an example of this from gevent which looks like it originally comes from a compatibility patch to support Python 2 and 3. The Python 2 support is gone but the iteritems function remains and is defined as iteritems = dict.items, this works on module.__dict__ currently but breaks if you replace this with a frozen dict[1].

  1. This prevents patch_all() from working before it even gets to trying to write to modules that the current PR branch freezes. ↩︎

17

And yet people clearly do want and benefit from immutability that is on public attributes. There are ways this could be used to get that without all the machinery of dataclasses (even dataclasses could benefit from it)

It’s not that you can’t touch it, it’s that you shouldn’t write something else into it. We don’t really have a read-only concept.

(we have property, but I think your java comparison holds more true there for some immutable record type needing 5+ properties to go along with 5+ _ prefixed fields vs a single declaration that the namespace shouldn’t be modified)

18

We have established pretty well over the years that there are users who want frozen datastructures. I count myself among them. I may be misinterpreting people’s replies, but I think the concern being raised is not about the usefulness of immutability in general, but rather of this particular proposal to make namespaces optionally immutable.

I’m unclear on what kind of use-case would benefit from freezing a module or class in this way.
I have seen bugs caused by accidentally writing into a shared dict which should have been frozen. I have not seen bugs caused by accidentally writing attributes to a class or module.

Absent some clearer use-case, I would be against introducing frozen class and module namespaces. There is a cost and I don’t understand the benefit.

8 Likes
19

I’m not completely against the idea, if the goal is not only to “seal” objects, but also to make them faster, at least in a future.

But I think we should retain the ability to monkeypatch things in some way. I don’t think this is possible if __dict__ will be a frozendict. At least not now.

BTW: Guido himself said:

for large programs you have to have a more disciplined approach and it helps if the language actually gives you that discipline, rather than telling you ‘Well you can do whatever you want’.

Guido van Rossum on dynamic vs static typing – Sébastien De Revière

The intrinsic dynamic nature of Python forced people that wanted “more discipline” to do “weird” things, like closures, del inside a module, cythonize the code and so on. Some of these tricks are in the stdlib too.

I don’t think these guys are not adults enough.

Minor consideration: I’d prefer __frozen__ instead of __frozendict__. It’s more generic and, most important, shorter :grin:

1 Like
20

would it be feasible to have a metaclass either pass a `dict` or `frozendict` to eventually `type.__new__` to determine the namespace type?

would a module level `__setattr__` be a fitting complement to `__getattr__` that’d be simpler than a `types.ModuleType` based proxy?