Inconsistent behavior with Python 3.11 enum mixin class behavior

Core Development
1

The changes to enums in Python 3.11 have resulted in some bizarre, undocumented, and (in my case) unwanted behavior.

One of the things that disturbs me most about the new behavior is that the data type of the enum values now depends on whether the constructor of the mixin type raises an exception. I view this as a bug.

Consider the following code:

import enum

class Base:
    def __init__(self, x):
        print('In Base init')
        raise ValueError("I don't like", x)

class MyEnum(Base, enum.Enum):
    A = 'a'

    def __init__(self, y):
        print('In MyEnum init')
        self.y = y

print(type(MyEnum.A))
print(type(MyEnum.A.value))



class Base2:
    def __init__(self, x):
        print('In Base2 init')

class MyEnum2(Base2, enum.Enum):
    A = 'a'

    def __init__(self, y):
        print('In MyEnum2 init')
        self.y = y

print(type(MyEnum2.A))
print(type(MyEnum2.A.value))

In Python 3.10, this produces:

In MyEnum init
<enum 'MyEnum'>
<class 'str'>
In MyEnum2 init
<enum 'MyEnum2'>
<class 'str'>

In Python 3.11, this produces

In Base init
In MyEnum init
<enum 'MyEnum'>
<class 'str'>
In Base2 init
In MyEnum2 init
<enum 'MyEnum2'>
<class '__main__.Base2'>

The code responsible is here.

Additionally as a consequence of this, MyEnum('a') works in both versions, but MyEnum2('a') works in Python 3.10, but not in Python 3.11.

In my specific case, I am creating a tokenizer. I want to use enums to represent symbol tokens. I have a base Token class, and I want my enum members to inherit from the Token class (ie isinstance(BoolLiteral.TRUE, Token) in the code below). The enum members should have a value field which is the string for the token. The enum members should not inherit from str, only from Token (so I do not want to use StrEnum).

In Python 3.10, this is trivial, eg:

class BoolLiteral(Token, enum.Enum):
    TRUE = 'true'
    FALSE = 'false'

In Python 3.11, I’m not sure what the proper way to do this is. The code above happens to work so long as Token lacks an __init__ method taking one parameter (because without such a method you get an exception, producing the old behavior). However it makes me uncomfortable to rely on that.

The documentation also references subclassing of the form

class EnumName([mix-in, ...,] [data-type,] base-enum):

It’s not clear to me how this is supposed to work. It doesn’t seem to work at all in Python 3.11 if the mix-in type has an __init__ method which doesn’t produce an exception. In Python 3.10, using str as the data-type makes the enum members be instances of str, which is not what I want. This behavior should be clarified in the documentation.

1 Like
2

What behavior is the Token class supposed to be providing? In other words, why does Token.__init__ exist?

3

I don’t currently have an __init__ for my Token class, so that code does work in Python 3.11 (but I don’t like relying on the fact that I don’t have an __init__ method). Token simply describes the interface for tokens. Mostly it exists for typing reasons – I want all my tokens to share a common base class. In principle I might add some methods common to all tokens.

What I had previously done though was introduce a SymbolToken class which had a constructor taking a symbol and set the attribute symbol on the object. I used that as the mixin for my enums. I expected the inherited constructor to act as if you defined such a constructor in the enum itself – ie it would get called for constructing enum members. Instead the inheritted constructor got called to wrap around the primitive enum member’s values in Python 3.11.

Ultimately, the idea of a SymbolToken class seems redundant, since it would be better just to use the enum member’s value rather than introduce a whole new attribute for that. However, it is very strange that an inherited constructor now works totally differently than one you define in the enum.

I think the new change conflates the notion of enum members and the enum member’s value. In my view, the mixin type should only define the base class of the enum members. The value you assign with = should be what gets stored in the enum member’s value field (and be used to initialize the enum member). That is currently conditionally violated depending on whether the mixin class’s constructor raises an exception.

4

I’ll update the docs as necessary, but basically if a data-type[1] is mixed in, then the members are also of that data type. Hence, each member of a class SomeEnum(int, Enum) is an int, and members of a class MyEnum(Base, Enum) is a Base (because Base is a data type, not just a mixin).

The ValueError being raised in the __init__ is a bug, though – thank you for finding it.

  1. a data-type being a superclass that defines either __new__ or __init__. ↩︎

5

There is a distinction between members and values. I agree that members of MyEnum should be of class Base – or more specifically that isinstance(MyEnum.A, Base) should be true, and that isinstance(MyEnum.A, MyEnum) should also be true. That is exactly what I am trying to accomplish.

What I completely disagree with is that the underlying enum values should also be an instance of the mixin type. This is inconsistent with the most basic enum you could create:

class Simple(enum.Enum):
    A = 'a'

# This holds in both Python 3.10 and Python 3.11 - it's good!
assert type(Simple.A) == Simple
assert type(Simple.A.value) == str

However, the current behavior in 3.11 is to try to wrap the values in the mixin type as well.

This line of code just doesn’t seem to make sense to try to do at all. Enum values have no inherent relation to the member type.

6

Wrote a reply earlier, but looks like it got marked as spam. But in short, there’s an important distinction between members and values in enums. I am in complete agreement that enum members should be instances of the mixin type. Not the values though. Values have no inherent relation to the member type, but Python 3.11 tries to wrap values in the mixin type, which makes no sense to me.

7

An enum’s member’s value has been the mixin type since Enum was introduced in 3.4 (current docs).

For example:

from enum import Enum

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y
    def __repr__(self):
        return "Point(%r, %r)" % (self.x, self.y)

class PE(Point, Enum):
    ORIGIN = 0, 0

repr(PE.ORIGIN)
# <PE.ORIGIN: Point(0, 0)>

isinstance(PE.ORIGIN, Point)
# True

isinstance(PE.ORIGIN.value, Point)
# True
8

That’s incorrect.

Your example is a great demonstration of the regression between Python 3.10 and Python 3.11. It worked differently in Python 3.10:

Point(0, 0)
True
False

Rule 3 in the current docs is a bit confusing, I’m not entirely sure what it means, but I’d point to rule 4:

When another data type is mixed in, the value attribute is not the same as the enum member itself, although it is equivalent and will compare equal.

9

The difference is advancement, not regression. :slight_smile:

Before Python 3.11 a mixin had to have a __new__ to be considered a data type; now it also considers an __init__ as defining a data type.

The meaning of rule 4 is that there is no is relationship; in other words, if the Point class above had a standard __eq__:

>>> PE.ORIGIN == PE.ORIGIN.value
True
>>> PE.ORIGIN is PE.ORIGIN.value
False

A standard __eq__:

def __eq__(self, other):
    if not isinstance(other, self.__class__):
        return NotImplemented
    return self.x == other.x and self.y == other.y
10

E.G. Once you’ve mixed in the Point class, every member of that enum will be a Point. Contrasted with a plain Enum where every member is a simple object, and one member’s value can by a Point, while another member’s value could be complex, and yet another member’s value could be a str.

11

I’d argue that the old behavior with __new__ (which I was not aware of, I thought you needed to set _value_ in __new__ to change the value) also didn’t make a whole lot of sense either, but I’d rather continue to support that than introduce breaking changes. I can’t understand why this behavior was extended to __init__ though. It’s a breaking change which is inconsistent with the way enums without mixins work.

In Python 3.10 you would have been able to do PE((0, 0)) and it would give you PE.ORIGIN. This is consistent with the example I showed earlier with Simple('a').

In Python 3.11 that doesn’t work. You need to do PE(Point(0, 0)) – except in the point class you originally wrote that doesn’t even work since you didn’t define __eq__.

If you wanted the values to be of type Point, you could have made them as such, but the thing on the right hand side of the equals sign in enums was the value (and also the intializer to __init__ for the members)

It doesn’t make a whole lot of sense to have a value at all if it is just another instance of essentially the same type.

I suppose there could be added a keyword metaclass argument as a wrapper class for the values, but I don’t see the point. If you want the value to share the same type as the member you can always do:

class PE(Point, Enum):
    ORIGIN = Point(0, 0)

    # only necessary because Point constructor wasn't defined to take points
    def __init__(self, pt):
        super().__init__(pt.x, pt.y)

In my case, I want my members to be instances of Token. I would like the values to be of type str.

12

I’m curious as to the history of why this notion of a “data type” was introduced in the first place. Would you happen to have any examples of where this is needed (or at least useful)?

In the common case of, eg class MyEnum(int, Enum) it essentially wouldn’t matter[1]. Very few user-defined classes directly define __new__, so they wouldn’t make use of it either. There seems no point to wrapping values in a “data type”, since this can simply be done explicitly. So while I can in some sense understand trying to make things more consistent, I can’t understand going in this direction rather than the other way.

But in either direction, a change which affects the data type of values within user-defined classes is a pretty major one and absolutely should have been documented in What’s New In Python 3.11 — Python 3.12.1 documentation.

  1. The docs do have an example that makes use of '__new__' in int.__dict__ in Python 3.10, but while it is an artificial example, if anyone actually did use ('11', 16) rather than 0x11, I can only imagine that they would actually want the value to be a tuple rather than an int, so that perhaps MyEnum(('17', 10)) and MyEnum(('11', 16)) give equivalent members with different names. ↩︎

13

The motivating factor for the change was the combination of dataclass and Enum; I have since found a different way to verify that a dataclass is being used, so the presence of __init__ is no longer checked for, and previous behavior has been restored: Unexpected behavior change in subclassing Enum class as of 3.11.

14

Good to hear! A couple questions though. In this test, why did you change from self.a to self._value_? In Python 3.10, self.a, self.value, and self._value_ all would have worked and done the same thing. Is that no longer the case in the new commit? If so, that should be fixed. I don’t have time right now to test it myself.

I also don’t really get the point of changing how dataclass mixins work. What was the concern with the previous behavior?

15

I don’t like f-strings, and it was the only one in the tests file.

Enums didn’t look like enums when inheriting from a dataclass.

16

Thanks for clarification, though I was wondering more why not 'Foo(a=%r)' % self.a?

What do you mean “look like”. Do you just mean the repr?

Personally I don’t see much benefit in changing the repr of enums with mixin dataclasses, but surely changing .value isn’t necessary to get the effect you want. If you want to change the repr, change __repr__. For example (using __init_subclass__ rather than metaclasses for brevity):

class Bar:
    def __init_subclass__(cls):
        if '__repr__' not in cls.__dict__:
            cls.__repr__ = lambda self: f'<Bar {super(cls, self).__repr__()}>'

@dataclass
class Foo:
    a: int

class FooBar(Foo, Bar):
    pass

print(FooBar(1))
17

Because I will personally go in and change the code if someone introduces new uses of % that are not testing that operator. :wink:

1 Like
18

The means of string formatting is irrelevant. I was inquiring as to why a test was changed from using self.a to self._value_ when both should work.

19

Ah, looking at it again I see your point – in the Foo class there is no self._value_, so that should be self.a.