Proposal: LiteralEnum — runtime literals with static exhaustiveness

Typing
1

Hello typing community​:waving_hand:,

I’d like feedback on a possible typing construct tentatively called LiteralEnum, aimed at a common gap between Enum/StrEnum and typing.Literal.

Problem
Many APIs use small, closed sets of scalar values (often strings: HTTP methods, event names, config keys). At runtime, these are most ergonomic as plain literals, but statically we want exhaustiveness checking.

Today this usually leads to duplication, e.g. a constants namespace plus a parallel Literal[...] union, or forcing callers to pass enum members instead of raw values.

Proposed idea
LiteralEnum defines a finite, named set of literal values that:

  • are plain runtime literals (str, int, bool, None, etc.),
  • provide a runtime namespace, iteration, and validation, and
  • are treated by type checkers as an exhaustive Literal[...] union.
  • This is not intended to replace Enum or Literal, but to cover the narrow case where literal values themselves are the API surface.

Minimal example:

from typing import LiteralEnum

class HttpMethod(LiteralEnum):
    GET = "GET"
    POST = "POST"
    DELETE = "DELETE"

def handle(method: HttpMethod) -> None:
    ...

handle("GET")          # accepted
handle(HttpMethod.GET) # accepted
handle("git")          # type checker error

At runtime:

  • HttpMethod.GET == "GET"
  • list(HttpMethod) == ["GET", "POST", "DELETE"]
  • "GET" in HttpMethod can be used to check if a string is valid
  • HttpMethod("GET") could optionally validate and return "GET" (acknowledging that callable classes usually construct instances)

At type-check time, HttpMethod is equivalent to:

Literal["GET", "POST", "DELETE"]

Subclass extension is explicit (extend=True) to avoid accidental widening.

Status

  • I have a small runtime prototype as a proof of concept (linked below, under 200 lines); it does not attempt to solve the type-checker side yet

  • I’m interested in whether this direction seems:

    • useful enough to justify checker support, and
    • compatible with existing typing model assumptions.

Draft PEP (early): LiteralEnum/PEP.md at master · modularizer/LiteralEnum · GitHub
Runtime prototype: LiteralEnum/src/typing_literalenum.py at master · modularizer/LiteralEnum · GitHub

I’m not attached to the name or the source code, I’m looking to validate the concept and scope before going further.

Questions for discussion:

  1. Do other people feel this pain point as much as me?
  2. Have you found yourself writing duplicate types: a (Literal plus an Enum or just a bare class with attributes)?
  3. Are there alternative designs that could provide a runtime namespace and an exhaustive type hint, without introducing a new core typing construct?

Thanks for any feedback,
Torin

8 Likes
2

So, it’s like a special case of the sum type?

3

I understand the use case and think it might be good for developer ergonomics.

Currently, I just use literals directly. I check unstructured input using cattrs.structure(method, Literal["GET", "POST"]). This won’t be typesafe until the major typecheckers implement TypeForm, but at least there’s a path forward.

I wonder if the problem could be solved without adding a new, in-between combination of literal and enum, with the following changes:

  • for each enum, flesh out the static argument type to __init__ (or I guess __new__, or whatever is called at runtime) to something like (for HttpMethod) HttpMethod | Literal["GET", "POST"]. This is the current reality, modulo some edge cases.
  • introduce a way to reference the type of another Callable’s parameter. So you’d annotate the first parameter of handle as (spitballing) method: ParamOf[HttpMethod]. Maybe this can already be done with a clever decorator?

Then, your handle function’s method parameter would essentially have the type HttpMethod | Literal["GET", "POST"].

1 Like
4
  1. Do other people feel this pain point as much as me?

YES

  1. Have you found yourself writing duplicate types: a (Literal plus an Enum or just a bare class with attributes)?

YES

  1. Are there alternative designs that could provide a runtime namespace and an exhaustive type hint, without introducing a new core typing construct?

See earlier discussion here: Amend PEP 586 to make `enum` values subtypes of `Literal` - #12 by randolf-scholz

My idea was to amend the typing spec so that — at least for IntEnum and StrEnum — they would be considered subtypes of literal int and str values.

Your proposal may actually be an easier way to get something like this accepted, though I still don’t get why we would need a special type for this rather than supporting it natively on IntEnum and StrEnum. @Jelle gave an example here why this is generally unsafe, but I tested marking the integer / string interface of IntEnum / StrEnum with @final and the mypy-primer results looked good, so I don’t get why we can’t just special case IntEnum and StrEnum rather than inventing a new type for this.

5 Likes
5

I do find myself often writing something like

from typing import Literal

type HttpMethod = Literal["GET", "POST", "DELETE"]
VALID_HTTP_METHODS: list[HttpMethod] = ["GET", "POST", "DELETE"]

def f(x: str):
    assert x in VALID_HTTP_METHODS
    reveal_type(x)  # Type of "x" is "Literal['GET', 'POST', 'DELETE']"

So, something like your proposal would be welcome.

One thing I’m maybe a bit worried about is that any code that evaluates type annotations at runtime will have to special-case this construct in order to understand that it is equivalent to Literal[...].

2 Likes
6

They could be more ergonomic - I have certainly felt this pain point before. I ran some benchmarks on the particular library though, and convinced myself (possibly mistakenly) that enums were slower anyway and dropped them, instead going with the highly contrived:

FieldTypeT = Literal["A", "B",]

class FieldType:

    A: Final = "A" 
    B: Final = "B"
    __members__: set[FieldTypeT] = {
        "A",
        "B",
    }
1 Like
7

You can also use the pattern

from typing import Literal, get_args

HttpMethod = Literal["GET", "POST", "DELETE"]
VALID_HTTP_METHODS: tuple[HttpMethod, ...] = get_args(HttpMethod)


assert VALID_HTTP_METHODS == ("GET", "POST", "DELETE")
6 Likes
8

Indeed, or with type HttpMethod = the slightly less ergonomic get_args(HttpMethod.__value__).

1 Like
9

It might be nice if we could just do this:

HttpMethod = Literal["GET", "POST", "DELETE"]

assert "GET" in HttpMethod
assert "GIT" not in HttpMethod

although, we would have to figure out what to do with something like this:

HttpMethod = Literal["GET", "POST"] | Literal["DELETE"]

since that should be equivalent according to the type system.

1 Like
10

Kind of, yeah! A sum type says “this value is one of these options.” LiteralEnum is basically that, but specifically for plain values like strings and ints. The difference is LiteralEnum also gives you a nice way to look them up by name (like HttpMethod.GET), loop through them, and validate them — things a general sum type doesn’t give you out of the box.

1 Like
11

Thanks for all the feedback! lots of great ideas here. I want to respond to a few points individually, then step back and frame what I’m personally looking for.

Quick responses:

@jorenham : yes, you can think of it like that. The extra value over a general sum type is primarily namespace, iteration, and validation out of the box

@Tinche : the ParamOf idea is creative. It seems very useful and I love that it is general. Is not quite the first-class treatment that I think is worth considering for LiteralEnum proposal, but I see its value

@randolf-scholz : I like your PEP 586 amendment and think it solves an important direction: using HttpMethod.GET where Literal["GET"] is expected. I also understand Jelle’s response and see value in having Literal typehints only match true literals. The gap is the other direction — passing a plain "GET" to a function typed method: HttpMethod still fails. LiteralEnum tries to make both directions work, which cannot be done by modifying Enum ( it would break stuff )

@peterc : the get_args pattern is clean for getting runtime iterable from a Literal. The thing it’s missing is namespace access — there’s no HttpMethod.GET to write in code, and you still end up with two separate names to maintain.

@tmk : your Final + __members__ class is almost exactly the pattern that motivated this proposal! I think it is a great pattern in the current ecosystem, but I would love to see a pattern supporting a single source of truth in future versions of Python

@Dutcho : good point about type HttpMethod = ... requiring get_args(HttpMethod.__value__). Another example of how current tools make you choose between “nice for the type checker” and “nice at runtime.”

@beauxq : making in work on Literal directly would be fantastic. It would not solve all pain points, but I see that as a step in the right direction

What I’m aiming for

Today, getting everything I want requires something like three separate constructs:

# A current pattern I think could be made MUCH better

FieldTypeT = Literal["A", "B"]

class FieldType:
    A: Final[Literal["A"]] = "A"
    B: Final[Literal["B"]] = "B"

FieldTypes: Iterable[FieldTypeT] = {"A", "B"}

…but it KILLS me that this has no single source of truth - every value is written three times.

I wanted to lay out my personal litmus tests for what I’d want from a solution. You are more than welcome to disagree with these goals and voice your own preferences. It would be interesting to hear where people draw different lines. For me:

  1. Namespace for member access, e.g. FieldType.A
    • Literal fails this, and I see it as Literal’s biggest weakness
  2. Raw literals accepted: "A" should be an acceptable input to a param typed as t: FieldType
    • Enum/StrEnum fail this and I see it as their biggest weakness for SOME use cases
    • NOTE: there are definitely times devs WANT to reject raw strings to avoid floating literals – I would not want to change Enum/StrEnum behavior here. I just think LiteralEnum should behae like a Literal here, not like an Enum
  3. Named members accepted: FieldType.A should be an acceptable input to a param typed as t: FieldType
    • comparing an Enum member against a Literal type hint currently fails, and I think this is one thing @randolf-scholz was trying to address with his PEP 586 amendment
  4. Iteration and containment: x in FieldType and for x in FieldType should be first-class
    • both Literal and Enum provide ways to do this, but it could be more natural (and as @beauxq suggested, even Literal could benefit from in support)
  5. Single source of truth: if I have to write "A" more than once when defining my type, it fails this test
  6. One type, not two: most current solutions require separate variables for different parts of the functionality, which I find error-prone

Here’s how the current approaches score:

Approach #1 Namespace #2 Raw lit #3 Member #4 Iterate #5 Single source #6 One type that does it all
Literal alone :cross_mark: :white_check_mark: :cross_mark: :cross_mark: :white_check_mark: :cross_mark:
Literal + get_args (@peter) :cross_mark: :white_check_mark: :cross_mark: :white_check_mark: :white_check_mark: :cross_mark:
StrEnum :white_check_mark: :cross_mark: :white_check_mark: :white_check_mark: :white_check_mark: :cross_mark:
StrEnum + PEP 586 amend (@Randolf) :white_check_mark: :cross_mark: :white_check_mark: :white_check_mark: :white_check_mark: :cross_mark:
Enum + ParamOf (@Tinche) :white_check_mark: :white_check_mark: :white_check_mark: :white_check_mark: :white_check_mark: :cross_mark:
Final class + Literal alias (@tmk) :white_check_mark: :warning: :white_check_mark: :warning: :cross_mark: :cross_mark:
Runtime in on Literal (@beauxq) :cross_mark: :white_check_mark: :cross_mark: :warning: :white_check_mark: :cross_mark:
LiteralEnum (proposed) :white_check_mark: :white_check_mark: :white_check_mark: :white_check_mark: :white_check_mark: :white_check_mark:

As far as why I’m suggesting a new type rather than modifying an existing one:

  • I don’t see a way to make Enum pass #2 without conflicting with code that intentionally requires enum members
  • My suggestion is much closer to “Literal with namespace features” than it is to being an actual Enum (certainly open to a rename if that’s a point of tension)
  • I don’t see a path to giving Literal namespace functionality. That would be a huge leap
12

There is an important property of StrEnum that’s not listed here: it subclasses str and enum members hash to the exact same values as the plain strings do. I’d want LiteralEnum to do the same.

In fact, if specced right (disallowing interface extensions on LiteralEnum), then this could allow nice things like type hinting a function argument with dict[Field, Value] or TypedDict with enum keys, while allowing callers to pass literal dicts satisfying the schema.

2 Likes
13

@randolf-scholz you make a great point and that is definitely a key discussion point I am eager to get feedback on.

Should FieldType.A be …
A. (Proposed) of type str (JUST the raw literal) or of
B. (similar to StrEnum) having a type which is a subclass like str, FieldType?

My thoughts:

  • I absolutely see value in what you are proposing, but for me, the value add is not enough to justify the added complication
  • Your proposal would make LiteralEnum lean towards being more like Enum and less like Literal, that is totally fine, just an observation
  • option B. has some niceties, but also may lead to a lack of trust and immediate recognition
    • when I see the subclassing, I immediately jump to “what broke?”
    • with a good implementation, the answer may very well be that absolutely nothing broke
    • BUT… if I do not have the time to verify myself, I don’t have the immediate trust
    • So… I wind up doing extra work like casting str(HttpMethod.GET), etc. until I read the source code or get convinced
  • I do not personally feel any pain points with the values not being an instance of the special type
    • If I wanted a StrEnum I could use one. It has value on its own, and yes maybe StrEnum could be improved by your PEP propsal or something similar
  • I am more focused on building off of the functionality of Literal than on extending the functionality of StrEnum

While I lean towards continuing to propose option A, I would still be thrilled if option B gained traction, and would help advocate for it.

Do others see advantages of the members having a special type as opposed to being more like a namespace that retains the true raw literal type?

1 Like
14

Just a note: I think you wanted to tag @jamesparrott instead of me in the two places I’m tagged.

15

Maybe it could be modeled after NewType:

from typing import SealedLiteralType

# single source of truth:
HttpMethod = SealedLiteralType("HttpMethod", ["GET", "POST", "DELETE"])

assert list(iter(HttpMethod)) == ["GET", "POST", "DELETE"]
assert "GET" in HttpMethod

def f(m: HttpMethod): ...
f("GET")  # type checker: OK
f("PUT")  # type checker: Error
16

I think the NewType-style factory suggestion is a clean mental model and would be an advancement for Python.

That said, the main thing I’m trying to preserve is a stable runtime namespace (Foo.BAR) as a single source of truth for the allowed literals.

For me that’s non-negotiable, because having an iterable typehint in no way eliminates the need to also have a namespace, and in current code it is not possible to have both using a single source of truth.

If the proposed type does not have namespace functionality, developers will inevitably need to create a second class or enum to maintain the namespace, leading to a second source of truth and lots of room for error.

So I’m very open to factory vs class vs hybrid as long as the result is a namespaced container whose members are the canonical literals.

Separately, just to avoid confusion about intent:

  • I’ve put a small reference implementation on GitHub (modularizer/LiteralEnum), with a PyPI package of literalenum mainly so people can poke at the proposed runtime behavior if that’s helpful.
  • I am not suggesting my source code gets adopted. I know that is not how this works and am sure someone else could right a narrower and cleaner implementation. However, I do like the runtime behavior which it shows
  • That code only handles runtime mechanics today : it does not attempt to solve typing or checker integration in a meaningful way

Happy to adapt or discard the implementation entirely if the group converges on a better API shape.

17

To get namespace support, would it be enough to enhance Literal so that you could do:

State = Literal["ON", "OFF"]
assert State.ON == "ON"

The earlier mentioned

assert value in State

would be handy too.

I’d also love being able to do

value: Literal = "ON"

to avoid repeating the value, but that’s not really related to this proposal.

2 Likes
18

@pekkaklarck This is interesting. Looking back at the litmus tests I defined, it could potentially meet every one if it was combined with a suggestion like that from @beauxq and in your use case and many of mine it would indeed be useful.

However, I see some major conflicts:

# 1. many sting literals are not legal variable names
TERMINAL_COLORS = Literal["\033[34m", "\033[0m"]
NUMBERING = Literal["1_a", "1_b"]

# 2. some string literals may conflict with important attributes and methods
DUNDER_NAMES = Literal["__add__", "__sub__", "__class__", "__dict__"]

# 3. Many times the purpose of a namespace is to make lookups easier, so requiring key == value would be non-ideal
TRANSLATIONS_OF_FOOD = Literal["comida", "nourriture"]

# 4. possibly unexpected/unnecessary attribute on single string literals
X: Literal["x"] = "x" # just a bit odd because following the logic we would also have X.x == 'x'

I think the obvious response to this is to only do it for uppercase non underscored literals, and even potentially limit to when there is more than one item.

But that feels… odd? a bit non-intuitive?

However, it did inspire this idea using the walrus operator:

HttpMethod = Literal[GET:= "GET", POST:= "POST"] # This SORTA works at runtime right now, defining locally-scoped non-namespaced variables of GET and POST, but fails on type checker's rules about literals

or using custom syntax

HttpMethod = Literal[GET="GET",POST="POST"]
# or 
HttpMethod = Literal(GET="GET",POST="POST")

which when paired with

assert "GET" in HttpMethod

could be pretty powerful

19

If I had to summarize thoughts here, it seems like a lot of the pushback centers around not wanting to add a new type to the standard library, and I think that is totally fair.

To clarify, the issue that I am trying to solve:

  • is not that LiteralEnum or a similar thing is not in the standard library: that is fine. Let it prove itself useful first.
  • I see the issue being that LiteralEnum or something similar is not possible in a way that behaves nicely with typehinting and maintains a single source or truth.

With that in mind:
Do you all see any potential support for a generic way to decorate or otherwise make a class with the behavior you wish to see from the type checkers? A solution which allows developers to solve this painpoint related to enums but also hopefully many other pain points?

For instance: when used as a value, treat it as its true type, but when used as a typehint treat it as some specified or inferred static type?

@static_type(Literal["GET", "POST"])
class HttpMethod:
    GET = "GET"
    POST = "POST"

That would not in and of itself allow a single source of truth, but might be extendable like

@static_type(spec=InferSpec(kind=Literal, params="values"))
class HttpMethod:
    GET = "GET"
    POST = "POST"

It is a much broader ask so I see it as less likely but I would happy to be wrong.

20

I feel like it would be nice to be able to just Unpack an into a Literal. Where the unpacked values must be marked as Final

Methods = ('GET', 'POST', 'PATCH', 'DELETE')
Method = Literal[*Methods] # Type Error

Methods: Final = ('GET', 'POST', 'PATCH', 'DELETE')
Method = Literal[*Methods] # Ok

I don’t really see a clean way of combining Literals and Enums though, since an Enum requires the members to be valid variable names like you said.

You could also just have a new LiteralEnum abstract that enforces that all literal values are mapped to enum names:

class MethodEnum(LiteralEnum[Method]): # Type Error (Missing 'DELETE')
    get='GET'
    post='POST'
    patch='PATCH'
1 Like