types.UnionType was merged with wrong ~class~ (not even a class)

Core Development
1

The typing module was introduced in Python 3.5 (Preliminary typing.py, anticipating provisional acceptance of PEP 484. · python/cpython@46dbb7d · GitHub). Initially, almost all these generic subscriptable things (List, Sequence, Callable) were classes (with special metaclass). The only exceptions were Union and Optional. Subscripting all of them returned an instance of the same class, so it was not easy to distinguish generic things from non-generic. There was a lot of hacking here.

In Python 3.7 it was re-designed (Implement PEP 560: Core support for typing module and generic types · Issue #76407 · python/cpython · GitHub) – now most of subscriptable things (except Generic) no longer classes, they were instances of _GenericAlias or _SpecialForm. Subscribing them returned an instance of _GenericAlias. Still a lot of hacking.

In Python 3.9 I refactored the whole hierarhy in more strictly typed way (Refactor typing._GenericAlias · Issue #84577 · python/cpython · GitHub), so List and List[int] became instances of different types. Subscribing any of subscrptable things now returns and instance of _GenericAlias or its more specialized subclasses (like _UnionGenericAlias). types.GenericAlias also was introduced in Python 3.9 (Implement PEP 585 (Type Hinting Generics In Standard Collections) · Issue #83662 · python/cpython · GitHub). It was similar to typing._GenericAlias and named alike. typling.List[int] returns typing._GenericAlias, list[int] returns types.GenericAlias.

In Python 3.10, types.UnionType together with type.__or__() was introduced (PEP 604 -- Allow writing union types as X | Y · Issue #85600 · python/cpython · GitHub). It was initially named types.Union, but was renamed in Rename types.Union to types.UnionType · Issue #88895 · python/cpython · GitHub to avoid confusion with typing.Union. It was similar to typing._UnionGenericAlias. typling.Union[int, str] returns typing._UnionGenericAlias, int | str returns types.UnionType.

It was planned to merge types.GenericAlias with typing._GenericAlias and types.UnionType with typing._UnionGenericAlias in distant future (after implementing the missing pieces in types.GenericAlias and types.UnionType).

types.UnionType has been merged during the 3.14 developing cycle, not with typing._UnionGenericAlias, but with typing.Union, which was not even a class. This breaks long planes for types.GenericAlias and types.UnionType and violates some principles.

  • typing.Union never was a class, from the beginning. Even when other generic types were classes. And there were reasons to make other generic types non-classes.
  • types.UnionType should not be subscriptable. It is not generic class, and classes should only be subscriptable to make a specialized type from generic class. The result of subscripting a class is always types.GenericAlias, not types.UnionType.

I suggest two options:

I am sorry that I only found out about this change three weeks ago and we need to make a decision so late. But if we do not make it, I suspect that we will need to spent next few years undoing this change. It will be much more painfully.

8 Likes
2

What concrete problems do you think the 3.14 approach will cause?

I made this change to unify the way union types are represented, so users writing introspection code don’t have to care about two kinds of unions. The model that is now in 3.14 means that introspection is simpler and users don’t have to reach to private classes like _UnionGenericAlias.

1 Like
3

It is interesting that @AlexWaygood made a similar argument to you in the original PR that merged Union and UnionType (gh-105499: Merge typing.Union and types.UnionType by JelleZijlstra · Pull Request #105511 · python/cpython · GitHub). You can read it, he used different words, and may be more convincing.

Two kind of unions are types.UnionType and typing._UnionGenericAlias. Two kinds of aliases are types.GenericAlias and typing._GenericAlias. typing.Union is not a kind of union, it is a special form to create a union. Many efforts was spent to ensure that special forms are distinct from classes. Other problem is that types.UnionType is not a generic class, so it should not be suscriptable, and that implementing subscription with different semantic than for all other generic classes is bad (it may even be forbidden by one of PEPs).

I initially thought that partial reversion may be enough, but after refreshing my memory while writing the above post, I think that complete reversion will be safer. Then we can return to initial plan of gradual changes. The first step iss to make typing._GenericAlias a subclass of types.GenericAlias and typing._UnionGenericAlias a subclass of types.UnionType. This will allow users to use public names for type checks. But there are some obstacles on that way, so this may take a time. Then we may need to deprecate some methods of typing._GenericAlias and typing._UnionGenericAlias. Finally, we will be able to remove typing._GenericAlias and typing._UnionGenericAlias.

8 Likes
4

FYI, I think there are some typos in your OP, which make it quite confusing.

5

Thank you, fixed.

6

With Python 3.14 being a RC already, how will reverting this go?

7

I can’t comment on this proposal as I don’t know the details, but in general, the goal of release candidates is to find and fix bugs before things are ‘frozen’ in the .0 final release. Of course we can fix things after that, but it would become subject to the deprecation policy and take much longer than reverting a change for now and waiting until the next minor version (3.15).

A concrete example from last year is the incremental cyclic GC, which was reverted during the 3.13 release candidates, but is present in 3.14.

A

4 Likes
8

I don’t know the implementation details well enough to vote for any of the possible outcomes, but a decision should be made rather sooner than later. The RC3 is coming in less than 3 weeks.

Some maintainers might be waiting for the result of this discussion before they prepare their codebases for 3.14.

1 Like
9

I see Serhiy’s opinion against Jelle’s, and no clear consensus. The Release Manager (@hugovk) or Steering Council can resolve that. But, time is running out for 3.14.

1 Like
10

I’m currently leaning towards the status quo, but would like to hear more opinions from the core team, especially those working in typing. Or I’d also welcome an SC decision (@storchaka: please contact them right away if you want to do this).

Time is indeed short; RC3 is Tuesday next week: 2025-09-16. If we do revert, let’s do it by then.

1 Like
11

Please file a ticket on the steering council tracker so it’s on our agenda. I don’t have a strong opinion and haven’t spoken with the other SC members, but if @hugovk is okay with it, and it can be changed before the last RC (either the one coming up, or a new RC slotted before final release), then I think it would be nice to fix this for 3.14.

3 Likes
12

To reiterate, I feel strongly that the behavior that is now in 3.14 is correct and desirable. The previous behavior relied on a confusing mix of private and public objects and made introspection harder. The new behavior makes it so unions are represented in a consistent, easily understandable way.

Previously, unions could be created either with typing.Union[int, str], creating an instance of the private class typing._UnionGenericAlias, or with int | str, creating an instance of types.UnionType. Now, both syntaxes create an instance of the same class, typing.Union.

The arguments that this is against some hidden design principle in typing is unconvincing to me. Unions are not all that much like types.GenericAlias conceptually, even if we used to use similar syntax to create them, and their runtime implementation doesn’t need to mirror GenericAlias.

6 Likes
13

If we do, I’d much prefer before next Tuesday’s RC3, as that gives 3 weeks to test before final, and this feels like a big change.

Slotting in an extra RC would make the test time very short: 1 or 2 weeks. RC3 is already an exceptional release due to the magic number bump; the original final RC would have given us 6 weeks.

2 Likes
14

Cross-referencing, Serhiy has made a formal request of the Steering Council:

A

1 Like
15

Most of the arguments to revert have been made. As a user of typing, I agree with most (not all) of those reasons, and don’t think these types should be merged as is.

Merging these more deeply intertwines typing with runtime python. I don’t see that as a good thing, it actually further complicates efforts to delay importing typing at runtime until actually needed.

Runtime introspection of annotations is happening more and more, but the libraries that do it are expected to be very familiar with python’s typing ecosystem.

Furthermore, attempting to simplify this by merging with UnionType directly, rather than the previously planned typing._UnionGenericAlias, doesn’t simplify anything for typing libraries!

typing-extensions exports typing.Union with a “forever forward” versioning because it exists for backports. types.UnionType is not a suitable replacement for all uses, so typing-extensions can’t export this instead currently. In fact, Until typing-extensions has a breaking change policy, it can’t be removed from there. All typing code still needs to branch on this behavior, and introspection code is only simpler if it ignores the blessed way to handle backports (and is therefore broken in my eyes) At worst for these libraries, it’s just selecting one more condition to go into this branch from.

16

I’m not sure what exactly this is referring to, but to be clear, in 3.14 code like “int | str” does not trigger an implicit import of the typing module. (Defining a generic using PEP 695 syntax does, since 3.12.)

_UnionGenericAlias wasn’t “planned”; it already existed. In 3.14 it still exists for compatibility with users who previously used this private class, but it’s deprecated and unused.

In 3.14, the way to check whether a runtime object is a union is isinstance(obj, typing.Union); previously it was isinstance(obj, types.UnionType) or typing.get_origin(obj) is typing.Union. So yes, it does simplify things for runtime typing libraries.

typing-extensions just re-rexports Union without changes. In 3.14, UnionType and Union are aliases for each other. typing-extensions has some special code for dealing with UnionType in earlier versions, which will become obsolete when 3.14 is the oldest supported version.

17

I’m sorry if you found it ambiguous, but “rather than” there would not mean that typing._UnionGenericAlias doesn’t already exist, but that that should have been what it was merged with rather than typing.Union.

See prior comments:

And a long trail of history that starts around this old bpo thread on intentionally Keeping UnionType seperate from Union, (it’s slightly easier to find related history from here)

It doesn’t help that the new repl results in typing being imported and that repr(type(int | str)) lies about it’s own type to claim it’s from typing, but that’s not what I was referring to.

This has people use isinstance on a special form provided by typing, and the repr intentionally leads them to use the reexport from typing, rather than from types, where it actually exists and prior to the merge was more clear in seperating annotation time meaning from runtime meaning.

There has been a concerted effort to intentionally separate the two more distincly and unambiguously, while also having the tools to communicate that “even though this is a runtime context, not a type expression, I want the static analysis to know I intend to use it as a type expression” TypeForm pep in progress.

Either directly introspecting the objects or using the functions from typing was the separation that existed in the other direction, where a runtime tool needs the static type information that was stored.

The more these are intertwined, the harder certain other possible improvements get. Taken from a recent conversation with @Sachaa-Thanasius (my apologies if I accidentally misrepresent some of intent below, please correct me if so) in another location, an example of inappropriately mixing these that is already a problem:

Bit random and idk if this makes sense, but I’ll take any feedback:

I was messing with isolating parts of typing so that it can be segmented, but there are a few ubiquitously used functions (e.g. _type_check()) which special-case a bunch of the special forms (Any, ClassVar, etc.) and even Protocol and Generic. Even the C code for typing and annotations stuff late-imports typing to call into these functions. I thought at least Generic could be isolated, but it calls back into, like, 5 different functions from typing which can’t be moved out because they specify specific special forms by name.

Any ideas for how one might go about breaking up/untangling this without giving up and doing a full rewrite in C?

The C code happens to late import already, but there’s a lot of added complexity by mixing runtime validation functions with what the language actually needs to provide, and I view merging like this to be alarmingly similar. Something I was thinking about proposing was actually to remove most of the runtime validation functions that run automatically upon creating certain types. The rationale for that from the same discussion:

If it’s not valid as a generic param, have static analysis (typecheckers already error for this statically) or the runtime tool consuming the annotation raise that, not the stdlib (edited)

separating things at the right level and location of concern also allows better expression even within just the standard library. The motivation for the above attempt at splitting typing that got complicated

The original motivation for this, uh, thought exercise, was the thought that reorganizing like this would allow separating Protocol from typing, which is a better representation of Python’s runtime typing than abc.ABC — the latter got a more fundamental place in the stdlib, Protocol should as well. Open the door to use it in a bunch of places that abcs are currently used where protocols are intended (see: abcs at runtime that are protocols in typeshed)

typing.get_origin performs normalization to the preferred cannonical form typing provided typeform equivalents and is documented as doing such some normalization. It already does this for the collections module. If the concern is to simplify this, I think you’ve taken the wrong approach, and should just add normalization in typing.get_origin from types.UnionType → typing.Union, This being separate as the actual implementation type, but having a way to normalize to a valid type information for typing users better matches other goals and decisions.

4 Likes
18

It’s worth noting that using the first version there (the one for 3.14+) is a type error prior. If anyone prefers the argument about prior plans rather than keeping unambiguous seperation going forward, you can find older history here on decisions by poking around the git blame for the __instancecheck__ on special forms that applied prior to this merge. It was added ~7 years ago, and there are specifically contemporaneous comments from the author and others about introspection functions for typing being planned (and we got those!) specifically instead of using isinstance.

I don’t particularly find the historical the more compelling argument over the clear delineation between looking at the details of a runtime value and a static typing value, but it exists (and more history than this)

1 Like
19

Having implemented runtime introspection a handful of times, I must agree that it seems like a mistake to revert this simplification. The arguments for reverting this seems theoretical and artificial to me.

1 Like
20

My vote is to revert. Alex and Serhiy’s arguments are compelling. Once released, this will be painful to fix.

1 Like