types.UnionType was merged with wrong ~class~ (not even a class)

Core Development
21

My own personal take on this having read through the thread and the original PR is that we should revert this for 3.14 as it is clear that there is not agreement and that resolving any of this becomes harder with this in place rather than keeping the code as it was in 3.13.

It is too late to be trying to craft new solutions such as fixes or partial reverts for 3.14 given the release candidate phase. Changes need to either be in or out as there is no further time for wider exposure and testing as happens during alpha and beta periods.

Reverting would not mean that it is gone for good, but what could be done around Union vs UnionType may take a different form in 3.15+ where time to work out what makes the most sense among everyone exists.

16 Likes
22

@storchaka is there public discussion of this plan? I haven’t seen any discussion of this, but it’s easy to miss these things.

I’m definitely torn on whether or not this should be reverted. Greg makes a great point that there’s disagreement and not really time to settle that, but I also don’t think it is healthy to have reverts right before release without strong arguments for reversion.

I think we should also not discount that the issue for merging Union has 26 thumbs up - I take that as a signal a number of users are pretty happy with this change, which is even more reason not to revert.

I would also say personally I like the change and think making Union a runtime type should fix an issue I have heard a number of typing users complain about in discussions. I also imagine it would simplify things for projects like pydantic.

4 Likes
23

I do not think it was explicitly writte. But when types.GenericAlias was introduced, it was named but private class typing._GenericAlias (I don’t think it was a great name, but it stuck). They performed a similar function, some effort was made to ensure their interchangeability (largely by me), and it was implied that in the future types.GenericAlias would replace typing._GenericAlias. The latter has additional methods and more strict type control, this is why it did not happen immediately. Something about this could have been clearly voiced in the discussions of those times.

When types.UnionType was introduced, it was initially named Union, but was immediately renamed to UnionType, precisely to avoid confusion with typing.Union. Any arguments for merging typing._GenericAlias with types.GenericAlias are applied to typing._UnionGenericAlias and types.UnionType.

What are these issues? How is this different from the fact that typing.List, typing.Final and typing.Any are not runtime types?

If you mean a runtime type check for a union type, isinstance(int|str, types.UnionType) always worked. It does not work with typing.Union[int, str] (as the types.GenericAlias does not work with typing.Lis[int]), but making typing._UnionGenericAlias an alias or a subclass of types.UnionType would solve this.

4 Likes
24

Some of this feedback was about the fact that we have both special forms and runtime types like list and they wished that e.g. List[T] would be the exact same as list[T]. Some is users being displeased that runtime and static types are different in e.g. isinstance checks or runtime type checking (this was much much worse before list[T] worked, I think this is a rarer complaint now). Not all of these complaints are things we should fix necessarily (e.g. I am sure some of these users want Any and Final in builtins, and that’s probably unlikely to happen, at least anytime soon). But I think making Union/a | b the same as runtime representation might have some value.

25

I see. To make typing.List[T] returning the exact same as list[T], you need to make typing._GenericAlias and alias for types.GenericAlias (and make some other changes). Making typing.List an alias for types.GenericAlias won’t help at all. This is the type of change I oppose.

The same for unions. Making typing._UnionGenericAlias an alias for types.UnionType is one of possible solutions of that issue.

Of course, this is a breaking change, so perhaps more gradual changes would be more appropriate. It takes time to develop a strategy that will please many. The latest release candidates are not the right time. Therefore, this change should be reversed.

4 Likes
26

I want to add a little to help the SC make an informed decision.

The controversial change has been in 3.14 since March, so it was in all betas and release candidates so far. That means people have already been testing it, may have made changes to adapt to the new behavior, and may even already be relying on it. Of course, we can still make changes during the RC phase, but at this point it’s not clear to me that reverting the change will make adoption of 3.14 easier rather than harder.

The arguments for reverting the change seem grounded in philosophical arguments around how typing constructs should behave, not in practical issues with how the new logic will behave. I haven’t seen concrete examples yet of what exactly will break with the new logic.

The PR making the original change does not revert cleanly. There is an open PR from Serhyi purporting to undo the merge, gh-137065: Unmerge types.UnionType and typing.Union by serhiy-storchaka · Pull Request #137069 · python/cpython · GitHub, but this PR actually implements a different variation that is also quite different from the pre-3.14 state. I would be opposed to merging it in the RC phase, because we’d be releasing a new variation of the logic that hasn’t gone through months of beta testing. A “clean” revert with fixing merge conflicts might be possible, but nobody has attempted it yet and it may cause other problems due to interactions with changes that have gone in since.

5 Likes
27

The full revert is available at gh-137065: Revert "gh-105499: Merge typing.Union and types.UnionType (#105511)" by serhiy-storchaka · Pull Request #138967 · python/cpython · GitHub.

4 Likes
28

Merge `typing.Union` and `types.UnionType` · Issue #105499 · python/cpython · GitHub was merged on Mar 4. I accidentally found out about it later and opened issue Merge 'types.UnionType' with `typing._UnionGenericAlias`, not `typing.Union` · Issue #137065 · python/cpython · GitHub and a PR for partial reversion on Jul 24. Since I couldn’t convince Jelle, I opened the general discussion here three weeks later, on Aug 13. Five weeks later, we still don’t have a final decision. The longer we delay, the more painful the changes will be in future.

29

An alternative here is to immediately add an admonition in the documentation saying to not rely on anything afforded by this change, except the high level functionality (just isinstance(obj, typing.Union) I think, right?).

Then for 3.15, the change will be reverted, some progress is made towards Serhiy’s long term plan, and crucially typing.Union’s __instance_check__ is updated to match the behaviour of 3.14.

2 Likes
30

You’ve probably noticed that the 3.14.0rc3 release has been made without reverting the UnionType changes.

The Steering Council took the concerns about this change seriously. We met, discussed, consulted with the 3.14 Release Manager, and ultimately decided not to revert this change so late in the 3.14 release candidate phase of the process, giving the go ahead for the rc3 release. What has shipped is the API going forwards. All problems with it are now normal APIs we need to support and can only change by following the normal change and deprecation policies going forward.

For us, the issue was both technical and process related. On the technical side, we are not necessarily agreeing with the change, and we do recognize that the design may have problems. We also acknowledge that there is disagreement within the core team as to whether this is the right change or not, and uncertainty in some of the future directions for typing support. On the process side, we also acknowledge that while concerns were raised early, it wasn’t until very late in the 3.14 rc phase that the concerns bubbled up to the Steering Council. While rc phase reverts have been approved in the past, they’ve always been for problems affecting Python “globally”, i.e. performance regressions, serious bugs affecting a wide-range of users, etc. We did not feel that this bug met the high bar for rc phase reversion, as it does not appear to fundamentally crash Python, break projects, or subvert user expectations. After weighing all ramifications, options, and outcomes, we decided not to revert this change.

It’s clear that there was a communications breakdown that led to all of this happening. I expect we’ll have suggestions around improving upon that later in order to avoid situations like this in the future. In the meantime, we urge members of the core team to engage with the Steering Council early when impasses cannot be resolved through normal collaboration and discourse. Don’t linger in frustration feeling that your opinions are not being heard! We will always have technical disagreements, and one’s positions may not always be adopted, but we hope that any frustrations you have with the process can be brought to the Steering Council’s attention, and we will do whatever we can to help to promote respectful resolution of such disagreements. We remind folks that you can contact the Steering Council through our GitHub issue tracker and private mailing list, and we encourage you to book a meeting with us using our Steering Council office hours.

We will undoubtedly conduct additional introspection into the process bugs that lead us here, so that we can avoid such outcomes in the future. We thank you for all your hard work in bringing Python 3.14 to fruition. Please reach out to the Steering Council in public or private if you have any additional concerns and feedback.

– Python Steering Council

26 Likes
31

Thank you for the detailed explanation of the Steering Council’s thinking. I think that reversion of this change so late in the release candidate phase or even in the post-release bugfix phase would bring less harm than keeping it for the period necessary for deprecation, but I accept your decision. Any chance to start deprecation in 3.14.1 instead of 3.15, considering it a bug fix?

The following operations should be deprecated:

  • types.UnionType[...]. Because types.UnionType is a class, but not a generic class.

    But typing.Union[...] should be kept – this is the purpose of typing.Union.

  • isinstance(..., typing.Union) and issubclass(..., typing.Union). Because typing.Union is not a type, it is a special form used to create type unions. Using it in the type checks is the same as using typing.Literal or typing.Final in the type checks.

    But isinstance(..., types.UnionType) and issubclass(..., types.UnionType) should be kept – I suppose this was the primary reason of exposing types.UnionType to public.

  • Some dunder attributes and methods like __origin__, __args__, __qualname__ or __mro_entries__() – but I cannot say right now which of them are harmless and are useless. We can leave this for later.

To do this, we need to separate types.UnionType and typing.Union and make the latter imitating the former. This itself is a breaking change.

7 Likes
32

Do these operations even need to be considered supported? Couldn’t they be explicitly documented as not supported already from 3.14.0? That seems like it could be a cheap documentation-only change.

33

Here’s another consequence of changing too quickly and skipping the normal process of deprecation.

3 Likes