Static type annotations in CPython

[srittau (Sebastian Rittau)]

@thomas

I want to encapsulate what was discussed and see if the wider core dev community agrees before trying to commit some of this to the devguide.

Not a core dev, but as a typeshed maintainer I have some comments. I had some previous thoughts in this GitHub issue: Thoughts about type hints in the standard library · Issue #5444 · python/typeshed · GitHub and this thread, linked before by Shantanu: Type annotations in the stdlib - #22 by srittau.

Generally, I see two different goals for annotations in the standard library with different concerns: Type checking the standard library itself and providing type annotations for third party libraries and applications (replacing the stdlib part of typeshed). (Although there is a certain overlap, as often one standard library module can be considered a third party to another one.)

Personally, long term I would like to see both. A fully type checked standard library that also provides type hint to third-party software.

Type checking the standard library

This is the more realistic short-term goal and can be implemented gradually or partially: Check all code that is already annotated, and require the checks to pass in CI. Allow annotations in new or changed code, but don’t require them.

The main concern is how exactly annotated code checks calls of unannotated functions/classes. Fall back on typeshed? Just assume Any?

@thomas

There’s a small pain point when the type-checked code uses features of the stdlib that haven’t been reflected in typeshed yet, but according to the typeshed maintainers they’re happy to take PRs for unreleased features, even if it means potentially rolling them back.

That’s actually the opposite of the policy we recently adopted:

We accept changes for future versions of Python after the first beta for that version was released.

That said, I’m sure we can amend the policy if this helps CPython development.

@hauntsaninja

I’m generally pretty happy with the status quo, but would like to remove unchecked annotations (the multiprocessing example you link isn’t even a valid type hint!)

Definitely, existing annotations should be valid and should be checked!

Providing type annotations for third-party software

@thomas

The typeshed setup is working well, and avoids some difficult problems (e.g. type checkers supporting different syntax and features, and thus needing slightly different annotations in a forked/vendored copy of typeshed). Given that typeshed is supposed to override the stdlib, having the same annotations in the stdlib itself wouldn’t gain us anything, and having different annotations might be confusing to users.

I agree that for the time being, typeshed seems to be the best solution. But long term – as both the type system and the annotations in typeshed mature, making changes less and less necessary, and more core developers gaining experience with typing – having the standard library annotations directly in CPython makes sense to me. This has several advantages:

• Less effort overall required, as the overhead of opening and managing separate PRs to typeshed won’t be necessary.
• Less chance of the type annotations diverging from the implementation.
• Removing the need for the awkward Python version handling in both type checkers and typeshed.

Also, how should extension modules be handled? We’d either need a mixture of inline annotations for pure Python modules and type stub files for extensions modules, or we could use type stub files for all types of modules consistently. The latter would have the advantage that the modules themselves don’t need to be type checked before this could be implemented.

But these are really only questions for the future that don’t need to be answered today.

Type annotation in documentation

@thomas

There’s also the question of how type annotations (in particular in the stdlib) should show up in the documentation. Sphinx has the ability to display them (which we don’t currently do), but especially for complex types it might not produce the most readable output. It is probably a good idea for well-annotated libraries with simple enough types, though.

I believe that using valid type annotation syntax in documentation makes sense and should be quite readable. But as you mention, it’s not always the best idea to use the actual type hints, which are written with type checking in mind.

One example: If I were to document the re.Match.groups function (actual documentation here), I would annotate it like this:

Match.groups(default: str = None) -> tuple[str | None, ...]
Match.groups(default: bytes = None) -> tuple[bytes | None, ...]

This differs from the actual annotations in several ways, and is less accurate, but is much more readable than this:

    @overload
    def groups(self) -> tuple[AnyStr | Any, ...]: ...
    @overload
    def groups(self, default: _T) -> tuple[AnyStr | _T, ...]: ...

Why Any in the returned tuple? What is _T? Why the overload? What’s the meaning of AnyStr here? All of these questions have good answers that are irrelevant for documentation.

Another example would be a signature with many arguments, two of which are mutually exclusive. In type annotations, you’d use an overload to ensure that only one argument is used. But instead of duplicating the long signature, in documentation it’s much more readable to just add the sentence “x and y are mutually exclusive”.

The _typeshed pseudo-module

Another concern with type annotations in the standard library is the _typeshed pseudo-module, which is only available at type checking time. This adds many useful type aliases and protocols. When transferring types into the standard library, these types would also need to copied somehow.