Version metadata -- `@typing.since(version="1.2.3")`

Ideas
1

This is a quick “Pre-PEP” for an idea I’ve had while working on pytest and its large ecosystem of plugins, where compatibility issues often crop up. Not fully fleshed out but hopefully gets the idea across.

Abstract

Add a @typing.since(version="1.2.3") decorator for classes and functions in a distribution package, to allows static analyzers to check for version compatibility.

Motivation

Packages evolve over time and add new APIs. It is often the case that users of the package want to support multiple versions of the package. For example:

  • An external pytest plugin wants to support multiple past versions pytest.
  • An external Django app wants to support multiple past versions of Django.

Such “addons” are most likely developed against the latest version of the package. Now, whenever the developer wants to use some class/function from the package, they need to go through a little process:

  • In which version of the package was this function added?
  • Is this version >= my minimum supported version of the package?

This proposal offers an improved workflow:

  • Developer tries to use the function.
  • The static analyzer (type checker, linter, code editor) puts up an error - function is too new.

Currently, type checkers offer this capability to the Python standard library as a special privilege. They do this by statically analyzing sys.version_info comparisons, and having “target python version” configurations. The stdlib type annotations (typeshed) utilizes this support. This is evidently very useful. Let’s democratize this capability.

Example

In Django version 100, a new function is added:

# In django

@typing.since(version="100.0.0")
def ask_skynet_to_write_my_website() -> None: ...

An external Django app django-wonderful-app supports Django>=4.2 and tries to use the new function:

# In django-wonderful-app
django.ask_skynet_to_write_my_website()

The static analyzer complains (the exact message is non-normative):

app.py:1: django.ask_skynet_to_write_my_website() is only available since django version 100.0.0, but this project declares support for Django>=4.2

Rationale

TODO

Backwards Compatibility

since will be added to typing_extensions to allow its use in older Python versions.

Specification

Add the following function to the typing module:

def since[F: Callable[..., Any]](*, version: str) -> Callable[[F], F]:
   def decorator(class_or_function: F) -> F:
      return class_or_function
   return decorator

Type checkers need to statically figure out a few pieces of information:

  • The distribution package to which each file being type-checked belongs to.
  • The minimal supported version of each distribution package in the current type-checking session.

Using this info, the type checker will attach a “since version” metadata to each class and function and check compatibility using standard Python version comparison at each call site in user code. If the class/function has no version metadata, it is considered compatible.

If @typing.since is used outside of a distribution package, e.g. on some standalone file, the static analyzer should ignore the annotation (TODO: Maybe better to warn).

Rejected Ideas

Sphinx versionadded directive

The Sphinx documentation generator supports a versionadded directive. Static analyzers can interpret this directive in docstrings.

I think static analyzer would prefer not to look at docstrings, since Sphinx syntax is not a Python-language standard, and an official Python “JavaDoc”-like docstring format is unlikely to happen at this point. Instead, Sphinx’s autodoc could start interpreting @typing.since and convert it to versionadded.

Provide the metadata in the form of if statements

In typeshed type stubs, mentioned in the Motivation section, versioning is achieved with if statements, for example:

# In itertools.pyi

if sys.version_info >= (3, 12):
    class batched(Iterator[tuple[_T_co, ...]], Generic[_T_co]): ...

We could generalize this pattern to arbitrary packages:

# Some file in django
if django.version_info >= (100, 0):
   def ask_skynet_to_write_my_website(): ...

I prefer typing.since for the following reasons:

  • There is no simple standardized version_info that the type-checker can statically rely on as far as I know.

  • Doing this in inline annotations rather than type stubs is not going to fly with developers, in my opinion.

  • As currently implemented in type-checkers (as far as I know), the error given is "itertools.batched is not defined` rather than a more informative “itertools.batched is only available since version 3.12”. This can probably be implemented, but might not be straightforward.

While the idea is rejected for the definition site, it can be very helpful for the call site, to allow for conditional use:

# Minimal required Django version: 4.2.0
if django.version_info >= (100, 0):
    # Static analyzer doesn't complain about version compatibility here.
    django.ask_skynet_to_write_my_website()
else:
    # Do it the old-fashioned way...
    ...

Minimal-versions static analyzer run

It is possible to address the issue by doing an extra run of the static analyzer in an environment which installs the minimal versions of packages rather than the maximal versions. This is possible and even desirable (certainly for tests), however it requires extra effort and does not offer the same developer user experience as a red squiggly in the code editor.

Prior art

There are many languages which support this feature for “platform” versions (e.g. language/stdlib version, Android SDK Level, etc.), but I’m not aware of any which support it for external packages. I’m sure there are some, references welcome!

Open Issues

  • Do modern Python packaging standards (pyproject.toml etc.) actually allow for obtaining the information described in the Specification statically?
  • Support “removed in version” – for e.g. stubs which support multiple versions, or a custom post-deprecation “use that instead” message?
  • Support variables and parameters? (e.g. Annotated[SinceVersion("1.2.3")]?)
  • Interaction with @overload?
  • Should the decorator make the info available at runtime?
1 Like
2

Have you seen PEP 702: Marking deprecations using the type system ? I think this is a similar idea, if I understood properly.

3

A nice thing about the if django.version_info >= (100, 0): variant is that it’s more powerful: it can be used not only to support new functions, but also type changes, new attributes, new parameters, and anything else. Type checkers could implement it by querying the package metadata for the installed version of Django. At runtime importlib.metadata.version returns this version, though it returns a string rather than an int.

The other two downsides for this idea are real, though: it is more verbose, and it is harder (though not impossible) for type checkers to give error messages that state that this value only exists in some specific version.

4

This doesn’t seem at all like something that should be classed as “type checking”. Certainly it’s potentially useful, and it’s something that a tool which statically analyzes Python code could do, but why does that make it “type checking”? Please don’t, by classing this as “type checking”, make it somehow seem inappropriate for other types of linter to implement a check like this.

(In case it’s not clear, I’m strongly against the OP’s since decorator living in the typing module for the same reason - this isn’t typing related).

3 Likes
5

This doesn’t need any support from the standard library at all if a type checker wants to do this kind of check, and I agree with it not belonging in typing. All it would take is a type checker being willing to implement it and grab each supported version of the library, then check that your usage is consistent with each supported version.

6

Indeed. And just as with PEP 702, static type checking is just one kind of static analysis, and I don’t think typing is the place for static analysis that has nothing to do with types.

1 Like
7

Maybe there could be a stdlib type somewhere (probably somewhere in importlib?) called something like VersionInfo that signals to type checkers and other tools that this is a constant used for these kinds of checks. An even stronger suggestion would be to have this be a typing special form so that you can write VersionInfo["django"] as an annotation and type checkers understand that they are supposed to look for the install version of this package without the need to analyiss improtlib calls or requiring developers to hardcode the versions.

8

The packaging ecosystem has an extensive infrastructure for specifying versions, recording what versions are installed in a given environment, and checking if versions match a given specification. You can get a package’s version at runtime from importlib.metadata, and you can do the same statically, if needed (all of the data is in standardised static files). I don’t think it would be sensible or advisable to duplicate all of that with a new facility designed from scratch.

1 Like