PEP 727: Documentation Metadata in Typing

PEPs
82

As a maintainer of Jedi I was asked to provide feedback here. Thanks for reaching out!

I generally agree that it would be nice to have documentation for params in a more structured
way. There are a few things that could use improvement:

  1. I feel like there should be some sort of typing.get_doc() call for annotations at runtime. It’s just very normal that one is able to extract information from the typing module. Unlike PyCharm/mkdocs/VSCode, Jedi isn’t just a static analyzer, it can also be used within IPython/Jupyter and extract runtime information.
>>> def f(a: int): ...
...
>>> a = inspect.signature(f).parameters['a']
>>> a.annotation
<class 'int'>
>>> typing.get_doc(a.annotation)
  1. Like others have pointed out: It feels a bit wrong to use Annotated for this. If this is a really a Python stdlib provided feature, I would probably prefer something like Doc[str, "The name of the user"], which is also shorter and more concise.
def foo(
    x: Doc[str, "Lorem Ipsum"],
    y: int,
) -> Doc[bool, "Lorem Ipsum Dolor"]:
    ...

I feel like that’s way more readable and for the tools like Mypy/Jedi/VSCode/etc, this is not a lot of extra work.

83

This feels like the key thing here. If I can indulge in a little history, originally annotations had no defined semantics, and they were explicitly for people to use for whatever they wanted, to encourage experimentation. Typing was always one anticipated purpose, but not the only one. But nobody really did much with annotations - there were a few experiments, but not much mainstream use.

So then the decision was made that annotations were for typing - it was always an intended use case, and no-one else had come up with other compelling uses, so let’s make the decision.

But once types became accepted, and common, people started finding other things that could be attached to variables, parameters, etc, and all those use cases that had never come up before started appearing. And so we have Annotated, marking things as deprecated, and discussions like this, about how we can cram non-typing annotations into the type system.

Maybe what we need to do is take a step back, and thing about how we can make non-typing uses of annotations sit more comfortably with type annotations? Not with Annotated, which frankly feels like a hack (and an unappealing, clumsy one at that…), but with some form of first-class way of allowing other use cases to grab a part of the annotation space for themselves, separately from typing.

Something like Doc works a bit like this, although it’s still not obvious how I’d use it to attach a docstring to something I didn’t want to declare a type for.

I guess what I’m really saying is that Annotated doesn’t really work, because it’s based on a presumption that anything that’s not a type is a “second class citizen”. And what we need is to re-think Annotated and produce something that’s less biased towards the “everyone uses types” mindset that tends to prevail in the typing community (for obvious and understandable reasons).

12 Likes
84

My current opinions:

  • document is not type. It would be nice if docstring is available without typing module.
  • It is really nice if docstring is available from both of runtime and statically (AST or CST).
  • I am worrying about annotation and Annotated are overused. I’d like annotation is just type hint.

So I prefer one of these approaches:

  • Add new syntax for function argument docstring.
    • e.g. def (a "comment a" : int, b "comment for b" : str)
  • Add some formalized text structure for function docstrings.
    • At least C# has this, although I don’t like XML.
    • PHPDoc looks like almost standard.
9 Likes
85

I’d also like new syntax for this, maybe

def add(a: int ("an integer"), b: int ("another integer")) -> int ("result"):
    return a + b
86

Those are parsed as function calls and you can’t avoid that due to runtime introspection potentially wanting to use function calls to produce some object that represents the type.

87

I like the idea of having documentation closer to the definition of the field. It’s easier when you have both of them in the same place.

I also like the idea of having a standardized convention of defining documentation per attribute and thus a standardized way to introspect these.

However, I have a few problems with the proposed syntax:

  1. It feels awkward to use the Annotated feature for this. The need to import something from typing in order to achieve this does not seem fun.

  2. Documentations may be quite long, writing them in the field definition will mostly “force” you to write the annotation over multiple lines (assuming you want to adhere to PEP8).
    It may disturb the ease of reading the actual definition.

While I assume it would be a harder implementation to write, may I suggest an alternative like expanding the current __doc__ feature to work in other contexts?

Something that will (roughly) look like so:

def foo():
    """
    Currently, this documentation will be automatically set on foo.__doc__
    """

class Bar:
    a: int
    """"
    The new feature allows this is documentation to be automatically set on Bar.a.__doc__
    """"
1 Like
88

Similar to the class attribute docstrings, “K&R style” declarations could be nice, as they separate the runtime and “static” (type, docstring) information, while being backward compatible. (It would still need a new place to store parameter docstrings, e.g. a new Documented[] in __annotations__.)

def frobnicate(widget, param, value):
    """Set widget's parameter to value."""

    widget: BaseWidget
    """The widget to frob."""

    param: str
    """The parameter to frob."""

    value: int
    """Parameter value to set."""
6 Likes
89

I released typing-extensions 4.8.0 yesterday with support for typing_extensions.Doc: typing-extensions · PyPI

90

I’ve updated this PEP 727 example to use typing-extensions: