PEP 727: Documentation Metadata in Typing

[tiangolo (Sebastián Ramírez)]

@PawelSwiecki

I’m curious, what are ways other (modern) programming languages deal with documentation metadata. Maybe it’s worth adding a comparison?

Yep, I’ll add a section with that. I’ve only seen variations of docstrings, in comments or so.

---

@erictraut thanks for the very detailed feedback, super useful, great to see you here.

I wish PyCharm and Jedi would also chime in, it would be ideal to build this having in mind all the interested parties, as much as possible.

@erictraut

I think it should be made clear that a doc string annotation has meaning only when used in conjunction with the declaration of a symbol, and it documents the intended meaning and use of that symbol.

I think this makes sense, and is the most correct/exact… nevertheless…

@erictraut

it says that to: Username “is equivalent to” to: Annotated[str, doc(…)]. That doesn’t make sense because the doc string for the type alias Username documents the symbol Username, which is a type alias

You’re right that it would technically make more sense to have this document just that specific symbol.

On the other hand, I think a possibly common use case could be wanting to share the same doc for the same parameter in multiple functions. And I can only imagine doing that with an alias.

Or for example, if there was a TypedDict for the params, e.g.:

class Params(TypedDict):
    first_name: Annotated[str, Doc("User's first name")]
    last_name: Annotated[str, Doc("User's last name")]
    age: Annotated[int, Doc("User's age")]

def create(**kwargs: Params): ...

def update(id: int, **kwargs: Params): ...

I would like to imagine tools/editors could show in the tooltip for each keyword argument parameter the respective doc string.

And if users are expected to interact with just create() and update(), I don’t think they would easily end up hovering over Params to get their tooltips.

So, on one side, I agree that what makes more logical sense is that it could only document that specific symbol. On the other hand, allowing them to transfer in some way would solve some use cases.

…still, I was thinking that adding that logic of transferring the doc from an alias to a parameter could be too involved for implementers (you) so I marked it as non-required. But you’re actually the one to tell me… what would be doable? Do you see a way this use case could be solved?

If not, or if the logic just doesn’t match and there’s no way to solve it, I would be happy to just update it to say it just documents whatever is the specific symbol it is attached to.

In any case, after clarifying that better in the “spec” part, I will remove the “Additional Scenarios” part. It was intended as clarifications and I can see it has only added confusion.

@erictraut

That implies that the argument can be specified either positionally or by keyword

Yep, thanks. I just updated the implementation in the typing_extensions PR and will update the PEP once a couple of other related PRs are done. It’s now gonna be just a Doc class, with a single positional-only argument.

@erictraut

Markdown

I would definitely prefer Markdown, but I thought that would be even more debated than this already is.

From what I’ve seen, seasoned experts (most here) are much more familiar and happy with reST, and most “newcomers” (including me) have not been able to get the hang of reST and have always preferred Markdown. So I would personally definitely prefer that. But I was delaying that to another PEP with its potentially long discussion…

Now, if no one is particularly opposed to default to Markdown (apart from being opposed to this PEP in general), I would make it the default.

Another option is to have another Doc() parameter with the language name, the same way we annotate here in Discuss a fenced code block. But then that’s even more stuff in this PEP that could be controversial, so, not sure.

Another option is to adopt another convention for this, like fenced code blocks, of saying that in a multi-line string, if the first characters are not a new line, those define the language. E.g.

"""markdown

Here's the **content**
"""

…but that lands in microsyntax as well, so, not sure.

If no one opposes directly to Markdown (specifically) I’ll make it the default in the PEP.

@erictraut

ParamSpec captures

Indeed, this is one of the cool things I have wanted/expected, I do several things with ParamSpec (e.g. Asyncer’s asyncify) and it would be very helpful to be able to transfer the tooltips/help from one function to the wrapped/decorated one.

I’ll add a note about it.

@erictraut

As others have already suggested, I think it would be useful to include a short section that provides a survey of other modern programming languages and how they deal with the problem of documenting parameters (and potentially type aliases and class-scoped variables).

I’ll add it.

@sirosen

Even if the approach doesn’t port directly, I think click’s existence as a library which has solved this problem fairly well

Indeed, Click is great. I built Typer on top of it, to add pretty much this that I’m proposing, on top of Click.

@brettcannon

I think I might be the “teams behind VS Code” that Sebastián is refer to since he emailed me.

Thanks for chiming in! (and for the early feedback months ago). …when mentioning VS Code and opinions I was actually referring to Pylance/Pyright, and Eric already replied above.

@AA-Turner

Sphinx would implement support for this, mainly because we’d have to. It would bring challenges though, including that people often put typing imports behind TYPE_CHECKING blocks & Sphinx uses a runtime importer, making extracting the documentation harder.

This is good to know.

I can imagine if people do that, they are also using from __future__ import annotations (or in the future when PEP 649 is already in).

This problem would also apply for libraries re-using this at runtime (e.g. FastAPI, Pydantic). Although, in those cases, the interest in using those libraries would be enough to update the code to not run into that. But maybe if they are using Sphinx they would also have interest in making that work…

Do you see a way this could be solved? Or anything that could be said in the spec about that that would make things easier for Sphinx?

@AA-Turner

I think this should be included as a rejected idea, with rationale. For example, why not lift up a string literal using the current rules for docstrings after any varable or parameter into a object.__doc_attrs__?

I should probably add that, around backwards compatibility with previous versions of Python (that would only work at runtime for newer versions), and return types.

@AA-Turner

I think there is a risk that by moving only parameter documentation out into “structured metadata” we relegate the docstring to a “legacy solution”, and loose the ability to capture the nuance we’re currently able to.

I think I should add a specific note about it, that a docstring should have higher priority than parameter docs, e.g. if choosing only one OR the other, a docstring should win.

That wouldn’t change how people would use it, which I can’t foresee, but at least show the intention of the PEP.

[AA-Turner (Adam Turner)]

@tiangolo

If no one opposes directly to Markdown (specifically)

As a data point, I would oppose a specific reccomendation of Markdown – PEP 287 still exists, and I (at least) find reST a more powerful language. Markdown additionally has a problem of which markdown you’re using – CommonMark, GFM, Pandoc Markdown, MyST, etc etc.

A

[AA-Turner (Adam Turner)]

@tiangolo

Do you see a way this could be solved?

PEP 649 will solve this, I believe, though only for the Python versions from which PEP 649 is implemented. For runtime users of annotations between now and then, I’m not sure, as e.g. import cycles or loading speed are strong drivers to guard imports. Sphinx tried to work around this by setting TYPE_CHECKING to True in a recent release, and this has caused several issues.

This is part of my hesitation with putting documentation into the type system in general – because of said import cycles, load times, etc, there’s been a multi-year history of an unhappy runtime-use settlement. Documentation (being just a static string) doesn’t have these problems and it does feel a bit odd to me that this is the route we’re advocating. MAL’s PEP 224 (mentioned earlier) I think could do with being re-read, as it covers a lot of the same ground as here.

Had PEP 649 been the status quo, I think I’d have no (very few) objections from the implementation perspective, as annotations wouldn’t be as special as they are now, but given that’s not the case I would like to at least properly explore putting parameter/attribute docstrings in some alternate location than the type system.

@tiangolo

I think I should add a specific note about it, that a docstring should have higher priority than parameter docs, e.g. if choosing only one OR the other, a docstring should win.

I’d agree with this – but in a hybrid model, what would the PEP reccomend? Should I describe caveats for a single parameter in that parameter’s doc(...) data, and information that covers multiple parameters in the docstring, or should I duplicate it across all doc(...) data? Where should documentation tools place the extracted doc(...) data when rendering a function’s description?

As a further aside (sorry!), how should such a documentation tool treat an Annotated[..., doc(...)] call? Taking Sphinx as an example, currently anything placed in a function parameter annotation is faithfully reproduced in the rendered HTML, LaTeX, etc, and potentially cross-referenced to the definition of the type used. If Annotated[..., doc(...)] became the norm, should such a documentation tool strip out all doc(...) instances?

@tiangolo

I should probably add that, around backwards compatibility with previous versions of Python (that would only work at runtime for newer versions), and return types.

If viewing this proposal as a standards-track Python improvement, rather than simply as a ‘typing’ feature, I think we should view compatability with previous versions of Python as a nice-to-have / bonus, rather than as a strong argument in favour of any solution – back compat can usually be made to work, even if it is more of a challenge. I’d prefer to have a settlement we all agree is good & Pythonic, rather than just the easiest ^[1].

---

Thank you again for engaging with all of our feedback so thoroughly – and sorry for having so much of it, I had no idea this PEP existed before it became a PR to the PEPs repo!

A

---

1. Of couse, if the easiest solution is the best, that’s a bonus for all! ↩︎

[anentropic]

the thread is quite long so I didn’t read past the first few replies, apologies if this idea has already been mentioned

most of the examples seem to be documenting the contextual meaning of an argument to a function, rather than the type itself

having that documentation attached to the type annotation associated to that argument seems conceptually wrong

and I don’t much like the way it reads, I concur with earlier comment that it obscures the type

[pf_moore (Paul Moore)]

@AA-Turner

Documentation (being just a static string) doesn’t have these problems and it does feel a bit odd to me that this is the route we’re advocating. MAL’s PEP 224 (mentioned earlier) I think could do with being re-read, as it covers a lot of the same ground as here.

I think this is an important point. Why is documentation being viewed as type information here (in the sense of being attached to the object’s type information)? Documentation and type information are conceptually two very different things, and it feels like this PEP is proposing to put docs in the type “because it’s a convenient place that we have available already” rather than because it’s the right place for this data to go.

I think one of the reasons I find this proposal so hard to understand is because for me, documentation and type information are conceptually almost completely distinct. Documentation is (IMO) important enough to be considered on its own merits, not as simply an adjunct to type information.

[pawamoy (Timothée Mazzucotelli)]

@AA-Turner

Sphinx tried to work around this by setting TYPE_CHECKING to True in a recent release, and this has caused several issues.

Our approach in mkdocstrings (Griffe really) is that we extract documentation statically, by parsing and visiting source code. We only import modules if we cannot find sources, and in that case objects either have proper annotations (not stringified ones, and that can be an issue too sometimes), or no annotations at all. I understand that if Sphinx does not work that way already, that would require huge changes. For the rare cases where users force modules to be imported (for various reasons) while sources are available, we’re in the same situation, and hope that PEP 649/else will come to the rescue.

@AA-Turner

Where should documentation tools place the extracted doc(...) data when rendering a function’s description?

That’s a good question, that PEP 727 should definitely ask, and maybe provide a few leads for. I’m considering supporting placeholders in docstrings, defined by users (with default values), used to insert the collected documentation:

"""Summary of my function.

<!-- parameters -->

Examples:
...
"""

Any markup is supported since users would be able to define the placeholders. The above example uses HTML comments because that’s how you add comments in Markdown.

@AA-Turner

how should such a documentation tool treat an Annotated[..., doc(...)] call?

In mkdocstrings, I’ve had success stripping everything around the actual type:

  {%- elif expression.classname == "ExprSubscript" and expression.canonical_path in ("typing.Annotated", "typing_extensions.Annotated") -%}
    {{ render(expression.slice.elements[0], annotations_path) }}

…but that means it would also strip Gt, Len or Predicate coming from annotated-types, which might not be what users want. It’s a bit more complex to remove just doc("..."), but feasible. Again, this is something that is worth mentioning in PEP 727

[thejcannon (Josh Cannon)]

@tiangolo one thing I didn’t see in the PEP and the discussion this far is how this is expected to play with -OO, which currently is documented as stripping docstrings.

I love the idea of this, especially the locality of the docstring, but I share the same concerns about mixing docstrings into type information. And I think -OO is a good example of where this breaks down.

[ofek (Ofek Lev)]

Another thing I think would be important to note is that (unless my experience is atypical) users far more often require looking at type information rather than semantics. That is to say, after the first use of a function you might be looking back and forth at the types but rarely do you need to immediately see what the arguments do because you have already internalized that.

So if my interpretation of how programmers generally operate is accurate, then the base case would become more cumbersome.

[tiangolo (Sebastián Ramírez)]

@AA-Turner

As a data point, I would oppose a specific reccomendation of Markdown – PEP 287 still exists

Thanks! I won’t change any of that in this PEP then.

@AA-Turner

For runtime users of annotations between now and then, I’m not sure, as e.g. import cycles or loading speed are strong drivers to guard imports

Yep, we get the same problems with FastAPI, Typer, Pydantic, etc.

@AA-Turner

MAL’s PEP 224 (mentioned earlier) I think could do with being re-read, as it covers a lot of the same ground as here.

Great read, thanks! Should I mention it on 727 as previous work?

I see the main drawbacks from that PEP 224 were:

1. The syntax you propose is too ambiguous: as you say, stand-alone string literal are used for other purposes and could suddenly become attribute docstrings.

This is definitely not the case, it’s actually too verbose because it’s very tightly coupling the documentation and the symbol.

1. I don’t like the access method either (__doc_<attrname>__).

This PEP doesn’t involve new dunders or anything else, so that should be fine.

It might be useful, but I really hate the proposed syntax.

There’s a high chance the BDFL, BDFL-delegate, or steering council just don’t like the syntax of PEP 727.

Nevertheless, then there was a clarification on what was the problem with the syntax:

It’s not the implementation, it’s the syntax. It doesn’t convey a clear enough coupling between the variable and the doc string.

Which this PEP covers, if anything, it’s too much coupling between the variable and the doc string.

Knowing about this PEP, I’ll also ping Marc-André, as I think he could have valuable input here as well.

@AA-Turner

I’d agree with this – but in a hybrid model, what would the PEP reccomend? Should I describe caveats for a single parameter in that parameter’s doc(...) data, and information that covers multiple parameters in the docstring, or should I duplicate it across all doc(...) data? Where should documentation tools place the extracted doc(...) data when rendering a function’s description?

I think what you imagine and mention would probably be sensible, but I’m not sure about putting recommendations around this in the PEP. I think current microsyntax doctsring formats don’t have too strict recommendations around this either: where should things be documented, in the prose or in the param-specific section, and how should tools render the prose and the param-specific section.

It think it’s also a bit more dependent on the tool and UI. E.g. I can imagine an editor that shows the doc for a specific parameter that is currently being filled right on top, and another tooltip on the side showing the function docstring (prose).

And I fear putting to strict restrictions/recommendations around these things would bring the same additional debates as enforcing Markdown/reST. So I would rather delay that for another PEP, if anything.

I think it’s okay, at least for now, to leave that decision to the tool… unless Sphinx, Mkdcostrings, Pyright (and hopefully Jedi and PyCharm if they came) agreed on what would be the best conventions and had the same idea…

@AA-Turner

As a further aside (sorry!), how should such a documentation tool treat an Annotated[..., doc(...)] call?

I think this is probably worth mentioning in the spec. I think, again, the tool can decide. I think Pyright (VS Code/Pylance) shows only the type and hide anything else in Annotated. But I guess it would be considerable for the tool to allow parameterizing this, e.g. for docs sites, if Annotated and contents should or should not be shown. It could also be an expandable/dropdown that by default is hidden. Anyway, I think it’s worth mentioning that this is something to consider (and it already is, regardless of this PEP).

@AA-Turner

Thank you again for engaging with all of our feedback so thoroughly – and sorry for having so much of it, I had no idea this PEP existed before it became a PR to the PEPs repo!

Thanks in particular to you for the technically focused and gentle feedback (as I also mentioned on the PRs), without negative emotions (sarcastic criticism, derogatory comments, etc.).

@thejcannon

@tiangolo one thing I didn’t see in the PEP and the discussion this far is how this is expected to play with -OO, which currently is documented as stripping docstrings.

I love the idea of this, especially the locality of the docstring, but I share the same concerns about mixing docstrings into type information. And I think -OO is a good example of where this breaks down.

Thanks for the feedback @thejcannon! I actually don’t understand, sorry, what does -OO mean?

@ofek

Another thing I think would be important to note is that (unless my experience is atypical) users far more often require looking at type information rather than semantics. That is to say, after the first use of a function you might be looking back and forth at the types but rarely do you need to immediately see what the arguments do because you have already internalized that.

So if my interpretation of how programmers generally operate is accurate, then the base case would become more cumbersome.

That’s a fair point. Although I would expect consumers to not need to go to the implementation, especially if it already has type annotations, I personally would normally rely on what the editor tells me are the types of the parameters.

Although it would indeed affect library authors that deal with those doc strings.

If for those authors it’s easier to deal with one of the microsyntaxes in docstrings than with this, then it would indeed be cumbersome.

For me, personally, I would be more comfortable with dealing with the verbosity in exchange of the certainty that if I remove a parameter it goes away with its docs, and if I add one, I would notice right away it doesn’t have docs yet. But again, personal taste…

And as this hasn’t been done before, who knows, I myself might hate it. (although I doubt it).

But once it’s in typing_extensions I’ll use it for FastAPI, Typer, SQLModel, and Asyncer, I think that’s a good testbed.

[thejcannon (Josh Cannon)]

@tiangolo

Thanks for the feedback @thejcannon! I actually don’t understand, sorry, what does -OO mean?

It’s a flag for Python itself:

1. Command line and environment

Do -O and also discard docstrings. […]

Or as an example:

josh@cephandrius:~/work/pants$ python -OO
Python 3.8.10 (default, May 26 2023, 14:05:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Spiderman:
...     """I don't feel so good Mr. Stark..."""
... 
>>> Spiderman.__doc__
>>>

[johnthagen (John Hagen)]

@ofek

So if my interpretation of how programmers generally operate is accurate, then the base case would become more cumbersome.

I was going to make this same point. Being able to very quickly scan the type hints of a function is currently a nice feature of the status quo.

Also, are there any other languages that put parameter docs into the syntax like this? The PEP could be supported by prior art, if it exists.

My guy feeling is this could start as something specific to FastAPI/Typer/etc to test out first.

[dimaqq (Dima Tisnek)]

@tiangolo

Strawberry

Sorry, what project is that? There’s straberry on PYPI, but that hasn’t had an update since 2011. There’s also strawberry-graphql and StrawberryFields but I’m not sure if either of those is being referred to.

[dimaqq (Dima Tisnek)]

From the PEP

firstname: Annotated[str | None, doc("The user's **first name**")] = None,

I imagine that this is meant to read “Optional user name” or “User’s first name, if they have one” or something like this?

After all, what’s annotated is either str or None…

This actually brings me to a more specific concern. Sorry if this is a bit long:

I feel that the proposal is a little weak with these inlined examples, however it gets stronger if the types are hoisted to the module level and reused, for example:

Surname = Annotated[str, doc("Person's family name, canonical spelling in native script. BOM is not allowed")]

async def register_birth(..., surname: Surname): ...

async def register_name_change(..., old: Surname, new: Surname): ...

async def register_historical_record(..., surname: Annotated[Surname|None, doc("Optional family name, if known")]: ...

What are the semantics of applying doc(...) again? Is the doc annotation replaces, or somehow merged?

Edit: I guess my exact case is covered by “nested annotations”, what if the type is same though, e.g.:

async def register_foreign_document(..., surname: Annotated[Surname, doc("restricted to Latin script")]: ...

[drunkwcodes]

I don’t like the idea which blends type hint and docstrings. It’s not elegant at all.

The docstring generation is simple by using github copilot or codeium. It is easy to make the docstring in sync with new code. What’s more, we can choose the style of docstrings with those AI.

I think this is a solved problem, and doesn’t need new typing solution or standard, thanks.

[flyinghyrax (Matt Seiler)]

I personally have mixed feelings about Annotated, and don’t like the mixing of type and other metadata, but annotations aren’t (technically^[1]) typing-specific.

I can also understand this needing some standardization. Although Annotated is explicitly for arbitrary metadata and there’s nothing stopping a library from adding doc(...) right now, the usage would be limited to cases like typer where the metadata would appear in things like CLI usage strings. Without some interoperability standard it doesn’t make sense for development tools to add support for doc(...) in all the other places docstrings get used.

What stands out to me is that there is some difference between annotations-as-types and annotations as other stuff. I feel the same way about PEP 702. Why would doc or deprecated go in the typing module? ^[2] It could just be me, but it would be nice to have some clarification of this in general.

---

1. An annotation might be a type hint but it could be any arbitrary data, right? ↩︎

2. …which makes me wonder about even Annotated being in the typing module… Not relevant since that’s already shipped. ↩︎

[malemburg (Marc-André Lemburg)]

@tiangolo

Knowing about this PEP, I’ll also ping Marc-André, as I think he could have valuable input here as well.

I have already answered in email. Just this much: I would much prefer a convention which relies on doc-strings and uses e.g. @-markers or perhaps f-string {markers} to let documentation tools know that I’m talking about a parameter from the function/method signature.

Example:

def create_user(lastname: str, firstname: str | None = None) -> User
    """
    Create a new user in the system, it needs the database connection to be already
    initialized.

    @lastname and @firstname are used to initialize the user record.

    Returns a @User instance.
    """
    pass

Here, the tool would see that @lastname, @firstname and @User are being referenced, check in the annotations of the signature for details on types, being optional, default values, etc. and then go ahead with giving me a nicely formatted help screen – well, at least in theory

Even without such a tool, a human would easily recognize the parameters in plain text and make correct associations.

Some other comments (YMMV):

In my experience, documentation is more concise when providing a more holistic view on what’s happening. Documentation of individual parameters isn’t always a good way to achieve this, since parameters are often used in combination, rather than on their own.

I also regard the source code doc-strings as information for fellow maintainers of the software, not really for the end user (API user). It’s good to be terse, to the point and only include what’s needed under the premises that the reader does already have a lot of context.

For the end user (API user), it’s better to have a hand written version of the documentation, similar to what we have in the Python stdlib docs.

[ajoino (Jacob Nilsson)]

I have great respect for your work with FastAPI and Typer, the latter of which I’m an avid user of, and you’ve inspired me to use type annotations for run-time use in my own code. However, I’m not a fan of this proposal for reasons that have already been brought up (mostly that it adds information in one of the most information dense parts of a function definition). It mirrors my least favorite part of Typer, that the help text goes right next to the parameters in the function header (not saying it’s a bad design choice, I honestly can’t think of any other way to do it). I think it’s ok in Typer, even if it makes working on documentation harder IMO, but I would not want to see that on the regular in other code.

For those wanting to discuss how the stdlib could be used to get the same information out of docstrings, I started a new topic. I see such an API as orthogonal to PEP 727, which would codify the existing approach to documentation.

[pradyunsg (Pradyun Gedam)]

Thanks for writing this PEP!

I haven’t double-checked that I’m not repeating something that someone else pointed out already. Apologies if I’m repeating something that’s already been discussed/mentioned. I am commenting on the PEP as written, having done a cursory read of the discussion here.

---

@tiangolo

About it being “yet another competing standard”, it’s true, it is, just like pyproject.toml.

I want to push back on this statement, albeit because nuances .^[1]

pyproject.toml as a whole wasn’t “yet another standard”. It came from a place of needing to determine build-time dependencies generically without needing code execution, a problem that we had no existing solution for already.

PEP 518 has an extensive rationale for why a novel approach and new file was proposed for it. PEP 517 has a similarly long rationale for enabling an ecosystem of build-backends, rather than forcing everyone to use setuptools which had proven difficult to evolve.

(I’m likely biased, since packaging changes tend toward really long rationales for design choices, but I was surprised at how short the rationale section is in this PEP )

The piece within this that I agree on being “one more way to do things” is the [tool] table within pyproject.toml. It originated as “let’s keep build configuration and allow build tool-specific configuration too” , and it evolved into “let’s allow all tool-specific configuration” since “what is a build tool” is an arbitrary and somewhat useless line of argument to have. I don’t think a similar line of argument applies here.

(and yes, with the power of hindsight, I do wish a better job was done of setting community expectations around this new file and the tool table. )

To use the parallel drawn, even though no one outlawed having tool-specific configuration files, or using non-TOML configuration files, nearly every popular development tool has had its users request/argue for having/contribute pyproject.toml’s [tool]-based configuration support. The expectation that having this standard would not mean that library/tooling authors will be pushed to adopt it, is at odds with that experience. It’s not a 1:1 situation, obviously, but I think some of the learnings transfer.

PS: The above is all based off of memory, so I might be wrong about some detail on this. It’s been 4-7 years, and I was still a curious teen back when the initial discussions were taking place. Please correct me if I got something wrong here!

---

@AA-Turner

@mikeshardmind

If the problem you are having is that these aren’t formalized enough, wouldn’t it be less friction to existing projects and not disrupt people already using those styles to reach out to each of these projects to more rigidly standardize the parameter section of their docstrings to the point where a formalized parsing is easy?

This should have a strong counter argument in the PEP, which currently I don’t think exists.

I’ll echo this sentiment.

This was my first reaction while reading this PEP as well as the initial discussion here. I think introducing one-more-way, especially as standard syntax/library code, ought to cover why improving existing solutions isn’t a viable option more thoroughly or, at least, have a stronger reasoning for that choice than what’s provided. For me, I’d set the bar at “we tried and it’s not feasible because (list of socio-technical reasons)” but I’m also cognisant that not everyone might think it needs to be that “high”.

To draw (again!) on the referenced parallel of pyproject.toml, the rationale for both PEP 517 and PEP 518 extensively discusses why the options are being considered over the existing solutions. Alternative approaches were considered and tried over the course of years before we got to the point of discussing using a new file for conveying that information.

---

At the cost of being reductive, the main selling points of this proposal (to me, anyway) seem to be programmatic access to the docstrings and the ability to reuse the arguments.

However, you need to either execute or pseudo-execute (i.e. implement all the type system “magic” of handling certain ifs, try-excepts etc; or, at least, keep logical track of aliases while respecting namespacing) to actually extract accurate type alias information. Compare that to using docstrings, which allows fetching all the information to be done from an AST directly; without needing to do a full execution of the module or handling any of the execution.

Both of these, of course, have a restriction: dynamic logic doesn’t work on either static analysis approaches. Arguably, there’s a tiny amount of dynamism allowed under the type system based model in exchange for significant complexity.

While Sphinx’s built-in autodoc executes code and it’d manage to adapt for this with some tractable reworking, the AST approach is how sphinx-autoapi works. (IIUC, mkdocstrings does this too – don’t quote me on that!) As currently written, this PEP would require AST-based API documentation generation tools to take on the complexity of extracting documentation information from types that might be aliased, and that those aliases might be conditionalized since the type system supports that.

Realistically, IMO this means that AST-based documentation generation tools won’t be supporting this PEP’s proposed model (not fully, anyway).

OTOH, removing support for it (type alias and/or all the conditional magic that come with the type system) would mean that the only thing this provides is an alternative syntax that could live outside the standard library and is a strictly equivalent alternative to standarding on a docstring format ecosystem-wide. It’d drastically weaken the argument for adopting this model as the standard model.

This could be, of course, reduced by adding restrictions on how the TypeAlias is assigned and managed, but that’s a symptom that we’re not using the right abstraction model here IMO.

While this isn’t a showstopper issue, it is certainly an argument against the proposed model IMO.

This actually leads me nicely into…

---

… The open issue about mixing documentation and typing. From the PEP:

It could be argued that this proposal extends the type of information that type annotations carry, the same way as PEP 702 extends them to include deprecation information.

Yes, but it’s not a strong argument IMO.

To use PEP 702: it discusses why the type system is a good vehicle to provide that information, type checkers are actually using this information, type checkers using that information results in useful behavious and the PEP includes references to prior art in other ecosystems showing that deprecation information leveraging the typing mechanisms isn’t a novel concept.

None of those are true for this PEP, IMO. To be clear: I’m not saying this PEP needs to do the same things or make the same arguments, but that it needs provide a stronger rationale for its design choices (especially, the socio-technical choice of not trying to settle/tackle the lack of standardisation problem).

---

Given that one of the motivating goals with this PEP is to provide richer errors/IDE experience, with the motivation explicitly calling out lack of ability to syntax check in IDEs, it’s interesting that it doesn’t bless a markup format. By not picking a markup format, we’d be punting the problem to a different layer (no invalid markup stuff in the strings you have in annotated). While “it’s unclear what to pick” is something that I agree with, I think the decision to not pick a markup format is a consequential one.

IMO, it’d be worth calling out in a more visible manner in the rationale (or rejected ideas, or wherever the authors deem appropriate). Even a position like “we don’t think it’s as important to have syntax corrections for the markup in those strings” would be fine here – it’s a judgement call, and I think the PEP makes the right one – but it seems like an important-enough detail for implementors and users to call out more visibly.

---

1. I’ve once again failed xkcd: Duty Calls because this statement stuck out to me and it’s now 1:50am. ↩︎

[sirosen (Stephen Rosen)]

@flyinghyrax

I personally have mixed feelings about Annotated, and don’t like the mixing of type and other metadata, but annotations aren’t (technically[1]) typing-specific.

I’ve seen more than one person mention that Annotated isn’t necessarily restricted to typing, and I feel that’s only half-true at best.

You mentioned one component of this – the fact that it comes from the typing namespace – but that’s mostly cosmetic. It could also be aliased into a new home if there were sufficient appetite for that.

The other thing is that Annotated requires a type annotation. So you can’t opt in to using a feature built on Annotated unless you also opt in to applying a type to whatever you’re annotating.

---

Is there no value in this being a tiny package on pypi? Most of the participants in this thread could create such a thing in an hour or two – it hardly needs a complex testsuite or other difficult infrastructure.

I’m not sure I follow why it should go into typing_extensions or typing, which seems to be a key part of the rollout plan here. A package can run ahead of the stdlib, and then transform into a rolling backport if it is pulled in.

---

Regarding rst vs markdown… As an older millennial (I can’t say I’m a younger developer anymore! ), markdown is way more comfortable to me. But I would still argue for rst, on the grounds that it is much better defined. Markdown is to rst as yaml is to toml.

If markdown is used, I think that also adds the burden of defining which flavor and subset of markdown is allowed. And I just can’t see all that effort as being worth it, in exchange for, mostly, single backticks rather than double.

[AlexWaygood]

Thanks for the PEP, @tiangolo!

I’ve used Annotated as a way of providing per-parameter documentation in the past, and I can see the attraction of standardising it so that third-party tools can understand this kind of pattern. However… I’m afraid I also just find the proposed syntax here much too verbose. It wouldn’t be something I’d consider using in my own code, unfortunately: I think readability of code is pretty important. I also think this sentence in the PEP isn’t entirely true:

Nevertheless, this verbosity would not affect end users as they would not see the internal code using typing.doc() .

End users would very much be affected by the increased verbosity if they called help() on a function to get documentation in the interactive REPL (unless the plan is to have inspect.signature or pydoc strip the documentation from the function signature before displaying it in the output of help()).

Similar to Laurie’s comments in PEP 727: Documentation Metadata in Typing - #50 by EpicWink, I wonder if a better solution might be adding new special forms to typing? Even if you don’t like the idea (there are lots of disadvantages – see below), it might be worth discussing in the “rejected ideas” section. Perhaps you could have something like this, where Param and Attr are both treated identically to Annotated when it comes to type-checking (the metadata is completely ignored by type checkers), but at runtime, you could enforce that only a single metadata element is provided, which is a string, with the expectation that people provide a documentation string for tools like Sphinx to use:

from dataclasses import dataclass
from typing import Param, Attr

def foo(
    x: Param[int, "must be less than 5"],
    y: Param[str, "must be four or more characters"]
) -> None: ...


@dataclass
class Foo:
    x: Attr[str, "needs to be at least five characters"]
    y: Attr[int, "Must be less than 10"]

Pros of this alternative idea:

• It’s less verbose; I find it much more elegant

Cons of this alternative idea:

• We’d need to add new special forms to typing that type checkers would need to add support for, whereas the current proposal works “out of the box” with type checkers
• Annotated[] is supposed to be the general-purpose way of adding metadata to annotations that’s irrelevant for type-checking; adding special forms for this purpose would be sort-of an implicit admission that Annotated is too unwieldy for some use cases
• The specification would probably be more complex – how would these new special forms interact with Annotated[], or with get_type_hints?