PEP 727: Documentation Metadata in Typing

[pawamoy (Timothée Mazzucotelli)]

Hello, maintainer of mkdocstrings here, and more specifically Griffe, which provides parsers for the common Google-style and Numpydoc-style docstring standards.

I personally like this proposal and already implemented basic support for it in a Griffe extension. I even had fun imagining what an extended version of the proposal could provide.

There’s an issue with docstrings standards like Google-style and Numpydoc-style: parsing them is pretty fragile, because they are not specific enough. They actually don’t even have a specification at all: both have a style guide (Google-style, Numpydoc-style) but no specs, so it’s up to each tool to guess and decide how they should be parsed. In Griffe, the ambiguous aspects are configurable with options, like whether to allow blank lines in Numpydoc-style sections. I recently had to make the Google-style parser more strict (blank line required before section, no blank line allowed after section title) because it had false-positives. None of this is officially specified.

These two styles, Google and Numpydoc, also have divergent features and designs: the Google-style seems to use/recommend Markdown, while Numpydoc is based on rST and designed only to work within Sphinx. Numpydoc supports “See also” section that should be parsed a certain way, while the Google-style does not. Numpydoc requires types to be added to Returns sections, forcing user to duplicate what’s already in the signature. Numpydoc supports multiple named items for Returns sections, while the Google-style does not. Many things are up to interpretation, and make it hard to elaborate a common ground of features and data structures to store the information.

There’s also what I call the Sphinx-style (not sure it has even a name), using :param foo: Hello. to describe parameters for example. This style does not support most of the sections supported by Numpydoc and the Google-style: no yields, no receives, no warns, no kwargs. It also seems to be based on / tied to rST. Not that rST is a problem, but a style/standard that is tied to a particular markup cannot IMO be reliably used by different tooling. The only truly markup-agnostic style that I know of is the Google-style, and that’s one of the reason I use it myself.

With type annotations in signatures, types are not required in docstrings anymore. It means that a pure docstring parser won’t provide all the necessary information to its consumers. It has to also run static/dynamic analysis to obtain types and default values from signatures (for example), or to put this responsibility on the consumer. It means tooling must constantly merge two sources of information: structured data obtained from static/dynamic analysis, and structured data obtained from parsing docstrings, and it’s sometimes hard to do right.

For these reasons, I like what PEP 727 proposes. It structures the information in the code itself (in annotations) so that we don’t have to design, write or parse custom syntaxes in docstrings. Docstrings can then use plain markup, chosen by maintainers (Markdown, AsciiDoc, etc.), in a consistent manner. If it is accepted, and later gets extended to support other kinds of information, it will also standardize what kind of information can be added to code, which I think is very important.

The PEP has one limitation I would like to mention: structuring information in the code won’t allow to intertwine prose and sections in docstrings anymore (like parameters, then text, then exceptions, then text again, then returns section), though it was never really specified either, and could be alleviated by tooling, for example with placeholders in docstrings that get replaced with rendered information.

I won’t comment on the verbosity aspects as I think the PEP addresses it, but will add that documentation generators can very well strip annotations of the surrounding Annotated when rendering docs. The data is there to collect, and can be hidden when exposed to users. Next version of mkdocstrings will do just that (I tried, it works)

[pawamoy (Timothée Mazzucotelli)]

@mdrissi

Another question is for libraries that do follow docstring format strictly how likely are they to be interested to move to this? Can some of the migration from numpy/google/etc docstrings → Annotated doc be automated and would they want it?

Very nice project idea. I think it is definitely possible to automate the migration from docstrings to annotations. The “get data from docstrings” part is already done, now we need the “write data back to annotations” part. There are multiple libraries allowing to transform concrete syntax trees, so it shouldn’t be too hard?

[pf_moore (Paul Moore)]

@methane

(s: str, a: int) looks much cleaner than (s: Annotated[str, "doc of s"], a: Annotated[int, "doc of a"]). This difference would become bigger when document is multiline.
I want to keep function signature simple and clean as possible.

This is my biggest concern as well. Functions where the arguments have type annotations can already be rather long, and Annotated on its own is rather verbose, so I’m generally glad it’s rare. By making it the recommended way of documenting arguments, this makes use of Annotated much more common, and furthermore, an argument docstring will itself be pretty long, making the problem worse.

Combined with the long-standing pressure we get from people wanting to be able to write “one-liners” in Python, we’re reaching a point where a typical Python function will consist of a huge declaration, combined with a highly compact body. And I can’t think of anything less readable than this.

Here’s an example from a recent function I wrote:

def download_pip(
    version: Annotated[str, doc("The version of pip to download")],
    hash: Annotated[str, doc("The SHA256 hash of the wheel file that will be doenloaded")],
    target: Annotated[Path, doc("The path where the downloaded wheel should be stored. Any directories in the path must already exist, and it is the caller's responsibility to use a valid wheel filename"]
):
    url = wheel_url("pip", version, "", "py3", "none", "any")
    with urlopen(url) as f:
        content = f.read()
    if hashlib.sha256(content).hexdigest() != hash:
        raise ValueError("Downloaded file has invalid hash")
    target.write_bytes(content)

Try quickly finding the parameter types from that - I deliberately marked the code as “text” to demonstrate the issue if you don’t have some sort of syntax highlighting support. Also, the restrictions on what you can pass as the target argument are pushed way off the screen to the right (at least on my display) meaning that the docstring for the parameter isn’t readable without scrolling the page. And have fun noticing the spelling mistake in one of the docstrings!

The version that you get if you run black on this is even worse:

def download_pip(
    version: Annotated[str, doc("The version of pip to download")],
    hash: Annotated[
        str, doc("The SHA256 hash of the wheel file that will be doenloaded")
    ],
    target: Annotated[
        Path,
        doc(
            "The path where the downloaded wheel should be stored. Any directories in the path must already exist, and it is the caller's responsibility to use a valid wheel filename"
        ),
    ],
):
    url = wheel_url("pip", version, "", "py3", "none", "any")
    with urlopen(url) as f:
        content = f.read()
    if hashlib.sha256(content).hexdigest() != hash:
        raise ValueError("Downloaded file has invalid hash")
    target.write_bytes(content)

The actual code is almost entirely lost, it’s barely more than half the number of lines of the function header…

And all of this is just with 3 arguments, all with basic types. Here’s a function from pip, which I picked at random so can be considered more typical of “real world” code:

    def send(
        self,
        request: PreparedRequest,
        stream: bool = False,
        timeout: Optional[Union[float, Tuple[float, float]]] = None,
        verify: Union[bool, str] = True,
        cert: Optional[Union[str, Tuple[str, str]]] = None,
        proxies: Optional[Mapping[str, str]] = None,
    ) -> Response:

Add docstrings to that, if you dare

Of course, the counter-arguments are “this is optional, you don’t have to use it” and “it’s up to the user to ensure their code is readable”. But honestly, those responses are at best naïve, and at worst dismissive of a real issue. People do demand policies like “your arguments must have types” and “your code must have docstrings” because IDEs can give a much better user experience to the user of the code if that data is available. It’s important, therefore, to make sure that including such information is possible without making things worse for the developer of the code, so that we encourage good practices.

If this were being proposed as an interim solution that could be used until a more developer-friendly solution was devised, then I’d be less concerned. But I don’t think we’re even close here to a solution that I’d want to advocate as the “one obvious way” to document function parameters.

[pawamoy (Timothée Mazzucotelli)]

@pf_moore

Add docstrings to that, if you dare

Dared!

def send(
    self,
    request: Annotated[
        PreparedRequest,
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ],
    stream: Annotated[
        bool,
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ] = False,
    timeout: Annotated[
        Optional[Union[float, Tuple[float, float]]],  
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ] = None,
    verify: Annotated[
        Union[bool, str],
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ] = True,
    cert: Annotated[
        Optional[Union[str, Tuple[str, str]]],
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ] = None,
    proxies: Annotated[
        Optional[Mapping[str, str]],
        doc(
            """Lorem ipsum dolor sit amet...
            
            ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
            Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
            Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
            Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
            """
        ),
    ] = None,
) -> Annotated[
    Response,
    doc(
        """Lorem ipsum dolor sit amet...
        
        ...consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
        Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
        Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
        Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
        """
    ),
]:
    ...

For functions with a lot more parameters, I think it does make the whole thing more readable. For example, in one of my project, I provide callables for common CLI tools. The pytest callable currently has 82 parameters. That’s without extensions, just pytest. When reading the code, the parameter list is so long that I have to scroll anyway to look at the docstring. So I have to scroll back and forth between the signature and the docstring to check the type and the description for a given parameter. Being able to write the description right next to the parameter would improve the readability a lot. Now I understand this is very rare case (I don’t hope to find more functions with 82 parameters in the wild), so this counter-argument does not have a lot of weight

[tiangolo (Sebastián Ramírez)]

Thanks for the input @Zomatree !

I don’t feel that Annotated should be used by other typing functions but rather be for 3rd parties to add additional metadata, leaning on Annotated for this type of feature feels like a hack.

As this is not altering the semantics of Annotated, I think it could be considered working in the same way as a third party library. The same way dataclasses are something that could perfectly be a third-party library (it was even inspired by one), but it makes sense to have a centralized option.

A side note: having a centralized option doesn’t mean enforcing it, the same way dataclasses are not enforced nor the only option, many people prefer to use attrs or pydantic.

If you’re concern is about using a space (Annotated) that should be somewhat reserved for third-party libraries that use Annotated, like FastAPI, Pydantic, Strawberry, Typer, SQLModel… we are the main ones asking for this, and we intend to support re-using this now (hopefully) “standardized” way to document parameters and types, instead of only relying on each of our custom solutions for this.

Furthermore I don’t see the appeal of this over a docstring, you are already adding this to your own code so it should be trivial to include this information inside the docstring instead.

I would encourage you to read the document. Most of the document is arguments for why this PEP would make sense.

In short, docstrings for parameters have several disadvantages, including:

• info duplication
• info resynchronization, because of the duplication… which in turn becomes a type of manual code cache invalidation for developers maintaining those docstrings
• editor support for editing this information
• runtime access to the information
• simplification for tooling implementers to extract the information (no custom syntax parser needed)
• No new micro syntax for developers to learn (not everyone knows all these microsyntaxes, but everyone knows Python)

This doesn’t feel like a feature which should be included in the stdlib.

There are several reasons why it would belong in the standard library (again, I encourage you to read the PEP).

Some reasons:

• Bring consensus and standardization, this would make it easier for developers to adopt it for new projects, and for implementers to decide to support it (the implementation would be minimal).
• No need for external libraries for this, developers of new packages could use it without installing something external (if they decided to do so).

[pf_moore (Paul Moore)]

@tiangolo

Bring consensus and standardization, this would make it easier for developers to adopt it for new projects, and for implementers to decide to support it (the implementation would be minimal).

I feel that you have this point backward. Putting this feature in the stdlib should only be done if there is already consensus. It doesn’t bring consensus, it acknowledges it.

In this case, I don’t think there’s consensus in the overall Python developer community that this is the best way to document function parameters. Apart from anything else, it doesn’t offer a usable approach for developers who don’t want to add type annotations in the first place…

[tiangolo (Sebastián Ramírez)]

Thansk for the feedback @mikeshardmind (also pre-discuss).

@mikeshardmind

Have other alternatives for additional syntax that does not place this inside of Annotated been considered?

The problem with another Python syntax being supported would be that it would be quite invasive, it would “reserve” that syntax exclusively for this purpose, it could require implementation changes for Python or the types themselves (e.g. supporting __matmul__ in types), and that would make it non-backwards compatible.

The idea with this proposal is that it would be as little invasive as possible, and as optional/opt-in as possible as well.

@mikeshardmind

I also don’t see that much a benefit over parsing existing docstring formats. If I use sphinx, my tests can use sphinx to pull it out as sphinx see it (And so on), so I’m lukewarm on the idea of yet another way to spell the same thing, even if I can appreciate the motivations for it.

Parsing docstring formats is non-trivial (more about it in a previous reply by @pawamoy). If in your libraries and use-cases you can afford to be tightly-coupled to sphinx, have it as a dependency and use it to extract the info, etc. that’s perfectly fine, and I would definitely not expect people in that position to adopt this, the problem is that it doesn’t apply to everyone else. For example, in FastAPI I can’t depend on Sphinx to extract parameter documentation (the same with Pydantic, Typer, etc).

I wouldn’t expect people that are already comfortable with their workflows to change or migrate. I consider it purely opt-in, as are for example TypeDicts, not everyone bothers to use them, but it’s there in the standard for those that want to use them.

Additionally, although yes, this is another way to spell the same thing, this would be the first one to be somewhat standardized, all the other docstring formats are microsyntax conventions for each specific tool that also require quite some learning. This would hopefully be fairly easy to learn, for those that want to adopt it, as it’s the same Python syntax.

Edit: I just realized I can reply/quote multiple things in a single post, so I’ll reply to the other things here.

@mikeshardmind

This is already 49 characters wide, and I didn’t use some longer, real-world types.

Yep, if you consider the docstring independent of the parameters, then the verbosity of the parameters would increase, at least for those using this.

This would be beneficial mainly for people writing and maintaining docstrings that have to scroll back and forth when dealing with and documenting a specific parameter.

At the same time, if this feels more cumbersome to use than to maintain the docstring independently, even with the info duplication, etc. I wouldn’t expect it to be used in those teams and projects.

---

I particularly don’t expect existing codebases and projects that are comfortable with their setup to migrate to this.

But I would expect new tools and projects to find this a bit simpler to use than to learn and acquire the expertise and workflows needed to use previous tools based on different microsyntax formats.

[tiangolo (Sebastián Ramírez)]

Thanks @EpicWink!

@EpicWink

typing.doc could be a subclass of str, reducing the new symbols introduced by the PEP (you get the “value” by taking the str of the instance). This is a minor optimisation, but I don’t see any drawback.

I think the main problem with this is that it would imply that finding a raw str in Annotated would now have a specific pre-defined meaning. By having a specific object (class), tools would be able to use raw strings there as they want (if they have been doing that), which they currently could.

[tiangolo (Sebastián Ramírez)]

@mdrissi

I’m less sure on standardization benefits. The biggest argument for me is on what tooling would be motivated to use this because it’s PEP vs without PEP?

Long before this I had contacted the teams behind VS Code and PyCharm, they had expressed their concerns about verbosity, but also that they would consider implementing support for this only if it was a PEP.

I just emailed them, it would be great to hear their opinions.

@mdrissi

Another question is for libraries that do follow docstring format strictly how likely are they to be interested to move to this? Can some of the migration from numpy/google/etc docstrings → Annotated doc be automated and would they want it?

I wouldn’t expect people that are comfortable with their current docstring formats to migrate. By the point they are fully using them, they probably already learned all the microsyntax details, adapted their workflows, etc. They might be more productive with their current system (each of them).

Nevertheless, when I’ve had to deal with and maintain/update docstrings for parameters I’ve always struggled, in particular when there are a lot of parameters and documentation, as it can end up pages below. For teams struggling with this, it could be interesting to have it.

Funny enough, writing a migration tool for each microsyntax would imply a parser for that syntax. Although writing valid code for this PEP would be much straightforward (as it’s standard Python). And for the same reason, it would be much more straightforward to migrate away from this proposal than to migrate away from each microsyntax.

@mdrissi

One benefit to this style is you can have aliases

I agree this would be quite useful, but I marked this use case as explicitly not required for conformant implementers, suspecting the implementation would be more difficult. But if editors and tooling thing this would be acceptable to be required, I would change it (e.g. @pawamoy’s Mkdocstrings, Sphinx, VS Code, PyCharm).

[mikeshardmind (Michael H)]

@tiangolo

The idea with this proposal is that it would be as little invasive as possible, and as optional/opt-in as possible as well.

Standards are rarely actually opt-in in the real world. The problem becomes “well, this became the standard, and now tooling is supporting this method, so we have to use this method to use modern tooling” This is fairly invasive even if the syntax was good and there was consensus on it being good, as it pushes the ecosystem towards churn on docs. I don’t believe this to be an improvement in the proposed form (or in the off-handed alternative I presented as an example of alternative means when asking about alternatives being considered) as it’s been shown above to get extremely verbose while interspersing code and non-code elements in ways that may not be quick and easy to review.

[tiangolo (Sebastián Ramírez)]

@Jelle

Looking at the PR implementing the PEP (Add doc and DocInfo from PEP 727: https://peps.python.org/pep-0727/ by tiangolo · Pull Request #277 · python/typing_extensions · GitHub), it does feel a little ugly that the doc() function doesn’t do anything other than create a DocInfo instance. Maybe we should combine the two?

This is a good point.

I had stronger opinions about having a function in previous versions when it was also expected to be used as a decorator, but that’s not the case anymore.

My main concern was that once this goes into stdlib it would be pretty much “set in stone”, and I considered a function would be more flexible, but again, most arguments are quite weak now.

My main reasons for a function were:

• It’s easier to tweak the implementation afterwards, but this makes more sense for things like typing.deprecated() that needs to do more things. I am not finding any imaginary future scenario where doc() could need to do something else than to return the class.
• It’s easier to change the type annotations of a function than those of a class, I had to change from classes to functions in FastAPI a long time ago for this, to override the return to Any for some things. Nevertheless, this would only affect here if Annotated acquired any additional semantics about the possible return types of the things inside. So, again, a weak argument.
• I considered it would be easier to extend doc() with multiple parameters, but this applied before when it had a **kwargs, now it would be equivalently “problematic” to add more fields than to extend the DocInfo class.

@Jelle does any of these cases seem potentially useful/valid for you in the future? If not, then I think we could just have a pure class and that’s it.

---

In that case, removing doc(), I would rename the class to Doc or to doc. Do you have (or anyone else has) opinions about this?

[PawelSwiecki (Paweł Święcki)]

Thanks for the PEP!

I’m curious, what are ways other (modern) programming languages deal with documentation metadata. Maybe it’s worth adding a comparison?

[tiangolo (Sebastián Ramírez)]

@kknechtel

and from what I can tell, the meaning of the string literal here is up to the tooling

That’s the main point to have something additional to a raw string. Currently a raw string is valid to mean anything. Having a function/class dedicated to this would affirm “this string is actually intended for docs”.

[tiangolo (Sebastián Ramírez)]

Thanks @baggiponte for the warm comments!

@baggiponte

• Is the doc function necessary? Couldn’t a doc param be added to Annotated? (I suppose you already thought of that, but I don’t catch why it couldn’t be feasible.)

Yeah, the problem with that is that it would require changing Annotated, it’s semantics and its meaning. By not changing anything but just using what is already there this would be less invasive.

Now, specifically about params in Annotated, I imagine you refer to something like Annotated[str, doc="the name of the user"], right? That would mean keyword arguments in generics, and it’s not supported, although there have been some discussions about that, but that’s a much larger topic than this.

@baggiponte

I am afraid function signatures will become really long. Of course, if that is the case, people should just use the regular docstring.

Indeed, signatures would be longer for those adopting this. But ideally, library authors would only adopt this if they are okay with the extra verbosity.

I think the point of view would also depend on if it comes from someone maintaining the docstring or only the signature, as this (potentially) improves things mainly for people maintaining docstrings.

[tiangolo (Sebastián Ramírez)]

@methane

If this PEP is accepted, this would be a recommended way to document parameters.

I wouldn’t expect/intend for this to be strictly recommended. I would consider it as opt-in, as with TypedDict, it’s already there, but not everyone is forced to use it (and not many do, but that’s fine).

@methane

When I read it first time, I was afraid about runtime cost of the documents. But PEP 563 and PEP 649 will save it.

Agreed, I think this would handle that concern.

@methane

(s: str, a: int) looks much cleaner than (s: Annotated[str, "doc of s"], a: Annotated[int, "doc of a"]). This difference would become bigger when document is multiline.
I want to keep function signature simple and clean as possible.

So I prefer to write the argument descriptions in the docstring of the function, except when the docstring is needed at runtime even if this PEP is accepted.
I don’t want to use Annotated in every functions. I want to use Annotated only if the function needs additional runtime metadata.

I agree that the version without Annotated is much cleaner. Nevertheless, the version without it also doesn’t have any docstring, I think it would be a bit more fair to compare the version with a docstring.

For functions that don’t need a docstring, I wouldn’t expect them to have doc() either.

At the same time, I wouldn’t expect this to be considered the only way to do things or the exclusively recommended approach, the same way I wouldn’t consider unittest the recommended library for testing. But it would be just one possible way to do it, and for now, the only way that doesn’t require an external tool and a microsyntax that requires learning and adapting.

I wouldn’t expect this to be heavily adopted by existing libraries and codebases that are already okay with their current internal docstring formats and workflows. But I would expect this to be a bit easier to learn and adopt for new codebases.

And I would hope this would be an easier approach to maintain docs for parameters than with the current docstring microformats.

I would only expect teams and codebases that are struggling with those issues to migrate to this, not everyone.

Maybe it would be useful to document that this is not the only way to document parameters, and maybe to show more the other options, not just by names but also with links?

@methane

Overall, I’m +0 to doc() and DocInfo in type_extensions, but -1 on discouraging argument document in function docstring.

Ah! I didn’t intend to discourage argument documentation in functions docstrings, only to show an alternative that I would consider more convenient in some cases, with the drawback of verbosity. But maybe I could paraphrase some of the content to not make it sound as strictly discouraging documenting parameters in docstrings if that’s how it’s sounding.

Which section of text do you consider would be directly discouraging documenting parameters in docstrings? And what alternative text would you think would be more aligned with that intent?

[effigies (Chris Markiewicz)]

IMO, while arg: type1 | type2 = defaultval is often a documentation improvement, arg: Annotated[type1 | type2, "docstring"] = defaultval is bloat. I would not use this. If I need more than the function name and protocol to document what it does, I write docstrings.

@tiangolo

ideally, library authors would only adopt this if they are okay with the extra verbosity.

Somewhat off-topic discussion of how "optional" this will be:

black doesn’t have a PEP, but its mere existence and standardishness has led people to submit “helpful” PRs where they completely reformat my projects^[1], or auto-format files they touch and turn 1-5 line PRs into messes where the semantic changes are buried.

If there’s a PEP saying “this is the way to document arguments”, I guarantee somebody will write a tool that parses numpydoc or similar docstrings and converts them to this format. And then someone will batch run it on every project that they vaguely interact with and submit PRs that I’ll have to reject, saying “This project doesn’t adhere to PEP 727.”.

As an outsider, I’ve seen maintainers of projects that don’t want to move their configuration into pyproject.toml, which is not a standard, get brigaded and accused of being rude, when the only alternative to rudeness is to respond with sufficient length to the most recent person to make a demand of you.

I’m at +/-0 on this practice, but -1 on this PEP.

---

1. I’m generally okay with black, and use it in a number of projects ↩︎

[baggiponte]

Now, specifically about params in Annotated , I imagine you refer to something like Annotated[str, doc="the name of the user"] , right? That would mean keyword arguments in generics, and it’s not supported, although there have been some discussions about that, but that’s a much larger topic than this.

Of course, makes sense.

This change would imply I either add a doc inside Annotate or a default parameter (e.g. with Depends(...) in FastAPI?

[tiangolo (Sebastián Ramírez)]

@pf_moore

This is my biggest concern as well. Functions where the arguments have type annotations can already be rather long, and Annotated on its own is rather verbose, so I’m generally glad it’s rare. By making it the recommended way of documenting arguments, this makes use of Annotated much more common, and furthermore, an argument docstring will itself be pretty long, making the problem worse.

I wouldn’t think this would be the exclusively recommended way to document arguments (I just commented more on that in reply to Inada).

Maybe I could rephrase some of the text to make more explicit that this is not the only way, and that it’s fine for projects to adopt a different pre-existing docstring microformat. Is there some way I could say that that you would feel is acceptable?

Now, nevertheless, although I agree it’s definitely more verbose, I would consider it much easier to maintain documentation for parameters this way than in docstrings, further away from the function signature. I’ve had to deal with long signatures and their docs in docstrings, far, far away from the actual signature, and I have to scroll back and forth.

I think that if writing and maintaining a docstring is considered a different job than writing the code itself, or in some way isolated from it, then this would affect negatively those in charge of only writing signatures and internal code and not docstrings. But for people writing and maintaining docstrings, I would hope this would be a net positive, despite the extra verbosity.

@pf_moore

Combined with the long-standing pressure we get from people wanting to be able to write “one-liners” in Python, we’re reaching a point where a typical Python function will consist of a huge declaration, combined with a highly compact body. And I can’t think of anything less readable than this.

I wouldn’t think that people wanting to write simple one-liners and would bother much writing docs for them, in docstrings or in Annotated, so I wouldn’t think this would affect them.

@pf_moore

Try quickly finding the parameter types from that - I deliberately marked the code as “text” to demonstrate the issue if you don’t have some sort of syntax highlighting support.

It’s true that if you don’t care much about the docstring, then having the docs intermixed with the signature would get in the way. Now, about having some sort of syntax highlighting, I haven’t seen many (or any?) use cases where there’s no syntax highlighting available, not recently (in the past 5 or 10 years).

Do you think it’s very common for people to have to deal with code without any syntax highlighting? In which use cases? Maybe I’m missing something or not having in mind some scenarios.

@pf_moore

Also, the restrictions on what you can pass as the target argument are pushed way off the screen to the right (at least on my display) meaning that the docstring for the parameter isn’t readable without scrolling the page.

For this reason multiline strings are allowed, the same as with docstrings.

@pf_moore

And have fun noticing the spelling mistake in one of the docstrings!

I wouldn’t consider it harder to notice a typo in Annotated than in a docstring, but I guess that depends on taste.

@pf_moore

The actual code is almost entirely lost, it’s barely more than half the number of lines of the function header…

How would you write the same function including a docstring that documents all the parameters? Wouldn’t the body of the function be dominated by a multiline string as well?

@pf_moore

Add docstrings to that, if you dare

I would consider @pawamoy’s example fairly readable.

@pf_moore

But I don’t think we’re even close here to a solution that I’d want to advocate as the “one obvious way” to document function parameters.

I agree I wish there was a better way to achieve all this, but dealing with docstrings for parameters in multiple formats in different projects, and always struggling with the same problems, I would be much happier dealing with the verbosity here than with what I had to deal in those cases.

I would consider this much better than current approaches, at least from the point of view of someone writing and maintaining docstrings.

But then again, how would you imagine a better way that could be achievable in Python? What would be the ideal in your point of view?

[tiangolo (Sebastián Ramírez)]

@pf_moore

I feel that you have this point backward. Putting this feature in the stdlib should only be done if there is already consensus. It doesn’t bring consensus, it acknowledges it.

You’re right. This wouldn’t (and shouldn’t) be accepted before it’s proven to be useful.

Having it in typing_extensions will help a lot to figure out if this is actually useful or not, while we continue the conversation here. But I had to start here to see if the parties involved would be interested (and benefited from this).

For now, I know that it would be adopted at least by FastAPI, Typer, SQLModel, Asyncer (from my side), Pydantic, Strawberry, and others (from other teams). And it would be supported by tools like Mkdocstrings, maybe others.

I would like to hear the opinions from Sphinx, VS Code, PyCharm, Jedi, Jupyter (I’ve contacted them all).

@pf_moore

Apart from anything else, it doesn’t offer a usable approach for developers who don’t want to add type annotations in the first place…

This is definitely true. And the same as with type annotations, it has to be marked as strictly optional and opt-in. Is there something in the text that you think could be changed (and how) to bring this point across better?

---

A good thing is that we can use my projects (FastAPI, etc.) as a testbed, and see if this has any real value. Probably others can try it as well, and we can see how the perception of this is.

But yeah, as this had never been done before, we don’t have a way to know how it feels to work with it, and if it improves or damages efficiency, developer experience, workflows, etc.

[mikeshardmind (Michael H)]

@tiangolo

I agree I wish there was a better way to achieve all this, but dealing with docstrings for parameters in multiple formats in different projects, and always struggling with the same problems, I would be much happier dealing with the verbosity here than with what I had to deal in those cases.

I would consider this much better than current approaches, at least from the point of view of someone writing and maintaining docstrings.

But then again, how would you imagine a better way that could be achievable in Python? What would be the ideal in your point of view?

This belies the problem right here. you value your use being easier over what other projects are already using. You keep calling this opt-in, but at the same time, you say that there is tooling and IDEs that are ready to jump at implementing whatever is standardized. This isn’t opt-in, no matter how much you claim it is, people familiar with the situation with typing who only adopted it due to IDE completions can attest as much.

The 3 most common standardized docstring styles are sphinx, numpy, and google. If the problem you are having is that these aren’t formalized enough, wouldn’t it be less friction to existing projects and not disrupt people already using those styles to reach out to each of these projects to more rigidly standardize the parameter section of their docstrings to the point where a formalized parsing is easy?

And as was pointed out prior to this discussion thread being made, other people have found it possible to parse all three of these for parameter docs: https://github.com/Rapptz/discord.py/blob/f74eb14d722aa1bc90f9d0478199250d2eb4e81b/discord/app_commands/commands.py#L168-L186