PEP 727: Documentation Metadata in Typing

That’s really not the case, we have real-world examples from use in FastAPI as a test bed that were brought up earlier in the thread.

And the fast API example also involves referencing other docs by URL, so if we were to hold what some people said is a goal of not doing that, it would be even worse.

Sorry that this was unclear. The old Google Style would be worst cast scenario +1 LOC per parameter vs this new style because old style requires the parameter name (and maybe type) to be repeated on two lines.

I think that the FastAPI init is quite an extraordinary case in which there are many parameters and each one has multiline documentation, going so far as to include example usage. Because this forces the type to be on its own line, and because the docstrings begin on a newline for clarity, I will concede that the FastAPI init would be fewer LOC in a traditional style vs the new style. Nevertheless, I find the doc style of that init signature to improve the readability of the function signature, source code, and documentation overall because the documentation is tightly coupled with the parameter.

1 Like

There is no standard way of making see:func: xyz jump to docs being referenced, no?
I want to put the docs for frequently used variables in a central place, explicitly not disrupting the flow of the code, but leveraging the IDE to just look at the variable if I want to, just like with any other variable definition. I don’t want to compile and jump back and forth between the IDE and the docs, that is a really bad dev experience IMO.

1 Like

see :func:`xyz` is Sphinx syntax. It won’t do anything clever for just calling help(some_function) in a REPL. I think PyCharm might try to render Sphinx docstrings (it’s a while since I used it) and therefore possibly cross references. The general answer though is no, it’s a genuine caveat that most of the ways you can look at a docstring are just plain text.

On the flip side though, mouse hovering over a function and a function help display dialog so large that it doesn’t fit on the screen popping up isn’t particularity great either.

If the alternative of standardizing the big 3 docstring formats happened instead of this, it’s possible that more IDEs would be willing to add support for following/viewing references in these standard formats, but for sphinx at least, this would mean that IDEs would either only support fully qualified references from the project + the standard library and builtins, guess when it comes to intersphinx, or need to be able to understand and look for intersphinx (not that this is difficult, but that it’s a consideration)

There are certainly other less disruptive ways here that we can look to for improving people’s ability to navigate documentation without needing to open a fully rendered docs site.

5 Likes

I was just made aware of “docments”: Docments – fastcore.

Without docments, if you want to document your parameters, you have to repeat param names in docstrings, since they’re already in the function signature. The parameters have to be kept synchronized in the two places as you change your code. Readers of your code have to look back and forth between two places to understand what’s happening. So it’s more work for you, and for your users.

Furthermore, to have parameter documentation formatted nicely without docments, you have to use special magic docstring formatting, often with odd quirks, which is a pain to create and maintain, and awkward to read in code.

That rings a bell :smile:

3 Likes

God help us if writing a parameter name more than once becomes a crime…

def do_nothing(x):
    return x  # Ahh! I wrote 'x' twice! The duplicity, it burns! 

I’ve been reading through the conversation with some mixed feelings. I currently use reST docstrings and don’t feel like switching to Annotated+Doc because of decreased readability (just my personal opinion).

Maybe, similarly to how | is now preferred over Union, there could be a way to simplify Annotated, leveraging type.__matmul__ to convert some_type @ (ann1, ann2, ...) to Annotated[some_type, ann1, ann2, ...]?

Then we could write docs as follows, letting the “real” type have its priority over the “annotations” (@ as in Annotation?) while reading this:

def some_function(
    some_parameter: SomeType
    @ (
        Doc("Some documentation goes here"),
    ),
    some_quantity: float
    @ (
        Doc("""
        Some multiline documentation
        """),
        Unit("m/s"),
    ),
) -> SomeReturn @ Doc("Some details about the return value")

Just wondering if this could stimulate the conversation further (given the scope, it could be a separate PEP as well).

Note: I’m using Unit from annotated-types just to show how this could play with other annotations.

I feel the need to keep beating the drum of: Insofar as this is useful for individuals, great, but it’s ultimately largely a stylistic choice that it feels like there’s not much point in debating. And to me, it feels like the it’s missing the primary value of the PEP.

But again, (imo) this PEP ultimately provides a standard location for this information to exist at runtime (for runtime introspection libraries, like pydantic/fastapi/sphinx) whether or not users ultimately are using that syntax or not. And it also enables one to document class attributes or module members, or other things that dont exist in a context where there is a signature with a __doc__ to attach to (again, useful only to programmatic introspection, otherwise a comment would suffice)

Perhaps I’m being too optimistic, but in my mind this PEP enables there to exist “off-the-shelf” adapters for “attribute docstring”, “google format docstring”, “numpy format docstring” that normalize those forms into Doc annotations, such that they’re more readily accessible during runtime introspection across tools.

If this PEP were accepted on those grounds, then you could conceive of future PEPs that made it more concise or ergonomic to use as a user. Certainly making Annotated more concise is a cross-cutting concern well beyond the scope of Doc annotations.

3 Likes

Now here’s the question, does Doc document the type itself or all variables of said type?

Thanks for sharing this. I quite like the documents simplicity! Now to add support for it into IDE’s… :grinning:

def add(
    a:int, # the 1st number to add
    b=0,   # the 2nd number to add
)->int:    # the result of adding `a` to `b`
    "The sum of two numbers."
    return a+b

Beating the drum on this does nothing if you aren’t going to address things that are seen as showstoppers for this pep.

  1. By placing documentation in Annotated, you undercut the ability to prevent docstrings from being loaded into memory (-OO) which is used in some embedded cases to limit memory use to what’s needed to run the application.

  2. It causes absurd git diffs and function parameter lists that span hundreds of lines. This isn’t a good user or maintainer experience in real world code (the pep author’s library was shown as an example of this)

  3. It creates ambiguity over what is being documented, the parameter or the type. Annotated is largely used to add additional information outside of the type system to a declaration, but some of the examples people have for this describe the parameter and not the type, while others want to reuse it documenting the type for all places it is used. This seems to indicate that we might be better off with standardizing a way to attach docs to type alias statements such that 1 and 2 arent an issue either

  4. It’s unclear why we need this, when standardizing the existing docstring formats can give the same benefits without any of the other issues, and without requiring every library using one of the big 3 existing docstring formats to have churn to support this.

5 Likes

A somewhat unrelated FR might be to add an option to strip annotations. This would also prevent dataclasses from working, but maybe there is some version of this that could work. (E.g. decorator names that make the compiler keep annotations or something)

  1. By placing documentation in Annotated, you undercut the ability to prevent docstrings from being loaded into memory (-OO) which is used in some embedded cases to limit memory use to what’s needed to run the application

I find this point less important. Individuals doing embedded work can simply work around it, as I am sure they are doing currently. In applications where -OO is necessary the individual is more than likely already taking extra care in what 3rd party libraries they can even use due the fact that many popular ones do runtime manipulation of docstrings and using -OO breaks the library.

1 Like

Of these points, i think only #3 is potentially relevant to the point I’m making. 1/2 are, again, about subjective per-user DX. Whereas my whole point is, none of that is relevant because this PEP doesn’t need to be a feature meant for end-users. it’s fundamentally providing space for a thing that doesnt exist.

  1. I dont find this especially damning personally. fastapi, pydantic, sqlalchemy (, my own libraries,) and other libs increasingly use Annotated to stash data relevant to the field, but which is irrelevant to the type. that seems largely the point of Annotated, ergo it makes sense for using it for this feature (or at least i dont think it’s confusing the purpose of anything)

a way to attach docs to type alias statements

How it that not just FooType = Annotated[T, Doc("Foo")]? I feel like this is very much the point of why this is a fundamental difference vs strings being attached at the much looser function/class scope.

  1. I really dont think any sort of docstring format gives you the same benefit. And i think it would be a net reduction in complexity for everything that needs to introspect docstrings if they could “just” use a set of docstring2doc/attribute_docstring2doc transformers that normalized everything, and tools for which this is a relevant concern wouldn’t need to all have their own bespoke handling of the many different ways fields can be documented.
2 Likes