PEP 727: Documentation Metadata in Typing

PEPs
102
Edit: tangent about how paramspec works differently for users and library devs collapsable for those uninterested

A little bit off-tangent here, but the problem is that the function using the ParamSpec has extremely limited knowledge of what is passed in, other than that it’s a callable, and (in some cases) the use of concatenate could reveal some number of parameters. kwargs are opaque to it, and any perceived lock-in actually only happens if the function using ParamSpec is tightly coupled in use with the function you are attaching it to. This makes it work great for this purpose in user code, but in library code, it does not quite serve this purpose.

So for @Liz 's described case with decorators in a library meant to be used by users, ParamSpec can only enforce via what’s present in concatenate, everything else comes from the user function which is to be decorated.

In user code where you write both the decorator and the function it wraps, then the decorated function has the expected type, but it isn’t the decorator enforcing it, it’s that the totality of both being present in the same code base means changes to either are immediately reflected in the resulting wrapped function.

These kinds of “Typing has different surfaces and impacts for application and library devs” come up in a few places.

It was included in @Liz’s comment indirectly as

But I think without a direct example of it or having been there before, it isn’t obvious to everyone why this is the case, hopefully the above explanation helps a bit. If you want a real-world example in public code, this shows how complicated this can get, and how it still only enforces the parameters added via Concatenate, not the ParamSpec. Note that this requires creating a lie for the type checker currently just to get even a little more usable for users, as ParamSpec is extremely limited.

103

+1 for making it clear that this

UserName: TypeAlias = Annotated[str, Doc("The user's name")]

would document the symbol being annotated instead of exclusively the type alias

(source).

[sorry if off-topic…] I believe it would be very useful to be able to add docs to types created using typing.NewType, too. Mind, it seems it would be possible out-of-the-box:

_UserName = NewType("_UserName", str)
UserName: TypeAlias = Annotated[_UserName, Doc("The user's name")]

Looks a bit hacky, and I’m not sure I like it. But will it work indeed? Should it be mentioned in the PEP?

related, but off-topic

This, on the other hand:

UserName = Annotated[NewType("UserName", str), Doc("The user's name")]

is invalid, at least in for mypy==1.5.1 (“error: Invalid type alias: expression is not a valid type”). This is definitely out of scope, so hiding this part.

1 Like
104

@tiangolo, this contradicts the feedback I gave you earlier. It introduces an inconsistency that I think is very problematic. Metadata associated with a type alias should not apply to symbols other than the type alias. A type alias is (by definition) an alias for a type. It does not carry with it other metadata that should be applied in the many situations where a type is used. If you provide documentation metadata for a type alias, it should apply only to the type alias. Annotated provides metadata about the symbol that you are declaring. If Annotated is supplied as part of a type alias declaration, then it should provide metadata about that type alias.

Types described by type aliases go through many transforms inside of a type checker. For example, you can use them as type arguments or combine them in unions or narrow them using conditional type guards. There are well-defined ways for type transforms to be applied. There are not well-defined ways for metadata to be applied because metadata doesn’t follow any formal system. It therefore doesn’t make sense to carry metadata along with types described by a type alias.

I urge you to update the PEP to make it clear that any documentation metadata provided for a type alias applies to a type alias only and will not be carried through to places where the type alias is used. I don’t think the PEP will be implementable as it’s currently drafted. And even if it, this inconsistency will create problems in the future.

5 Likes
105

Thanks @erictraut! I was waiting to hear your feedback on my response above about it.

Thought Process

Let me first explain my thought process, maybe we can catch something I’m misunderstanding.

Maybe the first thing I might be confusing is what a “symbol” would be.

I see that Annotated can be used in two main places:

  • Annotating a type hint: after the : in a parameter, variable, etc. Or in the return of a function. I’m considering all those type hints.
  • To create a type alias. After the =. In this case, a new object is created (the new type alias).

I would understand that the main use case for Annotated is type hints.

I also see that in PEP 593 they expected that people would normally create type aliases for the places where they would have used Annotated directly, just for the verbosity: PEP 593 – Flexible function and variable annotations | peps.python.org

Then, if we think about a “symbol”, I would think that Annotated creates a new symbol when used as a type alias (after the =, something is being assigned to), but I wouldn’t think it creates a new symbol for a type hint (after a :). When used in a type hint I would think it is only adding metadata to the same type information for that parameter symbol.

So, with respect to Annotated, I wouldn’t really think that a type alias symbol would be comparable to a variable with a type hint symbol.

Then I thought that the main use case would be to consider Doc() for the things that are annotated with type hints (e.g. after :). In that case, a type alias would actually not have documentation on its own, but only when applied to a type hint…

Nevertheless, I wrote that a single annotated type alias would have its own doc until used as an annotation in case that’s useful (although I don’t really see much use for it). Also because it’s not possible to add Annotated to the type of a type alias, i.e. this is not valid:

UserName: Annotated[TypeAlias, Doc("The type alias of a username")] = Annotated[
    str, Doc("The user's name")
]

Then, also, I don’t really imagine a use case where it would be useful to have documentation for an isolated type alias, only when used as a type hint.

Problems of Type Aliases

Now, considering the use cases you mention would be cumbersome because metadata doesn’t follow a formal system:

  • Type arguments
  • Unions
  • Type Guards

In all these cases, if a type alias was not used but a pure type instead, that same type could have been directly annotated with Annotated and with a Doc, and in the same way, the meaning of the documentation there would be more or less undefined, right?

So, would this be a problem of type aliases or of the way Annotated could be “freely” nested in other type structures and places?

Following that train of thought, maybe this would mean that Annotated with Doc would not really document the parameter of a function but only its type, or something like that, it’s just a pedantic difference, but being very formal, maybe that’s what it would mean… and in that case, that is not really very useful, it’s only useful when it’s documenting the parameter of the function.

Practical Use Cases

Now, apart from the strict correctness of where should it apply or not (and if I’m misunderstanding things or not), let’s consider the actual main use cases this would intend to cover.

I think the main use cases are the very simple ones. E.g. Independent of type aliases, I don’t see a reason why it would be helpful now to support nesting Annotated with Doc, for example in type parameters, unions, or type guards. At least not with the current state of the language and tools.

Maybe it would make sense to explicitly mention that Annotated with Doc only has a meaning when used at the top level of a type hint, e.g. right after the :. This would discard Annotated with Doc inside a union and other cases like that.

To clarify, this would be unspecified and would have no meaning because Annotated with Doc is inside a union:

def create_user(name: Annotated[str, Doc("The user's name")] | None = None):
    ...

Instead this would be valid because Annotated with Doc is at the top level:

def create_user(name: Annotated[str | None, Doc("The optional user's name")] = None):
    ...

Would that make sense?

Type Alias Use Cases

Now, about type aliases, as several have mentioned, it would have huge potential for this if there was a way to allow defining Annotated with Doc in a type alias that could be re-used.

For example, I could imagine many APIs could be simplified by re-using the same type alias for the same parameter in multiple functions. It would also help a lot with the discomfort of having long signatures. In fact, it would also reduce the function body as the docstring would also be smaller.

Or extending it further, I could imagine it would be very useful to declare some common parameters with their documentation in a typed dict, and then unpack it in **kwargs. This would not be strictly a type alias, but quite similar. And it would keep type correctness for the signature types, while providing docs for the parameters, and reducing a lot the code duplication in the library.

On the other hand, I personally don’t see much use in having a documentation string associated just with the same type alias symbol independent of anything else, I can’t imagine a use case for that yet.

Going Forward

Is there a balance we could strike here to allow Annotated with Doc in type alias for parameters? Or at least some of these scenarios?

The first thing I would think and suggest we do is limit the support of Doc in an Annotated type alias to only when it is used at the top level of a type hint, i.e. only when the type alias is used right after the : or in the return of a function. The Doc used in an Annotated type alias used anywhere else would have no meaning… I would think this would discard all the other cumbersome scenarios (e.g. type arguments, unions, type guards, etc.).

If you definitely don’t see a way this could be viable, the second thing I would suggest would be just removing anything specific to type aliases from the PEP, saying that this PEP only specifies the usage of Annotated directly in the type annotations, making it explicit that it leaves type aliases unspecified for now, just to leave the “open door” for future PEPs to extend (or restrict) it further in case there’s a way to make this work in the future.

Please let me know what you think of all this, I would like to know your thoughts before updating the PEP.

1 Like
106

Do we anticipate that this change will lead to parameter documentation beting taken common?

For example, do we see something like this happening?

type OptionalDType = Annotated[None | DTypeLike, Doc("The optional data type.   By default, the data-type is inferred from the input data.")]

def asarray(a: NDArray, dtype: OptionalDType, order: ... = None): ...

def asanyarray(a: NDArray, dtype: OptionalDType): ...

If so, this would be a giant advantage since it would both reduce typing and keep docstrings consistent.

From the last couple comments, this seems contentious, but is there a nice way to allow this type and metadata to be taken common?

107

Maybe the first thing I might be confusing is what a “symbol” would be.

A symbol is a named entity contained within a scope. Symbols are declared using various syntax in Python. A class symbol is declared using a class statement and a function symbol is declared using a def statement. A parameter or variable symbol is declared using the : token followed by a type expression. A type alias is declared using the : TypeAlias = form (PEP 613) or a type statement (PEP 695).

The “motivation” section for PEP 727 argues that there’s currently “no formalized way to provide documentation for parameters, return values, class-scoped variables, local variables and type aliases”. That’s stretching the truth because there’s already a well-defined and widely-supported mechanism for documenting parameters, return values, and class-scoped variables via docstrings. This PEP proposes to add a redundant mechanism for documenting those classes of symbols.

That leaves only local variables and type aliases that cannot be documented today. I’ll assert that docstrings for local variables are not important because they cannot be seen outside of their scope; simple comments suffice for local variables. The only net new value that PEP 727 provides is the ability to document a type alias. Now you’re arguing that this capability should be removed.

Why is it important to document a type alias? Some libraries (like numpy) make heavy use of type aliases. Currently, if you import one of these type aliases and hover over it in your editor, you get no documentation. By contrast, if you import a class, a function, or a constant, you will see documentation.

You’re arguing that a type alias symbol should carry with it non-type metadata, and this non-type metadata should be applied to other symbols when the type alias is used, but only for parameter declarations and only for certain type expression forms.

Type alias symbols can be used anywhere a type can be used. You’re arguing that documentation metadata associated with a type alias should be ignored unless that type alias is used in a “bare” form when annotating a function parameter (e.g. param: MyTypeAlias).

  • If you combine it in a union (e.g. param: MyTypeAlias | None), it is ignored. (But what if MyTypeAlias describes a union that includes None and the type checker internally simplifies MyTypeAlias | None to just MyTypeAlias?)
  • If you use it in a bound type variable (e.g. def func[T: MyTypeAlias](param: T): ...) it is ignored.
  • If you use it in a type alias (e.g. param: list[MyTypeAlias]) it is ignored.
  • If you use it to annotate a class variable or local variable (e.g. class Foo: x: MyTypeAlias) it is ignored.
  • It’s not clear what would happen if the type alias were generic and you explicitly specialize it (e.g. param: MyTypeAlias[int]). Should it be ignored in that case?

When adding new functionality to the type system, new features must compose with existing features. Composability breaks down when we start adding inconsistencies and special cases like what you’re proposing here.

I don’t want to spend too much time arguing about the details of this PEP. Based on the negative feedback it has received, I think it’s unlikely to be accepted by the SC in its current form.

I understand that you were motivated to write this PEP based on a real need you have in FastAPI. I also appreciate all the work you put into the draft PEP and your time advocating for it. That said, I think the stated motivation for the PEP is very weak. It provides almost no net-new capability to the language. Many respondents to this thread have already provided well-reasoned arguments against the proposal, and FWIW I largely agree with them.

In the spirit of identifying a path forward, let me suggest an alternative proposal that I think addresses your needs and avoids all of the problems and concerns about the current draft PEP.

Alternative proposal:

  1. Abandon the idea of using Annotated for documentation.
  2. Formalize the specification for function docstrings. Pick one of the three commonly-used standards and anoint it as the preferred docstring format. Or document and anoint all three.
  3. Create a library that parses the standard format(s) and extracts information (including parameter and return type documentation) from docstrings. This library should also validate the syntax of the docstring and its semantic consistency (e.g. do the documented parameter names match those of the function declaration?). This library can be used by all tools that need to extract this information.
  4. Create and publish a simple command-line tool based on the library that can be run on source files to verify function docstrings. In addition, work with the maintainers of popular IDEs and language servers to incorporate this capability so developers can receive immediate in-editor feedback (and perhaps completion and rename support) when editing their docstrings.

Unlike the current draft PEP 727, I think you could get the entire community to rally around this alternative proposal. It has the following benefits:

  • It avoids linking documentation and typing.
  • It avoids creating a new redundant mechanism for parameter and return type documentation.
  • It is consistent with the approach used in other popular languages (like Javascript). Refer to the JSDoc spec for details.
  • It builds on concepts that most Python developers are already familiar with rather than forcing them to learn a new, currently-unfamiliar mechanism.
  • It provides immediate value to maintainers of existing libraries. They can use the new tooling to validate their existing docstrings.
  • It works with functionality already implemented in all popular IDEs, language servers, and doc extraction tools.
  • It addresses a pain point faced by the maintainers of these tools — namely, that the lack of specification leads to the misinterpretation of docstring contents.
  • It results in even better tooling over time as IDEs and language servers add features to help developers create and maintain docstrings.
36 Likes
108

I find this disingenuous and too presumptuous. The has also been a lot of positive feedback, and a lot of constructive criticism.

The SC will accept our reject the proposals based on comparing the pros and cons of the proposal versus the pros and cons without it. Community consensus will guide that decision (usually fairly heavily), but I expect individual bias (which I see in much of the negative feedback) doesn’t force the decision. The SC has been good with that so far, so I expect the same.

This will require runtime parsing of strings. In my experience, slight mistakes can completely break this, and because it’s not enforced by the interpreter, those mistakes may not be seen until after release.

I much prefer a form enforced by the interpreter, such as documentation annotations (this proposal) or parameter doc-strings.

Adding another tool increases friction to development, which I don’t think is justified in this instance due to my arguments above.

I think typing is documentation

I don’t think the proposed mechanism is redundant, as it different benefits and drawbacks from the current system

I’ve still yet to see anyone in this thread provide a reason to ignore one of the two motivating pain points this proposal addresses: having a large distance between parameter definition and documentation in functions with a large number of well-documented parameters.

For that matter, the same goes for the other pain-point (duplicate parameter definition)

6 Likes
109

I will cover core features this proposal enables that are currently not covered well by docstrings and make standardizing docstring format not enough unless extensions are made.

  • Parameter/attribute level documentation instead of function level documentation. This one does partly exist for docstrings as attribute docstrings are mentioned in an old PEP although at runtime manipulation/extraction of them is not thing.
  • Sharing documentation for same parameter/attribute used in many (dozens/hundreds) of functions/classes.
NameT = Annotated[Optional[str], "The name scope used for any operations made in this layer/context."]

def foo(name: NameT):
  ...

def bar(name: NameT):
  ...

This second point is key benefit I see with this feature. For docstrings to support this they would need some reference/templating mechanism to be defined and standardized.

The first two points do not require that solution be connected to typing/Annotated. Annotated handles them well, but an alternative proposal that allows easy parameter/field sharing I would be happy to see.

Also this PEP being rejected does not mean the feature will not/can not be used. Only that it would become pypi package and libraries could still choose to use it. I’ve already been using Annotated for documentation this way for a couple years in internal codebases. There is no IDE support for it, but even without IDE support, sphinx/documentation generation integration is still doable. And for users of codebase they can click on parameter to jump function definition → click on type annotation to reach Annotated definition with docstring. I think PEP main value is trying to reach agreement on what Annotated documentation should look like for tools to support it. Are there rule clarifications/changes needed to make static analysis more convenient.

Questions of what does Optional[NameT] mean and how should documentation be shown can all be answered. My current implementation looks for Annotated any level nested in type (necessary for ClassVar/Required compatibility as they must be above Annotated) and expects only one documentation string to be found. So Optional[NameT] would just use documentation of NameT while Union[Annotated[int, “doc1”], Annotated[str, “doc2”]] is unsupported/rejected. Rules I use are partly helped by it not being standard/adjust annotations as needed so alternative rule of Annotated can not be nested in Union/must be top level would be fair choices.

edit: I’m neutral on runtime introspection benefits of Annotated vs docstring. I think Annotated is easier for runtime manipulation, but for docstrings if format was defined a linter could validate and enforce it.

edit 2: Regardless PEP being accepted/rejected I think agreement on rules for how Annotated aliases/Union/other types interact is still useful today. If feature goes on to be used as separate pypi package and rules are chosen independently couple years later if static analysis later wants to support it, there will be less ability to change rules. Right now what Union[Annotated[int, …], Annotated[str, …]] is still open thing and there’s little constraint to pick rules that are convenient for both static/runtime usage.

2 Likes
110

I’ll give one. The pep author’s own library needs about 600 lines for just parameters for a single class with this. The idea that moving the documentation closer is an improvement seems mistaken, I’d rather have a group of type declarations used by developers and a separate group of documentation meant for users based on this. I think someone also did bring up that duplication of these wasn’t a real issue because parameters rarely change in stable code.

18 Likes
111

I think that stance is highly subjective. To someone else (like myself) it really does strengthen the authors point. On mobile I can see:

  • the parameter name
  • the parameter type
  • the parameter docstring

All on my screen. No scrolling.

It takes up space, but not much more than if those docstrings were concatenated at the end of the function docstring. So I think arguing length is disingenuous.

5 Likes
112

It takes up significantly more space than something like sphinx docs. This is a level of extra indentation, removing some amount of horizontal space for what should be text, and this is less concisely presented than such.

The comparison was provided earlier in thread

The status quo also does this (see collapsed section for real-world docs that do this). This PEP isn’t providing those things over it, it’s arguably providing deduplication of some of this (see other posts about why it isn’t though), and see eric’s full post (but especially this section) for how this PEP is barely adding anything, and not necessarily an improvement.

1 Like
113

On the other hand, I can’t see the function’s signature (i.e., the list of valid arguments) without scrolling!

11 Likes
114

Please let’s not accuse people of being insincere when we disagree with them.

There are a number of use cases for making documentation a structured and reusable piece of data, visible at runtime. Arguments which reject those use cases as illegitimate are unlikely to find traction with the people who have this exact need.

That said, for me, as someone with these use cases (and, FWIW, we use FastAPI at work and I’m very appreciative of @tiangolo’s hard work), this doesn’t feel like the right solution.

I already use two tools which parse existing rst docstrings: sphinx and pylint (docparams). Other tools have been mentioned.
I think that a path which makes it possible to use rst docstrings is important.

I have specific ideas about this (a doc registry object, parsing tools for adding strings to that registry, and ideally sphinx, pylint, and/or other tool buy-in to help maintain diverse tooling on a common core framework), but they are another topic in their own right. Regarding this PEP, I think all I have to say is that I hope the SC defers or rejects it – I still don’t feel like I’ve seen an adequate answer for how and why it’s ready for the stdlib if there’s no package out there doing it.

9 Likes
115

This isn’t unique to this format in any way though.

Edit: this got posted eagerly, but you can achieve the same thing in a more condensed manner with existing tooling.

1 Like
116

Yes, but even with that, it is not implied that documentation is typing / type information.

Type information does belong in documentation, since it is a useful bit of information for humans (and machines!) to understand what is valid to pass in. That does not mean that detailed prose documentation should be a part of the type system.

21 Likes
117

Just jumping in with two comments after reading the PEP today and skimming, then becoming overwhelmed by the existing discussion thread here:

Syntax shorthand idea

Slice syntax allows for a : within [] which could be (ab)used to offer a shorthand to avoid needing to import and use typing.Doc directly.

    is_approved: Annotated[str: "The status of a PEP."]

Somewhat surprising to me who never writes code using slice objects… they don’t seem to care about their parameter types so the above could probably already be implemented?

>>> class X:
...   def __getitem__(self, bar):
...     print(bar)
... 
>>> x = X()
>>> x[str: "grail"]
slice(<class 'str'>, 'grail', None)

Feel free to send this straight to Rejected Ideas if it is unworkable or undesirable. =)

Editorial comment on the PEP’s examples

The current examples perpetuate the myths that “Western” cultures wrongly believe about people’s names by using a “first” and “last” name APIs in examples.

Just pick something else as the examples.

2 Likes
118

That’s a compelling argument and what I was hoping for to justify documenting type aliases, I didn’t know Numpy exported/exposed type aliases. Do you have a specific example where this is used?

The main argument for this PEP is to cover the use cases and problems that can’t be covered by current docstring conventions. I updated the document with short a bullet point list to make it easier to find all the individual points but I’ll copy it here in case it helps:

  • Editing would be already fully supported by default by any editor (current or future) supporting Python syntax, including syntax errors, syntax highlighting, etc.
  • Rendering would be relatively straightforward to implement by static tools (tools that don’t need runtime execution), as the information can be extracted from the AST they normally already create.
  • :sparkles: Deduplication of information: the name of a parameter would be defined in a single place, not duplicated inside of a docstring.
  • :sparkles: Elimination of the possibility of having inconsistencies when removing a parameter or class variable and forgetting to remove its documentation.
  • :sparkles: Minimization of the probability of adding a new parameter or class variable and forgetting to add its documentation.
  • :sparkles: Elimination of the possibility of having inconsistencies between the name of a parameter in the signature and the name in the docstring when it is renamed.
  • Reuse of documentation for symbols used in multiple places via type aliases.
  • :sparkles: Access to the documentation string for each symbol at runtime, including existing (older) Python versions.
  • :sparkles: No microsyntax to learn for newcomers, it’s just Python syntax.
  • :sparkles: Parameter documentation inheritance for functions captured by ParamSpec.

The items marked with :sparkles:, are not doable with current docstring conventions. That is the main motivation.

What I had understood was that the previous state of the PEP was unimplementable due to its complexity and special cases. That’s why I simplified it to only make it explicit what cases would be covered, and have only the minimum of that. In the current state, all those additional cases would not be defined yet, or would be “up to” the tool, as those are not the current main use cases.

It could be specified later in subsequent PEPs, accepting or rejecting additional details. But if I add/cover everything right now, I’m closing that door, so I’ll leave that door open for now.

If I specify what should a specific tool that transforms a type internally, or what to do with generic type aliases, I make things more cumbersome and strict for tools that would only be willing to support the simplest use cases, and those should cover 90% of the use cases already.

Indeed, it would be great to not have inconsistencies or special rules, like how Optional actually means Nonable and not “optional”, how type aliases have to be declared at the top level of a module, how contextvars also have to be declared of a module, how docstrings can’t be f-strings even though they are indeed strings, etc. But it’s probably an acceptable compromise.

In that sense, the question of how should Doc transfer and move, and your question about composable new functionality, would probably be more related to Annotated itself and how metadata in Annotated should transfer in all those cases. If there are already existing rules for Annotated (there aren’t as far as I know, at least not in its PEP), then those should probably apply as well to Doc. And maybe what I could do is just say that any rules that apply to Annotated would also apply to Doc, maybe only specifying that, at runtime, the last one available should be used, but not much more.

Thanks for the support @EpicWink!

Indeed, I feel most of the arguments rejecting this here disregard the actual motivation and use cases, it’s probably not a pain point for everyone, or not everyone has to maintain, manage, and write docstrings too frequently to have these pain points fresh, maybe. Anyway, I added the bullet point list to try and make it easier to compare the current systems and what is “unique” to this proposal.

You’re not the only one. :upside_down_face:

Ah! That’s super interesting. Not sure how tooling would take the first type argument now being a slice instead of the pure type, but that looks quite good.

The only drawback I see is that it would “reserve” using slices in types for docs, although I don’t know what other use cases could be there for them. I’ll wait to hear feedback about this idea before acting on it.

Thanks, I’ll do (I actually struggle with that, with two “first names” and two “last names”).

119

2 posts were split to a new topic: Comments to document attributes

120

I have to say that I’m fairly negative on this.

Most of my issues are to do with the general concept of placing potentially large blocks of additional text within function signatures but specific to this implementation I’d mention that using import typing alone makes some of my own libraries take anywhere from 5 to 18 times longer to import so I couldn’t justify using this over existing solutions.

In general though, I find some of the main things that this tries to make easier are less important to me and are more readily solved by tooling than the things I think it makes more difficult.

To address more specific points:

I think this is one place where a little duplication is better than putting too much in one place.

Using Annotated[..., Doc(...)] within a function signature along with the actual documentation for the parameter can put so much extra text between the parameter name and the default value that it takes significantly longer to read and connect the two.

It also takes much less time to see what arguments a function takes in a ~45 line signature than in a 700+ line signature (looking at the FastAPI example). The function definition can be read as a ‘short’ list of parameters and defaults and then names can be looked up in the docstring or the full documentation for more detail if needed. I use the ‘go to definition’ feature of my IDE fairly frequently when using libraries so this would definitely be something I’d see as a user.

I think these cases either can be or already are handled with appropriate tools.

PyCharm for instance can already warn about parameters missing from the docstring and documented parameters that aren’t in the function and can also rename a parameter in the docstring at the same time as renaming the parameter itself. It also already has the ability to auto-generate templates in multiple formats which handles some of the issues with learning syntax.

I can’t say what other editors do but pylint appears to have a parameter documentation checker so this doesn’t seem unsolvable for docstring based parameter documentation.

6 Likes
121

I’m not sure what other people think, but in my view motivations of the form “eliminate the possibility of X” are essentially unpythonic. They’re in the same category as things like “eliminate the possibility of mutating a class variable from outside the class”, which go against the flexibility and consenting-adults ethos of Python.

The current case is a bit different, since here we’re talking about annotations rather than “hard” barriers like strict private-member constructs. But I see them as pretty similar. “Eliminating the possibility” of things just isn’t something anyone should expect when they’re using Python. There may be potential to make certain desirable things easier (e.g., keeping documentation up to date), but I’m not convinced we can get there by thinking in terms of making undesirable things harder.

Moreover, precisely because these are annotations and not language behavior, we need to keep in mind that even with this proposal, those possibilities would not be eliminated! People would still be free to not use the new metadata and write docstrings the old way, so anything could still happen. People would also be free to use all the metadata in the world but not run a typechecker, or run one that doesn’t support this particular type of metadata, or change a type but forget to change the textual docstring next to it, or so on and on and on, and wind up still publishing code that has a confusing mismatch between the docs and the behavior.

There isn’t any way to avoid such a mismatch other than being careful and actually reading the docs and making sure they’re right. I realize that this proposal is an attempt to narrow the gap between typecheck-time and runtime, but I still don’t see that the potential gains there outweigh the potential costs in terms of code readability. It’s also not clear to me that type annotations are the way to do this at all (as opposed to just linters). If the goal is to make it easier for people to keep documentation up to date, I think we should consider that problem from multiple perspectives (e.g., structured docstrings) before even settling on the approach of using type annotations for this.

1 Like