PEP 746: TypedMetadata for type checking of PEP 593 Annotated

Typing
41

Hi! I read through the draft PEP and had a couple questions about the specification:

(1) All of the examples of type-checked metadata show an instance of a class with a __supports_type__ method, but the specification as written would also allow things like this:

class SpecialString:
  @classmethod
  def __supports_type__(cls, obj: str) -> bool:
    return isinstance(obj, str)

s: Annotated[str, SpecialString] = ""

(SpecialString is an object with a __supports_type__ method, and SpecialString.__supports_type__("") type-checks without error.) Is this intentionally allowed?

(2) What happens for a metadata object that has a __supports_type__ attribute that doesn’t fit the requirements? E.g., the attribute isn’t a method, or it is a method but has the wrong return type?

42

I’ve implemented provisional support for PEP 746 in pyright. This will give us a chance to play with the concept before the spec is cast in concrete. This functionality will be included in next week’s release of pyright (1.1.367). It is currently disabled by default but can be enabled by setting enableExperimentalFeatures to “true” in your pyright config file.

Here is the commit if you’re interested in the implementation details or test cases. Compared to most typing PEPs, this was very simple to implement.

As for the questions @rchen152 posed, I interpreted the wording in the spec to mean that the __supports_type__ attribute should be treated like most other magic methods in Python. If the object has an attribute named __supports_type__, the type checker binds it to its object and then simulates a “call” with a single positional argument. If that “call” fails because the object isn’t callable or the argument is incompatible with the callable’s signature, then the metadata type is considered incompatible.

If the “call” succeeds, then its return type is examined. If that type is statically determined to be falsy under all situations (e.g. Literal[False], None, Literal[0]), then the metadata type is considered incompatible. If the return type is possibly truthy (e.g. bool, int or Any), then the metadata type is considered compatible.

The current draft of the spec indicates that __supports_type__ should return a bool value. In my opinion, it’s fine for the spec to leave the behavior undefined in cases where the return type is not bool, but I’m not opposed to clarifying the behavior in cases where it’s not a bool.

3 Likes
43

Yes, I don’t see why not. The spec specifies a protocol (which is now given in code form) that this example would conform to, so it is allowed by the spec.

44

Amazing Eric, thank you so much! I’m glad it wasn’t too hard to implement.

I’ll add support to Pydantic and annotated-types in the next couple of days. I’ll try to also make a PR to msgspec but I’m not sure the timeline for that.

45

Looks like @erictraut already handled the latter :raised_hands:

I’m happy for us to discuss the name a bit more, it seems like it will be easy for everyone to change. Some ideas:

  • __accepts_type__
  • __is_applicable_to_type__
  • __is_compatible_with_type__
  • __apply_metadata__
  • __validate_metadata__
  • __check_metadata__
  • __validate_value__
  • __check_value__
46

Sure, I’m not opposed to it. I’m coming at this from the angle that because there are multiple type checkers that we don’t want to implement things in incompatible ways, peps should try to be as complete and unambiguous as possible. Hence the questions about supported cases and edge cases.

I suggested a small update to the draft based on what you and Eric described. Let me know what you think =)

Edit: forgot to comment on the proposed names. For what it’s worth, I don’t mind __supports_type__, but of the other proposals, __accepts_type__, __validate_value__, and __check_value__ all seem decent to me.

47

Thanks. I feel the _type suffix is slightly off if we don’t actually pass a type to the method. Therefore I’d favor __is_applicable__, which I think fits nicely with the purpose of the PEP: we ask whether a piece of metadata is applicable to a given type.

48

I like __is_applicable__. But don’t you think it should reference metadata or annotation? Otherwise it seems a bit generic of a name, just browsing code I don’t think I’d be able to guess what it does. __is_metadata_applicable__?

49

Thank you Rebecca!

50

That feels wrong since we’re asking the metadata whether the “base” type is applicable, not the other way around.

Aside: I feel we could use some better terminology around Annotated. There is no term for the first argument (I’d like to call it the base type) and the remaining arguments are called “annotations”, which is a really confusing term because we also have annotations in the Python syntax. We should call them “metadata” instead.

So I’d like the terminology to be:

Annotated[BaseType, Metadata1, Metadata2, ...]

If others agree with this terminology I can add it to the typing spec.

3 Likes
51

Agreed better terminology is needed. That sounds good to me.

Yeah, that makes sense. I guess __is_applicable__ is clear enough if you already know the class is meant to be Annotated metadata (and I generally discourage folks from re-using types as metadata so it should be obvious it’s metadata). And it will mainly be inside of libraries anyway where discoverability is less important than it is for something exposed to end-users. If you agree I can update the PEP tomorrow.

1 Like
52

@adriangb, I agree with your previous observation that __is_applicable__ sounds too generic.

Just throwing out a few additional ideas:
__metacheck__ (similar to __instancecheck__)
__check_metadata__
__metadata_check__
__apply_metadata__
__apply_meta__
__metadata__
__annotated_metadata__
__apply_annotated__

None of these strike me as great options, but I prefer most of them to __is_applicable__.

@Jelle, I like the idea of improving the terminology. The term “metadata” sounds good to me, but I have some misgivings about “base type”. Given the flexibility of Annotated, there’s no guarantee that the metadata applies to the type. Maybe we should avoid any qualifiers and simply refer to it as the “type”?

Annotated[Type, Metadata1, Metadata2, ...]

53

Here’s another edge case: What if the Annotated is annotating something that is not a type expression?

x: Annotated[Final[int], Gt(0)] = 0

This doesn’t make sense, since Final[int] is not itself a type, it’s an annotation expression.

It should probably be an error to apply something that implements the __supports_type__ protocol to an Annotated that isn’t wrapping a type.

1 Like
54

That makes sense. I assume Annotated[Annotated[int, "foo"], Gt(0)] is fair game?

55

Assuming your ... is a type expression, yes. The spec says that Annotated[Annotated[T, M1], M2] is flattened to Annotated[T, M1, M2].

1 Like
56

How about aliases? Will,

Positive = Annotated[T, Gt(0)]

x: Positive[int]

supported? This requires alias preserve metadata for later expressions potentially in other files.

1 Like
57

I think for this you should use a bounded type variable. Your example should be an error because T is not compatible with Gt(0)'s argument; it’s as if you had written def f[T](x: T): return x > 0.

Instead, you should write something like type Positive[T: SupportsGt[int]] = Annotated[T, Gt(0)].

5 Likes
58

That makes sense. While it’s somewhat verbose I expect use of aliases like this to be either libraries or advanced enough use cases that if users want it to work with the type system the verbosity is acceptable.

59

Personally I think the name __supports_type__ might be too vague and can be interpreted in more contexts than one might desire. May I suggest something like __annotatable_on__
or __annotatable_with__?
I think some form of the word “annotation” should exist to immediately give the reader the context of annotations.

60

I don’t particularly love that Annotated is called Annotated but I agree that there is value in including annotated or annotation in the name to reference back to Annotated.