Using __and__ for intersection is a much more obvious (I would even argue canonical!) choice than using __matmul__ for Annotated. I don’t think the two are equivalent.
5 Likes
I’d expect & to have the same rules inside and outside of annotations if intersections are accepted.
And again, & and | have their runtime meaning for types actually match the operator. An intersection is a logical and of types in set-theoretic typing. a union is a logical or. matmul here being used for annotations doesnt share that, so it does actually conflict and create an issue with reasonable use of what an operator was already defined for.
I’m fine with a new operator here too, I’m not dead set on this not being an operator, I’m opposed to specifically using matmul.
2 Likes
And not out of spite. It’s for not mixing @ in code bases where people are used to matrix multiplication with another meaning.
2 Likes
I have to admit, even though I love the proposal, that having to declare __matmul__ or __rmatmul__ methods, which literally say “matrix multiplication”, feels inelegant.
2 Likes
Both of you are missing my point. I wasn’t comparing the intuition of & to @; I was making the case that there’s a limit to what we should care about with forward references. I agree that & should be the operator of choice for intersections, but David’s argument would apply there too, so I was basically saying “if __matmul__ is too much, then __and__ will have the same problem”.
1 Like
Let’s take a deep breath. I’m going to step back and update the PEP, hopefully by this weekend. I will try to fairly capture the concerns raised here.
If anyone wants an early look at the draft, feel free to reach out. Once the update is pushed, if you think I missed your point or didn’t word it correctly, just let me know.
In the meantime, let’s try to reserve this thread for new points and keep things civil. I want to make it clear that I’m not aiming that at anyone specifically—at the end of the day, we are all just trying to make the typing ecosystem better
10 Likes
I think you misunderstand me here. annotationlib is not “maintaining weird, non-standard operators”. It’s just not making assumptions about what objects in annotations are.
The current implementation of Format.FORWARDREF works in the way that if we don’t know what an object in an annotation is we ‘refuse the temptation to guess’. It’s not assumed to be a type just because it’s in an annotation. A forward reference in this context is just a reference to some object that we can’t yet evaluate.
As we don’t know what it is we don’t know what the result of __matmul__(ref, other) is either, so we capture the whole ref @ other expression so it can correctly be evaluated later when we do know.
The practical issue that this results in is that this reference would be opaque and so it requires a change to how annotations are retrieved as partial information may be useful for Annotated (in a way that I don’t think it is for intersections[1]). This is what I was demonstrating with the cattrs example where it is able to make use of partial information to change the output of unstructure.
The most important thing is that I don’t want users switching to new syntax to have things silently break on them.
I also don’t want multiple ‘flavours’ of very similar annotation formats. dataclasses may not care about Annotated, but cattrs which will use the annotations it has previously gathered (it uses Field.type) does. Should dataclasses now make these assumptions too, even though they aren’t relevant for its own operation?
If you’re going to make the case for Format.FORWARDREF_STRUCTURAL and that essentially this is going to be the new standard format everyone’s going to have to use anyway, then just make Format.FORWARDREF that format rather than adding new formats and unnecessary decisions.
Again, I prefer adding something like DEFERRED that lets you get fully unevaluated references for something like dataclasses that avoids the need for a library that doesn’t care about Annotated to take a position on how an annotation should be evaluated if it’s going to be provided later to an unknown consumer.
Technically you could know if something should already be excluded but you can’t know what should be included without knowing all of the things involved in the intersection. ↩︎
3 Likes
I just read the PEP and I really like the Java Syntax but really dislike the inline @ syntax.
Would it be possible to make the annotation in python the same way, e.g.:
class Person:
@Field(ge=0, le=150)
age: int
Wouldn’t that also resolve some of the issues because it’s a new type of statement so there is no overlap?
4 Likes
This is the best proposal I’ve seen so far –[1] it feels very natural to me.
this emdash is written by a human ↩︎
I’ve looked through a code base I work on that uses Annotated for a place where this shorthand would actually help, and found nothing.
Context: we use Pydantic because we want to use fastapi, though most of us dislike Pydantic.
We use annotated for a couple of commonly reused types, like
UTCDateTime = Annotated[pt.AwareDatetime, pt.AfterValidator(convert_to_utc)]
and also the pattern
FooSettings = Annotated[X|Y|Z, pt.Field(discriminator="method")]
Because those definitions aren’t nested inside functions or classes, explicitly writing Annotated is fine.
For the most part we write Pydantic class attributes using the
class C(pt.BaseModel):
attr: float = pt.Field(..., default = 1.0)
pattern.
To me this pattern actually seems clearer than
class C(pt.BaseModel):
attr: float @ pt.Field(...) = 1.0
would be.
The Field object contains information about the constructor. All this information is stored together on the class in a way that can be understood fairly easily. There’s also very little visual noise, the = is a clear separator between the runtime attribute type and the constructor instructions.
When the args to Field are long enough to make it a multi-line expression, the = Field syntax still wins out over the @ Field syntax imo.
It also wins out over the “decorator” syntax
class Person:
@Field(ge=0, le=150)
age: int
as far as I’m concerned.
The only example from the PEP text which looks convincing is
from fastapi import FastAPI, Header, Depends
app = FastAPI()
@app.get("/secure")
async def secure_endpoint(token: str @ Header(description="Authentication token")):
return {"status": "authorized"}
but writing that as
Token = Annotated[str, Header(description="Authentication token")]
@app.get("/secure")
async def secure_endpoint(token: Token):
return {"status": "authorized"}
is better most of the time, because (in my limited experience) when I need a annotated type like this for one API endpoint, I almost always need the exact same annotated type for a couple of different API endpoints.
5 Likes
I find your arguments persuasive for this case. I think this option makes the most sense.
2 Likes
This example shows why I think this is unnecessary.
I don’t agree with the proposal’s method of fixing a supposed problem, but before even arguing that, the problem is self-inflicted. Dataclasses, and third party dataclass transforms already support not using Annotated at all and keeping the runtime type as the annotation, so the problem this solves shouldn’t exist in the first place, and only does if people choose to use annotated when they already don’t have to.
1 Like
Java has a pretty rich set of Target for annotations:
I would expect this synatx to translate to an annotation on the field (ElementType.FIELD) in java:
class Person {
@Field(ge=0, le=150)
int age;
}
Meaning that the decoration is attached to the field, not the actual type. It also has the ability to attach annotations to types (via ElementType.TYPE_USE). It looks something like List<@Field(ge=0, le=10) int>
If you wanted more of a one to one comparason for decorating the type itself:
class Person:
age: int @Field(ge=0, le=150)
And the java TYPE_USE equivalent:
class Person {
public @Field(ge=0, le=150) int age;
}
I think following the java route and supporting more targets would be a great follow up (c.f. some of my comments about static tracking etc…).
This looks pretty nice but how would that work when annotating function parameters?
If you wanted to annotate function parameters you could imagine something like this:
@app.command()
def greet(
@typer.Argument(help="The name of the user to greet")
name: str,
@typer.Option(help="Number of times to greet")
count: int = 1,
formal: bool = False
):
In java parlance, we would be annotating ElementType.PARAMETER, not a type.
1 Like
While it makes sense, it doesn’t feel like something that can be “easily” accomodated by the language syntax. That being said, it’s a question for a follow-up PEP I imagine.
That’s a fair point, but changing the default behavior of FORWARDREF is a big departure.
Instead, I’ll be replacing Format.FORWARDREF_STRUCTURAL in the draft with Format.TYPE. By following the typing-spec, operators are explicitly evaluated under typing rules. This guarantees | always produces a Union (solving the None | None issue) and @ always produces Annotated (solving None @annot).
We can always deprecate FORWARDREF later if TYPE and VALUE prove to be enough.
2 Likes
Interesting - was not aware about this since I don’t know java. Thank you for the excursion.
However python has no concept of a field so I would see no problem if python uses this syntax for a type annotation.
It would also work very well for type statements:
@Field(ge=0, le=150)
type AgeType = int
That looks actually okay. Not great, not terrible.
I think this is a bad idea. It’s one thing to assume undefined objects have certain behaviour but it’s another to make things valid that are already known to fail at runtime.
This also creates a corner case where Format.TYPE can now fully resolve something (no remaining ForwardRef instances) but VALUE will still always fail.
Note that None | None is currently broken enough that you can’t even get STRING annotations with it.
>>> from annotationlib import get_annotations, Format
>>> def f() -> None | None: ...
...
>>> get_annotations(f, format=Format.STRING)
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
get_annotations(f, format=Format.STRING)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
...
File "<python-input-1>", line 1, in __annotate__
def f() -> None | None: ...
~~~~~^~~~~~
TypeError: unsupported operand type(s) for |: 'NoneType' and 'NoneType'
1 Like
This actually makes the proposal more objectionable. It’s not correct to assume that everything in an annotation follows what’s valid as a type annotation. I’d rather tools that don’t understand actual matrix multiplication in an annotation and that need everything in an annotation to be a valid type annotation refuse to operate on it as invalid input to the tool than assume I must have meant to use Annotated and deviate from what’s actually defined on a type.