This discussion, the related draft PEP and similar draft PEP I had are expressing a similar idea.
FWIW there’s indeed overlap between your WIP proposals and the PEP.
type Movie = <{'name': str, 'location': <{'city': str}>}>
This is very sensible and can easily be the next step if this PEP is accepted or even incorporated.
Our PEP is more broad as it also focuses on enabling Protocols, which I think are very important to support as a lot of Python APIs revolve around classes.
I don’t think Sully is misusing the term. The PEP doesn’t add “Python operators”, it is adding type operators. The operands here are not Python objects and the results of the operations are not Python objects.
That’s why they are calling them type operators and they operate within the context of type annotations where you don’t expect arbitrary computation.
You created a problem by suggesting the use of a functional syntax that would look like Python but not be Python, and now you’re using that to say the terms in the PEP are a misuse of pre-existing Python terminology. They are not. The context changes the meaning. From Day 1 type annotations use indexing syntax for generics and claiming today that it’s confusing won’t get you far. Similarly, “binary or” in type annotations isn’t suggesting that the operands are integers. The separation of concerns proved to be a successful way to disambiguate matters.
PEP 827 type operators are not Python operators.
If somebody has a good proposal for an alternate word for “operator”, I don’t have any objection to reconsidering terminology, though I also don’t think it is a major problem: like Łukasz says, the context is important.
It’s not term unique to the PEP. It’s term that’d you find in general programming language theory/type theory material unassociated to any language. I think it is beneficial for the type system/type checking pep to not avoid terminology used to discuss types across languages.
I do think that this is a bit rough around the edges and has the potential I think to make usage of typing a significantly more frustrating experience and most of these would probably be served either by defining a protocol or by just typing it more loosely and relying on runtime behavior
It also has an effect of likely making error messages potentially incredibly difficult to follow as well (a problem with complex typing that both TypeScript and C++ have) as exploding the performance cost of type checking itself. Even if using these types are generally geared towards library authors, error messages will probably have to inevitably include data about them, which can potentially get pretty gnarly
Sure, theoretically some of these could be solved, but the developer experience with these types gets significantly more complex, especially in cases where there is judicious use of them, and I think there’s not much gain over library code that just defines the models for the queries like so
class UserFooQuery(table="user"):
# ...
select(UserFooQuery).where(...) # list[UserFooQuery]
or uses Any where appropriate. It feels like doing this is better than offloading all of this onto the type system
I’ve written a lot of both, and I’m so, so so much more frustrated when I bump into one of the ubiquitous labyrinth walls of what Python’s type system lets me do.
I want to be able to author function signatures without basically forgoing the type system and using code generation instead. I often wonder why I even bother trying to use the type system when I basically can’t use it for the library code I write daily?
I don’t think this is an issue of this PEP, this is an issue of all type checkers.
When I get a “call doesn‘t match any of the overloaded signatures”, the error message is basically useless. I’d want to see a list of signatures that’s almost matched, with an explanation of which type doesn’t match why, e.g. when I pass a value that has a long union type to a parameter accepting a long union type, I’d like to see what union members caused the mismatch.
The functionality in this PEP is something I’ve been reaching for again and again, basically every single time I try to do something not completely trivial with Python’s type system – which means basically every single time I’ve been handling numeric code.
I feel like if you all tried to implement a single numpy-style function (with a dtype argument influencing the output type), you’d yearn for this PEP as much as I do.
First review:
Link which is not an stdlib thing, thus it’s unclear what that does exactlyUser.id disappear if the result is a list?Unpackwell you explicitly want to avoid changing grammar… but if this gets accepted, you’re locking us out of **K and *L style expressions in typing hints possibly forever. This feels opinionated.ArgsParam: is it possible to * * List? Would **L syntax be allowed? If that’s undesirable, please call that out with a justification.Taking the proposed Slice as an example, there’s a trade-off between effectively designing a new language for types, and using plain Python for types.
The earlier means defining if e.g. Slice[SomeTuple, 0, -1, -1]is allowed to reverse a tuple, etc.
The latter would allow mroe generic things like type Foo = [e for e in SomeTuple if e.name.startswith("_")], though it gets a bit muddy wrt. what globals can be accessed in these expressions, how complicated an evaluation the type checker may be asked to perform.
Apologies if I’ve missed something in the PEP, but there are a few things I’m trying to work out how to express with these manipulations for dataclass-like decorators and base classes:
The PEP example __init__ is keyword-only. dataclasses has multiple ways that fields can be declared as keyword only including with the KW_ONLY sentinel ‘type’, is there any way to describe this?
attrs has converters which change the type of its __init__ fields. I see Attrs[T] can be used to obtain Member objects from a class, is there a way to obtain Params from a function/callable?
__post_init__ for the correct __init__ type.UpdateClass mentions possibly overriding old members, but dataclasses has the opposite behaviour of not overriding existing members, can this be expressed?
I’ll note separately that, having been part of the discussions here on the AST format proposal for annotations, I’ve soured on it pretty strongly the more I’ve thought about it.
While I’m not against making the annotations themselves simpler, I don’t think it should be at the cost of forcing every consumer of annotations to either defensively perform AST transformations at runtime or not support the new forms.
That said I am pro “Just store the strings”. Currently the string annotations are used in help() so it would be nice if these aren’t broken by the proposed changes[1].
Also getting STRING annotations currently is slow, having faster STRING annotations would also benefit the FORWARDREF format in the case where an error other than NameError occurs. ↩︎
A very interesting PEP, I’m still trying to get my head around it. I’ve wanted something like Pick for years now, mostly for database projections but also for REST-ish APIs. I’ve even given an online talk to the typing group a few years ago about leveraging the way attrs.fields() is typed in the Mypy attrs plugin; but that’s obviously non-portable.
The PEP contains an example where a model only has a few fields projected from it:
db.select(
User,
name=True,
email=True,
posts=True,
)
What would the example look like (at the site of use, mostly) if we wanted to deeply project posts? So we want the posts attribute selected, but mapped into a list[int](the IDs) instead of a list[Post].
It also has an effect of likely making error messages potentially incredibly difficult to follow as well (a problem with complex typing that both TypeScript and C++ have)
I think that is just the cost of trying to have more complicated machinery. There could be a separate typing primitive for every little use case (really lean into the “batteries included” nature of python) or you could have a “general” typing system. I think the general type-inclined python dev would rather have a general system so they can type their project now rather waiting for a future pep to maybe include their use case.
I also don’t think anything in this PEP precludes a future syntax update to simplify, similar to the original way you had to type unions and generic primitives (list, sets, etc).
It’s example-specific; meant to indicate that the field represents a link to another object in a database.
Because id wasn’t included in the select argument list that is deriving the result.
I’m not sure that I follow.
I’m not totally sure I follow what the proposal here is; the goal with this extended callables proposal is to have a very regular format for the parameters that can be manipulated easily.
Using “plain Python” is kind of a false economy here, because it would need to be very clearly defined what sublanguage is allowed, and programmers would need to keep that in mind when doing these things. Would/should startswithbe in that sublanguage? Hard to decide!
Yeah, we can produce positional-or-named arguments as well. Supporting the KW_ONLY sentinel is possibly but gnarly: currently it would involve a recursive type alias that walks down the list of the list of members.
Yeah, that should all be fine. You can get a tuple of Params out of a callable with GetArg[T, Callable, Literal[0]]. (Might be worth defining a type alias abbreviation for it too?)
Yeah, though it is somewhat manual: don’t put things in the UpdateClass if they already exist and you want to keep them.
Thanks for these answers, yes that would probably help. I’ll admit the description and demonstration for GetArg confused me on this somewhat. At the very least an example of using it on a function might be good.
Note that my own use case is probably a bit messier as it relies on matching names to get the correct type. The post init function acts as a partial init that can be used for conversion.
@prefab
class Example:
a: T1
b: T1
def __prefab_post_init__(self, b: T1 | T2): ...
annos = Example.__init__.__annotations__
annos['a'] # T1
annos['b'] # T1 | T2
I’m not sure how that would work for @dataclass style decorators, do you mean actually manually or just with some kind of conditional? Mostly relevant because dataclasses has the behaviour that it only adds methods if there isn’t already something in the class namespace.
For example here dataclasses won’t add an __init__ method for A, but will for B. From my understanding, the dataclass-like example in the PEP would always write.
from dataclasses import dataclass
@dataclass
class A:
a: int = 42
def __init__(self, a=54):
self.a = a
@dataclass
class B(A):
b: str = "H2G2"
A() # A(a=54)
B() # B(a=42, b="H2G2")
With a conditional.
Edit: Also, Members also includes the defining class as one of the elements, so you could check whether it was defined locally or inherited.
Regarding error messages, one thing we are doing to address this is the RaiseError operator, which will make it easy for libraries using type manipulation to produce more actionable type errors than might be achieved otherwise.
I guess more broadly I have issues with “a complicated type system” as a goal for the language and standard libraries and think this would be better suited as a mypy plugin or similar (and support can be a choice that a type checker makes) and the existing Annotated syntax and a 3rd party library (such as annotated_types)
If you want the library to support returning list ids for relations/links, you could make the library also support passing in some particular Literal enum value for ids, like:
db.select(
User,
name=True,
posts=Select.IDS,
)
It should also be possible to make it work recursively, though I haven’t built that example yet. (I should; it seemed to complicated for the PEP)
Something like this can definitely work:
db.select(
User,
name=True,
posts=Select(
title=True,
comments=True,
),
)
(Whether you can just do a regular dict there instead of a special object is a question worth thinking about a bit more.)
(Aside: I’m updating the PEP to add an age field to User also, since maybe it’s a little confusing that we select every field except id)