Typing: `NotRequired`-like for potentially-absent class attributes?

Typing
1

Briefly: I find myself wanting something like NotRequired for typing an attribute on a class that, when a user attempts to access it, throws AttributeError if it does not exist, and can be narrowed by hasattr, similarly to how a NotRequired field on a TypedDict throws KeyError on access if it does not exist, and can be narrowed by in.

Motivation

In stripe-python, we have a class StripeObject which is the parent class for all data objects in the library, that wrap JSON responses from the Stripe servers. StripeObject defines a custom __getattr__ such that accessing attributes obj.id, obj.description etc. throw an AttributeError if the JSON from the server did not contain the corresponding field.

While I wouldn’t choose this design if I were writing stripe-python from scratch today, it seemed reasonable ten years ago when the library was initially written, and changing it would be too breaking to our existing users for us to consider.

We have some workarounds in mind (see the github issue), but we’d really like to be able to provide our users with types that describe this behavior as it is. I’m fairly sure that this is impossible with the Python type system as currently designed, and I figured I would describe the problem as food for thought if nothing else.

Example

Here’s a simplified code example, to demonstrate what we’re trying to do.

from typing import Dict, Any, TYPE_CHECKING, Optional, Self, TypedDict, NotRequired

class StripeObject(Dict[str, Any]):
    if not TYPE_CHECKING:
        def __getattr__(self, k):
            try:
                return self[k]
            except KeyError as err:
                raise AttributeError(*err.args)

    @classmethod
    def init_from_json(cls, json) -> Self:
        obj = cls()
        for k, v in json.items():
            super(StripeObject, obj).__setattr__(k, v)
        return obj


class Customer(StripeObject):
    id: str
    description: Optional[str]  # Optional is a lie. We'd rather give this something like NotRequired.


def only_strings_allowed(x: str):
    print(x)


cus = Customer.init_from_json({"id": "cus_xyz"})

if cus.description is not None:  # Raises an attribute error
    only_strings_allowed(cus.description)

if hasattr("description", cus):  # It would be great if this narrowed the type
    only_strings_allowed(cus.description)

Thoughts

One line of thought I explored – could I define our own narrowing function using TypeGuard? For instance


def stripe_has_attr(obj) -> TypeGuard[???]: 
  # ...

if stripe_has_attr("description", cus):
    only_strings_allowed(cus.description)

What would TypeGuard narrow to? Some sort of generic protocol HasAttr[TAttrName]? I think this doesn’t work out, because protocols have to define a fixed set of methods – you can’t have a “dynamic” protocol with a method named after a generic type variable like TAttrName.

Questions

Anyway, my questions are:

  • Can anybody think of a way to type this that I am just missing?
  • If not, is extending Python’s type system in a way that would allow this something that would be considered?
2

Narrowing isn’t really feasible here with the current type system because it doesn’t distinguish between “A Customer instance that is known to have a valid description attribute and one that doesn’t”. The type of cus is Customer prior to the if statement and is still a Customer in the body of the if statement. There’s no “narrower type” in the type system than Customer in this case.

Pyright does have some special-case logic for TypedDicts where it internally tracks which NotRequired items have been assigned a value and/or have been tested for presence. It sounds like you’re looking for this sort of mechanism but applied to attributes on normal (non-TypedDict) classes.

Here’s an example of how this works in pyright with a TypedDict.

class Movie(TypedDict):
    name: str
    director: NotRequired[str]

def func(m: Movie):
    reveal_type(m)  # Movie
    print(m["director"])  # Type error: reportTypedDictNotRequiredAccess

    if "director" in m:
        reveal_type(m)  # Movie
        print(m["director"])  # No type error

To my knowledge, no other Python type checker implements this check today. There is an open feature request in the mypy issue tracker.

One possible solution in your use case is an intersection type in conjunction with NotRequired. Intersection types are not yet part of the Python type system but may be added at some point in the future. If intersection types were available, a type checker could narrow the type of cus to something like SupportsDescription & cus where SupportsDescriptor is a protocol that defines a description attribute that is known to be present.

1 Like
3

Ah, good to know! Didn’t fully realize that the narrowing on in was pyright-only – before I wrote my post I was reading through the PEPs and was surprised not to see mention of the in-narrowing there, but now things make much more sense.

The ability to intersect a non-typeddict with a typeddict seems close to solving this, but based on my very naive understanding of what this would mean, I worry a little bit as with

class Foo(TypedDict):
  a: NotRequired[str]

foo["a"] isn’t quite the same as foo.a.

Perhaps related, I recently learned about the object shapes feature in PHPStan (type checker for PHP). They have two separate element structural types for classes vs “arrays” (analogous to python’s dictionaries), I think for this reason.

4

While I wouldn’t choose this design if I were writing stripe-python from scratch today, it seemed reasonable ten years ago when the library was initially written, and changing it would be too breaking to our existing users for us to consider.

I can’t say I’ve used the library, but the examples you have here seem to have public types that allow the user to modify the data as both a dict and an object. I think you’re out of luck even with intersections to an extent, at least without some further additions beyond them, as you would need TypedDicts to synthesize the correct overloads for __getitem__ / __setitem__ for them to work in an intersection, which is out of scope for the ongoing Intersection work (it’s a reasonable request for type checkers to do so with the presence of intersections, but intersections won’t mandate it or be held up for this to happen)

If both of those were to be added, your example could have a public type of

class CustomerTypedDict(TypedDict):
    id: str
    description: NotRequired[str]

class IdStr(Protocol):
    id: str

class DescriptionStr(Protocol):
    description: str

# Things a customer always is/has (getitem/setitem already handling NotRequired on the dict interface)
type CustomerBase = Intersection[CustomerTypedDict, IdStr, StripeObject]
# Composed with the optional part that 
type CustomerType = CustomerBase | Intersection[DescriptionStr, CustomerBase]

# it looks like the StripeObject class is dynamic enough that this works moving all your type info to type contexts only could work.
Customer: type[CustomerType] = StripeObject  # may need a type ignore here as the only one

Seeing this need here though, I’m making a note to discuss hasattr and narrowing of intersections involving protocols. I think it’s the cheapest possible runtime check, so we should (if reasonably accomodatable) allow hasattr(obj, "description") to narrow the union in the example above, or at least I need to be writing the language for this in a way that leaves narrowing on that as an option to type checkers.

1 Like
5

There has been some previous discussion here: Optional class and protocol fields and methods · Issue #601 · python/typing · GitHub.

1 Like