PEP 827: Type Manipulation

A colleague created a function similar to bufferCount in rx js (RxJS) and this got me wondering how to better type this. I got this, but it feels weird. Is there something better I’m missing? I couldn’t really see how to make a tuple of T of size based on a parameter.

@overload
def batch[TT](
    batch_size: Length[TT],
) -> Operator[GetArg[TT, tuple, Literal[0]], TT]:
    ...

@overload
def batch[T](
    batch_size: int,
) -> Operator[T, tuple[T, ...]]:
    ...

def batch[T](
    batch_size: int,
) -> Operator[T, tuple[T, ...]]:
    ...

I’ve wondered the same before, and although I found a pretty simple solution for the runtime typing aspect, adding support for checkers would be hard of course. I found the best way (as a somewhat temporary solution) is to just overload the like 5 most used cases, and hint the rest as tuple[T, ...].

The same problem also exists for not passing any size, just a tuple of n elements, and returning a tuple of n elements of a new type by the way (e.g. def fn(arg: tuple[T, ...]) -> tuple[Something[T], ...]).

Wouldn’t the later be possible using:

def fn[Ts: tuple[object, ...]](arg: Ts) -> tuple[Something[T] for T in Iter[Ts]]

At first I wanted to say that you can’t really do this, because you can’t do arithmetic on int literals (intentionally)… but I think you can do something like:

type _TupleHelper[El, Len, Tup] = Tup if IsEquivalent[Length[Tup], Len] else _TupleHelper[El, Len, tuple[El, *Tup]]
type MakeTuple[El, Len] = _TupleHelper[El, Len, tuple[()]]

def batch[T, L: int](batch_size: L) -> Operator[T, MakeTuple[T, El]]:
    ...

Some general feedback, based on a personal project I’m doing where PEP827 sounds like it should address my use case, but it does not entirely.

So, I’m building an actor library in Python. The idea is that on one hand, we have actor classes with behaviors defined. This actor class can have an __init__ with state and normal methods. Only the behavior methods are exposed to other actors. Actors can only talk to other actors through a proxy class that only exposes behavior methods (no other methods or internal attributes).

Deriving such a proxy class from an actor class is possible with using NewProtocol.

The problem here is that an actor reference itself needs to be serializable and deserializable. And this is a problem, because we need a nominal type. If the actor is named MyActor, and the derived type is Ref[MyActor] (which is a NewProtocol), then we can’t do Ref[MyActor].deserialize(…). Maybe we can do MyActor.deserialize_ref(…), but that’s not the point - ideally something like Ref[MyActor] would end up in a Pydantic datastructure and should be deserializable as such. So, I’d like the derived Ref[]to be a subclass of a Pydantic BaseModel.

This is where __init_subclass__ sounds useful, but trying Michael’s mypy branch, I didn’t manage to get this to work. The __init_subclass__ approach does not seem to work for generic types and is not able to iterate over the fields of the type parameter passed to this generic type.

I’m looking for something along the lines of:

class Actor:
    pass

class Ref[A: Actor]:
    def __init_subclass__[T](cls: type[T]) -> UpdateClass[
        *[
            Member[p.name, p.type]
            for p in Iter[Attrs[A]]
            # if IsAssignable[p.type, Behavior]
        ],
    ]:
        pass

    actor_id: int

Ideas welcome :slight_smile:

1 Like

While looking for other approaches, something else that seems missing is a way to derive sum types. (NewTypedDict/NewProtocol are product types, right?). I would expect to also have a NewUnion[...].

# Assume I have an actor defined using behavior methods:

class MyActor(Actor):
    " Actor with two behaviors defined. "
    @behavior
    async def set_value(self, input: int) -> None: ...
        # @behavior turns method into `Behavior[Input, Output]` instance.
        # In this case, Behavior[int, None]

    @behavior
    async def get_value(self, input: None) -> int: ...

# This is the collection of behavior messages that can be send to that actor.
type ActorBehavior[A: Actor] = NewUnion[
    *[p.type for p in Iter[Attrs[A]]
    if IsAssignable[p.type, Behavior]]
]

class Ref[A: Actor]:
    def call_behavior[Input, Output](self, behavior: ActorBehavior[A] & Behavior[Input, Output]) -> Output: ..

ref: Ref[MyActor] = ...  # Reference to `MyActor` on which behavior methods can be called.
ref.call_behavior(MyActor.set_value(4))

The above is not functional. I’m not sure whether this would work out. But it should illustrate the problem space.

Anyway, I hope this is useful feedback. I really appreciate the effort around this PEP.

(Thinking and responding to the full message, but this is a quick answer)

You can use Union[…] in the way you want

1 Like

Oh, thanks! I didn’t realize that.

I need to think a bit more about how to specify this more precisely in the document, but my intention with __init_subclass__ is that the shape of the class is “fixed” at class declaration time, which means that it can’t depend on inspecting the class’s type parameters to do it.

What does your desired interface look like exactly? You will be calling Ref[MyActor].deserialize(…), and deserialize will be a classmethod on ref?

What does this look like in the non-typed version of this? I’m not super familiar with the actor model, I suppose; do you have a link to what you are building or something similar?

Thanks!
deserialize() will be a classmethod on the Ref indeed. I have a repository. Let me clean up and come back in a couple of days.

Update with links to working code using NewProtocol for type safety. I think I have something that works sufficiently well.

The actor library I’m working on is called Actorium and can be found here:

It’s a work in progress, in search for what works best within the constraints of the Python environment.

Let me quickly explain actors in this context.

An actor is an entity that can receive messages and processes them sequentially. It’s like a lightweight thread that has no shared state with other actors. An ActorSystem is created at the beginning and is responsible for the scheduling and the event loop. ActorSystem instances from multiple processes (or sub interpreters) can be connected so that actors can find each other across processes. Spawning an actor produces an actor reference, a proxy that allows for sending messages to that actor. The proxy object itself is serializable so that it can be passed as part of a message to another actor as a return address, which is useful for implementing an RPC mechanism.

Now, having only actor_reference.tell(...) syntax is cumbersome to deal with. So, Actorium implements behavior actors, inspired by the Pony language. For a behavior-type actor, the methods that we want to expose to other actors are decorated by the @behavior decorator. As a result of that, the actor-proxy is an object that exposes those behavior methods and those only.

This is where NewProtocol becomes essential. We need to derive the proxy class from the actor class. Without that, we’d have close to zero type coverage for Actorium code, because actor programming is doing behavior calls across actors all over the place.

from actorium import Actor, BehaviorActor, Mailbox, behavior, run, spawn

class Calc(BehaviorActor):
    @behavior
    async def double_it(self, value: int) -> int:
        return value * 2

    @behavior
    async def plus_one(self, value: int) -> int:
        return value + 1

class Main(Actor[None]):
    async def run(self, mailbox: Mailbox[None]) -> None:
        async with spawn(Calc) as calc_ref:
            result = await calc_ref.be.double_it(4)
            print("Double of 4 is", result)

if __name__ == "__main__":
    run(Main)

The be attribute, which stands for “behaviors” on calc_ref is a NewProtocol derived object that only contains the public part of the actor interface. I would have liked if we could do calc_ref.double_it(4) and have the behavior methods available straight on the reference, but found no way to achieve that in a type safe way.

Note that calc_ref here is a serializable object that we can pass to other processes so that they can call these behavior methods. It’s of type BehaviorRef[Calc] and only contains an actor address. The type parameter defines which methods become available under the be attribute.

The NewProtocol definition can be found here: actorium/src/actorium/core/actor/behaviors.py at main · actorium/actorium · GitHub For me what I got works sufficiently well. No further questions from my side.

If you’re interested, there’s a related type challenge related to how the spawn() method in the above example is typed. (spawn accept __init__ arguments from the actor, but the actor reference is inferred as a return type.)

Looks like IsAssignable does not support variadic generics. Is that intentional or an oversight or maybe something for later? Right now I have the following workaround:


type BehaviorRefMethods[A: BehaviorActor] = NewProtocol[
    *[
        # Take the `behavior_method` from the `_BehaviorMethod`
        # attributes from a `BehaviorActor`.
        Member[p.name, GetMemberType[p.type, Literal["behavior_method"]]]
        for p in Iter[Attrs[A]]
        if IsAssignable[p.type, _BehaviorMethod[Any, Any, Any]]
        or IsAssignable[p.type, _BehaviorMethod[Any, Any, Any, Any]]
        or IsAssignable[p.type, _BehaviorMethod[Any, Any, Any, Any, Any]]
        or IsAssignable[p.type, _BehaviorMethod[Any, Any, Any, Any, Any, Any]]
    ]
]

See: actorium/src/actorium/core/actor/behaviors.py at 052e349bac250244cdbc22cf067340bf4f992308 · actorium/actorium · GitHub

There is a problem with the current proposal for evaluating if expressions in annotations: in a post-PEP 827 world, runtime inspection of an annotation via the standard library (get_type_hints / .__annotations__) can no longer be trusted for type analysis.

The reason is the default hook for typing.Bool, which makes it evaluate to False:

int if typing.Bool[Literal[True]] else Literal["UNREACHABLE"]

A naive check reveals Literal["UNREACHABLE"], while the intended reduction is int.

Moreover, this behavior is not even consistent at runtime — it depends on the order in which the operations are applied.

First a naive annotation lookup, then eval_typing:

def foo() -> int if typing.Bool[Literal[True]] else Literal["UNREACHABLE"]:
    raise NotImplementedError

assert foo.__annotations__['return'] == Literal['UNREACHABLE']
# ✅ typemap-unaware lookup resolves `typing.Bool[...]` to False
# expected for naive evaluators

assert eval_typing(foo).__annotations__['return'] == Literal['UNREACHABLE']
# ⚠️ typemap-aware evaluator returned the cached value instead of reducing

# RESULT: foo() -> Literal["UNREACHABLE"]

First eval_typing, then the naive lookup:

def bar() -> int if typing.Bool[Literal[True]] else Literal["UNREACHABLE"]:
    raise NotImplementedError

assert eval_typing(bar).__annotations__['return'] is int
# ✅ typemap-aware evaluator resolved first — correct type put in cache

assert bar.__annotations__['return'] is int
# ⚠️ typemap-unaware lookup used the cached typemap-aware (reduced) value

# RESULT: bar() -> int

I genuinely see only two solutions:

  • Bool and Iter are fully reduced at annotation lookup — which is equivalent to including a full evaluator (typemap) in the standard library, as reduction in general requires the full subtyping engine.
  • Bool and Iter raise an error when retrieved via a naive .__annotations__ / get_type_hints() lookup.

Honestly, I didn’t really understand why a runtime evaluator couldn’t be included in the standard library in the first place, but both solutions should be feasible.

Playground: 99_if_order_dependency.py

1 Like

Based on PEP 827, I built tysql — a small, statically-checked subset of PostgreSQL.

class Post(Table):
    id: PrimaryKey[int]
    author: ForeignKey[User, Literal["id"]]
    created_at: datetime
    text: str


stmt = Select[
    InnerJoin[
        User,
        Post,
        On[Eq[Col[User, Literal["id"]], Col[Post, Literal["author"]]]],
    ],
    Cols[
        Col[User, Literal["id"]],
        As[Count[Col[Post, Literal["id"]]], Literal["n_posts"]],
    ],
    GroupBy[Col[User, Literal["id"]]],
    OrderBy[Col[User, Literal["id"]], Literal["asc"]],
]

if TYPE_CHECKING:
    rows = run(stmt, data=None)
    reveal_type(rows[0]["n_posts"])  # int

Take a look at the live playground, with tysql and PEP 827 installed:

5_join.py — tysql and PEP 827 playground

While developing it, I found a lot of bugs in the implementations of typemap and the mypy fork. I also stumbled onto quite a nice testing setup:

Instead of a mypy fixture, you just run mypy directly over your codebase. Mark what’s expected to fail with # type: ignore, and (with warn_unused_ignores) mypy ensures it actually fails.

def mypy_test_loud_keys() -> None:  
    # static: mypy checks it, pytest skips it
    if TYPE_CHECKING:
        p: Loud = {"X": 1, "Y": 2.0}
        assert_type(p["X"], int)
        assert_type(p["Y"], float)
        p["x"]  # type: ignore[misc]  
        # negative: the lowercase key must be rejected


def test_loud_keys() -> None:  
    # runtime: pytest checks the evaluated class
    D = eval_typing(Loud)
    assert D.__annotations__ == {"X": int, "Y": float}
    assert D.__required_keys__ == frozenset({"X", "Y"})
6 Likes

I would appreciate if the PEP could add some consideration for API docs. This PEP seems to be written entirely around type-checking and runtime use of annotations. Type-checkers and libraries using annotations at runtime are not the only users of type annotations: users of these libraries expose documentation to their users, so tools that extract API docs will have to be able to understand the syntax proposed in this PEP. I’m not too afraid to say that API docs is a big part of Python development and library maintenance.

If FastAPI/Pydantic start using the syntax proposed by this PEP, docs extraction tools will have to support it too, because users of FastAPI/Pydantic will want to document their models. These tools are not equipped for this I believe. I’ll speak for myself: Griffe is not equipped for this. Supporting what this PEP proposes requires a proper type-checker, and Griffe and other documentation extraction tools are not type-checkers.

Currently, extracting API docs from Python source is straight-forward enough: parse source with ast or similar, introspect compiled modules, iterate on the AST or in-memory objects. Just a few tricks here and there to understand concepts like descriptors or typed dicts and return good results to documentation renderers. This PEP stops that: tools would now need to evaluate type annotations to understand what’s going on and return results that documentation readers are satisfied with.

Taking the FastAPI example:

class Hero(NewSQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)

    name: str = Field(index=True)
    age: int | None = Field(default=None, index=True)

    secret_name: str = Field(hidden=True)

type HeroPublic = Public[Hero]
type HeroCreate = Create[Hero]
type HeroUpdate = Update[Hero]

I will leave aside the fact that this is better documented as an HTTP API rather than a Python API, because I guess not all users of this PEP will be HTTP API projects.

Here the developers will probably want to expose and document HeroPublic, HeroCreate and HeroUpdate since they are public symbols, used in public functions/endpoints. They will want their users to know what is in each of these classes. They have three options for that: explicitly list that in docstrings, rely on docs extraction tools, or do nothing and let readers do the work.

Option 1, manual docs:

type HeroPublic = Public[Hero]
"""Output class for getting heroes.

Has `id`, `name` and `age` set.
"""
type HeroCreate = Create[Hero]
"""Input class for creating heroes.

Must have `name`, `age` and `secret_name` set. 
"""
type HeroUpdate = Update[Hero]
"""Input class for updating heroes.

Can have `name`, `age` and/or `secret_name` set. 
"""

This is already very redundant and doesn’t even show field types or descriptions.

Option 2, expect the tool to handle it:

The docs tool would have to understand how Public, Create and Update work (by the way, the example could make it more clear that these types would be imported from fastapi, as in the current code block they are NameErrors).

Understanding what Public does (essentially keeping only fields that are not hidden) means going and finding it in FastAPI’s source code (which might not be available in the current docs-building environment), and understanding the computation that Public does. That’s literally impossible for a static analysis tool that is not a type-checker. The PEP says:

the implementation of Public[], Create[], and Update[] computed types is relatively complex

To me, “relatively complex” here sounds like it’s hard to write, harder to read, harder yet to type-check, and IMO simply impossible to understand without a type-checker. Docs extraction tools that are not type-checkers (I believe none of them are) are effectively excluded from the party. Unless they start using proper type-checkers internally (I expand on that after option 3).

Option 3, left as an exercise to the reader:

We could say that if Public and Hero cross-link to the symbol in FastAPI’s docs and the symbol in the current docs respectively, users can read the docs for both symbols and gather that HeroPublic is Hero without hidden fields. This could be more than enough. But this example is just one simple example. I’m sure users of this PEP will find more advanced use-cases, with more elaborate type computations, for which asking readers to do the mental work wouldn’t be decent. Developers will want docs extraction tools to ease the documentation task, to make docs easy to read and understand for their users.


Supporting this PEP in the API docs world would require cooperation between type-checkers and documentation tooling. Ideally, type-checkers would expose (Python) APIs for querying their internal data/state/model based on symbols’ canonical paths. Exactly what the PEP mentions in the FastAPI example: “given the code in mypkg and fastapi, what shape do I get with fastapi.Public[mypkg.Hero]” (or simply “what is the shape of mypkg.HeroPublic”). And the answer would be “this type, evaluated, would look something like: […]”. There would probably be many more kinds of query that would need to be supported. I realize this is quite vague, and requires quite some thinking and coordination between type-checkers and documentation tools, and that’s exactly why I think it deserves some consideration in the PEP :slightly_smiling_face: Otherwise docs tools are left on their own, with feature requests accumulating and no expected support from type-checkers.

The PEP mentions Typescript, and it’s true that it makes type-checking a required part of API data extraction: just like how Typedoc has to run the Typescript compiler (up to type-checking and before emitting JS) to extract API data.

3 Likes