Is it ok to use async __new__?

Python Help
1

This example runs and linters don’t complain. Other than it being unexpected for a class constructor to behave like an async generator, is there anything fundamentally wrong here?

import asyncio
from collections.abc import AsyncGenerator


class A:
    async def __new__(cls) -> AsyncGenerator[int, None]:
        yield 1
        yield 2

async def main():
    async for elem in A():
        print(elem)

if __name__ == "__main__":
    out = asyncio.run(main())
2

At one level there is nothing special about __new__ as a classmethod so the question is no different from asking about calling A.some_class_method() instead of A().

There is a subtlety though that A() does not call A.__new__ directly but rather type(A).__call__ which (ignoring metaclasses) goes on to call A.__new__ and possibly a.__init__(). In your case this subtlety does not come up though and you can think of A() as being equivalent to A.__new__().

Regardless this is a confusing way to write the code.

1 Like
3

Thanks for the comment. Although weird, it does allow one to create specialized versions of “functions” that have fewer generics. I think this is not possible today without classes.

import asyncio


class my_generic_function[A, B]:
    async def __new__(cls, a: A, b: B) -> tuple[A, B]:
        return a,b

class my_specialized_function[B](my_generic_function[str, B]): ...

async def main():
    await my_specialized_function('hello', 1) # OK
    await my_specialized_function(1, 1) # Type checker error

if __name__ == "__main__":
    out = asyncio.run(main())

The async aspect is really irrelevant in this example, but it is good this trick also works for async functions.

4

Note that using an async function here is almost surely not covered by the intent language design (unless you can find language to the contrary) and since __new__ is a special name, any behavior you rely on that isn’t explicitly documented may break in a future update.

1 Like
5

That is my question. Is this intentionally supported or does it just happen to work? Note that using async on __init__ is explicitly disallowed, so some thought went into this already. If this is “safe” today, I doubt it will break. __new__ is special but it is also public interface.

Edit: it seems this that

  • Not returning a subclass of A is quite common (see this and related issues) and
  • People have been using async __new__ extensively
6

Nothing in this violates the data model, and I don’t think it’s possible to change the language to forbid this in a way that wouldn’t be classified as a breaking change as this falls entirely within the documented behavior of __new__.

I wouldn’t write the class you have there this way and personally consider it bad form, but it’s legal code with a well defined meaning that happens to align with your usage.

The idiomatic way (and therefore, probably the way that ends up easier to maintain and collaborate with others on) to write this depends on some semantic behaviors that your example doesn’t differentiate between, and I imagine there’s a reason you are asking about this being a class that isn’t captured in your simplified example.

1 Like
7

The other part of the context is this discussion PEP 718: subscriptable functions - #84 by rsdenijs. I have a generic async function, and want to specialize it. Without that PEP, it is not possible to specialize, but using the “bad form” used here, it is possible today. This thread just shows the trick also works for async generic functions.

8

As a workaround for two separate limitations, one in the type system, one in type checkers, this is probably fine, but I wouldn’t be shocked if some other tooling (such as a linter) complains about this, or even if some typechecker did at some point despite being legal.

You may want a note somewhere in internal documentation about intentionally using new this way if it becomes used in multiple places in the code base, or a quick link or mention in places it is used if used rarely.

There are other ways to do this that are a little more idiomatic, but I can sympathize with these feeling like being overly verbose while working around limitations in the type system, and it sounds like you are aware of them. If you’d like me to go into more detail on other ways I’d consider writing this, I’d be happy to, but if you’re just looking for whether or not this use is fragile, in terms of language stability: you’re in bounds on data model, in terms of tooling stability, it depends on the level of understanding your tools have of what’s allowed by the language + what else the authors believe is worth warn for and/or supporting.

9

Thanks. Yeah my main concern was language stability or gotchas. This seems to be covered. Tooling-wise, this is compatible with pyright and ruff. The main NO for this solution is the surprise factor for other developers upon seeing this pattern. Hopefully PEP 718 or some alternative gets accepted and allows to do this in a natural way.

Regarding alternatives, the only “standard” alternative I am aware of is to write a wrapper function with the specialized signature. This is very verbose.