Use function annotations to improve bytecode specialization?

Ideas
1

CPython 3.11 introduced adaptive bytecode specialization. It can specialize operations after observing runtime behavior.

For example:

def add(x, y):
    return x + y

Initially uses BINARY_OP, then after enough executions it may become an integer addition specialization.

But what about:

def add(x: int, y: int) -> int:
    return x + y

Here the compiler already has information that both arguments are expected to be integers.

Currently, this information seems to be ignored by CPython’s optimizer. The interpreter still starts from the generic BINARY_OP and needs to rediscover the types at runtime.

Would it make sense to use annotations as an optimization input?

For example, emit a more specialized operation directly:

LOAD_FAST x
LOAD_FAST y
BINARY_ADD_INT
RETURN_VALUE

instead of:

LOAD_FAST x
LOAD_FAST y
BINARY_OP
RETURN_VALUE

I understand that annotations are not runtime-enforced in Python today. This does not necessarily require changing that. Similar to static languages, this could be an optional optimization based on the programmer-provided contract.

The main question is whether CPython should make use of existing typing information instead of throwing it away during compilation and performing type discovery again at runtime.

1 Like
2

If the compiler can infer the exact types at compile time, it can also infer whether __add__ *will fall back to __radd__. In that case, the runtime if check for radd is unnecessary. The generated code can call the corresponding C function directly.

3

That would break dynamic addition / deletion of attributes though…

class mytype:
     def __add__(self, other: object) -> mytype: ...

# Implementation is irrelevant 

m1 = mytype()

m1 + 1 # Uses __add__

# Might be called after some n calls of the previous behavior, in an if-case, ...
del mytype.__add__

m1 + 1 # Should use int.__radd__, raises an error
2 Likes
4

I think the language allows for the identifier int to have other values than the type builtins.int.
Reading

def add(x: int, y: int) -> int:
    return x + y

doesn’t yet tell that the arguments are both integers.

5

Type annotations are not enforced at runtime, by design. The following is perfectly valid code:

def add(x: int, y: int) -> int:
    return x + y

print(add("foo", "bar")) # "foobar"

Optimizing the bytecode in advance would break this snippet. That’s a massive breaking change that has been discussed several times but I seriously doubt will ever happen.

5 Likes
6

True, and if you really want it, there’s Cython, although it uses its own types.

(disclaimer: I don’t have hands-on experience with Cython, but my understanding is it uses the annotations for faster code, just like OP proposed)

1 Like
7

I really, really wish we could do this, but unfortunately & has already said, this would be a massive breaking change. Sure, my own very personal opinion is that dynamic attribute setting is a massive code smell anyways, etc… but python is designed at the core to support this. The only option would be to have a config option to activate these types of optimizations, but I’m pretty sure this has already been discussed and would be some huge work anyway. If you do need to speed up python with guaranteed types, I can’t recommend enough trying rust compiled modules with pyo3. Rust is very approachable when coming from python, despite it’s reputation, and you will see a lot of perf improvements.

You can also try mypc, it’s closer than what you are thinking (using type hints infos to compile code), altough using mypy has a type checker is meh in my experience (VS basedpyright for example)

Finally, cython, has this has already been recommended. However developper experience is much more clunky, you lose memory safety (guaranteed by both python and rust) once you go low level with ctypes, and good luck making it work seamlessly with type checker/linter/formatter/LSP.

2 Likes
8

That would be the job of something like mypyc, or Nuitka (to a limited extent).

3 Likes
9

The existing optimization from BINARY_OP to BINARY_ADD_INT is heuristic, based on some kind of runtime statistics, anyway, no? I’m assuming that it in any case we can never be 100% sure that BINARY_ADD_INT will always be called with ints, so it has to have some kind of error-detection and fallback already? Maybe it’d be ok to use the type-hints and start out with BINARY_ADD_INT, and 99% of the time it will be fine, and in the remaining 1%, where the programmer has decided to do something “funny”, we rely on the (presumed) existing error detection and fallback?

2 Likes
10

The heuristic you refer to is already sophisticated, and in my view, more reliable than type hints, since those can always be wrong (for example, I once saw an official typeshed stub having an annotation of a dictionary subscripted with dictionary keys) or abused by downstream. Ironic.

1 Like
11

If Python is already checking types at runtime (via operator dispatch), why can’t it use the annotations to check at the call site? It’s the same check, just earlier.

12

this is still a hinting for the adaptive intrepter to know that like this can be type of X, instead of this must be type of X.

13

it should still be inherited from int, or it’s not int.

14

it’s still a hinting, not a contract, which means that it only hints the JIT that it can be X.
and for the code you provieded, in my opinion that if you write the type hints, you should follow it, or why you write it?

15

if i can use python, why i use cython

16

Finally, cython, has this has already been recommended. However developper experience is much more clunky, you lose memory safety (guaranteed by both python and rust) once you go low level with ctypes, and good luck making it work seamlessly with type checker/linter/formatter/LSP.

i agree with that.
that’s why i want to use the originally provided type hints to optimize the adaptive intrepter.

17

but we have been using RTTI to check like that does a class provides a operation
so why not use it to optimzie

18

so we need a deopt that looks like C2JIT

19

so that’s why the type hints can only still be hints =)

20

This comes up from time to time; I’ve been guilty of asking about it, several years ago :‍)

The current adaptive interpreter optimizes after two iterations:

import dis

def add(a, b): return a + b

for i in range(3):
    print(f'after {i} calls:')
    dis.dis(add, adaptive=True)
    add(1, 2)

after 0 calls:
  3           RESUME                   0
              LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP                0 (+)
              RETURN_VALUE
after 1 calls:
  3           RESUME_CHECK             0
              LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP                0 (+)
              RETURN_VALUE
after 2 calls:
  3           RESUME_CHECK             0
              LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP_ADD_INT        0 (+)
              RETURN_VALUE

Speeding up two iterations is not worth the implementation complexity of passing hints to the optimizer.

4 Likes