Implement __class_getitem__ for ctypes.pointer

Ideas
1

Proposal

Make ctypes.pointer support __class_getitem__. As it is currently a function, we can change it to a class, with __new__ implementing the current features of the function. This should not affect existing behavior significantly.

  1. We could make pointer[...] return a GenericAlias as usual
  2. Or alternatively, it can, at runtime, return a call POINTER(...)
    With this option we can effectively eliminate the need for the other POINTER type.
POINTER(POINTER(c_uint)) == pointer[pointer[c_uint]]

Rationale

Currently, ctypes.pointer is used as a generic for type hinting purposes (mypy, pyright, pycharm, etc.) as well as for runtime-libraries that use type hinting.

from ctypes import pointer, py_object

def fn() -> pointer[py_object[str]]:
    ...

But this is not evaluable at runtime since ctypes.pointer is not a function that supports __class_getitem__. So this does not work as a type alias nor with get_type_hints.

We also have another type, ctypes.POINTER, to create a typed pointer, but this doesn’t accept generic aliases, so it also doesn’t work in typing.

from ctypes import pointer, py_object

def fn() -> POINTER(py_object[str]):
    ...
1 Like
2

Pointer types are subclasses of ctypes._Pointer with a given _type_, such as ctypes.c_int. The exceptions are function pointer types, which subclass _ctypes.CFuncPtr, and simple pointer types, which subclass ctypes._SimpleCData for a given _type_. The latter includes ctypes.c_void_p (type “P”), ctypes.c_wchar_p (type “Z”), and ctypes.c_char_p (type “z”).

The builtin POINTER and pointer functions exist to facilitate creating and caching _Pointer types and instances.

The POINTER function creates and caches a _Pointer subclass for the given type, if one isn’t already cached. For example:

>>> ctypes.c_int in ctypes._pointer_type_cache
False
>>> c_int_p = ctypes.POINTER(ctypes.c_int)
>>> c_int_p._type_ is ctypes.c_int
True
>>> ctypes.c_int in ctypes._pointer_type_cache
True

The pointer function returns a pointer to its argument. It calls POINTER to create and cache a new pointer type, if one isn’t already cached. For example:

>>> b = ctypes.c_ubyte()
>>> ctypes.c_ubyte in ctypes._pointer_type_cache
False
>>> pb = ctypes.pointer(b)
>>> type(pb)._type_
<class 'ctypes.c_ubyte'>
>>> ctypes.c_ubyte in ctypes._pointer_type_cache
True
2 Likes
3

The main issue is for typing here. We have class getitem for ctype types like Array for typing

from ctypes import Array, c_uint

x = Array[c_uint]
print(x, type(x))
>> _ctypes.Array[ctypes.c_uint] <class 'types.GenericAlias'>

The wrapping of pointer as a class makes it finally possible to type hint generic pointers, like Array.

1 Like
4

The base pointer class is ctypes._Pointer. It works similarly to a ctypes.Array subclass, except the latter includes a _length_ in addition to a _type_. For example:

>>> c_int_Array_3 = ctypes.c_int * 3
>>> c_int_Array_3.__bases__
(<class '_ctypes.Array'>,)
>>> c_int_Array_3._type_ is ctypes.c_int
True
>>> c_int_Array_3._length_ == 3
True

We have no operator for creating and caching a ctypes._Pointer subclass. Thus we have the builtin ctypes.POINTER function instead.

1 Like
5

Are you using the latest version of your type checker? The typeshed stubs were updated a few months ago in `ctypes`: `pointer` is a function, not a class by AlexWaygood · Pull Request #8446 · python/typeshed · GitHub, to reflect the fact that at runtime, pointer is actually a function that returns instances of _Pointer. If you’re using the latest version of mypy or pyright, it should be no longer valid to use pointer as a type annotation, since functions can’t be used as type annotations. You’ll need to start using _Pointer instead of pointer in your type hints. So perhaps the question should be: "Should we add __class_getitem__ to ctypes._Pointer?

Having said that, it’s not ideal that users have to access a private class in order to annotate their code. It might be nice to make ctypes._Pointer public.

1 Like
6

One reason that ctypes._Pointer and ctypes._CFuncPtr are private is to discourage manual subclassing that bypasses _pointer_type_cache, _c_functype_cache, and _win_functype_cache, which are used by ctypes.POINTER(), ctypes.CFUNCTYPE(), and ctypes.WINFUNCTYPE(). Since type checking is still based on the Python class, the caches are important in order to avoid creating multiple pointer types for the same _type_, or multiple function pointer types for the same (_restype_, _argtypes_, _flags_). The ctypes.Array type uses the * sequence repeat operator, which has an internal cache.

Possibly the cache support could be relocated to the corresponding metaclasses: PyCPointerType, PyCArrayType, and PyCFuncPtrType. In this case, using the cache would have to be disabled if additional class attributes are defined other than, respectively, ("_type_",), ("_type_", "_length_"), and ("_restype_", "_argtypes_", "_flags_").

2 Likes
7

What is the canonical way of type-hinting ctypes types then? Perhaps it’s just a documentation problem?

If it’s not a (lack of) documentation issue, then I think it’d be good to come up with solutions.

How would you see ctypes type hinting for generic-like concepts, say POINTER(foo) and such? In my own projects, I just monkey patch things to make both mypy and the runtime happy.

The problem is not isolated, there’s this StackOverflow question with no good answer.

POINTER(foo) is a type-generator, for better or worse. The syntax predates typing annotations. As far as I’m concerned, POINTER[foo] should do what POINTER(foo) does and that would solve the problem, wouldn’t it?

The original idea of __class_getitem__ for ctypes.pointer of course should be for ctypes.POINTER instead. pointer is the equivalent of the C & operator - it doesn’t manipulate types. But POINTER(foo) is like C++ POINTER<foo> - a type-maker.

To get my legs wet with getting changes into Python, I volunteer to implement this and get it through the process, including documentation work of course. But there would need to be some discussion first. Perhaps I should start a new thread, with more complete of an idea?