API for special attribute lookup

Core Development C API
1

Special attribute lookup is something I’ve been thinking about, and collecting notes, off and on for a while.
Now Victor proposed to make _PyType_Lookup public, which I think is something of a special case, so let me turn my notes into a proposal: let’s add a function called PyObject_LookupAttribute.

int64_t PyObject_LookupAttribute(
    PyObject *obj,
    PyObject *attr_name,
    int64_t flags,
    PyObject **result,
    PyObject **defining_class);

Get the attribute named attr_name from obj.

If the attribute is not found, set *result to NULL and return 0 (without an exception set).
If the attribute is found, set *result and return output flags (a positive number).
On failure, set *result to NULL and return a negative value with exception set. Note that the bits of the result are meaningful; the function does not return -1.

*defining_class is set to NULL unless flags contains Py_ATTRIBUTELOOKUP_DEFINING_CLASS, see below.

Flags are common for the flags argument (“input”) and the return value (“output”):

Py_ATTRIBUTELOOKUP_FOUND:

  • input: Ignored.
  • output: Set if the attribute was found, even if getting its value (for example a descriptor call) raised an exception.

Py_ATTRIBUTELOOKUP_SPECIAL:

  • input: Do not look in the instance. Python does this when looking up special methods.
  • output: Always set if requested.

Py_ATTRIBUTELOOKUP_DEFINING_CLASS:

  • input: Set *defining_class to a new reference of the class that the attribute was found on, or NULL if such a class wasn’t found.
  • output: Set if *defining_class was set. (Like _FOUND, this may be set on error; even in that case the caller owns the new reference to *defining_class.)

Py_ATTRIBUTELOOKUP_DESCRIPTOR:

  • input: Do not call descriptors; return them directly.
  • output: Set if *result is a descriptor that would be called if the flag wasn’t set.

Py_ATTRIBUTELOOKUP_METHOD:

  • input: Return unbound methods with Py_TPFLAGS_METHOD_DESCRIPTOR.
  • output: Set if *result is such a method that would be bound if the flag wasn’t set.

Py_ATTRIBUTELOOKUP_ERROR (the sign bit):

  • input: Must not be set.
  • output: An error occurred; an exception has been set.
2

How would this work for classes whose’s tp_getattr doesn’t defer to PyObject_GenericGetAttr at some point?

The Py_ATTRIBUTELOOKUP_DEFINING_CLASS of your proposed function seems to assume tp_getattro is implemented using PyObject_GenericGetAttr (I’d have a similar issue with exposing _PyType_Lookup.

1 Like
3

I’m not sure about this API, it looks complex to use. I may prefer specialized functions for each task, like _PyType_Lookup() ( Py_ATTRIBUTELOOKUP_SPECIAL) and lookup_method() ( Py_ATTRIBUTELOOKUP_METHOD) which have a simpler API.

What is the use case for Py_ATTRIBUTELOOKUP_DEFINING_CLASS?

Py_ATTRIBUTELOOKUP_SPECIAL: I ran a code search for _PyObject_LookupSpecial() in PyPI top 15,000 projects and I didn’t find any user. But I’m not sure if _PyObject_LookupSpecial() has the same use case: or is it more like _PyType_Lookup()? Cython implements its own __Pyx__PyObject_LookupSpecial() using _PyType_Lookup() and Py_TYPE(res)->tp_descr_get.

4

I ran a code search for _PyObject_LookupSpecial() in PyPI top 15,000 projects and I didn’t find any user.

NumPy has its own versions, although we should maybe just use the internal Python one instead for speed…
We do have both LookupSpecial (type) and LookupSpecial_OnInstance, although I would probably have a type lookup if it wasn’t for backcompat for some of these (although I know it does surprise some).

I think I tend to agree that these flags feel too complicated. One small detail I wonder about is whether defining_class can’t be borrowed (as obj keeps it alive), maybe I don’t have enough sensitivity for avoiding borrowed references yet, but returning a valid one on error seems also surprising.

Not sure where I am heading, so a bit random maybe:

  • For type lookups, the PyType_Lookup could pass out defining_class (optionally, especially if not borrowed). I am not sure it is useful, but it also seems harmless if Petr thinks it’s useful.
  • For non-type lookups, I meander in two directions for a distinct function:
    1. One that uses defining_type != NULL to figure out if it is unbound to avoid flags (mostly?) EDIT: Ahhh… defining_type also make sense for an “bound”/instance attribute, although I am not sure when it might be valuable information
    2. Just int PyObject_LookupOnIntanceOnly(obj, attr, **out). I.e. if I care to allow both bound and unbound, I would have to just first do an instance only lookup and then do a type lookup if I don’t find it. (Since I tend to be happy to nudge towards type-only, that seems perfectly fine to me. Who knows… maybe someone will even use it because they know they don’t care to even check the types for something.)
      (And now that I wrote that, my initial feeling is that this is a bit funky, but maybe quite nice, unless I am missing how some __getattr__ details work?)
5

Rather obviously, I haven’t tried to implement this – just wrote down what I’d like at the API level.
Thanks for your concerns :slight_smile:

Good point. Those classes can’t honor DESCRIPTOR/METHOD/DEFINING_CLASS requests.

For calling unbound methods with METH_METHOD, for example.

It looks like one resolves descriptors and one doesn’t. Can you tell which one is which? And can you tell if projects are using the correct one for their needs? :‍)

IMO, lookup is not a good name for public API, especially if we need to add more variants of the operation.
GetAttr is clear enough – you get a single value that you’d get in Python – but for other variations the name should probably be more descriptive. Especially if we need to add several of the variants, either now or later.

You can change types at runtime, so even Py_TYPE(obj) isn’t sound. Too late to change Py_TYPE, but let’s not add new ones.