Make _PyObject_LookupAttr public

Core Development
1

In Speed up and clean up getting optional attributes in C code · Issue #76752 · python/cpython · GitHub I added private C API _PyObject_LookupAttr as a convenient function for getting optional attribute which combines PyObject_GetAttr(), PyErr_ExceptionMatches(PyExc_AttributeError) and PyErr_Clear().

int _PyObject_LookupAttr(PyObject *obj, PyObject *attr_name, PyObject **result);

It returns one of three values, 0, -1 or 1 and stores the result or NULL in *result.

  • Return 1 and set *result != NULL if an attribute is found.
  • Return 0 and set *result == NULL if an attribute is not found;
    an AttributeError is silenced.
  • Return -1 and set *result == NULL if an error other than AttributeError
    is raised.

It allows to replace the following code

PyObject *result = PyObject_GetAttr(obj, attr_name);
if (result == NULL) {
    if (PyErr_ExceptionMatches(PyExc_AttributeError)) {
        PyErr_Clear();
        // handle "no such attribute" case
    }
    else {
        // handle other errors
    }
}
else {
    // handle "has attribute" case
}

as

PyObject *result;
if (_PyObject_LookupAttr(obj, attr_name, &result) < 0) {
    // handle other errors
}
else if (result == NULL) {
    // handle "no such attribute" case
}
else {
    // handle "has attribute" case
}

Advantages of using this function:

  • It simplifies the code, especially if “no such attribute” and “has attribute” cases are handles the same.
  • Due to this, it allowed to fix incorrect code which ignored other errors or treated them the same as AttributeError.
  • It avoids raising and catching the AttributeError exception at first place, if the attribute is looked up in object’s __dict__.

Currently this function is used around 350 times in the CPython code. I think that it can be useful for use in third-party code too, because it makes easier to write correct code. While the function can be re-implemented using other public C API (PyObject_GetAttr(), PyErr_ExceptionMatches(), PyExc_AttributeError and PyErr_Clear()), it will not provide the performance gain.

Now, I have a question about the name. I chose different verb to differentiate from PyObject_GetAttr, but I am not sure that it is the best option. I considered other variants with Find, Search, GetOpt, TryGet, etc. Lookup looked good to me, but it would conflict with _PyType_Lookup() and _PyObject_LookupSpecialId() if we made them public too. Whatever name we choose, we should perhaps use the same verb in names of other similar functions (getting optional item in mapping [2] and specifically dict [3]).

Do you support adding such function to the public C API? What is the best name fot it and for similar functions?

[1] Speed up and clean up getting optional attributes in C code · Issue #76752 · python/cpython · GitHub
[2] Add _PyMapping_LookupItem() · Issue #106307 · python/cpython · GitHub
[3] C API: Add PyDict_GetItemRef() function · Issue #106004 · python/cpython · GitHub

6 Likes
2

About the interface, I modeled it (returning a tri-state code for “success”, “not found” and “error” and storing the result by the provided address) by _PyObject_GetMethod() and some static functions. PyIter_Send() added later was modeled by the same scheme.

It allows to avoid using PyErr_Occurred() which adds some overhead to differentiate between “not found” and “error” cases.

3

Maybe call it PyObject_GetOptionalAttr?

This feels similar to getattr(obj, “attr”, None) in pure Python, except with a unique singleton default. There is no special verb there, so maybe we should consider it as a modifier here too.

2 Likes
4

I’m in favor of making the API public, I like the proposed API (error cases, ignoring AttributeError, etc.). I have no preference for the method name.

4 Likes
5

Would not it be confusing that its interface is so different from other Get functions which returns PyObject*. _PyObject_LookupAttr replaced _PyObject_GetAttrWithoutError which was the same as PyObject_GetAttr, but did not raise AttributeError. The users can expect this from PyObject_GetOptionalAttr. I searched for a new verb to emphasize the fact that it not only returns an optional value, but has a completely different interface.

If PyObject_GetOptionalAttr does not cause confusion, would it be good to name other function PyMapping_GetOptionalItem?

And what about the name for a new dict-specific function? Victor initially proposed name PyDict_GetItemRef, but it does not look descriptive to me (suffix Ref has very different meaning in other names). PyDict_GetOptionalItem would be in line with PyMapping_GetOptionalItem, but existing PyDict_GetItemWithError already returns an optional item, the only difference is that it returns a borrowed reference.

6

Just a :+1:, this would be useful both for speed and probably also small simplification in NumPy. I somewhat like the Optional suggestion in the name, the different signatures are a bit of a downer, but to me they seem different enough in practice to be obvious when reading code (and an obvious error when writing).

One similar function is: PyContextVar_Get which uses Get. A difference is that it also has the *default_value as input.

In NumPy practically every function call has to go through getattr(obj, "special_attr", NULL) in C; I only see such dispatching schemes to increase in the future.
We do skip this for most builtin Python types and our own ones, so it isn’t usually a performance hit in practice. But there are examples where it is a significant overhead (I know one example where that overhead contributed to downstream moving to lower overhead but subtly incorrect functions instead – although NumPy should provide new API to avoid that also).

(I am a little curious if there might be a way to allow these fast-paths for custom getattr’s. But we don’t use them for the moment, even if I can see that changing at some point.)

1 Like
7

Maybe I am overthinking this. Here is an issue:

8

While ironically previous decade meme worthy… We could adopt a convention of appending Maybe to such API names going forward: PyObject_GetAttrMaybe

Such a name does imply to the reader that the result needs checking before use.