There’s a recurring need in the standard library to look up special methods (like __enter__, __exit__, etc.). Currently, libraries like contextlib implement their own logic using getattr_static.
A recent issue (#144386) discovered a behavioral difference between the with statement and contextlib.ExitStack.enter_context() when context manager methods are defined via __slots__. The fix (#144420) added descriptor handling to contextlib, but as noted in the review, using getattr_static is “heavy machinery” and we need a C-level function exposed at the Python level.
This would also resolve the #62912 where operator.index()'s pure Python lookup doesn’t match the C implementation due to the lack of a proper special method lookup API.
Proposal
Expose _PyObject_LookupSpecialMethod (the function used by the LOAD_SPECIAL bytecode) as a Python function.
Open Questions for Discussion
1. Function naming
What should the function be called? (Current PR (#144990) uses lookup_special_method).
2. Module placement
Which module should the function be placed in? (Current PR puts it in types module).
3. Return value semantics
The _PyObject_LookupSpecialMethod has an optimization: for method descriptors (regular functions), it returns the unbound function rather than a bound method to avoid creating temporary objects. This means the caller must pass the instance as the first argument.
Should the public API:
Option A: Follow the C function exactly (return unbound for functions, call __get__ for other descriptors)
Option B: Always return a bound method (simpler for users, but less efficient and inconsistent with LOAD_SPECIAL)
(Current PR chooses Option A).
4. Error handling
Return None if the special method is not found?
Raise AttributeError?
(Current PR returns None if the special method is not found).
Related Issues
#62912: Issue about operator.index doesn’t match the C version
#144989 Issue about exposing _PyObject_LookupSpecialMethod
#144990: PR to expose _PyObject_LookupSpecialMethod
In most cases we need not _PyObject_LookupSpecialMethod, which is an optimization for C and requires the caller to handle the case when self should be passed explicitly during the call, but _PyObject_LookupSpecial, which just returns a callable.
Yes, it adds some overhead, but handling the special case to avoid that overhead in Python adds more overhead (it is different in C).
So this part should be changed. We may need a function that looks up the name without using the descriptor protocol, but this is a different case.
The function should not return None if the special method is not found, otherwise how can we distinguish this from None assigned to the special name? The C code does not interpret None as “not found”. I think the function should have interface similar to getattr().
What module? I don’t like putting more stuff which is not builting types in the types module. The operator module is other candidate. It already contain the _compare_digest() function which can only be implemented in C. We should accept that the experiment of implementing operator in Python failed.
It does not contain _compare_digest. Only _operator does and this function is not imported inside of operator.
_operator is “misused” to provide a function for hmac.py. This isn’t an argument for exposing this function in operator.py.
Independent of that I think operator is a good candidate. The other alternative is inspect, but that’s expensive to import (although maybe lazy imports fix this?).
_operator is “misused” to provide a function for hmac.py. This isn’t an argument for exposing this function in operator.py.
More precisely, it’s there because we need an unconditional module that is always shipped with the interpreter (we can’t put it in _hashlib because the latter is conditioned to OpenSSL, and other modules may have been too costly to import at that time or even less relevant). HMAC did not have a built-in module either so we couldn’t put it there.
If we need a module where we expose some C functions directly, did we consider a new dedicated module for that? Or an alternative builtin (though it may be too niche…). Maybe a getattr_static builtin could be reasonable? Or offer a ctypes-like safe interface for exposing interpreter-level functions without creating a module just to hold them?
Is returning None here really a smart play? If I am performing a lookup, it could very well result in some attribute being None.
I.e. imagine this:
class C1:
attr = None
lookup(C1, "attr") # Returns `None` (valid)
lookup(C1, "typo") # Returns `None`, so I don't notice the typo unless I change the value of attr
I think it’s pretty simple to see where I’m going with this, but lMO raising an error is the most logical thing here to do (similar to getattr and getattr_static).
Imo a _capi module could work well here, although it would be hard to argue in favor of one for non C implementations. In that case, maybe a _pyapi module would be more fitting.
The workaround for private implementation APIs is off-topic, since we’re talking about a new public API.
I’m personally OK with using the types module, as that has long been about exposing the type system machinery in addition to exposing specific implementation types. The main alternative I can see would be a static method on type, but I don’t see any compelling advantage to that over a module level function.
For the technical details:
I agree with Serhiy that the subtle C optimisation of special method lookup should not be exposed, since the convenience/speed trade-off is worth it in C, but would be far more dubious in a public Python API where special casing returned functions involves a much more expensive type check than the optimised check that C consumers can perform.
I also agree we should raise an exception for failed lookups
Quick update: modified the PR based on the consensus reached by now:
Switched to _PyObject_LookupSpecial instead of _PyObject_LookupSpecialMethod and accordingly renamed (for now) the function from lookup_special_method to lookup_special.
Raise AttributeError if the special method is not found.
Add optional default parameter similar to getattr.
I’ve looked at the implementation in types.py a bit, and IMO caling the last argument args is missleading. Apart from that, using *args allowes for users to do lookup_special(obj, "doesnt_exist", 1, 2, 3) which is a bit wired considering that we should only allow one single default argument. Perhaps we could change the signature of lookup_special to def lookup_special(object, name, default=__sentinel): ... and filtering out __sentinel lateron. Sure in the C code its called *args, similar to builtin_getattr’s definition in bltinmodule.c, but IMO, calling it default (like in the stubs) would make more sense.
I could quickly make a PR to your branch, so you can see what i mean in more detail.
with a , / missing. Both C code and docs specify that the parameters are positional only. Following the spirit of PEP 399, the signatures and actually also the docstrings of corresponding C and Python functions should agree.