I implemented a decorator as a class with the relevant dunder methods:
def __init__(self, *args: Any, **kwargs: Any) -> None:
def __call__(self, *args: Any, **kwargs: Any) -> Any:
def __get__(self, obj: Any, objtype: Any = None) -> Any:
The debugger shows that the flow through the above is different if the name of the decorated method starts or not with an underscore.
Is the difference in behavior documented anywhere?
Could you please be more specific with what difference you are seeing? Ideally with a small example that uses print to show the different order of operations?
Is this due to name mangling?
I haven’t been lazy investigating this, but I did get stuck in “can’t happen” mode.
I’ll add a trace to demonstrate what I’m talking about.
As it stands, my desired workflow fails for methods with no leading underscores in their name, and it succeeds if the underscore is present. There may be bugs in the decorator, but that doesn’t explain the different outcomes depending on the presence of the leading underscore.
Here’s the code for the decorator, while work on a trace:
class rule:
def __init__(self, *args: Any, **kwargs: Any) -> None:
self.impl: Callable[..., Any] | None = None
self.params: tuple[Any, ...] = ()
self.kwparams: dict[str, Any] = {}
self.ruleinfo: RuleInfo = RuleInfo(
name='<none>',
obj=None,
impl=lambda: None,
is_leftrec=False,
is_memoizable=True,
is_name=False,
params=(),
kwparams={},
)
# If the first argument is a callable and no other args exist,
# it was used as @rule. Otherwise, it was @rule(...).
if len(args) == 1 and callable(args[0]) and not kwargs:
self.impl = args[0]
return
self.params = args
self.kwparams = kwargs
def __call__(self, *args: Any, **kwargs: Any) -> Any:
# If impl is None, we are in the @rule(...) phase
if self.impl is None:
impl: Callable[..., Any] = args[0]
new = rule(*self.params, **self.kwparams)
new.impl = impl
functools.update_wrapper(new, impl)
# Return a new instance with the implementation bound
return new
return self
@functools.wraps(self.impl)
def wrapper(obj: Any = None, ctx: Any = None) -> Any:
return self._run(obj, ctx, args, kwargs)
return wrapper
# Otherwise, this is the actual function call
# return self._run(None, None, args, kwargs)
def __get__(self, obj: Any, objtype: Any = None) -> Any:
if obj is None:
return self
# Return a wrapper that binds 'self' (obj)
if not callable(self.impl):
return self
self.ruleinfo = self._ruleinfo(obj, self.impl)
# return self._run
@functools.wraps(self.impl)
def wrapper(obj: Any, ctx: Any = None) -> Any:
return self._run(obj, ctx, (), {})
return wrapper
def _ruleinfo(self, obj: Any, impl: Callable[..., Any]) -> RuleInfo:
name = impl.__name__ # type: ignore
is_leftrec = getattr(impl, 'is_leftrec', False)
is_memoizable = getattr(impl, 'is_memoizable', True)
is_name = getattr(impl, 'is_name', False)
return RuleInfo(
name=name,
obj=obj,
impl=impl,
is_leftrec=is_leftrec,
is_memoizable=is_memoizable,
is_name=is_name,
params=self.params,
kwparams=self.kwparams,
)
def _run(
self,
obj: Any,
ctx: Any = None,
_args: tuple[Any, ...] | None = None,
_kwargs: dict[str, Any] | None = None,
*args: Any,
**kwargs: Any,
) -> Any:
assert obj, f'{obj=!r} {ctx=!r}'
if isinstance(ctx, ParseCtx):
return ctx._call(self.ruleinfo)
else:
return obj._call(self.ruleinfo) # legacy case
Seems like your RuleInfo class is doing something with the name, is the issue possibly there?
RuleInfo is just a NamedTuple that doesn’t mangle with names.
I validated every occurrence of "_" in the code base, and added the debug traces to the decorator class. This it what I discovered:
The Python compiler will reuse instances of a decorator that is a class.
From what I saw in the traces the reuse probably occurs through something like @cache over decorator instances, so the same instance may get called over different function/methods if the parameters used on the decorator are the same.
Problems that occur:
__init__() stores information on the instance, then those attributes may get reused on subsequent calls to __call__() or __get__().If data in decorator is required by the wrapper, then the data should be copy.copy() to variables local to the function returning the wrapper so they are unique (and not shared) across calls to a reused decorator. If the decorator notices it’s being reduced, then it can return a new instance of itself instead of the final wrapper.
As to the decorator instance reuse, it seems it varies depending on leading underscores in the decorated method/function name, but that was not the problem.
This is the decorator that’s working and handling all cases:
class rule:
def __init__(self, *args: Any, **kwargs: Any) -> None:
# note: we may be new instance created by self.__call__()
self.func: Callable[..., Any] | None = kwargs.pop('func', None)
debug(f'__init__ {id(self)=} {fn(self.func)!r} {args=!r} {kwargs=!r}')
self.params = args
self.kwparams = kwargs
def __call__(self, *args, **kwargs) -> Any:
debug(f'__call__ {id(self)=} {args=!r} {kwargs=!r}')
func = args[0] if args else None
if callable(func):
return rule(*self.params, func=func, **self.kwparams)
else:
# note: this self may be being reused
return rule(*args, **kwargs)
def __get__(self, instance: Any, owner: Any = None) -> Any:
debug(
f'__get__ {id(self)=}'
f' { fn(self.func)=!r} {instance=!r} {owner=!r}'
f' {tn(instance)=!r}'
f' {self.params=!r} {self.kwparams=!r}'
)
owner = owner or type(instance)
func = self.func
params = copy(self.params) or ()
kwparams = copy(self.kwparams) or {}
assert isinstance(func, Callable)
ruleinfo = RuleInfo.new(instance, func, params, kwparams)
if issubclass(owner, ParseContext) and isinstance(instance, Ctx):
# NOTE:
# v5.16 <= parser <= v5.17.1 may use @rule on methods
# defined inside a ParseContext
debug(f'__get__ LEGACY {instance=!r} {owner=!r}')
return self._rules_in_ctx(id(self), ruleinfo)
else:
return self._rules_in_obj(id(self), ruleinfo)
...
???
Whatever effect you are seeing, this is an incorrect way of describing it.
It would really help if you could show a small, self-contained example demonstrating the problem.
The posted code for __call__ doesn’t look right, since it returns the decorator instead of making the call (I’m presuming this was edited for debugging exploration and not reverted before posting).
See the implementation of __call__() in the new version of the decorator.
def __call__(self, *args, **kwargs) -> Any:
debug(f'__call__ {id(self)=} {args=!r} {kwargs=!r}')
func = args[0] if args else None
if callable(func):
return rule(*self.params, func=func, **self.kwparams)
else:
# note: this self may be being reused
return rule(*args, **kwargs)
When __call__ is issued there isn’t enough information about the context to invoke the decorated method correctly.
I’ll add a trace by rule instance to figure out reuse and dunder method call order, just for the sake of this discussion.