@extends / @copy_signature to add arguments to an overriden method of a derived class

Typing
1

It has been a while since I started looking for a way to implement this. I have already come across several discussions and ideas about it, with the most relevant/recent one being in this topic:

The topic clearly explains what we are trying to implement, but I would like to expand upon the example given there. I believe that the @dataclass decorator already solves the example concerning the __init__ function. My proposal is a decorator that functions as follows:

class A:
    def fn(self, param1: int, param2: str):
        """
        :param param1: Documentation for param1
        :param param2: Documentation for param2
        """
        # do something

class B(A):
    @extends(A.fn)  # or only @extends
    def fn(self, param3: int,  *args, **kwargs):
        """
        :param param3: Documentation for param3
        """
        super().fn(*args, **kwargs)  # args/kwargs should contain param1 and param2

The fn method in B expects to receive exactly the same arguments as fn in A, plus the specific arguments added in B. This would even be a generalization of what is discussed on this topic:

The @copy_signature that is discussed there would only be the case when no extra arguments are added.

Currently, we can implement this to work in runtime environments, like Jupyter Notebooks, by manipulating the signature of each function. However, this approach does not generalize well to static type checkers in IDEs.

As I mentioned, it seems to me that the @dataclass decorator has successfully implemented this behavior. For example:

@dataclass
class A:
    param1: int = 1
    param2: float = 0.4

@dataclass
class B(A):
    param3: int = 2

An instance of class B will have self.param1, self.param2, and self.param3 as attributes, and when instantiating it in an IDE, it will correctly show that it takes param1, param2, and param3 to be instantiated. Unfortunately, it appears that the documentation is still not fully functional, but avoiding the need to copy/paste some parameters is already a significant improvement!

So, as of today, do we have any workaround for this pattern? Do you think we could implement something similar to the @dataclass decorator?

2

This doesn’t look like it is a valid subclass implementation: foo.fn(param1, param2) must be a valid call by Liskov substitutability, but it is not. param3 would need a default.

As a user, I would also expect the new parameter to come last, not first, i.e. for foo.fn(param1, param2, param3) to work. Is the decorator intended to make this work, i.e. reorder the unnamed input arguments, and work out how they line up with any keyword arguments?

1 Like
3

Yes, you are right, I think I got a little bit ahead of myself trying to generalize too much. I think that this pattern mostly emerges when we are exclusively extending method with **kwargs.

ActualIy, I can think of two distinct cases:

  • adding some functionality with optional **kwargs to a parent method
  • forwarding all the required_parameters / args / kwargs (or an entire tuple/dict) from a function to another

I think that both cases are fairly common and could be addressed by some kind of decorator, but you are right that we should be more careful when using it to extend a method.

An example of the first case that I think should be addressed is when we have something like:

class A:
    def fn(self, param1: int, param2: str):
        """
        :param param1: Documentation for param1
        :param param2: Documentation for param2
        """
        # do something

class B(A):
    def fn(self, param1: int, param2: str, param3: int=2):
        """'
        :param param1: Documentation for param1 (that is the same as before)
        :param param2: Documentation for param2 (that is the same as before)
        :param param3: Documentation for param3 (new)
        """
        # do something and possibly call super().fn(param1, param2), but not necessarily

which with this decorator would be refactored to

class B(A):
    @extends(A.fn)
    def fn(self, *args, param3: int=2):
        """'
        :param param3: Documentation for param3 (new)
        """
        # do something and possibly call super().fn(*args), but not necessarily

This decorator that I have in mind should be able to tell the function being extended that the *args are the parameters from A.fn. Sure, there are still some cases we have to consider, like, how do we distinguish between the required parameters, *args and **kwargs from A.fn? Maybe, by telling explicitly to the decorator which one is which? Maybe something like

`@extends(A.fn, required_args, args, kwargs)`
def fn(required_args, args, kwargs, param3: int=2):

While I am still unsure, I believe it is crucial to avoid the copy-and-paste scenario. This would allow one to modify A.fn (or have the function modified when we are not responsible for the code base where A lives) without the need to modify B.fn by adding/removing the new parameters from A.fn.

The second example that I have mentioned is when we have some function that is forwarding all the kwargs or an entire dict to another function, which is also extremely common, for example when using matplotlib:

import matplotlib.pyplot as plt

def fn(x, y, **kwargs):
    # do something
    plt.plot(x, y, **kwargs)

Here we usually do not have the documentation for the **kwargs part, we usually have to tell the user to refer to the documentation of the function that is using the **kwargs. The decorator would be able to tell fn that the kwargs used there are the kwargs from plt.plot, for example by writing:

@extends(plt.plot, kwargs=kwargs)
def fn(x, y, kwargs):

Although these are two distinct usage cases, it seems to me that both could be addressed by the same decorator which is responsible for telling the new function that it is extending another function that has required_args, args and kwargs that are the same being used in this new function.

4

This also breaks LSP. It’s no longer valid to pass the passed through args by name.

For kwargs at least you can type it as (**kwargs: Unpack[SomeTypedDict]) and reuse the typed dict.

5

We can do

@extends(A.fn)
def fn(self, *args, param3: int=2, **kwargs):
        """'
        :param param3: Documentation for param3 (new)
        """

and what the decorator does is tell the function that the *args and **kwargs of the new function are the *args and **kwargs of the extended function. We will not be able to use new *args and **kwargs for the new function, but I think that still covers the pattern. The original function can have its args and kwargs, and we are just extending it with some functionality. We may need to always use *args and **kwargs when using the decorator, but I do not think that this is too much of a trouble.