PEP 570: Python Positional-Only Parameters

#34

The problem is not in the “normal” common arguments, but in the “exceptional” additional ones that the dual-use enables.

x=[1, 2]; y= [3, 4]; z = [5, 6]; np.add(x, y, z) would be clearer as np.add(x, y, out=z) since 99% of users meant np.add(np.add(x, y), z) but mistakenly thought they could use many operands at once. But all that is water under the bridge the real problem is

any problem that happen now will keep happening with this PEP and without

True, but the PEP would enable and encourage package maintainers to use more advanced syntax out-of-step (in my opinion) with users’ (as I percieve them) needs, complicating the language and raising barriers for new users. As others have said, the docstring for np.add requires quite a bit of mental effort to parse, exactly what is lacking when trying to debug an already confusing error. I have had students look at that dsl and prefer to try various code snippets rather than understand what it means.

#35

The main problem with using / in the help output for builtins which use Argument Clinic is that this syntax is not documented. If it would be documented, there would be less questions (not more than about * and **). I don’t particularly like the / syntax, but no better syntax was proposed for last 7 years.

I am in a great favor of PEP 570, no matter what syntax will be used. Currently we need to write complex and cumbersome code. Two examples:

class MutableMapping(Mapping):
    def update(*args, **kwds):
        if not args:
            raise TypeError("descriptor 'update' of 'MutableMapping' object "
                            "needs an argument")
        self, *args = args
        if len(args) > 1:
            raise TypeError('update expected at most 1 arguments, got %d' %
                            len(args))
        if args:
            other = args[0]
            ...
        ...

class Formatter:
    def format(*args, **kwargs):
        if not args:
            raise TypeError("descriptor 'format' of 'Formatter' object "
                            "needs an argument")
        self, *args = args  # allow the "self" keyword be passed
        try:
            format_string, *args = args # allow the "format_string" keyword be passed
        except ValueError:
            raise TypeError("format() missing 1 required positional "
                            "argument: 'format_string'") from None
        return self.vformat(format_string, args, kwargs)

And help() outputs a useless signature for these functions: (*args, **kwargs). It is even confusing, because it does not contain self as signatures of other methods.

With PEP 570 the above examples can be written as:

class MutableMapping(Mapping):
    def update(self, other=(), / , **kwds):
        ...

class Formatter:
    def format(self, format_string, /, *args, **kwargs):
        return self.vformat(format_string, args, kwargs)

This feature will be useful not only for experts, but it can reduce the confusion of newbies. First, the will no longer be confused by / in signatures of some builtins. Second, they will no longer be confused by nonavailability to pass self as a keyword argument to unbound methods.

3 Likes
#36

It’s not a feature. Readability Counts, and writing len(obj=configurations) is not something we want to encourage.

1 Like
#37

There’s a big difference between encouraging it and changing the grammar in a way that lets someone else decide whether I can or not. We can still say it’s ugly, un-Pythonic, and plenty of other things without having to make it literally impossible.

(Edit - reminder, my position is to fix up the builtins to get rid of / completely)

1 Like
#38

Regarding the argument that the / is hard to teach, there are tons of advanced features that beginners don’t need to be taught. OTOH presumably one of the problems with the current usage in Argument Clinic is that it generally produces function signatures that look just like what you can use with def, and / is the exception here. Things will be more regular once we also support it in Python syntax.

I definitely see the need for this feature, and I haven’t seen a reasonable alternative proposal, so I am strongly in support of PEP 570.

2 Likes
#39

Adding a docstring would help. Or overriding signature to make it appear to the user in one way while it actually implements it differently (I think this one needs a change to work, but it’s better than adding another type of parameter).

The suggestion to let named parameters be provided twice - once in the actual one and once in a **kwargs - could also be interesting, but likely a cause of bugs since it’s currently a useful error. Though maybe we can achieve that with a decorator for the few cases where it’s useful?

#40

Literally the same argument applies to accepting the PEP – you don’t have to use the new feature.

When it comes to who can control how a function can be called, obviously that should be the author of the function, not the caller.

#41

The fact that this is not obviously true in Python has been one of my favourite things about the language for years. Callers can do so much that the callee didn’t plan for, design, or implement. Proper first-class functions are great. Functools.partial is great. Binding via __get__ is great. Two argument iter is great. Compared to the other languages I get to use, Python is such a breathe of freedom in this area.

And all of those things are truly optional for a beginner (in a well-designed library). Even the semantics added by __get__ are natural and help users to avoid issues, despite the implementation complexity.

Introducing an error when an argument is provided by name just doesn’t fit into the category of making functions easier to call.

#42

I didn’t want to bring this up, but in the type system that is slowly evolving (PEP 484 etc.) positional-only parameters are occasionally important.

In particular, we’ve found that people sometimes override a method in a subclass but disregard the parameter name. (Typically when it’s “obvious” to the users of the class that it’s a positional parameter.)

There is debate possible over whether this should be considered a Liskov violation. There are good arguments on both sides: it’s unsound t let this pass, but OTOH we’ve found much code written like this.

If the base class were to declare the argument as positional-only such debate would not be needed.

Why it’s unsound:

class Base:
    def meth(self, arg: int) -> str: ...

class Sub(Base):
    # Is this a valid override or not?
    def meth(self, arg1: int) -> str: ...

def func(x: Base):
    x.meth(arg=12)

func(Sub())  # Runtime error
1 Like
#43

This whole line of reasoning feels like a a strawman argument. There are plenty of constraints, e.g. you can’t pass an argument (by name or position) that’s not expected.

And I have encountered plenty of situations where this was a pain for the API authors – it constrains API evolution in undesirable ways.

#44

I mean, it’s clearly not a valid override in Python, as you demonstrate, so why isn’t that the determining factor for correctness here? “Technically correct but doesn’t work” is a very academic result, so it’s a shame if type checking is heading that way. (Besides, this wouldn’t be the first time that using type checking revealed invalid code that was just never hit before :wink: )

#45

One thing I’d like to bring up with respect to the “adds complexity to the language” question is that this syntax is already in use in the documentation (via argument clinic) and in third parties (numpy, etc), so to the extent that it has to be taught, you already have to teach it as a documentation convention.

In some sense, this PEP makes it easier to teach, because instead of saying, “Sometimes you’ll see a / in the documentation, that means that everything before that is positional-only, but that’s just a documentation convention, the actual implementation would need to be in C or use *args, **kwargs”, you can say, “Just like * specifies everything after is keyword-only, / specifies that everything before it is positional-only.”

Another nice thing about elevating this documentation convention to the level of syntax is that it creates an obvious standard way to indicate that an argument is positional-only. As it stands now, / is the mostly widely-used and simplest, but as the current state of documentation section mentions, other conventions are used as well. Once there’s a standard syntax for this, you’ll likely see less variation in documentation conventions, which again makes it easier for newcomers who won’t have to learn 3-4 different ways of reading function signatures.

3 Likes
#46

Good points Paul. It would be helpful to add user guidance documentation about the options available and when to use. Something in the spirit of Trey Hunner’s post.

3 Likes
#47

Rejecting such code in the type checker was considered – IIRC it was what mypy did initially – but eventually we realized that this would cause a lot of false positives for a purely academic reason. The code was obviously designed to be called positionally, and there were no calls using keywords instead (my example of a runtime error was meant to highlight why technically it has to be considered unsound).

Fixing the code by renaming the argument would be tedious (you have be careful to fix all uses of that argument in the function body) and would render the code less readable – sometimes the base class has a good reason to name a argument x, and the subclass has just as good a reason to name it y. (For example, when subclassing Mapping to implement a DNS lookup cache, you may not want to use the generic argument names ‘key’ and ‘value’ but rather ‘host’ and ‘address’.)

If there had been a way to declare explicit positional arguments (other than the **args hack) this whole argument would have been unnecessary.

FWIW the *args hack is common enough that I really don’t buy the “there’s no use case” feedback. There is a clear and common use case, and so far we’ve only seen two proposals to support it: PEP 570, or some kind of marking for individual arguments, which surely is going to have all the same problems that opponents bring up for PEP 570 (unteachable, ugly, no use case etc.).

#48

Indeed, continuing @storchaka example there is this bug when this was a real problem and positional only arguments are needed:

https://bugs.python.org/issue9137

import collections
x = collections.OrderedDict()
x.update(red=5, blue=6, other=7)
Traceback (most recent call last): 
...
line 490, in update for key, value in other:
TypeError: 'int' object is not iterable

Using the *args hack makes the signature obscure and non informative and you lose the power of tools that automatically generate documentation based on the signature.

1 Like
#49

Amaury Forgeot d’Arc asked an interesting question:

And what about this?
>>> x.update(self=5)

https://bugs.python.org/issue9137#msg109093

Should self be a positional-only parameter? :slight_smile:

I was curious to reproduce the bug. You can reintroduced it in the master branch using the change below.

update() fails with self and other arguments:

$ ./python
Python 3.8.0a3+ (heads/run_main-dirty:0be720f87d, Mar 29 2019, 14:50:44) 
>>> import decimal
>>> import collections.abc
>>> traps = decimal.Context().traps
>>> isinstance(traps, collections.abc.MutableMapping)
True

>>> traps.update(other=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vstinner/prog/python/master/Lib/_collections_abc.py", line 837, in update
    for key, value in other:
TypeError: 'int' object is not iterable

>>> traps.update(self=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: update() got multiple values for argument 'self'

Patch to introduce the MutableMapping.update() bug:

    diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
    index c363987970..aef6352ff0 100644
    --- a/Lib/_collections_abc.py
    +++ b/Lib/_collections_abc.py
    @@ -821,30 +821,21 @@ class MutableMapping(Mapping):
             except KeyError:
                 pass
     
    -    def update(*args, **kwds):
    +    def update(self, other=(), **kwds):
             ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
                 If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
                 If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
                 In either case, this is followed by: for k, v in F.items(): D[k] = v
             '''
    -        if not args:
    -            raise TypeError("descriptor 'update' of 'MutableMapping' object "
    -                            "needs an argument")
    -        self, *args = args
    -        if len(args) > 1:
    -            raise TypeError('update expected at most 1 arguments, got %d' %
    -                            len(args))
    -        if args:
    -            other = args[0]
    -            if isinstance(other, Mapping):
    -                for key in other:
    -                    self[key] = other[key]
    -            elif hasattr(other, "keys"):
    -                for key in other.keys():
    -                    self[key] = other[key]
    -            else:
    -                for key, value in other:
    -                    self[key] = value
    +        if isinstance(other, Mapping):
    +            for key in other:
    +                self[key] = other[key]
    +        elif hasattr(other, "keys"):
    +            for key in other.keys():
    +                self[key] = other[key]
    +        else:
    +            for key, value in other:
    +                self[key] = value
             for key, value in kwds.items():
                 self[key] = value
#50

See https://github.com/python/cpython/pull/12620 for changes that can/should be made after accepting and implementing PEP 570.

The code of functions that use the *args hack will be significantly simplified. But the PR also fixes much more functions that need the *args hack if do not accept PEP 570. Do you want to add this hack in over 60 functions?

3 Likes
#51

That’s an interesting question. I think we can all agree that it should be treated as such (and type checkers do). But I don’t think we will ever need to write

class C:
    def meth(self, /, arg): ...

It just means that if there are positional arguments we needn’t worry that using / affects self.

#52

Hence, my position is that we should stop using them in the documentation :slight_smile:

I don’t think “we dislike / in documentation so we should add more of them” is a compelling argument (though it does seem like most people making this argument weren’t around before the /'s were added).

#53

Many of these changes are breaking, since people may be passing the existing arguments by name. How much more difficult do we want to make it to port to 3.8? (Edit: or whichever version this lands in)