PEP 570: Python Positional-Only Parameters

PEPs
41

The fact that this is not obviously true in Python has been one of my favourite things about the language for years. Callers can do so much that the callee didn’t plan for, design, or implement. Proper first-class functions are great. Functools.partial is great. Binding via __get__ is great. Two argument iter is great. Compared to the other languages I get to use, Python is such a breathe of freedom in this area.

And all of those things are truly optional for a beginner (in a well-designed library). Even the semantics added by __get__ are natural and help users to avoid issues, despite the implementation complexity.

Introducing an error when an argument is provided by name just doesn’t fit into the category of making functions easier to call.

42

I didn’t want to bring this up, but in the type system that is slowly evolving (PEP 484 etc.) positional-only parameters are occasionally important.

In particular, we’ve found that people sometimes override a method in a subclass but disregard the parameter name. (Typically when it’s “obvious” to the users of the class that it’s a positional parameter.)

There is debate possible over whether this should be considered a Liskov violation. There are good arguments on both sides: it’s unsound t let this pass, but OTOH we’ve found much code written like this.

If the base class were to declare the argument as positional-only such debate would not be needed.

Why it’s unsound:

class Base:
    def meth(self, arg: int) -> str: ...

class Sub(Base):
    # Is this a valid override or not?
    def meth(self, arg1: int) -> str: ...

def func(x: Base):
    x.meth(arg=12)

func(Sub())  # Runtime error
1 Like
43

This whole line of reasoning feels like a a strawman argument. There are plenty of constraints, e.g. you can’t pass an argument (by name or position) that’s not expected.

And I have encountered plenty of situations where this was a pain for the API authors – it constrains API evolution in undesirable ways.

44

I mean, it’s clearly not a valid override in Python, as you demonstrate, so why isn’t that the determining factor for correctness here? “Technically correct but doesn’t work” is a very academic result, so it’s a shame if type checking is heading that way. (Besides, this wouldn’t be the first time that using type checking revealed invalid code that was just never hit before :wink: )

45

One thing I’d like to bring up with respect to the “adds complexity to the language” question is that this syntax is already in use in the documentation (via argument clinic) and in third parties (numpy, etc), so to the extent that it has to be taught, you already have to teach it as a documentation convention.

In some sense, this PEP makes it easier to teach, because instead of saying, “Sometimes you’ll see a / in the documentation, that means that everything before that is positional-only, but that’s just a documentation convention, the actual implementation would need to be in C or use *args, **kwargs”, you can say, “Just like * specifies everything after is keyword-only, / specifies that everything before it is positional-only.”

Another nice thing about elevating this documentation convention to the level of syntax is that it creates an obvious standard way to indicate that an argument is positional-only. As it stands now, / is the mostly widely-used and simplest, but as the current state of documentation section mentions, other conventions are used as well. Once there’s a standard syntax for this, you’ll likely see less variation in documentation conventions, which again makes it easier for newcomers who won’t have to learn 3-4 different ways of reading function signatures.

3 Likes
46

Good points Paul. It would be helpful to add user guidance documentation about the options available and when to use. Something in the spirit of Trey Hunner’s post.

3 Likes
47

Rejecting such code in the type checker was considered – IIRC it was what mypy did initially – but eventually we realized that this would cause a lot of false positives for a purely academic reason. The code was obviously designed to be called positionally, and there were no calls using keywords instead (my example of a runtime error was meant to highlight why technically it has to be considered unsound).

Fixing the code by renaming the argument would be tedious (you have be careful to fix all uses of that argument in the function body) and would render the code less readable – sometimes the base class has a good reason to name a argument x, and the subclass has just as good a reason to name it y. (For example, when subclassing Mapping to implement a DNS lookup cache, you may not want to use the generic argument names ‘key’ and ‘value’ but rather ‘host’ and ‘address’.)

If there had been a way to declare explicit positional arguments (other than the **args hack) this whole argument would have been unnecessary.

FWIW the *args hack is common enough that I really don’t buy the “there’s no use case” feedback. There is a clear and common use case, and so far we’ve only seen two proposals to support it: PEP 570, or some kind of marking for individual arguments, which surely is going to have all the same problems that opponents bring up for PEP 570 (unteachable, ugly, no use case etc.).

48

Indeed, continuing @storchaka example there is this bug when this was a real problem and positional only arguments are needed:

https://bugs.python.org/issue9137

import collections
x = collections.OrderedDict()
x.update(red=5, blue=6, other=7)
Traceback (most recent call last): 
...
line 490, in update for key, value in other:
TypeError: 'int' object is not iterable

Using the *args hack makes the signature obscure and non informative and you lose the power of tools that automatically generate documentation based on the signature.

1 Like
49

Amaury Forgeot d’Arc asked an interesting question:

And what about this?
>>> x.update(self=5)

https://bugs.python.org/issue9137#msg109093

Should self be a positional-only parameter? :slight_smile:

I was curious to reproduce the bug. You can reintroduced it in the master branch using the change below.

update() fails with self and other arguments:

$ ./python
Python 3.8.0a3+ (heads/run_main-dirty:0be720f87d, Mar 29 2019, 14:50:44) 
>>> import decimal
>>> import collections.abc
>>> traps = decimal.Context().traps
>>> isinstance(traps, collections.abc.MutableMapping)
True

>>> traps.update(other=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vstinner/prog/python/master/Lib/_collections_abc.py", line 837, in update
    for key, value in other:
TypeError: 'int' object is not iterable

>>> traps.update(self=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: update() got multiple values for argument 'self'

Patch to introduce the MutableMapping.update() bug:

    diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
    index c363987970..aef6352ff0 100644
    --- a/Lib/_collections_abc.py
    +++ b/Lib/_collections_abc.py
    @@ -821,30 +821,21 @@ class MutableMapping(Mapping):
             except KeyError:
                 pass
     
    -    def update(*args, **kwds):
    +    def update(self, other=(), **kwds):
             ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
                 If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
                 If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
                 In either case, this is followed by: for k, v in F.items(): D[k] = v
             '''
    -        if not args:
    -            raise TypeError("descriptor 'update' of 'MutableMapping' object "
    -                            "needs an argument")
    -        self, *args = args
    -        if len(args) > 1:
    -            raise TypeError('update expected at most 1 arguments, got %d' %
    -                            len(args))
    -        if args:
    -            other = args[0]
    -            if isinstance(other, Mapping):
    -                for key in other:
    -                    self[key] = other[key]
    -            elif hasattr(other, "keys"):
    -                for key in other.keys():
    -                    self[key] = other[key]
    -            else:
    -                for key, value in other:
    -                    self[key] = value
    +        if isinstance(other, Mapping):
    +            for key in other:
    +                self[key] = other[key]
    +        elif hasattr(other, "keys"):
    +            for key in other.keys():
    +                self[key] = other[key]
    +        else:
    +            for key, value in other:
    +                self[key] = value
             for key, value in kwds.items():
                 self[key] = value
50

See https://github.com/python/cpython/pull/12620 for changes that can/should be made after accepting and implementing PEP 570.

The code of functions that use the *args hack will be significantly simplified. But the PR also fixes much more functions that need the *args hack if do not accept PEP 570. Do you want to add this hack in over 60 functions?

3 Likes
51

That’s an interesting question. I think we can all agree that it should be treated as such (and type checkers do). But I don’t think we will ever need to write

class C:
    def meth(self, /, arg): ...

It just means that if there are positional arguments we needn’t worry that using / affects self.

52

Hence, my position is that we should stop using them in the documentation :slight_smile:

I don’t think “we dislike / in documentation so we should add more of them” is a compelling argument (though it does seem like most people making this argument weren’t around before the /'s were added).

53

Many of these changes are breaking, since people may be passing the existing arguments by name. How much more difficult do we want to make it to port to 3.8? (Edit: or whichever version this lands in)

54

I think it’s time to take a break with this discussion. Nobody is going to get anyone to change their mind by adding more “refutations” like what’s developing here.

What I would like to see at this point is someone volunteering to write up end user docs along the lines of what Carol suggested:

55

See https://github.com/python/cpython/pull/12620 for changes that
can/should be made after accepting and implementing PEP 570.

I don’t think adding ‘/’ to function definitions such as these
adds readability to the stdlib:

  • def runcall(self, func, *args, **kw):
  • def runcall(self, func, /, *args, **kw):

IMO, the number of cases Serhiy found is actually more a hint at
perhaps creating a decorator specifically for this use case,
rather than adding new syntax.

1 Like
56

This is not about adding readability, but about fixing bugs. Do you think that

  • def runcall(*args, **kw):

is more readable than

  • def runcall(self, func, /, *args, **kw):

?

Many of these changes are breaking, since people may be passing the existing arguments by name.

Most are not. To make the transition more smooth we must to add temporary even more complex code, similar to the code in UserDict.__init__(), in some functions.

57

Not sure if this is what you mean but if accepted we can indeed add a page with the different kind of arguments and when to use them in the python docs. It is a fair point :).

58

It would really help if someone had a draft ready.

59

Thanks for writing that PR! Definitely interesting to see the potential impact of this PEP on preexisting code.

To summarize for myself what I’m hearing, the objections are that it’s surprising/non-obvious or that it’s unnecessary because the need isn’t great enough to warrant coming up with a solution to avoid the *args hacks and we can just add named parameters to all the C code to avoid the need for this.

The support seems to be that it’s non-obvious because it’s been a documentation convention and not syntax so people are not prioritizing teaching it and that the *args hack and pre-existing C code is expansive enough that it warrants its own syntax.

Personally, I fall in the support camp. Take keyword-only arguments. I’m not sure if that syntax is obvious to anyone who isn’t told how it operates, but once you explain it then you understand it and you move on. I also think it’s an instance of Python putting the API designer in charge and not the API user, otherwise promoting the use of keyword-only parameters for things where usage of literals would be ambiguous when provided positionally wouldn’t by considered a motivator for keyword-only parameters.

So with me thinking that Python is about putting API designers first, that leads me to ask whether there is enough *args hacks and such to warrant the syntax. Serhiy’s PR tipped me over as I wasn’t expecting that many instances in the stdlib. Plus I don’t think we are about to say “hey C extensions, update all your code” in order to do away with the need (their performance desires are probably too great, but the fact we have never done it in the stdlib also suggests ). And without this it makes Python shims inconsistent (e.g. I suspect PyPy wouldn’t mind this feature).

2 Likes
60

We can prepare a draft for said document. One question: are you referring to a document explaining when and how to use all possible kind of parameters or just positional-only?