PEP 570: Python Positional-Only Parameters

Indeed, continuing @storchaka example there is this bug when this was a real problem and positional only arguments are needed:

https://bugs.python.org/issue9137

import collections
x = collections.OrderedDict()
x.update(red=5, blue=6, other=7)
Traceback (most recent call last): 
...
line 490, in update for key, value in other:
TypeError: 'int' object is not iterable

Using the *args hack makes the signature obscure and non informative and you lose the power of tools that automatically generate documentation based on the signature.

1 Like

Amaury Forgeot d’Arc asked an interesting question:

And what about this?
>>> x.update(self=5)

https://bugs.python.org/issue9137#msg109093

Should self be a positional-only parameter? :slight_smile:

I was curious to reproduce the bug. You can reintroduced it in the master branch using the change below.

update() fails with self and other arguments:

$ ./python
Python 3.8.0a3+ (heads/run_main-dirty:0be720f87d, Mar 29 2019, 14:50:44) 
>>> import decimal
>>> import collections.abc
>>> traps = decimal.Context().traps
>>> isinstance(traps, collections.abc.MutableMapping)
True

>>> traps.update(other=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vstinner/prog/python/master/Lib/_collections_abc.py", line 837, in update
    for key, value in other:
TypeError: 'int' object is not iterable

>>> traps.update(self=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: update() got multiple values for argument 'self'

Patch to introduce the MutableMapping.update() bug:

    diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
    index c363987970..aef6352ff0 100644
    --- a/Lib/_collections_abc.py
    +++ b/Lib/_collections_abc.py
    @@ -821,30 +821,21 @@ class MutableMapping(Mapping):
             except KeyError:
                 pass
     
    -    def update(*args, **kwds):
    +    def update(self, other=(), **kwds):
             ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
                 If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
                 If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
                 In either case, this is followed by: for k, v in F.items(): D[k] = v
             '''
    -        if not args:
    -            raise TypeError("descriptor 'update' of 'MutableMapping' object "
    -                            "needs an argument")
    -        self, *args = args
    -        if len(args) > 1:
    -            raise TypeError('update expected at most 1 arguments, got %d' %
    -                            len(args))
    -        if args:
    -            other = args[0]
    -            if isinstance(other, Mapping):
    -                for key in other:
    -                    self[key] = other[key]
    -            elif hasattr(other, "keys"):
    -                for key in other.keys():
    -                    self[key] = other[key]
    -            else:
    -                for key, value in other:
    -                    self[key] = value
    +        if isinstance(other, Mapping):
    +            for key in other:
    +                self[key] = other[key]
    +        elif hasattr(other, "keys"):
    +            for key in other.keys():
    +                self[key] = other[key]
    +        else:
    +            for key, value in other:
    +                self[key] = value
             for key, value in kwds.items():
                 self[key] = value

See https://github.com/python/cpython/pull/12620 for changes that can/should be made after accepting and implementing PEP 570.

The code of functions that use the *args hack will be significantly simplified. But the PR also fixes much more functions that need the *args hack if do not accept PEP 570. Do you want to add this hack in over 60 functions?

3 Likes

That’s an interesting question. I think we can all agree that it should be treated as such (and type checkers do). But I don’t think we will ever need to write

class C:
    def meth(self, /, arg): ...

It just means that if there are positional arguments we needn’t worry that using / affects self.

Hence, my position is that we should stop using them in the documentation :slight_smile:

I don’t think “we dislike / in documentation so we should add more of them” is a compelling argument (though it does seem like most people making this argument weren’t around before the /'s were added).

Many of these changes are breaking, since people may be passing the existing arguments by name. How much more difficult do we want to make it to port to 3.8? (Edit: or whichever version this lands in)

I think it’s time to take a break with this discussion. Nobody is going to get anyone to change their mind by adding more “refutations” like what’s developing here.

What I would like to see at this point is someone volunteering to write up end user docs along the lines of what Carol suggested:

See https://github.com/python/cpython/pull/12620 for changes that
can/should be made after accepting and implementing PEP 570.

I don’t think adding ‘/’ to function definitions such as these
adds readability to the stdlib:

  • def runcall(self, func, *args, **kw):
  • def runcall(self, func, /, *args, **kw):

IMO, the number of cases Serhiy found is actually more a hint at
perhaps creating a decorator specifically for this use case,
rather than adding new syntax.

1 Like

This is not about adding readability, but about fixing bugs. Do you think that

  • def runcall(*args, **kw):

is more readable than

  • def runcall(self, func, /, *args, **kw):

?

Many of these changes are breaking, since people may be passing the existing arguments by name.

Most are not. To make the transition more smooth we must to add temporary even more complex code, similar to the code in UserDict.__init__(), in some functions.

Not sure if this is what you mean but if accepted we can indeed add a page with the different kind of arguments and when to use them in the python docs. It is a fair point :).

It would really help if someone had a draft ready.

Thanks for writing that PR! Definitely interesting to see the potential impact of this PEP on preexisting code.

To summarize for myself what I’m hearing, the objections are that it’s surprising/non-obvious or that it’s unnecessary because the need isn’t great enough to warrant coming up with a solution to avoid the *args hacks and we can just add named parameters to all the C code to avoid the need for this.

The support seems to be that it’s non-obvious because it’s been a documentation convention and not syntax so people are not prioritizing teaching it and that the *args hack and pre-existing C code is expansive enough that it warrants its own syntax.

Personally, I fall in the support camp. Take keyword-only arguments. I’m not sure if that syntax is obvious to anyone who isn’t told how it operates, but once you explain it then you understand it and you move on. I also think it’s an instance of Python putting the API designer in charge and not the API user, otherwise promoting the use of keyword-only parameters for things where usage of literals would be ambiguous when provided positionally wouldn’t by considered a motivator for keyword-only parameters.

So with me thinking that Python is about putting API designers first, that leads me to ask whether there is enough *args hacks and such to warrant the syntax. Serhiy’s PR tipped me over as I wasn’t expecting that many instances in the stdlib. Plus I don’t think we are about to say “hey C extensions, update all your code” in order to do away with the need (their performance desires are probably too great, but the fact we have never done it in the stdlib also suggests ). And without this it makes Python shims inconsistent (e.g. I suspect PyPy wouldn’t mind this feature).

2 Likes

We can prepare a draft for said document. One question: are you referring to a document explaining when and how to use all possible kind of parameters or just positional-only?

I think Carol suggested that we should augment the docs with guidance about all parameter options available (read the linked post).

@pablogsal One request for the PEP: add a “How to teach this” section (per the template in PEP 12 or the list of suggested sections in PEP 1). Doesn’t have to be a lot of text. Perhaps a suggested change to the Tutorial in the docs? Surely there’s a place there where keyword args are discussed, that would be a good place to discuss positional args as well, leading with a realistic example.

2 Likes

-1 from me as well. This is not an “advanced feature”, this is just a distraction.

2 Likes

I definitely think, “We shouldn’t have positional-only arguments at all” is a valid position to take here, though as a library maintainer the arguments in the PEP are pretty compelling.

What I don’t think is a very strong position is continuing to allow positional-only arguments via *args and in the C API, but maintaining the asymmetry between the C API and Python. If you take as a given that Python will continue to support positional-only arguments through other means, then I think my point stands that keeping this out of the language only leads to more confusion, because without support in the Python language itself, these things are hard to explain and hard to document.

I also understand the position that adding this syntax can be seen as an endorsement that would encourage more of this kind of API design. It’s almost certainly true that even if the PEP explicitly says that this is just to remedy the asymmetry and not an endorsement, people will still take it as an endorsement. However, at the end of the day, I don’t think that’s a very good reason to reject this change, because it means end users will be suffering from awkward implementations and documentation (and inconsistency between C and Python implementations).

3 Likes

range() has never been converted to use Argument Clinic, presumably because the existing implementation is fine and it wouldn’t really gain us anything. Argument Clinic could theoretically handle the funny semantics of range() by using “optional groups”, which are groups of optional parameters enclosed in square brackets. That’d look something like this[1]:

range([start,] start_or_stop, [step], /)

When there are optional groups on both the left and right side of required parameters, and it’s ambiguous as to which one to choose first, Argument Clinic gives priority to optional groups on the left side. I designed that specifically to handle range().

In my opinion, range() is a special case that is unfortunately breaking the rules. It’s convenient but surprising. Certainly I wouldn’t want to try to contort Python syntax to accommodate it–beyond saying "implement it yourself with *args and len()". I would not propose, and doubt I would ever like, an extension to Python’s function declaration syntax that made it directly expressible. I guess that makes me a syntactic reactionary.

.

[1] Argument Clinic would actually require the declaration to be spread across multiple lines, to make parsing easier or something.

4 Likes

FWIW I’m +0.2 on the PEP. It’s always been a little surprising that Python has so many functions whose semantics are not implementable in Python without rolling it yourself using *args and len(). And it would make the dream of “reimplement the Python standard library in pure Python” a lot easier to realize. But I could live without it, and I can understand why some folks say it’s unneeded and not worth the semantic overhead.

1 Like

Wouldn’t this work fine?

def range(start, stop=None, step=1):
    if stop is None:
        start, stop = 0, start
    <continue as normal>

That reduces it to a single “weird” case, which is that your positional argument goes into “start” but is interpreted as “stop” (and providing “start=” without “stop=” is also a bit weird, but at least it’s visibly weird at the call site). All other uses, positional named or otherwise, work the same as today, no?