PEP 570: Python Positional-Only Parameters

The pep does not propose anything different and tries to mirror the argument clinic (the examples in the “Syntax And Semantics” are actually from Larry). One of the main ideas of the PEP is preserving precisely the interface of the argument clinic to satisfy PEP 399.

1 Like

Interesting…I see how this can be beneficial for some similar APIs, but I think it can be a significant jump in complexity for the syntax of argument declaration. Currently typedargslist is the more complicated rule that Python has (at least it generates the biggest DFA) and I would suggest preserving the “once you use a default argument in your declaration all the rest of the arguments have to be default as well” rule, as this can explode in complexity very fast.

I will try to play a bit with this idea to see how complicated is to use or what advantages can have.


As a curiosity, here is the current DFA for typedargslist :

2 Likes

Sounds like we’d better skip it then – that API design is never going to win any prizes. And as long as it’s possible to propose this in a future PEP. :slight_smile:

I see. That would be a bit odd. I’m wondering what you think is the preferred design for parameters in new APIs. My gut instinct is that keyword parameters should always be passed by keyword and that positional args should always be passed without the keyword.

If we were going to change it so without ‘/’ or ‘*’ you get positional only, we could have something like

from __future__ import keyword_only_parameters

Then you could have a 2to3 like tool that would fixup code (i.e. add ‘*’ to the end of function parameter lists). That is a lot of code churn but the change is purely mechanical. After the feature is the default, the ‘/’ in the parameter list becomes unneeded and could be left out.

I’m trying to think in the very long term and how we wish novices would write Python code. Remembering to put ‘/’ in the parameter list seems like an extra burden. If we did go with the __future__ idea, we would have to give people a long time to update their code, like 10 years or something.

1 Like

Please put me down for a strong -1.

The / notation in the docstrings has been a failure with users. It is confusing and distracting.

The problem been solved is mostly an imagined problem. Real users almost never have a problem with CPython as it has been for almost 30 years.

Python has been getting harder and harder to teach. Instructors with a time limit have to pick and choose the topics to drop entirely. This proposal will make it worse.

Lastly, the notation looks gross (to my eyes at least). This is a language where we say beauty counts but then seem to be willing to throw it away for something that isn’t an actual problem in userland (or is at least a very, very minor problem that doesn’t interfere with day to day usability). I really don’t want to have to see, write, or teach something like this:

def name(p1, p2=None, /, p_or_kw=None, *, kw):

Please don’t do this.

6 Likes

This is not true, as we said in the PEP major 3rd part libraries are already using the “/” notation implementing the feature on their own, like numpy:

https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs

This is covered in the PEP:

Currently users are surprised when first encountering this notation, but
this is to be expected given that it has only recently been adequately
documented [#document-positional-only], and it is not possible to use
it in Python code. For these reasons, this notation is currently an
oddity that appears only in CPython’s APIs developed in C. Documenting
the notation and making it possible to be used in Python code will
certainly eliminate this problem.

I think this is highly subjective, as you are making an argument based on the perception of “beauty” and that will change massively from person to person. This PEP tries to solve several problems that are described in the document, and we firmly believe that these problems are real and important. The fact that a feature is focused mainly for library developers does not make it less important, as having flexible and stable interfaces is very important, especially for helping with future changes as the PEP describes.

Interfaces evolve over time, and maintaining backwards compatibility for example is something that library developers struggle continuously. Right now there is the danger that every name that you use in your interface needs to be forever or you need to consider breaking users. Not only that, but moving to more flexible interfaces using *args is also a possibility that positional-only parameters allow, all of this with descriptive and explicit signatures (instead of uninformative functions that only receive *args and **kwargs).

The PEP makes a great effort in describing not only all the situation were positional only arguments are important and solve real problems, but also mentions other very important aspects like the fact that they are needed for compliance with PEP 399. The problems that CPython currently solves with the argument clinic are not unique of CPython, the standard library or C functions, so I don’t see why Python functions in the standard library or external users cannot benefit from it.

The “problems” in argument clinic only exist because they predate argument clinic. Larry tried very hard to avoid adding anything new, but had no choice because some existing built in functions essentially dispatch on argument count and cannot reasonably support named parameters (such as range).

I’m sympathetic to the “func, **{“func”: Value}” case, but it feels very contrived - I’ve never seen an actual complaint about this. And this is easily achieved with the tricks already listed here, along with plenty of other approaches (such as using a parameterless lambda with a closure).

If/when you get to counting, count me as -1. And honestly I’d go as far as changing the builtins like range() to support named arguments no matter how ugly the implementation gets. Knowing that every argument can be specified by name is a feature, and I’m not convinced abandoning that is worth it.

1 Like

Calling something “subjective” is a way of dismissing it, but language design and usability don’t live in a metric space – there are no “objective” measures. That doesn’t mean that usability and beauty are unimportant. They are in fact the most salient features of the language.

Please don’t blow-off my concerns. For many years, I have spent nearly every working day explaining Python to groups of engineers and scientists across multiple companies and multiple disciplines. This makes me keenly aware of what challenges people face, what problems they care about, and what parts of the language either help or hinder them. I can report that the “/” notation in the help() output and tooltips has been an abject failure. It is not user friendly, necessary, or communicative.

Over time, we’ve had a trend of adding unnecessary, low-payoff complexity to the language. Cumulatively, it has greatly increased the mental load for newcomers and for occasional users. The help() output has become less self-explanatory over time and makes the language feel more complex. The proposal at hand makes it worse.

1 Like

Evidence that there isn’t an important problem to be solved:

  • It doesn’t come up in Python training courses at all
  • There don’t seem to be any prominent blog posts on the subject
  • It doesn’t show up in Hacker News comments
  • There don’t seem to be any recorded talks on the subject
  • On StackOverflow’s 2,000,000+ Python questions, this comes up almost never
  • I spend my life answering questions about Python and have never been asked about this
  • It is rare to see a bug report or tracker issue where this is a root cause.
  • Most Python programmers don’t even know what the “/” means in help() output – just ask around
  • If you ask Python consultants what problems they are asked to solve, it won’t be mentioned
  • Other prominent languages haven’t felt the need to add a “/” notation
  • Ask full time code reviewers (like Jack D) whether this is a recurring or important issue (the answer is no)

The PEP isn’t wrong that there aren’t some issues. It is wrong that there about the significance being enough to warrant a grammar change that will be inflicted on 100% of the users in a way they mostly won’t be able to ignore when reading code.

3 Likes

If NumPy was already dragged in as an example, I can confirm that PEP 457 positional_or_keyword_parameters are confusing to users. NumPy has open prs trying to untangle errors users encounter when they meant one thing but ended up doing another. In the instance quoted - ufuncs - I have had users mistakenly use an out parameter when they thought it was an input operand, with no recourse to easily analyze the unexpected change in the “operand” value (out is modified in-place). If they were forced to explicitly use a keyword value, code readability would increase.

2 Likes

How does np.add(x1=[1,2], x2=[3,4]) increase readability?

But out can already be passed by keyword so any problem that happen now will keep happening with this PEP and without, this would only affect x1 and x2.

I’ve been bitten by this a few times (admittedly rarely), typically when pass-through keywords are used like

def somefunc(foo, bar, **kwargs):
    ...do something with foo and bar...
    somefunc(**kwargs)

This doesn’t work as expected if one wants to pass foo or bar as keyword arguments.

But a much simpler solution to this problem would be to allow putting keyword arguments in **kwargs if they have the same name as a mandatory positional argument. In other words, make it such that somefunc(x, y, foo=z) is actually allowed, by setting foo=x and kwargs={'foo':z}.

1 Like

The problem is not in the “normal” common arguments, but in the “exceptional” additional ones that the dual-use enables.

x=[1, 2]; y= [3, 4]; z = [5, 6]; np.add(x, y, z) would be clearer as np.add(x, y, out=z) since 99% of users meant np.add(np.add(x, y), z) but mistakenly thought they could use many operands at once. But all that is water under the bridge the real problem is

any problem that happen now will keep happening with this PEP and without

True, but the PEP would enable and encourage package maintainers to use more advanced syntax out-of-step (in my opinion) with users’ (as I percieve them) needs, complicating the language and raising barriers for new users. As others have said, the docstring for np.add requires quite a bit of mental effort to parse, exactly what is lacking when trying to debug an already confusing error. I have had students look at that dsl and prefer to try various code snippets rather than understand what it means.

The main problem with using / in the help output for builtins which use Argument Clinic is that this syntax is not documented. If it would be documented, there would be less questions (not more than about * and **). I don’t particularly like the / syntax, but no better syntax was proposed for last 7 years.

I am in a great favor of PEP 570, no matter what syntax will be used. Currently we need to write complex and cumbersome code. Two examples:

class MutableMapping(Mapping):
    def update(*args, **kwds):
        if not args:
            raise TypeError("descriptor 'update' of 'MutableMapping' object "
                            "needs an argument")
        self, *args = args
        if len(args) > 1:
            raise TypeError('update expected at most 1 arguments, got %d' %
                            len(args))
        if args:
            other = args[0]
            ...
        ...
class Formatter:
    def format(*args, **kwargs):
        if not args:
            raise TypeError("descriptor 'format' of 'Formatter' object "
                            "needs an argument")
        self, *args = args  # allow the "self" keyword be passed
        try:
            format_string, *args = args # allow the "format_string" keyword be passed
        except ValueError:
            raise TypeError("format() missing 1 required positional "
                            "argument: 'format_string'") from None
        return self.vformat(format_string, args, kwargs)

And help() outputs a useless signature for these functions: (*args, **kwargs). It is even confusing, because it does not contain self as signatures of other methods.

With PEP 570 the above examples can be written as:

class MutableMapping(Mapping):
    def update(self, other=(), / , **kwds):
        ...
class Formatter:
    def format(self, format_string, /, *args, **kwargs):
        return self.vformat(format_string, args, kwargs)

This feature will be useful not only for experts, but it can reduce the confusion of newbies. First, the will no longer be confused by / in signatures of some builtins. Second, they will no longer be confused by nonavailability to pass self as a keyword argument to unbound methods.

3 Likes

It’s not a feature. Readability Counts, and writing len(obj=configurations) is not something we want to encourage.

1 Like

There’s a big difference between encouraging it and changing the grammar in a way that lets someone else decide whether I can or not. We can still say it’s ugly, un-Pythonic, and plenty of other things without having to make it literally impossible.

(Edit - reminder, my position is to fix up the builtins to get rid of / completely)

1 Like

Regarding the argument that the / is hard to teach, there are tons of advanced features that beginners don’t need to be taught. OTOH presumably one of the problems with the current usage in Argument Clinic is that it generally produces function signatures that look just like what you can use with def, and / is the exception here. Things will be more regular once we also support it in Python syntax.

I definitely see the need for this feature, and I haven’t seen a reasonable alternative proposal, so I am strongly in support of PEP 570.

2 Likes

Adding a docstring would help. Or overriding signature to make it appear to the user in one way while it actually implements it differently (I think this one needs a change to work, but it’s better than adding another type of parameter).

The suggestion to let named parameters be provided twice - once in the actual one and once in a **kwargs - could also be interesting, but likely a cause of bugs since it’s currently a useful error. Though maybe we can achieve that with a decorator for the few cases where it’s useful?

Literally the same argument applies to accepting the PEP – you don’t have to use the new feature.

When it comes to who can control how a function can be called, obviously that should be the author of the function, not the caller.

The fact that this is not obviously true in Python has been one of my favourite things about the language for years. Callers can do so much that the callee didn’t plan for, design, or implement. Proper first-class functions are great. Functools.partial is great. Binding via __get__ is great. Two argument iter is great. Compared to the other languages I get to use, Python is such a breathe of freedom in this area.

And all of those things are truly optional for a beginner (in a well-designed library). Even the semantics added by __get__ are natural and help users to avoid issues, despite the implementation complexity.

Introducing an error when an argument is provided by name just doesn’t fit into the category of making functions easier to call.