PEP 570: Python Positional-Only Parameters

storchaka · March 31, 2019, 7:33am

Many of these changes are breaking, since people may be passing the existing arguments by name.

You are right. We should add deprecation warnings similarly as in the UserDict constructor.

pf_moore · March 31, 2019, 10:50am

Some people (e.g. me!) will think the first one is clear and obvious and the second is opaque and confusing

I could live with the / notation, but it is as ugly as sin. I’d likely learn to live with it, and as @guido pointed out, no-one has come up with a less ugly solution - but I think that’s only true if you restrict yourself to syntax based options (and more specifically, options that are appropriate in the context of Argument Clinic, because that’s where the / notation was introduced, not as Python syntax although my recollection is that it was intended as Python-compatible).

IMO, the decorator syntax is attractive enough (although so far only in isolated examples intended to “sell” the idea ) to be worth at least exploring further.

storchaka · March 31, 2019, 4:10pm

Is there a sample implementation of PEP 570? I afraid that the implementation of the decorator idea will be more complex.

gpshead · March 31, 2019, 6:22pm

This didn’t strike me as odd at all. It wasn’t Python language syntax. It only appeared in help text due primarily to CPython internals-only argument clinic adoption. There has really been no good reason for most people to care about it. It was much easier to just willfully overlook the / as being a weird glitch in documentation rendering and move on with practical work.

If we adopt this PEP with /, people will learn what it is.

gpshead · March 31, 2019, 7:01pm

(quote above edited to use markdown – the asterisks were disappearing).

I’m understand that. I just don’t love the /. Regardless of token, the unique delineation between all three potential arglist sections makes perfect sense.

A * has had meaning within Python arguments forever both in single and double form, so adopting it stand alone for keyword only syntax was very natural. / has never been used in any context other than division in Python, seeing it gain an additional meaning slightly raises some eyebrows. But so would using any of the other single character punctuation tokens :;.@^%|&+-~.

Maybe “;” instead of “, /,” would be neat for this purpose as it would read nicely. Yet another alternative proposal I’ll pre-emptively reject:

def sink_swamp_castle(burn; *, who_doesnt_leave="He"): ...
def meet_wizard(name;): ...

That use of semicolon in place of the comma seems somewhat englishy but I suspect would ultimately lead to more confusion than a / in the list.

Agreed.

gpshead · March 31, 2019, 7:31pm

An additional reference to add - A pure python decorator from 2007 exists:

https://code.activestate.com/recipes/521874-functions-with-positional-only-arguments/

If we were to seriously consider a decorator implementation we’d need to do it such that it was a special one ala @classmethod that has no startup time or runtime cost and leaves no room for error on specifying what exactly was positional. I’m -1 on decorators for this which appear already rejected so I won’t dive in on that (others are already doing so).

pganssle · March 31, 2019, 9:05pm

I’m sorry if my suggestion came off this way, I certainly didn’t want to suggest that it would be somehow shameful to use this functionality. As a library maintainer concerned about API evolution, I’m already in the target audience for this, so I definitely don’t think people should be shamed for using it.

I was hoping there might be a more neutral compromise wording that would satisfy people (like me) who think this is very useful and good functionality, and people who would be willing to see it added to improve the user experience for people who need this but are not willing to go so far as to promote its use. With additional thought, I’m not sure that there is a good wording that wouldn’t be interpreted as shaming the user for using this functionality, though, so unless someone has a specific request to add some compromise wording to the PEP, I will drop this line of argument.

larry · March 31, 2019, 10:46pm

*args is the exception, not the rule. And I suspect there are many cases where one can correctly intuit what *args is used for, e.g. max(). Anyway your point only (slightly) helps the case for the PEP: giving users more expressive power to design their APIs makes it easier for them to express themselves, and thus less likely to use *args hacks. Though I suppose it’s debatable about how much additional expressive power positional-only parameters affords the user.

larry · March 31, 2019, 11:00pm

For what it’s worth, I just re-read PEP 457, my PEP proposing a positional-only parameters syntax. Six years on I find it a little bizarre. Yes, it revived Guido’s 2012 idea for / as the positional-only parameter delimiter. But I also blithely launch into “optional groups”, a concept I needed in Argument Clinic to express some funny functions that count their positional arguments and require some arguments in groups, e.g. curses.subpad() which either takes two or four arguments.

Let me say this about that: simply adding / to delimit positional-only parameters is not sufficient to express all Python library functions without *args and len() hacks. To cover everything you really do need something like “option groups” where one can have multiple parameters that must be specified or omitted as a unit. Supporting positional-only parameters in Python would let us express the semantics of some Python library functions more elegantly in pure Python, e.g. the dict constructor. But it falls short of getting us all the semantics to implement crazy things like curses.window.overlay(), which accepts either one or seven (!) arguments.

njs · April 1, 2019, 2:48am

There’s a tendency in OSS discussions for things to turn really combative with everyone staking out positions early and punching back and forth, or assuming that other people are doing this and interpreting every comment as an all-out attack. I think it’s an unpleasant way to do things that promotes burnout and bad decisions, and try to avoid it So please don’t assume that because I mentioned something about how people respond to / that it means I’m arguing that / is terrible and trying to destroy it… my priority is to understand what the trade-offs actually are between the different options.

In this case, I was specifically replying to @pablogsal’s claim that using / in docs would be more understandable to people than using decorators in docs, which seems extremely unlikely to me I’m just saying, one of the trade-offs with / is that it’s pretty opaque the first time you see it – as you put it, it looks like a “weird glitch”. That’s not necessarily a bad thing; there are plenty of situations where everyone agrees that using terse punctuation is the best approach (e.g. infix arithmetic). But I hope we can at least agree that this is a trade-off!

This is part of why I think a crucial issue is how widespread the usage will be.

Arbitrary punctuation is especially great for common features, and especially bad for rare features. For example, it’s pretty arbitrary that Python uses {} for dicts and [] for lists, instead of the other way around. In an alternate world, we could have avoided this by requiring that everyone write out dict(...) and list(...) every time. But that would suck, b/c everyone uses dicts and lists constantly. So the arbitrary punctuation doesn’t hurt too much:

yeah, you have to memorize it, but since you use it constantly, it’s like a little flash card in every line of code, so the memorization is easy
memorizing is a one-time cost, and since the feature is commonly used, this cost is amortized over lots of usage
in the mean time, the benefit of terseness gets multiplied by all these uses

But for rarer, more niche features, the calculation is totally different:

you rarely see it, so learning/remembering it is harder
the cost of memorizing doesn’t get amortized much, both because the feature is rare, and because you have to keep reminding yourself what it means
And if you rarely use it, then who cares how terse it is?

toonarmycaptain · April 1, 2019, 3:05am

Absolutely love this! I’m relatively new to coding, but just this week I was designing a method for which I’d like to do this!

pf_moore · April 1, 2019, 7:47am

Hi @toonarmycaptain, and welcome!

As someone relatively new to coding, maybe you could explain why you feel that positional-only parameters would be particularly important to your use case? One of the debates here is over the possibility that the feature will be over-used (it’s generally considered to be a relatively niche need) and it would help a lot to understand what motivates the use of the feature.

In all the years I’ve used Python, I’ve never really noticed the lack of positional-only parameters. My instinct says that it’s more likely to be something that people familiar with languages that don’t have named arguments want, but your comment makes me wonder if that intuition is wrong.

steve.dower · April 1, 2019, 1:12pm

I’d implement the decorator to just add the number of positional only arguments as an attribute of the callable (rather than being a wrapper). Which is exactly what the / would result in, but without the grammar/parser changes. (Maybe someone could come up with a clever backport that implements one of the “tricks” automatically, but we don’t have to use that if we’re just going to make this a feature of the calling convention.)

storchaka · April 1, 2019, 1:45pm

So the following code works?

@positional_only_args(1)
def foo(fmt, **kwargs):
    return fmt.format_map(kwargs)
assert foo('x={fmt}', fmt=42) == 'x=42'

steve.dower · April 1, 2019, 1:47pm

Yeah, that’s kinda where I was heading at that point in my thought process. Though I think I was thinking more about “well they’ll have to read the docs anyway so why not put a not-quite-honest signature there and explain the semantics in text”. Which we already do in the actual documentation in places, and I’ve certainly done it in docstrings.

But maybe some people feel like they need permission to have information like this “below the fold”, as it were?

steve.dower · April 1, 2019, 1:50pm

In the same hypothetical world as the / variant working, sure.

Though it may be more expressive with a decorator to pass the names of arguments that “if passed by name, will always go into kwargs and never into a concrete parameter”?

toonarmycaptain · April 1, 2019, 1:53pm

FIrstly, it’s plausible I’m doing something unwise. I have two classes, let’s call them Class and Student. You can add students to a class by using a method Class.add_student(). Student currently takes two arguments - name and picture_path, but will later take more. I want to design Class's add_student() method to not require changes every time Student has features added, eg arguments for position, weight, height arguments etc.

My first solution was to

def add_student(self, name: Union[str, Student], picture_path: Union[Path, str] = None):
        if isinstance(name, Student):
            self.students.append(name)
        else:
           self.students.append(Student(name, picture_path))

Which works fine, but I don’t feel it particularly helpful to have name be either a Student object or the student’s name, and this implementation is fragile to changes in Student's signature (is that the term?), which will happen as I continue developing.

My current solution is this:
.add_student can take two kinds of arguments - a Student object or the arguments to construct one. The way I’m doing this is to have Student's __init__ method take a name positional parameter, and **kwargs, since I’m fine with adding Students with just a name and adding more info later, but then I have to remove the name parameter from kwargs, lest I get a TypeError: init() got multiple values for argument ‘name’:

def add_student(self, student: [Student] = None, **kwargs: Any):
           if student and isinstance(student, Student):
        self.students.append(student)
    else:
        # if name := kwargs.get('name') isinstance(name, str): Python 3.8
        name = kwargs.pop('name', None)
        if isinstance(name, str):
            self.students.append(Student(name, **kwargs))

It would be nice if I could make the name parameter to Student positional only, since it’s the only argument that would be supplied, and thus the name key in kwargs would be ignored.

The other advantage in this situation that I see is that if I make the student parameter in the .add_student method positional only, I won’t have a clash if I add a student kwarg to the Student class (for what reason I don’t know, maybe I want to be able to add a student with self.name ='class_average', or self.name='always in trouble', and self.student = False), which would insulate the Class.add_student method to any changes to Student's additional arguments (decoupling, is that the term?).

steve.dower · April 1, 2019, 2:15pm

I think this may be a misunderstanding of the proposal, or at least the point where using it would end up making your code more complicated.

Today, you could simply pass all of kwargs as Student(**kwargs) and it will expand out name just as you have (though without the typecheck). And if name is missing, you’ll get the same error as if you called Student without providing it.

If you were to make name positional-only, you would be forcing yourself into having to extract it as you are here. And any additional positional-only parameters would mean changing all of these calls to ensure that the arguments are correctly forwarded. (Using**kwargs as you have is actually a fairly common pattern to forward “known” arguments in a kwargs to avoid having to make changes in many places.)

Your case for potentially wanting to use “student” as an argument to Student is legitimate. That’s the fmt and self examples that we’re figuring out how to deal with. Making “student” in “add_student” be positional only would be one way to deal with this, but I’m not prepared to say that’s the best way.

toonarmycaptain · April 1, 2019, 2:22pm

Ok, so while I was researching and double checking my reply here I found that I could use:
self.students.append(Student(**kwargs)) and it works fine (this is after finding I could use dict.pop(key, default) instead of assigning the value then using del dict['key'] to delete the key. So I am learning today!

I believe my point about insulating positional and kwargs from name clashes, particularly when taking arguments that might be reliant on other code, still stands.

…and, I see you replied with what I found while I was writing my reply! Cheers.

steve.dower · April 1, 2019, 2:29pm

Honestly, I think this is the best thing to have happened so far in this thread (and I’m even more glad it was self-learning and not just from my reply)