Allow identifiers as keyword arguments at function call site (extension of PEP 3102?)

Ideas
1

It is common to see keyword arguments passed values from identifiers (typically local variables) with the same name as the argument. Here’s an example from the pytorch library:

func(params,
     d_p_list,
     momentum_buffer_list,
     weight_decay=weight_decay,
     momentum=momentum,
     lr=lr,
     dampening=dampening,
     nesterov=nesterov,
     has_sparse_grad=has_sparse_grad,
     maximize=maximize)

My proposal is to allow these keyword arguments to be passed as e.g. weight_decay instead of weight_decay=weight_decay

This would apply only where positional arguments are not allowed e.g. arguments which are forced to be keyword-only in the function definition or arguments following at least one explicit keyword argument at the call site

I would also propose allowing * at the call site to force keyword arguments, analogous to the use of * to force keyword only arguments in the function definition, added by PEP 3102

The principal benefit would be to eliminate redundant typing in function calls like the example above, which now becomes:

func(params,
     d_p_list,
     momentum_buffer_list,
     # following arguments are all keyword, due to * in the function definition after momentum_buffer_list
     weight_decay,
     momentum,
     lr,
     dampening,
     nesterov,
     has_sparse_grad,
     maximize)

A particular application would be the case of packing variables into a dictionary, where we could then abbreviate:

dict(a=a, b=b, c=c)

as:

dict(*, a, b, c)

Note that this proposal does not change the behaviour of any existing valid python code. It only gives a valid meaning to some previously invalid syntax, therefore no ordinary existing code should be impacted

There could be extraordinary cases which are impacted e.g. automatic code generation which speculatively tries to generate possibly invalid code and tests it empirically with eval/exec/compile

I feel that this change naturally extends PEP 3102 and with hindsight could have been included as part of that PEP

It also relates indirectly to the support for = in format strings, added in Python 3.8. The similarity being that in both cases we are allowing that the form of an expression in code, not just the value of the expression at runtime, can be impactful. This is aspect is fairly rare, so I thought it’s worth pointing out that this isn’t unprecedented in Python

I realise there is a very high bar for this kind of language change! In any case I am interested to hear the views of others, especially any problematic situations for this proposal which I haven’t thought of

4 Likes
2

This has been brought up at least once that I can remember (this being the elimination of a = a, and the general argument against it that still holds is that it’s much too magical. It would do your case well to look up those earlier discussions, find the arguments for and against and argue what has changed to again make this an option to consider.

2 Likes
3

Hi Jacob - thanks for replying on my post. It doesn’t surprise me that this idea has been raised before. I would be very interested to read past discussions on this topic

I have searched for this topic here and on google and so far come up empty handed. It’s hard to think of search terms that will catch this. If you are able to find a link to the discussion you’ve seen before I would be very grateful

4

Finding old discussions is annoying because they are spread over multiple possible forums, here, github issues, the python-ideas mailing list and the python-dev mailing list. I’m on my phone now but I’ll post the links after work unless somebody else does so first.

Regarding the idea itself, I’m not a huge fan. Apart from it being a bit too implicit, I can also see potential confusing situations like this:


def foo(x, y): ...

x = 1
y = 2

# Will this be called with args or kwargs semantics?
foo(y, x)
1 Like
5

This part, unfortunately, is way too magical. A call MUST be well-defined without reference to the function being called, otherwise all kinds of things become near-impossible.

But if done with a syntactic marker, that would be a bit more reasonable. Your second example:

seems far more plausible to me. This eliminates the “is this positional or keyword semantics?” problem and is purely and simply syntactic sugar.

1 Like
6

Major issues:

  1. When code is generated at the call site, it doesn’t know anything about the function being called (because it will be looked up at runtime). If I write func(params, d_p_list, momentum_buffer_list, weight_decay) then Python cannot determine that weight_decay is a keyword-only argument of func. It can’t even determine that func is callable (maybe there will be a TypeError at runtime).

  2. What would happen if the function also accepts var_positional arguments, or positional arguments with defaults? How do we know when the implicitly-keyword arguments start?

def func(*x, y=None):
    pass

func(y) # is this equivalent to `func(y, y=None)`, or to `func(y=y)`?
  1. What about unpacking arguments with *? Could those results end up accidentally being keyword arguments?

I think having a syntactic marker at the call site - such as a bare * argument (mirroring how it works in parameters) - solves all these problems, but I’m not sure:

func(params, d_p_list, momentum_buffer_list, *, weight_decay) # etc.

It would also be needed explicitly in all cases, even the ones where * is used to unpack positional arguments - because currently, func(x, *y, z) can mean that z is intended as a positional argument as well.

Another option would be to say that ** unpacking can unpack from a set as well as a mapping, and have it implicitly look up “values” for those “keys’” from the local namespace. Then we could write func(a, b, **{'c', 'd', 'e'}) as equivalent to func(a, b, c=c, d=d, e=e). This could even become a more general unpacking rule - e.g.

# with this hypothetical feature implemented
normal_meal = {'bacon': 1, 'eggs': 2}
spam = 3
baked_beans = 4
viking_meal = {**normal_meal, **{'spam', 'baked_beans'}}
assert viking_meal == {'bacon': 1, 'eggs': 2, 'spam': 3, 'baked_beans': 4}
2 Likes
7

Previous discussions: Shorthand notation of dict literal and function call

1 Like
8

many thanks!

This case would be unambiguous because (under the proposal) kwargs semantics would apply only when positional arguments are illegal. However, I agree that one could get tripped up on this by not being sure about the keyword-only argument status of the function (moreover subsequent changes to the keyword-only argument status of the function would have unexpected impact on the caller here)

This is a very good point. I suppose this is necessary in order to be able to compile the calling code to bytecode without locking down the function definition code we are calling into

It seems then that the * marker is really necessary both for the compiler and for clarity of intention for human readers of code. And as per comments from Karl, it is important that this is a bare * and not *args since the latter doesn’t preclude more positional args

1 Like
9

Thanks Chris

I guess in answer to Jacob’s question “what has changed to again make this an option to consider” all that’s different is an alternative syntax proposal to address the same issue

I guess that’s a matter of taste but maybe folks find this:

dict(*, a, b, c)

more palatable than this:

dict(a=, b=, c=)

or:

dict(:a, :b, :c)

as previously proposed

Apologies for having raised this issue as a new topic. Hopefully at least the discussion of call-site unambiguity is helpful for others reading this as it was for me

4 Likes
10

Just to adapt my list of advantages from the previous proposal (var=) to the * proposal:

  1. You get dict(*, key1, key2) for free.
  2. An API changing an argument with a default to keyword-only can be dealt with (if parameters and variables match) by changing f(a, b, c) to f(a, b, *, c).
  3. It encourages using variables with names that match parameters. f(*, long_variable_name) is cleaner than f(long_variable_name=longvar), which some might prefer to f(long_variable_name=long_variable_name).

You obviously already covered 1.

I do prefer this approach to the previous suggestions, as it closely mirrors the function signature, which is a well-established concept. I agree with others in this thread that this should require an explicit *,, and never be implicit. Regardless of whether the technical ambiguities could be resolved, I would still want a clear syntactic marker for visual clarity.

I still find the idea of a shorthand appealing, for whatever that’s worth. The main argument I see against it is that positional-only and keyword-only arguments are advanced concepts that beginners can go quite a while before needing to understand. You can read code that calls functions with keyword-only arguments with no new concepts, and it’s only when you need to write that code that it matters. This would shift the need to understand these to the first time you encounter something like dict(*, a, b, c), which has the potential to become a common idiom.

2 Likes
11

It’s worth noting that p-only and kw-only are actually irrelevant here; the arguments passed in this way will always be passed as keywords, regardless of what’s defined at the other end.

1 Like
12

I like the idea of the func(*, a, b, c)-like syntax as an equivalent to func(a=a, b=b, c=c), yet I am not sure the * character is the most appropriate choice of the marker – because in the case of the definition syntax, the * and *somevar constructs are consistent with each other in that they have the same consequences for any following parameters (that they will be keyword-only ones), e.g.:

def func1(*args, kwonly, kwonly2): ...
def func2(*, kwonly, kwonly2): ...

…but, when it comes to the new imaginary call syntax, there would be no analogous consistency:

func(*args, a, b, c)  # existing syntax, we cannot (and don't want to) change its meaning
func(*, a, b, c)  # new syntax, supposed to be equivalent to `func(a=a, b=b, c=c)`

…so it might be confusing.

I believe that a lone = character would be a much better choice.

func(=, a, b, c)  # new syntax, supposed to be equivalent to `func(a=a, b=b, c=c)`

Not only the aforementioned confusion would be avoided, but also the = character it this role seems quite suggestive. I believe, it would be easy to learn and remember (or maybe even guess?) what this shortcut is supposed to mean.

3 Likes
13

The hash character already has meaning, this won’t work. Sorry.

14

@Rosuav looks like you fired before reading. His proposal is to use =, not #.

(Not that I think Python needs this feature.)

1 Like
15

This approach would essentially prevent code refactoring. If you were to rename a parameter in a function definition, it would cause issues in the parts of the code where this function is employed.

1 Like
16

Dangit, I think I was messed up by a dirty screen there. My bad. I swear it looked like a hash sign.

TBH though, I think the asterisk is still sufficiently parallel in meaning that it’s a better choice, and the equals doesn’t have much to brag of. If this feature were to be implemented (far from certain), the equals sign would be appropriate alongside a specific argument, but the asterisk would be a viable separator. So these two notional syntaxes make at least some sense:

func(a=, b=, c=)
func(*, a, b, c)

But I’m not convinced that they’re offering enough here.

2 Likes
17

Me neither. But for the sake of argument (or to show off how devious my mind works :slight_smile: ), I would prefer the former (a=) because that draws the reader’s attention to the unusual syntax for each keyword. The * having an “effect at a distance” on all following arguments in the call is more easily missed when skimming code.

Also, we should at least consider how this would combine with keyword arguments that do specify both the keyword and value – in case you have a bunch of keywords following a similar pattern but one exception. Again, I think here a= wins – we can easily understand what f(a=, b=1, c=) would mean, given a minimal explanation of a=.

Also, someone previously worried about what happens if a refactoring of the callee causes a keyword to change. Again, I think the a= notation, augmented with the idea of mix-and-match, would win out here, since you could just change the call site from f(a=, b=, c=) to f(a=, new_b=b, c=).

Still, this is all an intellectual puzzle that we can try to solve, but it leaves the larger question unanswered – does Python really become a better language with this feature? I don’t think it does – this isn’t like something I’ve seen in other languages (maybe Perl? :slight_smile: ) and if you don’t already know what it means it’s hard to look up (you’d probably end up finding the trailing = in f-strings, which has totally different grammar and meaning). It also feels a bit too easy to overlook. This can all be litigated endlessly (“but what if we …”), but in the end it’s not me you have to convince but the Steering Council, and they’ve rejected much less niche proposals.

11 Likes
18

Thanks for the feedback, Guido!

I agree with the benefits you point out

The improvement is of convenience and readability. It’s not a huge deal of course. But I feel like there are already dozens (hundreds?) of areas of Python where convenience and readability wins compared to other languages have contributed to the popularity of Python. This would just add to that list

Is the meaning totally different? To me there is something similar here:

f"{long_identifier=}" is shorthand for f"long_indentifier={long_identifier}"

func(long_identifier=) is shorthand for func(long_identifier=long_identifier)

One relevant example I know of from another language is implicit in Scala. A good explanation is here. implicit in Scala is way more magical than the feature we are discussing - it sources parameters not just by matching to the call site identifier but actually searching for any suitably typed variable in the calling scope to plug in :slight_smile:

I can believe this. I would argue though that this proposal is, an a sense, not that niche. It is arguably a minor convenience add. But it is widely applicable - a large percentage of significant codebases presumably include func(x=x, y=y, z=z)-like calls. My guess is many Python users would take up the opportunity to use the proposed concise syntax going forward if this feature were available in the language

2 Likes
19

Hmm, can we replace that “presumably” with an actual list of examples?

20

Systemverilog has a similar feature. It is called “implicit port connection” there. Its syntax is dot. f(.a, .b, c=z).

I think this feature can make Python code a lot less verbose when applicable. I would then be more inclined to use keyword arguments instead of positional arguments. This can make Python code safer.