PEP 570: Python Positional-Only Parameters

PEP
(Pablo Galindo Salgado) #1

Hello everyone, let me present PEP 570!

This topic serves to host the discussion for the PEP.

You can find the full document here:

Overview

This PEP proposes a syntax for positional-only parameters in Python.
Positional-only parameters are parameters without an externally-usable
name; when a function accepting positional-only parameters is called,
positional arguments are mapped to these parameters based solely on
their position.

Rationale

Python has always supported positional-only parameters. Early versions
of Python lacked the concept of specifying parameters by name, so
naturally, all parameters were positional-only. This changed around
Python 1.0 when all parameters suddenly became positional-or-keyword.
This change allowed users to provide arguments to a function either
positionally or referencing the keyword used in the function’s
definition. However, this is not always desirable, and in fact even in
current versions of Python many CPython “builtin” functions still only
accept positional-only arguments.

Users might want to restrict their API to not allow for parameters to be
referenced via keywords, as that exposes the name of the parameter as
part of the API. If a user of said API starts using the argument by
keyword when calling it and then the parameter gets renamed, it will be
a breaking change. By using positional-only parameters the developer can
later change the name of any arguments or transform them to *args
without breaking the API.

Even if making arguments positional-only in a function can be achieved
by using *args parameters and extracting them one by one, the solution
is far from ideal and not as expressive as the one proposed in this PEP,
which targets providing syntax to specify accepting a specific number of
positional-only parameters. Also, it makes the signature of the function
ambiguous as users won’t know how many parameters the function takes by
looking at help() or auto-generated documentation.

Additionally, this will bridge the gap we currently find between builtin
functions that can specify positional-only parameters and pure Python
implementations that lack the syntax for it. The ‘/’ syntax is already
exposed in the documentation of some builtins and interfaces generated
by the argument clinic.

Making positional-only arguments a possibility in Python will make the
language more consistent and since it would be a normal feature of
Python rather than a feature exclusive to extension modules, it should
reduce surprise and confusion by users encountering functions with
positional-only arguments. Notably, major third-party packages are
already using the “/” notation in their interfaces^1.

Positional-only arguments may be useful in several situations. One of
the more extreme situations is in a function that can take any keyword
parameter but also can take a positional one. Well-known examples for
this situation are Formatter.format and dict.update. For instance,
dict.update accepts a dictionary (positionally) and/or any set of
keyword parameters to use as key/value pairs. In this case, if the
dictionary parameter were not positional-only, the user could not use
the name that the interface uses for said parameter or, conversely, the
function could not distinguish easily if the parameter received is the
dictionary or one key/value pair.

Another important scenario is when argument names do not have semantic
meaning. For example, let’s say we want to create a function that
converts from one type to another:

    def as_my_type(x):
        ...

The name of the parameter provides no value whatsoever, and forces the
developer to maintain its name forever, as users might pass x as a
keyword.

Another good example is an API that wants make it clear that one of its
parameters is the “main” argument through positional-only arguments.
For example, see:

    def add_to_queue(item: QueueItem):
        ...

Again we get no value from using keyword arguments here, and it can
limit future evolution of the API. Say at a later time we want this
function to be able to take multiple items while preserving backwards
compatibility:

    def add_to_queue(items: Union[QueueItem, List[QueueItem]]):
        ...

or to take them by using argument lists:

    def add_to_queue(*items: QueueItem):
        ...

we will be forced to always keep the original argument or we would
potentially break users. By being able to define positional-only
arguments, we can change the name of the parameters at will or even
change them to *args as in the previous example. There are multiple
interfaces in the standard library that fall into this category, for
example the “main” argument of collections.defaultdict (called
default_factory in its documentation) can only be passed
positionally. One special case of this situation is the self parameter
for class methods: it is undersired that a user can bind by keyword to
the name “self” when calling the method from the class:

    io.FileIO.write(self=f, b=b"data")

Indeed, interfaces from the standard library implemented in C usually
take “self” as a positional-only argument:

    >>> help(io.FileIO.write)
    Help on method_descriptor:

    write(self, b, /)
        Write buffer b to file, return number of bytes written.

Another essential aspect to consider is PEP 399^3, that mandates that
pure Python versions of modules in the standard library must have the
same interface and semantics that the accelerator modules implemented in
C (). For example, if collections.defaultdict were to have a pure
Python implementation it would need to make use of positional-only
parameters to match the interface of its C counterpart. A more detailed
discussion about this topic can be found in the
Motivation section.

Positional-Only Parameter Semantics In Python Today

There are many, many examples of functions that only accept
positional-only parameters in the standard library. The resulting
semantics are easily experienced by the Python programmer – just try
calling one, specifying its arguments by name:

    >>> help(pow)
    ...
    pow(x, y, z=None, /)
    ...
    >>> pow(x=5, y=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pow() takes no keyword arguments

pow() clearly expresses that its arguments are only positional via the
/ marker, but this at the moment is only a documentation convention,
Python developers cannot write such syntax.

Besides, there are some functions with particularly interesting
semantics:

  • range(), which accepts an optional parameter to the left of its
    required parameter.^4
  • dict(), whose mapping/iterator parameter is optional and
    semantically must be positional-only. Any externally visible name
    for this parameter would occlude that name going into the **kwarg
    keyword variadic parameter dict^5

One can simulate any of these in pure Python code by accepting
(*args, **kwargs) and parsing the arguments by hand. However, this
results in a disconnect between the Python function signature and what
the function accepts, not to mention the work of implementing said
argument parsing and the lack of clarity in the resulting signature.

As mentioned before, this syntax is already being used outside the
CPython code base for similar use cases^6, remarking that these
scenarios are not exclusive to CPython and the standard library.

Currently users are surprised when first encountering this notation, but
this is to be expected given that it has only recently been adequately
documented [#document-positional-only], and it is not possible to use
it in Python code. For these reasons, this notation is currently an
oddity that appears only in CPython’s APIs developed in C. Documenting
the notation and making it possible to be used in Python code will
certainly eliminate this problem.

Motivation

The new syntax will allow developers to further control how their API
can be consumed. It will allow restricting certain arguments to be
positional-only, so they cannot be passed with a keyword.

A similar PEP with a broader scope (PEP 457) was proposed earlier to
define the syntax. This PEP builds partially on top of that, to define
and provide an implementation for the / syntax in function signatures.

In addition to the API benefits outlined earlier in this document,
positional-only arguments are also faster, as demonstrated in this
thread about converting keyword arguments to positional: ^8. In fact,
because of these benefits there has even been a recent trend towards
moving builtins away from keyword arguments: recently,
backwards-incompatible changes were made to disallow keyword arguments
to bool, float, list, int, tuple.

Providing a way to specify positional-only arguments in Python will make
it easier to maintain pure Python implementations of C modules and will
allow users to take advantage of these benefits even in code written
only in Python. It will also encourage users to start with
positional-only arguments when they believe that passing a keyword
argument provides no clarity; unlike making a keyword argument
positional-only, allowing a positional argument to be passed
positionally is not a breaking change.

This is a well discussed, recurring topic on the Python mailing lists:

Positional-only parameters have also the (minor) advantage of enforcing
some logical order when calling interfaces that make use of them. For
example, the range function takes all its parameters positionally and
this disallows forms like:

    range(stop=5, start=0, step=2)
    range(stop=5, step=2, start=0)
    range(step=2, start=0, stop=5)
    range(step=2, stop=5, start=0)

at the price of disallowing the use of keyword arguments for the
(unique) intended order:

    range(start=0, stop=5, step=2)

Another critical aspect that motivates positional-only arguments is PEP
399^11: Pure Python/C Accelerator Module Compatibility Requirements.
This PEP states that:

This PEP requires that in these instances that the C code must pass
the test suite used for the pure Python code to act as much as
a drop-in replacement as reasonably possible

It is clear that if the C code is implemented using the existing
capabilities to implement positional-only parameters using the argument
clinic and related machinery, it is not possible for the pure Python
counterpart to match the provided interface and requirements. This also
creates a disparity between the interfaces of some functions and classes
in the CPython standard library and other Python implementations. For
example:

    $ python3 # CPython 3.7.2
    >>> import binascii; binascii.crc32(data=b'data')
    TypeError: crc32() takes no keyword arguments

    $ pypy3 # PyPy 6.0.0
    >>>> import binascii; binascii.crc32(data=b'data')
    2918445923

Other Python implementations can, of course, reproduce the CPython APIs
manually, but this goes against the spirit of PEP 399^12 that intends
to avoid duplication of effort by mandating that all modules added to
Python’s standard library must have a pure Python implementation
with the same interface and semantics.

A final argument in favor of positional-only arguments is that they
allow some new optimizations like the ones already present in the
argument clinic since said parameters must be passed in strict order.
For instance, CPython’s internal METH_FASTCALL calling convention
has been recently speciallized for functions with positional-only
parameters to eliminate the cost for handling empty keywords. Similar
performance improvements can be applied when creating the evaluation
frame of Python functions thanks to positional-only parameters.

The Current State Of Documentation For Positional-Only Parameters

The documentation for positional-only parameters is incomplete and
inconsistent:

  • Some functions denote optional groups of positional-only arguments
    by enclosing them in nested square brackets.^13
  • Some functions denote optional groups of positional-only arguments
    by presenting multiple prototypes with varying numbers of
    arguments.^14
  • Some functions use both of the above approaches.^15

One more important idea to consider: currently in the documentation
there is no way to tell whether a function takes positional-only
parameters. open() accepts keyword arguments, ord() does not, but
there is no way of telling just by reading the documentation.

Syntax And Semantics

From the “ten-thousand foot view”, and ignoring *args and **kwargs
for now, the grammar for a function definition currently looks like
this:

    def name(positional_or_keyword_parameters, *, keyword_only_parameters):

Building on that perspective, the new syntax for functions would look
like this:

    def name(positional_only_parameters, /, positional_or_keyword_parameters,
             *, keyword_only_parameters):

All parameters before the / are positional-only. If / is not
specified in a function signature, that function does not accept any
positional-only parameters. The logic around optional values for
positional-only arguments remains the same as for positional-or-keyword
arguments. Once a positional-only argument is provided with a default,
the following positional-only and positional-or-keyword arguments need
to have defaults as well. Positional-only parameters that do not have a
default values are required positional-only parameters. Therefore the
following are valid signatures:

    def name(p1, p2, /, p_or_kw, *, kw):
    def name(p1, p2=None, /, p_or_kw=None, *, kw):
    def name(p1, p2=None, /, *, kw):
    def name(p1, p2=None, /):
    def name(p1, p2, /, p_or_kw):
    def name(p1, p2, /):

While the followings are not:

    def name(p1, p2=None, /, p_or_kw, *, kw):
    def name(p1=None, p2, /, p_or_kw=None, *, kw):
    def name(p1=None, p2, /):

Origin of the “/” as a separator

Using the “/” as a separator was initially proposed by Guido van
Rossum in 2012[^17] :

Alternative proposal: how about using ‘/’ ? It’s kind of the
opposite of ‘*’ which means “keyword argument”, and ‘/’ is not
a new character.

Full grammar specification

A draft of the proposed grammar specification is:

    new_typedargslist:
      tfpdef ['=' test] (',' tfpdef ['=' test])* ',' '/' [',' [typedargslist]] | typedargslist

    new_varargslist:
      vfpdef ['=' test] (',' vfpdef ['=' test])* ',' '/' [',' [varargslist]] | varargslist

It will be added to the actual typedargslist and varargslist, but
for more relaxed discussion it is presented as new_typedargslist and
new_varargslist. Also, notice that using a construction with two new
rules (new_varargslist and new_varargslist) is not possible with the
current parser as a rule is not LL(1). This is the reason the rule needs
to be included in the existing typedargslist and varargslist (in the
same way keyword-only arguments were introduced).

Implementation

An initial implementation that passes the CPython test suite is
available for evaluation^18.

The advantages of this implementation involve speed, consistency with
the implementation of keyword-only parameters as in PEP 3102 and a
simpler implementation of all the tools and modules that will be
impacted by this change.

Rejected Ideas

Do Nothing

Always an option, just not adding it. It was considered though that the
benefits of adding it is worth the complexity it adds to the language.

After marker proposal

A complaint against the proposal is the fact that the modifier of the
signature impacts the tokens already passed.

This might make it confusing to users to read functions with many
arguments. Example:

def really_bad_example_of_a_python_function(fist_long_argument, second_long_argument,
                                            third_long_argument, /):

It is not until reaching the end of the signature that the reader
realises the /, and therefore the fact that the arguments are
position-only. This deviates from how the keyword-only marker works.

That said we could not find an implementation that would modify the
arguments after the marker, as that will force the one before the marker
to be position-only as well. Example:

    def (x, y, /, z):

If we define that / makes only z position-only, it will not be
possible to call x and y via keyword argument. Finding a way to work
around it will add confusion given that at the moment keyword arguments
cannot be followed by positional arguments. / will, therefore, make
both the preceding and following parameters position-only.

Per-argument marker

Using a per-argument marker might be an option as well. The approach
adds a token to each of the arguments that are position only and
requires those to be placed together. Example:

    def (.arg1, .arg2, arg3):

Note the dot on arg1 and arg2. Even if this approach might look easier
to read, it has been discarded as / goes further in line with the
keyword-only approach and is less error-prone.

Some libraries use leading underscore^19 to mark those arguments as
positional-only.

Using decorators

It has been suggested on python-ideas^20 to provide a decorator
written in Python as an implementation for this feature. This approach
has the advantage that keeps parameter declaration more easy to read but
also introduces an asymmetry on how parameter behaviour is declared.
Also, as the / syntax is already introduced for C functions, this
inconsistency will make it more difficult to implement all tools and
modules that deal with this syntax including but not limited to, the
argument clinic, the inspect module and the ast module. Another
disadvantage of this approach is that calling the decorated functions
will be slower than the functions generated if the feature was
implemented directly in C.

Thanks

Credit for most of the content of this PEP is contained in Larry
Hastings’s PEP 457.

Credit for the use of ‘/’ as the separator between positional-only and
positional-or-keyword parameters go to Guido van Rossum, in a proposal
from 2012.[^21]

Credit for discussion about the simplification of the grammar goes to
Braulio Valdivieso.

Copyright

This document has been placed in the public domain.

[^10]: Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

[^17]: Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

[^21]: Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

11 Likes

(Gregory P. Smith) #2

Nice writeup! This has been a long time coming. :slight_smile:

This PEP is critical to have for to be able to duplicate CPython extension or builtin positional only behavior APIs in pure Python code without today’s gross slow hacks of direct *args and **kwargs processing.

4 Likes

(Victor Stinner) #3

Can you please edit your first message to paste the full PEP content? So it can be quoted.

0 Likes

(Jeroen Demeyer) #4

A good example where this would be useful is inspect.getcallargs. It’s currently implemented as

def getcallargs(*func_and_positional, **named):
    """Get the mapping of arguments to values.

    A dict is returned, with keys the function argument names (including the
    names of the * and ** arguments, if any), and values the respective bound
    values from 'positional' and 'named'."""
    func = func_and_positional[0]
    positional = func_and_positional[1:]
    ...

The reason for handling func like this is because otherwise we couldn’t use func as keyword argument. With PEP 570, this would become

def getcallargs(func, /, *positional, **named):

For me, this is the strongest motivation for PEP 570, so it would be good to mention this kind of example and that it’s allowed to use a keyword in **kwargs with the same name as a positional-only argument.

1 Like

(Jeroen Demeyer) #5

Is there any chance to relax the condition

Once a positional-only argument is provided with a default, the following positional-only and positional-or-keyword arguments need to have defaults as well.

I’ve had use cases (admittedly rarely) for a required keyword argument after an optional positional argument.

This might be hard to implement, but I just wanted to drop this as suggestion.

0 Likes

(Paul Ganssle) #6

If I understand you correctly, that use case is actually not forbidden by the requirement you quoted, the requirement there is saying that once you have a positional argument that is allowed to be elided, it may not be followed by a required argument that may be passed as positional. So in order to have an optional positional argument followed by a required keyword argument you’d do:

def f(x=None, /, *, y):
   ...

I think it would create some ambiguity to have something like this:

def f(x=None, /, y):
   ...

Because in order to pass y positionally, you’d need to also pass x, but reading the syntax, it seems ambiguous as to whether f(1) would be valid and equivalent to f(None, 1), or invalid, since y is not passed.

I do think it is logically possible to accept a definition like this, though, where the behavior is that y is required, and the function can either be called with f(x, y) or with f(y=blah), but I’m not sure it’s worth the confusion.

2 Likes

(Pablo Galindo Salgado) #7

Is sort of implied from this example:

Positional-only arguments may be useful in several situations. One of
the more extreme situations is in a function that can take any keyword
parameter but also can take a positional one. Well-known examples for
this situation are Formatter.format and dict.update . For instance,
dict.update accepts a dictionary (positionally) and/or any set of
keyword parameters to use as key/value pairs. In this case, if the
dictionary parameter were not positional-only, the user could not use
the name that the interface uses for said parameter or, conversely, the
function could not distinguish easily if the parameter received is the
dictionary or one key/value pair.

But we can make that more explicit :slight_smile:

1 Like

(Pablo Galindo Salgado) #8

Indeed, with the current implementation:

>>> def f(x=None, /, *, y):
...     print((x,y))
...
>>> f(y=4)
(None, 4)
>>> def f(x=None, /, y):
...     print((x,y))
...
  File "<stdin>", line 1
SyntaxError: non-default argument follows default argument

2 Likes

(Stéphane Wirtel) #9

Hi Jeroen,

I am not sure if your example is the best one because you could rewrite
getcallargs like that

Python 3.7.2 (default, Mar 21 2019, 10:09:12) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def getcallargs(func, *positional, **named):
...     print(func)
...     print(positional)
...     print(named)
...     return func(*positional, **named)
... 
>>> getcallargs(print, "hi", "hello", "world", sep='-')
<built-in function print>
('hi', 'hello', 'world')
{'sep': '-'}
hi-hello-world
>>> 

or then I have not understood your example.

0 Likes

(Pablo Galindo Salgado) #10

Then you cannot call getcallargs with a keyword named func:

>>> getcallargs(print, "hi", "hello", func=open)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: getcallargs() got multiple values for argument 'func'
0 Likes

(Stéphane Wirtel) #11

Sure, I confirm

>>> getcallargs(func=print, sep='-')
<built-in function print>
()
{'sep': '-'}

>>> getcallargs(func=print, 'hi', 'hello', sep='-')
  File "<stdin>", line 1
SyntaxError: positional argument follows keyword argument
>>> 

Thank you for the idea. I will read your PEP in details for this week-end, I have not yet read your PEP in details,

0 Likes

(Pablo Galindo Salgado) #12

But with this PEP you can:

>>> def getcallargs(func, /, *positional, **named):
...   print(func)
...   print(named)
...   print(positional)
...   print(f"Calling {func} with args={positional} and kwargs={named}")
...
>>> getcallargs(print, "hi", "hello", func=open)
<built-in function print>
{'func': <built-in function open>}
('hi', 'hello')
Calling <built-in function print> with args=('hi', 'hello') and kwargs={'func': <built-in function open>}

Notice there are multiple interfaces like this, for example dict.update or Formatter.format.

1 Like

(Stéphane Wirtel) #13

yep, before your PEP, this capacity was not possible. ok, I will read your PEP :snake:

0 Likes

(Serhiy Storchaka) #14

The PEP mentions a per-argument marker as an alternate syntax.

There is an interesting use case for a per-argument marker combined with keyword-only parameters. If the parameter is marked as both positional-only and keyword-only, it cannot be passes neither as positional, neither as keyword argument. But if it has a default value, you will get a function local name initialized with the value created at function creation time. There are uses cases for this feature:

  1. Microoptimization.
def f(x, *, .int=int):
    return int(x)

The name is looked up as a local variable instead of a global variable of built-in. This is faster.

  1. Caching.
def f(x, *, .cache={}):
    try:
        return cache[x]
    except KeyError:
        pass
    y = cahce[x] = g(x)
    return y

Currently we use names that starts with an underscore for these porposes, but it is just a convention, and nothing prevents the user to pass such arguments. In addition, the code with underscored names (_int) looks uglier, and help() exposes the implementation detail.

2 Likes

(Mario Corchero) #15

Interesting usecase! But the “/“ syntax seems more in line with the current syntax of having a marker that splits types of arguments (like for keyword only arguments).

Additionally, even if that was added it will still be exposed in ‘help’, unless we say by “convention” an argument that is both positional only and keyword only is basically some kind of “private. I think that belongs to a different conversation (or PEP) about having arguments that are accesible to the function but private to the user.

If we just focus on the intent of the PEP, providing positional only arguments, I think the “/“ syntax is not only the “most natural” but also the most logical, as it is already spread and used in C APIs and documentation.

2 Likes

(Neil Schemenauer) #16

I fully support the idea of this PEP. We should have a way to say a parameter is position-only.

I wonder, do we really need to support “positional_or_keyword_parameters” for new APIs? If we don’t care to support all four kinds of parameters at once, we could just allow changing ‘*’ to ‘/’ to get the new semantics. I.e. for backwards compatibility, allow:

def name(positional_or_keyword_parameters, *, keyword_only_parameters):

For new APIs, allow:

def name(positional_only_parameters, /, keyword_only_parameters):

I’m not sure this is a good idea but I think it at least worth considering. Add it to “Rejected Ideas” if we don’t want it. I think it simplifies the grammar, which is a good thing. I worry about how complicated the Python grammar as become. It could simplify the implementation too. I.e. have a flag on the code object that says if positional_or_keyword_parameters are allowed.

1 Like

(Paul Ganssle) #17

I think this is super cool (and now I wish I had a way to specify something like that), though I agree with Mario that enabling that use case doesn’t provide enough value to overwhelm the downsides of the per-parameter specification version.

If the grammar were tweaked to allow * to come before /, I suppose an uglier version of this could be possible with the current proposal:

def f(x, *, int=int, /, y):
   ...

But that would also have the downside that in order to use that pattern, you must have only positional-only and keyword-only parameters, and no positional-or-keyword parameters, which is not particularly palatable.

1 Like

(Neil Schemenauer) #18

Rather than having:

def f(x, *, .int=int):
    return int(x)

Would it be nicer to have a “static” binding statement that is internal to the function? Sort of like ‘global’ or ‘nonlocal’ but behaves like your keyword parameter does. E.g.

def f(x):
    local int = int  # RHS evaluated at function definition time
    return int(x)

Since we probably don’t want a new keyword, could use some grammar based on existing keywords. Using keyword arguments for this performance trick was a neat hack but I don’t think the parameter list is actually the place to be putting it (if we are going to be designing new syntax to support it).

2 Likes

(Pablo Galindo Salgado) #19

I also think that said use case deviates from the purpose of the PEP. That is more or less inclined to add some private attributes to functions that live as long as the function and I do not think function signatures should be the place for that. (Imagine the documentation or the help section).

Even if this could be useful, I think is somehow independent of this PEP.

2 Likes

(Victor Stinner) #20

I never liked the int=int “hack” (but I use it, especially for del destructor, to kept some objects alive). In the C language, there is a dedicated syntax for that: “static type var = value;” ensures that the variable is only initialized once and the variable value is kept between function calls. We need something like that in Python.

Currently, I use sometimes:

def func():
   if func.cache is None: func.cache = init_cache()
   ... use func.cache ...
func.cache = Noe

I dislike this syntax because Python has to fetch “func” variable from globals, it’s inefficient. And the the function must stay in globals under the same name: “del func” breaks the code even if the function is stored something.

I don’t think that the syntax of function parameters should be “abused” to get “static variables”.

0 Likes