Hello everyone, let me present PEP 570!
This topic serves to host the discussion for the PEP.
You can find the full document here:
Overview
This PEP proposes a syntax for positional-only parameters in Python.
Positional-only parameters are parameters without an externally-usable
name; when a function accepting positional-only parameters is called,
positional arguments are mapped to these parameters based solely on
their position.
Rationale
Python has always supported positional-only parameters. Early versions
of Python lacked the concept of specifying parameters by name, so
naturally, all parameters were positional-only. This changed around
Python 1.0 when all parameters suddenly became positional-or-keyword.
This change allowed users to provide arguments to a function either
positionally or referencing the keyword used in the function's
definition. However, this is not always desirable, and in fact even in
current versions of Python many CPython "builtin" functions still only
accept positional-only arguments.
Users might want to restrict their API to not allow for parameters to be
referenced via keywords, as that exposes the name of the parameter as
part of the API. If a user of said API starts using the argument by
keyword when calling it and then the parameter gets renamed, it will be
a breaking change. By using positional-only parameters the developer can
later change the name of any arguments or transform them to *args
without breaking the API.
Even if making arguments positional-only in a function can be achieved
by using *args
parameters and extracting them one by one, the solution
is far from ideal and not as expressive as the one proposed in this PEP,
which targets providing syntax to specify accepting a specific number of
positional-only parameters. Also, it makes the signature of the function
ambiguous as users won't know how many parameters the function takes by
looking at help()
or auto-generated documentation.
Additionally, this will bridge the gap we currently find between builtin
functions that can specify positional-only parameters and pure Python
implementations that lack the syntax for it. The '/' syntax is already
exposed in the documentation of some builtins and interfaces generated
by the argument clinic.
Making positional-only arguments a possibility in Python will make the
language more consistent and since it would be a normal feature of
Python rather than a feature exclusive to extension modules, it should
reduce surprise and confusion by users encountering functions with
positional-only arguments. Notably, major third-party packages are
already using the "/" notation in their interfaces[1][2].
Positional-only arguments may be useful in several situations. One of
the more extreme situations is in a function that can take any keyword
parameter but also can take a positional one. Well-known examples for
this situation are Formatter.format
and dict.update
. For instance,
dict.update
accepts a dictionary (positionally) and/or any set of
keyword parameters to use as key/value pairs. In this case, if the
dictionary parameter were not positional-only, the user could not use
the name that the interface uses for said parameter or, conversely, the
function could not distinguish easily if the parameter received is the
dictionary or one key/value pair.
Another important scenario is when argument names do not have semantic
meaning. For example, let's say we want to create a function that
converts from one type to another:
def as_my_type(x):
...
The name of the parameter provides no value whatsoever, and forces the
developer to maintain its name forever, as users might pass x
as a
keyword.
Another good example is an API that wants make it clear that one of its
parameters is the "main" argument through positional-only arguments.
For example, see:
def add_to_queue(item: QueueItem):
...
Again we get no value from using keyword arguments here, and it can
limit future evolution of the API. Say at a later time we want this
function to be able to take multiple items while preserving backwards
compatibility:
def add_to_queue(items: Union[QueueItem, List[QueueItem]]):
...
or to take them by using argument lists:
def add_to_queue(*items: QueueItem):
...
we will be forced to always keep the original argument or we would
potentially break users. By being able to define positional-only
arguments, we can change the name of the parameters at will or even
change them to *args
as in the previous example. There are multiple
interfaces in the standard library that fall into this category, for
example the "main" argument of collections.defaultdict
(called
default_factory in its documentation) can only be passed
positionally. One special case of this situation is the self parameter
for class methods: it is undersired that a user can bind by keyword to
the name "self" when calling the method from the class:
io.FileIO.write(self=f, b=b"data")
Indeed, interfaces from the standard library implemented in C usually
take "self" as a positional-only argument:
>>> help(io.FileIO.write)
Help on method_descriptor:
write(self, b, /)
Write buffer b to file, return number of bytes written.
Another essential aspect to consider is PEP 399[3], that mandates that
pure Python versions of modules in the standard library must have the
same interface and semantics that the accelerator modules implemented in
C (). For example, if collections.defaultdict
were to have a pure
Python implementation it would need to make use of positional-only
parameters to match the interface of its C counterpart. A more detailed
discussion about this topic can be found in the
Motivation section.
Positional-Only Parameter Semantics In Python Today
There are many, many examples of functions that only accept
positional-only parameters in the standard library. The resulting
semantics are easily experienced by the Python programmer -- just try
calling one, specifying its arguments by name:
>>> help(pow)
...
pow(x, y, z=None, /)
...
>>> pow(x=5, y=3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: pow() takes no keyword arguments
pow()
clearly expresses that its arguments are only positional via the
/
marker, but this at the moment is only a documentation convention,
Python developers cannot write such syntax.
Besides, there are some functions with particularly interesting
semantics:
range()
, which accepts an optional parameter to the left of its
required parameter.[4]dict()
, whose mapping/iterator parameter is optional and
semantically must be positional-only. Any externally visible name
for this parameter would occlude that name going into the**kwarg
keyword variadic parameter dict[5]
One can simulate any of these in pure Python code by accepting
(*args, **kwargs)
and parsing the arguments by hand. However, this
results in a disconnect between the Python function signature and what
the function accepts, not to mention the work of implementing said
argument parsing and the lack of clarity in the resulting signature.
As mentioned before, this syntax is already being used outside the
CPython code base for similar use cases[6][7], remarking that these
scenarios are not exclusive to CPython and the standard library.
Currently users are surprised when first encountering this notation, but
this is to be expected given that it has only recently been adequately
documented [#document-positional-only], and it is not possible to use
it in Python code. For these reasons, this notation is currently an
oddity that appears only in CPython's APIs developed in C. Documenting
the notation and making it possible to be used in Python code will
certainly eliminate this problem.
Motivation
The new syntax will allow developers to further control how their API
can be consumed. It will allow restricting certain arguments to be
positional-only, so they cannot be passed with a keyword.
A similar PEP with a broader scope (PEP 457) was proposed earlier to
define the syntax. This PEP builds partially on top of that, to define
and provide an implementation for the /
syntax in function signatures.
In addition to the API benefits outlined earlier in this document,
positional-only arguments are also faster, as demonstrated in this
thread about converting keyword arguments to positional: [8]. In fact,
because of these benefits there has even been a recent trend towards
moving builtins away from keyword arguments: recently,
backwards-incompatible changes were made to disallow keyword arguments
to bool
, float
, list
, int
, tuple
.
Providing a way to specify positional-only arguments in Python will make
it easier to maintain pure Python implementations of C modules and will
allow users to take advantage of these benefits even in code written
only in Python. It will also encourage users to start with
positional-only arguments when they believe that passing a keyword
argument provides no clarity; unlike making a keyword argument
positional-only, allowing a positional argument to be passed
positionally is not a breaking change.
This is a well discussed, recurring topic on the Python mailing lists:
- September 2018: Anders Hovmƶller: [Python-ideas] Positional-only
parameters - February 2017: Victor Stinner: [Python-ideas] Positional-only
parameters,
discussion continued in
March - February 2017:[9]
- March 2012:[10]
- May 2007: George Sakkis: [Python-ideas] Positional only
arguments - May 2006: Benji York: [Python-Dev] Positional-only
Arguments
Positional-only parameters have also the (minor) advantage of enforcing
some logical order when calling interfaces that make use of them. For
example, the range
function takes all its parameters positionally and
this disallows forms like:
range(stop=5, start=0, step=2)
range(stop=5, step=2, start=0)
range(step=2, start=0, stop=5)
range(step=2, stop=5, start=0)
at the price of disallowing the use of keyword arguments for the
(unique) intended order:
range(start=0, stop=5, step=2)
Another critical aspect that motivates positional-only arguments is PEP
399[11]: Pure Python/C Accelerator Module Compatibility Requirements.
This PEP states that:
This PEP requires that in these instances that the C code must pass
the test suite used for the pure Python code to act as much as
a drop-in replacement as reasonably possible
It is clear that if the C code is implemented using the existing
capabilities to implement positional-only parameters using the argument
clinic and related machinery, it is not possible for the pure Python
counterpart to match the provided interface and requirements. This also
creates a disparity between the interfaces of some functions and classes
in the CPython standard library and other Python implementations. For
example:
$ python3 # CPython 3.7.2
>>> import binascii; binascii.crc32(data=b'data')
TypeError: crc32() takes no keyword arguments
$ pypy3 # PyPy 6.0.0
>>>> import binascii; binascii.crc32(data=b'data')
2918445923
Other Python implementations can, of course, reproduce the CPython APIs
manually, but this goes against the spirit of PEP 399[12] that intends
to avoid duplication of effort by mandating that all modules added to
Python's standard library must have a pure Python implementation
with the same interface and semantics.
A final argument in favor of positional-only arguments is that they
allow some new optimizations like the ones already present in the
argument clinic since said parameters must be passed in strict order.
For instance, CPython's internal METH_FASTCALL calling convention
has been recently speciallized for functions with positional-only
parameters to eliminate the cost for handling empty keywords. Similar
performance improvements can be applied when creating the evaluation
frame of Python functions thanks to positional-only parameters.
The Current State Of Documentation For Positional-Only Parameters
The documentation for positional-only parameters is incomplete and
inconsistent:
- Some functions denote optional groups of positional-only arguments
by enclosing them in nested square brackets.[13] - Some functions denote optional groups of positional-only arguments
by presenting multiple prototypes with varying numbers of
arguments.[14] - Some functions use both of the above approaches.[15][16]
One more important idea to consider: currently in the documentation
there is no way to tell whether a function takes positional-only
parameters. open()
accepts keyword arguments, ord()
does not, but
there is no way of telling just by reading the documentation.
Syntax And Semantics
From the "ten-thousand foot view", and ignoring *args
and **kwargs
for now, the grammar for a function definition currently looks like
this:
def name(positional_or_keyword_parameters, *, keyword_only_parameters):
Building on that perspective, the new syntax for functions would look
like this:
def name(positional_only_parameters, /, positional_or_keyword_parameters,
*, keyword_only_parameters):
All parameters before the /
are positional-only. If /
is not
specified in a function signature, that function does not accept any
positional-only parameters. The logic around optional values for
positional-only arguments remains the same as for positional-or-keyword
arguments. Once a positional-only argument is provided with a default,
the following positional-only and positional-or-keyword arguments need
to have defaults as well. Positional-only parameters that do not have a
default values are required positional-only parameters. Therefore the
following are valid signatures:
def name(p1, p2, /, p_or_kw, *, kw):
def name(p1, p2=None, /, p_or_kw=None, *, kw):
def name(p1, p2=None, /, *, kw):
def name(p1, p2=None, /):
def name(p1, p2, /, p_or_kw):
def name(p1, p2, /):
While the followings are not:
def name(p1, p2=None, /, p_or_kw, *, kw):
def name(p1=None, p2, /, p_or_kw=None, *, kw):
def name(p1=None, p2, /):
Origin of the ā/ā as a separator
Using the "/" as a separator was initially proposed by Guido van
Rossum in 2012[17] :
Alternative proposal: how about using '/' ? It's kind of the
opposite of '*' which means "keyword argument", and '/' is not
a new character.
Full grammar specification
A draft of the proposed grammar specification is:
new_typedargslist:
tfpdef ['=' test] (',' tfpdef ['=' test])* ',' '/' [',' [typedargslist]] | typedargslist
new_varargslist:
vfpdef ['=' test] (',' vfpdef ['=' test])* ',' '/' [',' [varargslist]] | varargslist
It will be added to the actual typedargslist
and varargslist
, but
for more relaxed discussion it is presented as new_typedargslist
and
new_varargslist
. Also, notice that using a construction with two new
rules (new_varargslist and new_varargslist) is not possible with the
current parser as a rule is not LL(1). This is the reason the rule needs
to be included in the existing typedargslist and varargslist (in the
same way keyword-only arguments were introduced).
Implementation
An initial implementation that passes the CPython test suite is
available for evaluation[18].
The advantages of this implementation involve speed, consistency with
the implementation of keyword-only parameters as in PEP 3102 and a
simpler implementation of all the tools and modules that will be
impacted by this change.
Rejected Ideas
Do Nothing
Always an option, just not adding it. It was considered though that the
benefits of adding it is worth the complexity it adds to the language.
After marker proposal
A complaint against the proposal is the fact that the modifier of the
signature impacts the tokens already passed.
This might make it confusing to users to read functions with many
arguments. Example:
def really_bad_example_of_a_python_function(fist_long_argument, second_long_argument,
third_long_argument, /):
It is not until reaching the end of the signature that the reader
realises the /
, and therefore the fact that the arguments are
position-only. This deviates from how the keyword-only marker works.
That said we could not find an implementation that would modify the
arguments after the marker, as that will force the one before the marker
to be position-only as well. Example:
def (x, y, /, z):
If we define that /
makes only z position-only, it will not be
possible to call x and y via keyword argument. Finding a way to work
around it will add confusion given that at the moment keyword arguments
cannot be followed by positional arguments. /
will, therefore, make
both the preceding and following parameters position-only.
Per-argument marker
Using a per-argument marker might be an option as well. The approach
adds a token to each of the arguments that are position only and
requires those to be placed together. Example:
def (.arg1, .arg2, arg3):
Note the dot on arg1 and arg2. Even if this approach might look easier
to read, it has been discarded as /
goes further in line with the
keyword-only approach and is less error-prone.
Some libraries use leading underscore[19] to mark those arguments as
positional-only.
Using decorators
It has been suggested on python-ideas[20] to provide a decorator
written in Python as an implementation for this feature. This approach
has the advantage that keeps parameter declaration more easy to read but
also introduces an asymmetry on how parameter behaviour is declared.
Also, as the /
syntax is already introduced for C functions, this
inconsistency will make it more difficult to implement all tools and
modules that deal with this syntax including but not limited to, the
argument clinic, the inspect module and the ast module. Another
disadvantage of this approach is that calling the decorated functions
will be slower than the functions generated if the feature was
implemented directly in C.
Thanks
Credit for most of the content of this PEP is contained in Larry
Hastingsās PEP 457.
Credit for the use of '/' as the separator between positional-only and
positional-or-keyword parameters go to Guido van Rossum, in a proposal
from 2012.[21]
Credit for discussion about the simplification of the grammar goes to
Braulio Valdivieso.
Copyright
This document has been placed in the public domain.
https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs ā©ļø
https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaln.html ā©ļø
http://docs.python.org/3/library/functions.html#func-range ā©ļø
https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs ā©ļø
https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaln.html ā©ļø
https://mail.python.org/pipermail/python-ideas/2016-January/037874.html ā©ļø
https://mail.python.org/pipermail/python-ideas/2017-February/044888.html ā©ļø
Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html ā©ļøhttp://docs.python.org/3/library/curses.html#curses.window.border ā©ļø
http://docs.python.org/3/library/functions.html#func-range ā©ļø
http://docs.python.org/3/library/curses.html#curses.window.addch ā©ļø
Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html ā©ļøhttps://mail.python.org/pipermail/python-ideas/2018-September/053319.html ā©ļø
https://mail.python.org/pipermail/python-ideas/2017-February/044888.html ā©ļø
Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html ā©ļø