Should `None` defaults for optional arguments be discouraged?

[mdickinson (Mark Dickinson)]

I recently merged a PR authored by Sergey Kirpichev that fixes inspect.signature (which previously failed with a ValueError) for math.log and cmath.log. A side-effect of that change is that math.log and cmath.log now accept the Python value None for the base argument. That is:

>>> from math import log
>>> log(2.3, None)
0.8329091229351039

Previously, this was a TypeError.

In comments on the original issue, Raymond Hettinger alludes to the new API as being “damaged”:

there seems to be willingness to damage the function rather than on improving the capabilities of signature objects […]

and

people should be working on that problem rather than damaging the underlying functions […]

I’m not seeing the damage here, and accepting None seems to me like a reasonable trade-off for the benefit of having usable signatures. The None default is a common idiom, and at least one other function in the math module already accepts None in this way (as do many non-math functions, of course):

>>> from math import perm
>>> perm(10, None)
3628800

More generally, I’d expect that having functions with signatures expressible in standard form would aid consistency and compatibility with other Python implementations, as well as helping tools that need to do inspection of signatures for one reason or another.

Is there a general principle that we should avoid these sorts of changes? What are the downsides of allowing None in this kind of situation?

[uranusjr (Tzu-ping Chung)]

In theory, I wish arg being optional is less expressed with arg=None, but without a good way to express sentinels^[1], None is the most reasonable compromise to me.

---

1. Sentinel values in the stdlib ↩︎

[pitrou (Antoine Pitrou)]

@mdickinson

log(2.3, None)

This looks more surprising to me than simply log(2.3). The latter I parse immediately, the former it takes some seconds to wonder what that None is for.

So in this particular case of well-known math functions I think that Raymond is right.
That doesn’t mean that passing None can’t be useful (for example it probably makes writing proxies/wrappers easier), but it doesn’t result in very readable code when done directly as in your example.

[mdickinson (Mark Dickinson)]

@pitrou

This looks more surprising to me than simply log(2.3). The latter I parse immediately, the former it takes some seconds to wonder what that None is for.

Right - I wouldn’t expect anyone to be writing code that way in practice, and it definitely wouldn’t be the recommended way to write a log with default base; the change is simply that that’s now permitted.

[mdickinson (Mark Dickinson)]

@mdickinson

it definitely wouldn’t be the recommended way to write a log with default base

To give an analogy, it’s similar to the built-in round function: it’s already permitted to write round(2.575, None), but I wouldn’t expect anyone to do so in practice, and I’ve seen no evidence that this permission causes people to deliberately write their round calls in that form. It seems to have been pretty harmless in the round case, and I can’t imagine why it would be any less harmless in the math.log case.

>>> round(3.25, None)
3

[arhadthedev (Oleg Iarygin)]

If the single-argument version means a natural logarithm (implying base=math.e), can we declare log as follows and not allow None?

log(x, base=math.e)

[storchaka (Serhiy Storchaka)]

I concur with Raymond. We should accept that inspect.signature() is not able to represent signatures of all extension functions (for example dict.pop, constructors of int, str).

None is convenient as a default value in most functions implemented in Python, but it is not so conventional for functions implemented in C (it is not accepted as optional int or double argument), and in some cases passing None and not passing an argument both have valid but different semantic (for example in dict.pop).

The correct solution is to implement support of alternate signatures. For example:

dict.pop(key, /)
dict.pop(key, default, /)

str()
str(object, /)
str(bytes, /, encoding, errors='strict')
str(bytes, /, errors)

[pitrou (Antoine Pitrou)]

As a sidenote, I don’t understand why math.perm’s second argument is optional.

[mdickinson (Mark Dickinson)]

@pitrou

I don’t understand why math.perm’s second argument is optional.

For context, there’s some discussion at One argument form of math.perm() · Issue #81359 · python/cpython · GitHub

[guido (Guido van Rossum)]

Contrary to most reactions, I find it natural and desirable to accept None as an alternative to omit an optional parameter in a wide variety of APIs. Not because I like to read or write log(2.3, None) but because I like to be able to define a simple wrapper, e.g.

def my_log(x, base=None):
    # <extra stuff here>
    return math.log(x, base)

This allows both my_log(x) and my_log(x, base) to be called, and my_log() doesn’t have to hard-code knowledge about the default base. While it is possible to write a wrapper that accepts an optional extra argument and passes that on only when present, it is uglier and harder to read, e.g.

def my_log(x, *args):
    assert len(args) in (0, 1))
    # <extra stuff here>
    return math.log(x, *args)

This is slower too (even if you take out the assert) because handling *args takes a slower path in the interpreter. And why should you have to do it this way?

(EDIT: The version with *args also makes it more awkward to access base in the wrapper in case the “extra stuff” wants to intercept a certain base.)

[gpshead (Gregory P. Smith)]

I agree with Guido here. There is nothing wrong with accepting None as a proxy for “the default value” on arguments where there otherwise would never be a meaningful interpretation of None.

There is very minor consequence for static analysis in that it cannot infer that the value of a variable being passed into such an API must not be None after the call as the type signature of the function must naturally be declared as float|None. But this really doesn’t matter in practice.

Even when writing pure Python code, it may seem easier to write a function with its default value in the parameter list at first… But when it is something commonly wrapped or overridden, it can be worth the extra hoop to use None as the default and add an x = "default" if x is None in the method. Just to make the lives of other wrappers or overriders easier to avoid redeclared default copying or wacky conditional hoops.

[gpshead (Gregory P. Smith)]

One thing we do need (which I believe has been noted above) is a way to get the type signature showing up in the docstring (and thus IDE help text as Raymond noted in the issue) to display the underlying default.

That None is accepted is more of an implementation detail, not the way you want to document the API.

[storchaka (Serhiy Storchaka)]

This does not work for a dict.pop() and range(). It would be nice to have either a special syntax for optional parameters without default value, e.g:

def mapping_pop(mapping, key, default=?):
    try:
        res = mapping[key]
    except KeyError:
        if not isset default:
            raise
        res = default
    else:
        del mapping[key]
    return res

or a way to overload function by the number of arguments, e.g:

def mapping_pop(mapping, key):
    res = mapping[key]
    del mapping[key]
    return res

@overload
def mapping_pop(mapping, key, default):
    try:
        res = mapping[key]
    except KeyError:
        res = default
    else:
        del mapping[key]
    return res

The former option may be more convenient in many simpler cases, but the latter option is more powerful.

[skirpichev (Sergey B Kirpichev)]

@arhadthedev

If the single-argument version means a natural logarithm (implying base=math.e), can we declare log as follows and not allow None?

log(x, base=math.e)

It was declared in this way before and this is wrong: it makes an impression, that this is an ordinary case for a generic base (“calculated as `log(x)/log(base)” (c) rst docs).

But this is not the case: math.e is a float number, not a real number e. Same case we have, for example, in the exp() function, and that is noted in the rst docs: “Return e raised to the power x, where e = 2.718281… is the base of natural logarithms. This is usually more accurate than math.e ** x or pow(math.e, x).”. (BTW, for same reason some other docstrings/rst docs looks wrong, e.g. the exp: “Return e raised to the power of x.” Better: “Return the exponential of x.”)

The None seems to fit this case well just as for the perm: in the later function the default value depends on the first argument. In the math.log case - we can’t represent the default value by some other standard type (that might be possible with a come CAS, like the sagemath, but not with the current stdlib).

[skirpichev (Sergey B Kirpichev)]

@gpshead

One thing we do need (which I believe has been noted above) is a way to get the type signature showing up in the docstring (and thus IDE help text as Raymond noted in the issue) to display the underlying default.

How about this fix for the math.log docstring (and for cmath.log):
Return the logarithm of x to the given base or the natural logarithm of x
(with the base=None patch, of course)?

That’s how look IDLE with this:

BTW @mdickinson, here is a new pr, that address some minor issues like above.

[skirpichev (Sergey B Kirpichev)]

@storchaka

The correct solution is to implement support of alternate signatures. For example:

dict.pop(key, /)
dict.pop(key, default, /)

Unfortunately, such syntax (like c++ function overloading) isn’t possible with the CPython interpreter. I.e. I can’t reproduce pure-Python function with the current signature of the math.log, right?

[tjreedy (Terry Jan Reedy)]

I sympathize with Raymond’s objection, but also Mark’s unease.

For round, the absence of the 2nd argument is not the same as the presence of the default. A signature with ndigits=0 would be misleading.

>>> round(3.33)
3
>>> round(3.33, 0)
3.0

The summary line does not explain this difference, but the rest of the docstring does.

For math.log, the difference between no base and base=math.e is more theoretical than practical as long as log(math.e) == 1.0. But the unease is that (I think) it does not have to be, and aside from the time waste, we would not want an implementation of log to actually divide by log(e).

[hugovk (Hugo van Kemenade)]

See also this thread:

Signatures, a call to action

I saw the other thread on math.log() and wanted to start a new one to shift the focus to what I think is the underlying problem than needs to be solved. If the playful story telling style doesn’t fit your tastes, please try and look past the style and focus on the substance of the post. I tried rewriting this a few times but found that the parallel construction form of comparison and contrast best communicated where work needs to be done. There once was little scripting language called Py…

[SnoopJ (James Gerity)]

I’m not seeing the damage here

In my reading of the issue, the “damage” is the expansion of the type of the base argument from “always a float” to “a float or maybe not-a-float,” which seems like a more “zoomed-in” concern than the question this thread is asking.

For whatever a user’s opinion matters here, I don’t see any reason that the stdlib should in general discourage the use of None to indicate an optional argument, but there may be individual cases where it muddies the waters.

[CAM-Gerlach]

This can work both ways, though, as even if something is syntactically a float, int, etc (like using 0, -1, or even math.e for a flag value, as some has proposed), semantically the meaning and behavior is a special case, different from that with any other int/float. Using None (or a dedicated sentinel type) ensures the syntax clearly communicates that this is a semantic special case, and can allows type checkers and other tools to statically verify it is being handled inside the function (or wrappers) and better introspect usage outside of it.