About type-conversion special methods

Core Development
1

There are special methods used in implicit and explicit type conversions. __str__, __bytes__, __int__, __index__, __float__, __complex__.

  • __index__, __float__ and __complex__ – for implicit convertion to int, float and complex. Most of the C API which convert these Python types to corresponding C types accept also objects that implement these special methods and use them if necessary.
  • __str__, __bytes__ and __int__ – only for explicit conversion by str(), bytes() and int().

Now, there are some questions: when to the special method and how to handle its result.

Look, for example, at PyFloat_AsDouble(). If the argument is a Python float, it gets the C double value directly from the ob_val field of the PyFloatObject structure. This works also for float subclasses, becaus ethey have the same structure. If the argument if not a Python float, then the __float__ method is used, and the C double value is obtained from its result. It can also fall back to using __index__, but this is not relevant here.

If the result of __float__ is not a Python float, then this is error. __float__ is not called recursively, this could be unsafe.

Note two things:

  1. __float__ is not called for float subclasses. It is not necessary, we already have PyFloatObject. Calling __float__ would be a waste of time. It is even more true for long integers, strings, etc.
  2. Returning a float subclass from __float__ is not an error, fro the same reason. We only need to read PyFloatObject.ob_val, it is the same for subclasses as for exact float.

However, there was one problem. If str() just returns the result of __str__, it may return a subclass of str, and this is not whet we expect. On Python level we expect that str() always returns an exact str, this is the way to convert the object to exact str. There was a simiular issue with operator.index() and __index__.

The decision taken to solve this problem was two-step. If the result of the special methos is not an exact base type, but its subtype:

  1. Convert it to the base type ignoring its type and only using its content.
  2. Emit a deprecation warning.

BTW, __str__ and __bytes__ were overlooked, so we still have a problem of returning an str subclass from str(), bytes() and repr().

I think the deprecation was wrong. Silent conversion in step 1 was enough (it can be omitted if we need not a Python object, but just its content, as in PyFloat_AsDouble()). Deprecation just adds an inconvenience for users. For example, let we have an int-like object which implents __index__ that returns a calculated value. If the value happens to be not exact int, but int subclass (for example bool or IntEnum), the user is forced to convert it to int before returning from __index__. In worst case they will use int() for conversion, which can silently truncate the non-integer result, hiding the real bug. In any case, explicit conversion to int takes time and creates unnecesary object. Even if the exact int is needed (for example if we use operator.index()), it is faster to create in C than call operator.index() or int() in Python. And in most cases it should only be converted to C integer, so no need to create an intermediate Python object.

So my suggestion – just remove the deprecation. The code that already worked will continue to work. No new errors will occur. Most users will not notice anything. But the future user code could be cleaner and faster.

3 Likes
2

This is not true in case of PyNumber_Float() (equivalent of float(obj)). Here we call the __float__ dunder for float subclasses too. Also, we have here a specific check for “broken” float subclasses:

    /* A float subclass with nb_float == NULL */
    if (PyFloat_Check(o)) {
        return PyFloat_FromDouble(PyFloat_AS_DOUBLE(o));
    }

see also gh-112636: remove check for float subclasses without nb_float by skirpichev · Pull Request #112637 · python/cpython · GitHub

I would rather support such decision.

In principle, we can imagine that __float__ got overridden in the subclass and it returns something different from the ob_val, say:

>>> class FloatSpam(float):
...     def __float__(self):
...         return 42.
...         
>>> x = FloatSpam(1.25)
>>> float(x)
42.0
>>> float.from_number(x)
1.25

Though, it’s just a broken float subclass, isn’t?

Edit:
I didn’t traced down all initial arguments for deprecation of subclasses. Here is the discussion thread that seems to be related. As far as I can understand, the main reason to reject subclasses is API. Docs says:

object.__float__(self ) […] Called to implement the built-in functions complex(), int() and float(). Should return a value of the appropriate type.

I.e. float() essentially just call the __float__() dunder, it’s all (modulo type-checking of it’s output). In proposed version we have to document how the float constructor process the return value of the dunder. See this Guido’s comment.

CC @mdickinson

3

Maybe it will be useful to emit warnings/errors for those classes? One can use _float_ dunder or float subclassing but not both.

4

Sorry, can you provide an example? Where you would like to emit errors/warnings?

5

I’m not sure about existing mechanisms on related topic. But basically defining this dunder method for the subclasses of float can emit warning/error.

6

This topic is not about deprecating using __float__() for float’s subclasses, but rather about return value of such dunder methods.

7

I understand this. Yes, it’s a bit off-topic, but not very much, IMO.

I’m not sure about removing deprecation warning.
In general, arguments look fine.
But it feels a little awkward.