About type-conversion special methods

Core Development
1

There are special methods used in implicit and explicit type conversions. __str__, __bytes__, __int__, __index__, __float__, __complex__.

  • __index__, __float__ and __complex__ – for implicit convertion to int, float and complex. Most of the C API which convert these Python types to corresponding C types accept also objects that implement these special methods and use them if necessary.
  • __str__, __bytes__ and __int__ – only for explicit conversion by str(), bytes() and int().

Now, there are some questions: when to the special method and how to handle its result.

Look, for example, at PyFloat_AsDouble(). If the argument is a Python float, it gets the C double value directly from the ob_val field of the PyFloatObject structure. This works also for float subclasses, becaus ethey have the same structure. If the argument if not a Python float, then the __float__ method is used, and the C double value is obtained from its result. It can also fall back to using __index__, but this is not relevant here.

If the result of __float__ is not a Python float, then this is error. __float__ is not called recursively, this could be unsafe.

Note two things:

  1. __float__ is not called for float subclasses. It is not necessary, we already have PyFloatObject. Calling __float__ would be a waste of time. It is even more true for long integers, strings, etc.
  2. Returning a float subclass from __float__ is not an error, fro the same reason. We only need to read PyFloatObject.ob_val, it is the same for subclasses as for exact float.

However, there was one problem. If str() just returns the result of __str__, it may return a subclass of str, and this is not whet we expect. On Python level we expect that str() always returns an exact str, this is the way to convert the object to exact str. There was a simiular issue with operator.index() and __index__.

The decision taken to solve this problem was two-step. If the result of the special methos is not an exact base type, but its subtype:

  1. Convert it to the base type ignoring its type and only using its content.
  2. Emit a deprecation warning.

BTW, __str__ and __bytes__ were overlooked, so we still have a problem of returning an str subclass from str(), bytes() and repr().

I think the deprecation was wrong. Silent conversion in step 1 was enough (it can be omitted if we need not a Python object, but just its content, as in PyFloat_AsDouble()). Deprecation just adds an inconvenience for users. For example, let we have an int-like object which implents __index__ that returns a calculated value. If the value happens to be not exact int, but int subclass (for example bool or IntEnum), the user is forced to convert it to int before returning from __index__. In worst case they will use int() for conversion, which can silently truncate the non-integer result, hiding the real bug. In any case, explicit conversion to int takes time and creates unnecesary object. Even if the exact int is needed (for example if we use operator.index()), it is faster to create in C than call operator.index() or int() in Python. And in most cases it should only be converted to C integer, so no need to create an intermediate Python object.

So my suggestion – just remove the deprecation. The code that already worked will continue to work. No new errors will occur. Most users will not notice anything. But the future user code could be cleaner and faster.

3 Likes
2

This is not true in case of PyNumber_Float() (equivalent of float(obj)). Here we call the __float__ dunder for float subclasses too. Also, we have here a specific check for “broken” float subclasses:

    /* A float subclass with nb_float == NULL */
    if (PyFloat_Check(o)) {
        return PyFloat_FromDouble(PyFloat_AS_DOUBLE(o));
    }

see also gh-112636: remove check for float subclasses without nb_float by skirpichev · Pull Request #112637 · python/cpython · GitHub

I would rather support such decision.

In principle, we can imagine that __float__ got overridden in the subclass and it returns something different from the ob_val, say:

>>> class FloatSpam(float):
...     def __float__(self):
...         return 42.
...         
>>> x = FloatSpam(1.25)
>>> float(x)
42.0
>>> float.from_number(x)
1.25

Though, it’s just a broken float subclass, isn’t?

Edit:
I didn’t traced down all initial arguments for deprecation of subclasses. Here is the discussion thread that seems to be related. As far as I can understand, the main reason to reject subclasses is API. Docs says:

object.__float__(self ) […] Called to implement the built-in functions complex(), int() and float(). Should return a value of the appropriate type.

I.e. float() essentially just call the __float__() dunder, it’s all (modulo type-checking of it’s output). In proposed version we have to document how the float constructor process the return value of the dunder. See this Guido’s comment.

CC @mdickinson

3

Maybe it will be useful to emit warnings/errors for those classes? One can use _float_ dunder or float subclassing but not both.

4

Sorry, can you provide an example? Where you would like to emit errors/warnings?

5

I’m not sure about existing mechanisms on related topic. But basically defining this dunder method for the subclasses of float can emit warning/error.

6

This topic is not about deprecating using __float__() for float’s subclasses, but rather about return value of such dunder methods.

7

I understand this. Yes, it’s a bit off-topic, but not very much, IMO.

I’m not sure about removing deprecation warning.
In general, arguments look fine.
But it feels a little awkward.

8

Honestly, I don’t understand well this well. It’s quite subtle.

I’m more comfortable with examples (see below).

IMO __float__() should only return exact float instance, so the DeprecationWarning makes sense and we should now convert it to an error. Otherwise, the result is unclear and there are bad surprises, especially if the float subclass has a __float__ method. If __float__() returns a float subclass, __float__() is not called again which gives surprising results: see example 1 and example 2.

Is there any project in the wild which is impacted by the DeprecationWarning?

numpy is not affected by this issue:

$ python
>>> import numpy

>>> f64=numpy.float64(1.5)
>>> isinstance(f64, float)
True
>>> type(f64).__bases__
(<class 'numpy.floating'>, <class 'float'>)
>>> type(f64.__float__())
<class 'float'>

>>> f16=numpy.float16(1.5)
>>> isinstance(f16, float)
False
>>> type(f16.__float__())
<class 'float'>

Example 1:

class MyFloat(float):
    def __float__(self):
        return 42.

class NonFloat:
    def __init__(self):
        self.value = MyFloat(1.5)

    def __float__(self):
        return self.value

# (A) __float__() returns float
print(float(MyFloat(1.5)))
print()

# (B) __float__() returns MyFloat(float): emit DeprecationWarning
print(float(NonFloat()))

Output 1:

42.0

x.py:16: DeprecationWarning: NonFloat.__float__ returned non-float (type MyFloat).  The ability to return an instance of a strict subclass of float is deprecated, and may be removed in a future version of Python.
  print(float(NonFloat()))
1.5

It’s quite surprising that I get 42.0 in case (A) but 1.5 in case (B). It looks inconsistent to me.

Example 2:

class MyFloat(float):
    def __float__(self):
        return MyFloat(42)

# __float__() returns MyFloat(float): emit a DeprecationWarning
print(float(MyFloat(1.5)))

Output 2:

x.py:6: DeprecationWarning: MyFloat.__float__ returned non-float (type MyFloat).  The ability to return an instance of a strict subclass of float is deprecated, and may be removed in a future version of Python.
  print(float(MyFloat(1.5)))
42.0

It works as expected but it’s a little bit strange that the float() calls __float__() but then doesn’t call __float__() again on the float subclass. I’m not sure if I expect 1.5 or 42 result in this case :slight_smile: The deprecation warning sounds like “hey, something is wrong”: I agree :slight_smile:

If the float subclass has no __float__() method, obviously, things are simpler.

Example 3:

class MyFloat(float):
    pass

class NonFloat:
    def __init__(self):
        self.value = MyFloat(1.5)

    def __float__(self):
        return self.value

# (A) __float__() returns float
print(float(MyFloat(1.5)))
print()

# (B) __float__() returns MyFloat(float): emit DeprecationWarning
print(float(NonFloat()))

Output 3:

1.5

x.py:16: DeprecationWarning: NonFloat.__float__ returned non-float (type MyFloat).  The ability to return an instance of a strict subclass of float is deprecated, and may be removed in a future version of Python.
  print(float(NonFloat()))
1.5

@storchaka says that the DeprecationWarning doesn’t bring any value in this case. The conversion should be done silently.

Example 4:

class MyFloat(float):
    pass

# __float__() returns float
print(float(MyFloat(1.5)))

Output 4:

1.5
1 Like
9

This proposal doesn’t change this.

But instead of issuing a warning or raising an exception — Serhiy proposed to use “known value” of the base class type, available for a subclass. I.e. just use ob_val for float or float subclasses. So, in your examples we get same values, but all of the float type. And no warnings.

In short, if you return a subclass instance in the __float__() dunder — this value should be processed by consumers of this API (e.g. float() constructor) to get first the base class value.

My objections rather (1) explicit is better than implicit, (2) I’m not sure this will be easy to document, especially for alternative Python implementations.

This is not something, proposed to change.

On another hand, assuming this proposal, it might be then not clear why e.g. in your 1st example PyNumber_Float() shouldn’t ignore provided __float__() method and just use ob_val…