Options for a long term fix of the special case for float/int/complex

Typing
145

Adding StrictFloat and also # type: strict_float seems confused.

It could be worth adding a future import as well so that the message is:

  • Use from __future__ import strict_float in 3.15 upwards.
  • Use # type: strict_float in older versions.
  • In a future Python version strict_float will be the default.

The future import would do nothing at runtime but it sends the message that this is officially expected to be default behaviour in future.

If that sequence is the situation though then I don’t see why anyone would use StrictFloat because the future way is to write float and you can use that already if you add # type: strict_float.

The proposal to add StrictFloat makes sense if the plan is not for strict_float to become the default behaviour in future but I think in that case it would be hard to argue for changing anything else rather than just adding StrictFloat.

These things are not entirely mutually exclusive but I think if you knew confidently whether or not you were going to make float mean float by default in future then you would not come up with a plan to add both StrictFloat and # type: strict_float.

1 Like
146

I’m going to be direct here, I really dislike that there isn’t a policy for how to handle “breaking” changes for typing. It’s not a language level break and has no effect on runtime, so the normal CPython policies don’t seem to cleanly apply. This is especially so when typechecking was made intentionally an “external system” that can iterate independently.

To have the ability to fix issues be more rigid and painful than just a standard deprecation period to me indicates that the specification being separate isn’t serving a good purpose.

Which leads me to what I think the best two options we have are, and I can’t reccomend one over the other since typechecker authors haven’t contributed to this thread in a way that indicates which they would prefer. I don’t see any option that leaves a basic type like float ambiguous in the long term as worth pursuing.[1] (say, 3.14 EOL as a target date for latest I think reasonable to push this as process)

Lean into the path that was picked for specification

Just make the change in the specification. Let typecheckers decide how to handle it. This is the path that matches the seperation between language and typing that was intentionally made, and there are already parallels to this with mypy’s bytes promotion rules being planned to change. The path forward for typecheckers is simpler here, they only have to support one set of rules at a time, not keep a transition rule/setting unless they opt into it for their users. The path forward for users if their typechecker doesn’t is likely “pin your typechecker till you and your dependencies (or stubs for them) are updated for this” or enable/disable flags if it is handled by your typechecker like mypy is handling bytes promotion deprecation.

Take the time to create a deprecation policy for typing

I don’t want to push a design here. I also don’t like the idea of editions and mandating that typecheckers maintain multiple sets of rules, that’s a lot of implementation complexity to mandate on them. If the major typechecker authors all agree they prefer this path, then so be it, but I can’t in good conscience ask them to maintain editions or even per-file flags for this, especially since this isn’t something I’m going to spend time to contribute to any of them. [1:1]

  1. To be very blunt, I’m more inclined to work on a competing typechecker that’s intentionally non-compliant with the standards that prevent soundness at this point and show that it’s possible to have soundness without it being too hard to use. ↩︎ ↩︎

7 Likes
147

You need a way to express the idea of “float and not int” at a single call site without forcing the entire file/module/package to update to strict float formatting. The alternative is that “strict float” mode becomes viral: client code would be unable to even express 'list[float]` when calling a strict-float library function unless they update the entirety of their own code base. Libraries that implement strict float typing would then get bug reports complaining that they can’t be used without errors by clients that are not yet ready to implement strict float themselves.

I’d could live with a line-specific typing directive to use strict float interpretation on a specific line, but this seems more confusing and prone to errors than just creating a new type.

To try to make a concrete example:

# library.py
# type: strict_float

def lib_func(x: list[float]) -> None:
    ...

# client_code.py
from typing import reveal_type
import library

def client_func(x: list[float]) -> None:
    reveal_type(x)  # list[float | int]
    # Runtime checking is required
    for item in x:
        if not isinstance(item, float):
            raise TypeError

    library.lib_func(x)  # Type error: Argument of type `list[float | int]` cannot be assigned to a type of `list[float]`.

Without some escape hatch, it’s impossible for client.py to eliminate the type error - there’s no syntax to express list[float & ~int] without client.py also implementing strict_float mode.

Using line-scoped directives, client.py changes to this:

# client_code.py
from typing import TypeGuard
import library

def is_strict_float_list(arg: list[float | int]) -> TypeGuard[list[float]]:  # type: strict_float
    for item in arg:
        if not isinstance(item, float):
            return False
    return True


def client_func(x: list[float]) -> None:
    reveal_type(x)  # list[float | int]
    if not is_strict_float_list(x):
        raise TypeError

    reveal_type(x)  # list[float]
    library.lib_func(x)

Using a StrictFloat type, you end up with:

# client_code.py
from typing import TypeGuard, StrictFloat, reveal_type
import library

def is_strict_float_list(arg: list[float]) -> TypeGuard[list[StrictFloat]]:
    for item in arg:
        if not isinstance(item, float):
            return False
    return True


def client_func(x: list[float]) -> None:
    reveal_type(x)  # list[float | int]
    if not is_strict_float_list(x):
        raise TypeError

    reveal_type(x)  # list[float]
    library.lib_func(x)

This is a simple enough example that both work OK. The main difference is that the line-scoped typing directive required changing both annotations on is_strict_float_list, which isn’t too bad. Things get a bit more confused if is_strict_float_list has a multi-line definition:

def is_strict_float_list(
        arg: list[float],
) -> TypeGuard[list[float]]:  # type: strict_float
    for item in arg:
        if not isinstance(item, float):
            return False
    return True

Now we have a use of an old-style “loose” float and a strict float in the same function definition! Using a StrictFloat token at least means there is never ambiguity about what float means within an annotation for any single python file.

Yes: adding a StrictFloat type opens the possibility that some users may opt to simply never enable stict_float mode on their type checkers. This requires concerted messaging from the typing community as a whole to say “strict_float is the future, get on board.” The alternatives are either more complicated or user-hostile.

Part of the challenge here is that the goal is to move from a situation where it is literally impossible to express the type “float and not int” to making it the default meaning of one of the most common types. I don’t see any way to navigate that transition smoothly without making that type expressible on a per-annotation basis.

148

Presumably a type checker would understand that float(obj) returns a real float so e.g. lib_func([float(a) for a in x]) would work. That actually does convert int to float which may be what is actually needed in this situation. In the other direction a shallow copy of the list is enough:

def lib1(x: list[str]):
    pass

def lib2() -> list[str]:
    return []

def client1(x: list[str | int]):
    lib1([str(i) for i in x])

def client2() -> list[str | int]:
    return list(lib2())

I see your point that StrictFloat would still be useful though. You might actually need to share a mutable list across the boundary without converting or copying. There are other general escape hatches as well like list[Any] or # type: ignore but I’m sure many would prefer StrictFloat if available.

It could still be contagious if it makes more sense for client_func to change its own parameter to StrictFloat and then that would propagate backwards to callers and so on. I can imagine wanting to chase this all the way back rather than doing this conversion right before calling lib_func but that is why I would just set strict_float everywhere.

It is hard to know what the more problematic issues would be. I don’t think list[float] is a common parameter annotation for public API although. The only examples I’ve seen are a few cases of list[int] | list[float] in typeshed’s tkinter stubs.

I have just grepped some scientific/mathematical libraries and I haven’t actually seen list[float] being used as any part of any annotation outside of tests. The libraries I have checked are majority unannotated though. Returning vanilla float seems much less common than I expected and is likely not problematic anyway. I can see many cases where function parameters are annotated float but should really be SupportsFloat which is where the false positives and churn would be.

149

I’d love float to stop being float | int, but it seems gradual transition only makes sense with introduction of StrictFloat.

Otherwise inevitably there will be a very confusing period when you’re seeing float in function signature and it may mean multiple things and you’d have to go to source (search file for some # type: strict-float directive or check py.typed or check something else).

Here’s the plan I had in mind for the gradual transition:

  1. Introduce typing.StrictFloat (or OnlyFloat or other name, perhaps something even shorter) to allow everyone to annotate the actual floats right now. The most motivated users will adapt it and will help introducing it to standard library and other popular libraries, gradually spreading it across the community.

  2. Projects that adapt it, will explicitly include a flag to their pyproject.toml / py.typed / elsewhere that they support it (use-strict-floats=true), so linter or type checker could suggest replacing all float with StrictFloat | int.
    It shouldn’t change the way type checkers interpret usual float - it should be possible to reason about the code without knowing that internal setting to make transition as simple as possible. When user see annotation def test() -> float they shouldn’t go to the file to check if some setting is active.

  3. After some time, we’ll be able to see that StrictFloat is adapted enough (and maybe we find more caveats we’d need to be aware of when actually changing meaning of float) and we can start moving from float being float | int - we’ll be able to set a date after which old behaviour will be considered deprecated. E.g. linters and type checkers may warn you about float being used in annotations in all cases unless use-strict-floats=false is set to turn off the warnings.

  4. Then the day X comes and we say that StrictFloat is now deprecated too and it’s safe to be replaced with just float everywhere.

150

If it becomes at least optionally possible to have float means float as something like a per-file setting (even if it never becomes default) then I would not want to change float to StrictFloat anywhere. What I would want to do is keep annotations as float or change some to SupportsFloat or float | int as needed and then enable the setting that says float means float as soon as possible.

For one thing I would prefer to avoid a situation where the annotations get changed from float to StrictFloat and later changed back to float. Apart from needing to change the annotations twice though what I really don’t like about using StrictFloat is that I don’t want it to be visible to users anywhere even temporarily. There are various places where the annotations are visible:

  • IDEs show type annotations for both functions and the variables that are returned from functions.
  • Interactive help like help(func) or ??func in ipython shows annotations from the code.
  • HTML docs show annotations.

I want all those places to show actual float as float. I don’t want a situation where a user hovers over a variable that is unambiguously a float but their IDE displays it as StrictFloat:

# Library function:
def lib_func(x: float | int) -> StrictFloat: ...

# User code:
t = lib_func(1)
reveal_type(t) # I don't want users seeing StrictFloat

I’m sure some people in this discussion who are aware of the special case would find that confusing. I think most users are unaware of the special case though and would be more confused by seeing StrictFloat. Especially if the plan is to have float mean float by default one day then I want most users to never need to know about this ambiguity.

1 Like
151

Perhaps IDEs could show something like “float (strict)”. Type checkers that are also language servers already do a lot of special processing for this kind of hover information, I believe. So, special-casing this as well is probably not too much work.

152

I don’t think we need a strict float transition type.

Here’s a way to make this work without it.

# strict_aliases.py
# type: strict_float

type StrictFloatList = list[float]

# elsewhere, where there isn't strict_float use yet
from strict_aliases import StrictFloatList

If it’s based on where the type is defined, then it’s possible for libraries to introduce an alias if they want to not update a whole file at once, without making this something that every user of typing has to migrate twice.

153

I’m wondering now which of these two is better/easier to manage as a transition:

  • Having per-file or per-project or whatever settings so that float means different things in different files or codebases simultaneously under the type checker.
  • Having a global option that makes float mean float that you can add e.g. --strict-float when running a type checker and it applies to all codebases simultaneously.

If you have a global option then you don’t have any issues with mismatching types and invariance and you don’t need StrictFloat or anything else. Instead just some annotations like float would be overly restrictive in some cases when the option it enabled.

It would not be hard to turn on the global option and replace float with float | int in a few places until it type checks with the option either on or off. I think that the amount of actual change needed in annotations/code for this is a lot smaller than some might expect but it would be easier to explore that if you could turn the option on globally when running a type checker.

2 Likes
154

  • Having per some code unit option doesn’t allow anyone to reason about what someone meant by float without checking the code unit setting and keeping it in mind or using some other tool for it (e.g. IDE type hints or notes in documentation).

  • Having global option basically creates a situtation when there’s no transition period as type checker would assume everyone suppose to migrate to new float behaviour immediately.

In both cases it becomes less of a transition period but more of a confusion period, for this period things will be even more confusing than they currently are.

155

I think we are currently in the confusion period.

It is not clear what the annotations written now were intended to mean. The type checkers don’t handle float consistently. The types will in many cases be confused at runtime as well i.e. there will be actual runtime bugs where the types really are wrong.

It is very easy to make a mistake and accidentally return int:

def f(nums: list[float]) -> float:
    return sum(nums)

Did the author of this truly want to allow f([]) to be an int?

If I wrote that code then the answer is most definitely no: I wanted the inputs and output to be unambiguously float and not int. This is precisely why a type checker would be useful but current type checkers have allowed these things to be confused which means that the types really will be confused at runtime in many cases.

The parallel thread is trying to standardise the idea that float means float | int precisely because it is not a consistent behaviour of type checkers right now. I have only really tested mypy and pyright but I would say that pyright does consistently treat float as meaning float | int in most cases but mypy often does not. A lot of codebases use mypy as their CI type checker and don’t bother with pyright though.

Here is a difference:

def f(x: float):
    if not isinstance(x, float):
        # mypy thinks this is unreachable
        # pyright thinks it is int
        reveal_type(x)

x = 1.0
if not isinstance(x, float):
    # mypy thinks this is unreachable
    # pyright thinks this is int
    reveal_type(x)

In the first case you could say that if float means float | int then pyright is correct and mypy is wrong. However that means that mypy does not implement float means float | int correctly. In the second case it is unambiguously correct that the code is unreachable so mypy is correct and pyright is wrong.

Thinking that actually unreachable code is reachable is not necessarily a soundness hole but it shows that pyright is not consistently distinguishing between “this is a real float” and “this is annotated as float which means it is float | int” because if it did then it would know that the branch is unreachable.

Here is an example for Jelle’s unsoundness collection:

def func(x: int) -> str:
    y: float | str = x
    if not isinstance(y, float):
        return y
    else:
        return str(y)

This checks fine under mypy but not pyright.

These are the overloads involving int and float that I said are fundamental for many scientific and mathematical libraries and that don’t work because the special case causes unsafe overlap:

from typing import overload

class Int: ...
class Float: ...

@overload
def convert(x: int) -> Int: ... # type: ignore
@overload
def convert(x: float) -> Float: ...

def convert(x: int | float) -> Int | Float:
    if isinstance(x, int):
        return Int()
    elif isinstance(x, float):
        return Float()
    else:
        raise TypeError

reveal_type(convert(1))   # Int
reveal_type(convert(1.0)) # Float

Only pyright requires the type: ignore. According to mypy this type checks fine. In the other thread Jelle said that this was because of a bug in mypy implying that the intention would be to break this in future.

Both mypy and pyright infer the types of Int and Float correctly at the bottom. I have already written overloads like this and just used type: ignore in various places because in context the unsafe overlap practically never matters and it isn’t possible for a type checker to do much else useful if it can’t infer the most basic types. The other thread is proposing to standardise the idea that this code is wrong but this is not something that type checkers are currently consistent about.

We are in the confusion period right now. Type checkers do not handle float means float | int consistently. Codebases do not consistently use float as meaning either literally float or float | int. Actual floats and ints are mixed up in types at runtime in situations where it was not intended.

It is by checking codebases under --strict-float that you can resolve the confusion:

  • Type checkers can easily implement --strict-float consistently.
  • Codebases can be fixed so that they really have floats at runtime in the places where they should.
  • Annotations can be made accurate to their meaning under the --strict-float option while still being consistent in the default mode (writing float | int works either way if you want that).

It will take time to remove the confusion and you will never know when that process is complete. That is true of everything though in a world of gradual typing where you have things like Any in the typeshed for even the most basic types like int and float.

4 Likes
156

That’s a good point! I withdraw my suggestion that a StrictFloat type be added to the standard library.

1 Like
157

The problem with this approach that it disregards users who may be relying on float being float | int as unimportant, that it’s okay to suddenly change existing behavior everywhere - from day 1 of implementation of this everyone has to make their code comply to it to avoid confusing situations interacting with other libraries.

I understand it can be confusing right now already and, since this convention is implicit, type checkers don’t handle it perfectly in all situtations, but for users who were okay with how it worked previously (and it worked this way for a really long period of time), it’s just a source of new typing errors in yesterday’s perfectly working code.

So it’s an approach to maximize breakage to make sure transition goes as fast as possible.

158

I don’t think that this disregards those users. Firstly you don’t suddenly change existing behaviour for anyone by adding --strict-float as an optional flag. It would definitely need to be introduced as something that is off by default and time would be needed while annotations are changed before anyone could contemplate it becoming an on by default thing. During that time it is useful that you can test what it would be like by turning --strict-float on everywhere rather than separate libraries slowly sprinkling StrictFloat around.

There are not two groups of users here who do or do not depend on this: with no changes to type annotations everyone is relying on float meaning float | int in various places. The annotations for float itself in typeshed depend on the special case e.g.:

class float:
    def __add__(self, __x: float) -> float: ...

You would first need to change that:

class float:
    def __add__(self, __x: float | int) -> float: ...

Now those annotations are correct[1] with or without --strict-float. I assume that typeshed already has type tests that are run with different type checkers and configurations so this would be another configuration to test.

It would be pointless for anyone else to try out --strict-float without typeshed being changed to have consistent annotations for at least the float and complex classes. If that happens though then it becomes possible to try this with other codebases and get a more meaningful estimate of the actual change impact.

I think that the actual changes needed are much smaller than people might think and that often float | int will not be the correct replacement. Provided typeshed and perhaps some other basic dependencies are compatible with --strict-float I am sure that many codebases would use it regardless of it ever becoming the default much like some would use other options like mypy --strict.

I suspect though that type checkers and typeshed etc would not want to start any process of adding --strict-float and changing annotations to match unless it was agreed from the outset that the intended outcome was for it to become the default one day. That is the main difference between --strict-float and StrictFloat. Adding StrictFloat can make sense as an incremental improvement that makes minimal changes and could fix some issues now. If you see float means float as the eventual future though then it is clearly better just to go there directly with --strict-float rather than wasting time with StrictFloat.

  1. Here float.__add__ is actually one of the few places where float | int really is the exact correct type. Arithmetic with float needs to be not greedy so that other types can coerce float (1.0 + np.float64(3)). SupportsFloat is not right here because float will only coerce the exact int type. ↩︎

1 Like
159

Maybe we could have a strict typing with :: alongside with smooth typing using :, like :

def func(num1:: float=0., num2: float=1):
    # num1 is strictly float, num2 is whatever works as a float
    ...

This could preserve existing behavior, be used for automatic type guarding or static typing in the future, and serve as an hint to end users whether the type is indicative or required.
… And would apply to any type.

Like a function requiring a hashable sequence could strictly require a tuple :

def iterate_hashable_sequence(seq::tuple): ...
160

Iirc that has already been suggested a long time ago, but people said syntax like :: is unpythonic, and ambiguous.

161

There might be different syntaxes, probably a more pythonic one involves a new keyword only (that might be recognized only in typing statements).

def func(num1: only float=0., num2: float=1): ...

But whatever… Isn’t the problem addressed here only about a pinpointed case of the lack of strict typing on just the type float ? I think the proposal is not getting anywhere as is… but a generalized way of providing strict typing would solve this problem and fit new use cases and would not break anything.

1 Like
162

For me it’s important to have an easy way to type “Real” (currently float), and acceptable/useful to have a way to type “StrictFloat”.

Because the ease of use of functions is important, so calling a function as f(float_var=0) should remain possible. At the same time, it should be possible to write a function whilst thinking only about the primary use case.

If you want to transition away from a state where the type annotation float is ambiguous, I don’t see how you can do that without transitioning to a state where no-one type-hints with float. So you need both “Real”/“SupportsFloat” and “StrictFloat”/“OnlyFload”.

163

We got into the current situation due to a well-intentioned, but ultimately misguided, attempt at reflecting in typing the Python “numeric tower” philosophy of “int mostly working as float”. This philosophy mostly worked in pure Python (except a small number of cases like hex()), but the ecosystem has then changed with the massive rise in popularity of fastmath / tensorflow / whatever vector libraries in which things like list[float] need to mean just that, if an int sneaks in some of them are OK, some change behavior and some crash.

We can’t just ask people to add an if not isinstance(foo, float): foo = float(foo) to it. Not only is this a behavior change, but all that C / GPU cuda magic framework lives outside our control. Deprecations or re-interpreting existing (perhaps valid, perhaps not) uses of bare float to mean “float-only” will be quite disruptive. There are realities that won’t change no matter how we type things.

But apart from that, we should examine the several categories of use cases that the current design of bare float mixes together:

  1. float means numeric tower behavior – it is safe (at least, no less unsafe) to pass in int. I assume most basic arithmetic use cases (whether written in Python or C) are like this, since this was the intention of the language design back then.
  2. float means accepts both float and int, but the behavior differs in a way that does not conform to the numeric tower. Say, serialization libraries which dispatch on types and where float and int are considered distinct in behavior.
  3. float means float only. This likely represents a narrower set of typing use cases, but where this occurs it really matters – type hints of this sort are widely read due to popularity of the frameworks and libraries.

All of the above are valid use cases, but I expect most existing use cases of bare float to belong to case 1. The goal of designing a solution is to minimize disruption to case 1 while providing a pathway for cases 2 and 3 to adapt. My suggestion is to:

  1. Keep bare float and reemphasize that current bare floats mean that this function/attribute respects numeric-tower behavior. Its use as a return type does not preclude returning an int.
  2. Make float | int (perhpas StrictFloat | int as well?) mean expect possible non-numeric-tower behavior. This would ideally be typed as float | int to make the fact of differing behavior explicit, but current linters do not encourage this.
  3. Add StrictFloat. This, and int, should be considered a subtype of bare float at type-checking time. What behavior it has at runtime requires further consideration.

After this is added, it should be considered a type hinting bug if a function in use case 2 uses bare float but exhibits non-numeric-tower behavior (such behavior should already have been documented in docstrings etc.), and for functions that accept float to reject int. Indeed this distinction based on numeric tower behavior is just overloading type hints with more semantic behavior, but IMO this is likely a pragmatic option.

2 Likes
164

Maybe some generic type typing.Strict could be added to the Typing module? The implementation could look like e.g.: NotRequired, meaning we will have a Special form which type checks all params and returns a _GenericAlias. Therefore users could do x: strict[float] = 3.14 and type checkers will infer the type to be float instead of some int | float | complex.

1 Like