Constrained type vars with `Any` default

Typing
1

I originally brought this up in Constrained TypeVars with Any default · Issue #20184 · python/mypy · GitHub , but @sterliakov pointed out that mypy’s behavior is correct according to the typing spec:

In python/typeshed#13683 I’m trying to change a type var to the following:

_I = TypeVar("_I", int, bytes, float, bool, default=Any)  # possibly memoryview types

Unfortunately, mypy complains: error: TypeVar default must be one of the constraint types [misc]. While this is of course generally sound advice, a default of Any should always be permissible, as this allows memoryview to stand for any possible memoryview type. Without it, users are forced to write memoryview[Any] in every instance, which is unreasonable.

This is what the spec currently says:

For constrained TypeVar s, the default needs to be one of the constraints. A type checker should generate an error even if it is a subtype of one of the constraints.

I suggest to change the typing spec to allow Any as default for constrained type vars, for the reasons given above.

1 Like
2

Wouldn’t it work like this?

_I = TypeVar("_I", int, bytes, float, bool, Any, default=Any)

edit: ah no nevermind; that would permit everything

Or otherwise:

_I = TypeVar("_I", bound=float | bytes, default=Any)
3

The second looks like it should work, I’ve used that before.

1 Like
4

Please note that bounds are not equivalent to constraints, as the former also allow subtypes. (Although I seem to remember some discussion whether constraints are really necessary and couldn’t be replaced with bounds.)

5

Any, with it’s current definition, as a default breaks the purpose of constraints.

To update the spec here such that users don’t need to write memoryview[Any], you would need a new symbol meaning “Any One of the valid constraints”.

Some prior stuff I’ve brought up about fixing the meaning of Any, as it currently causes problems by allowing more than only things that are otherwise valid within gradual typing, would fix this and allow Any to be a reasonable default itself, but there was little appetite among typechecker authors last time.

You’ll also run into some typecheckers erroring for implicit Any, so users are going to have to write that anyway unless you can convince typecheckers to stop doing that (also harmful to gradual typing)

6

My assumption is that silencing these errors must be the underlying motivation for explicitly specifying default=Any on a typevar?

Otherwise it seems redundant, since Any is already specified as the “default default” for a typevar.

I don’t agree with this. The meaning of Any is “an unknown static type.” This is compatible with “…but only some of the possible materializations would be valid in this position.” We already support, for example, specializing a TypeVar to Any (and/or giving it a default of Any), even if it has an upper bound, meaning that there would be some materializations of that Any that would not be valid specializations for that typevar.

I don’t see why Any should be invalid as a default specialization of any typevar for which it is a valid explicit specialization. So if it is allowed to say C[Any] explicitly, where C is generic over a constrained typevar, I think it should also be valid to give Any as the default for that constrained typevar.

7

I don’t find that that definition actually avoids the issue with constraints (specifically, the full definition found in the specification), but even if we bend a few things to say that it also strongly requires the things this would imply to be necessary, without directly spelling out their requirement, to work everywhere within the type system, then, type checkers today aren’t actually following the behavior outlined there.

It’s not possible for a typechecker to adhere to this and allow the use of Any without tracking bounds separately on every instance of Any in types, something that I don’t believe typecheckers currently do, as that was a blocker on actually requiring support for negation of gradual types. If I’m wrong about this and the situation has changed, please do correct me as that would simplify efforts elsewhere.

8

Can you clarify why this prevents a default-specialization of Any for a constrained typevar, if it does not prevent an explicit specialization of Any for that same constrained typevar (which is already allowed)?

And if it’s a problem for a constrained typevar, can you clarify why it’s not a problem for a bound typevar (for which it is also true that only a subset of possible types would be a valid specialization)?

I’m not able to extract any concrete meaning from anything in this paragraph. Could you be more specific in your referents?

9

The explicit use of Any there runs into the same problems as the default and both should be allowed eventually and shouldn’t be allowed until the issues with not tracking Any are fixed.

Ideally, that would be by removing this line in the specification:

In Python’s type system, we don’t take the gradual guarantee as a strict requirement, but it’s a useful guideline.

and then fixing all of the places that violate the gradual guarantee.

I don’t have my notes with me on this as I’m currently traveling. I remember that it had something to do with the pairing of mutual exclusivity in constraints, with the places the type system is currently lossy due to not tracking certain things. It essentially leads to situations where the mutual exclusivity is lost.

Sorry if that’s unsatisfying, but reworking through the places that the definitions being inconsistent leads to unsoundness is not something I want to be spending my time on right now

I also remember that more consistent definitons without exceptions in them would remedy this. It’s possible I’ve missed an improvement that enables it, or missed that an improvement in definitions I was aware of was enough to enable it, but I have a nagging feeling this is not the case.

10

I strongly suspect that disallowing explicit specialization of a constrained TypeVar to Any is a non-starter, given ecosystem impact. Specializing TypeVars to Any is quite common. So if there are issues with that, those are pre-existing (and long-standing) issues. Regardless of what we do in this thread, we have to live with those issues or find solutions to them.

Given that explicit specialization to Any is already allowed, I don’t see how allowing the default specialization to be Any introduces new issues, so I don’t see any reason here to oppose the proposed change. It’s inconsistent to allow an explicit specialization and not allow the same specialization as a default.

1 Like
11

While this is common, constrained type variables, rather than simply bound ones, are significantly less common.

I worry that it seems we’re frequently collecting more and more cases where certain things don’t fit together well within the type system, and rather than fix or disallow the things that aren’t working well until they can be fixed, we just say that the ecosystem impact would prevent that.

It may create new wider impact for existing typesystem issues, as this is only coming up in trying to change the typeshed from a place not currently using a constrained typevariable, to one that does, in a type that will have a lot of impact (memoryview)

12

BTW, this is yet another use-case for AnyOf (AnyOf - Union for return types · Issue #566 · python/typing · GitHub), i.e.

type _NotABouncyCastle = AnyOf[bool, int, float, bytes]
_I = TypeVar(
    "_I",
    bool, int, float, bytes, _NotABouncyCastle,
    default=_NotABouncyCastle,
)

I suppose it’d also be a better “default default” than the current Any.

1 Like