Previously, this section of the spec was not asserted in conformance tests or implemented by any type checkers: Callables — typing documentation
I propose a change to the typing spec and conformance tests to update the rules to account for PEP 728 (closed/extra items).
It is only safe to unpack a non-closed TypedDict in a function call if that function has **kwargs in its signature, and any extra items are assignable to the type of **kwargs.
If the function being called has **kwargs, checkers should error if the TypedDict’s extra items are not assignable to the type of **kwargs. For this rule, open TypedDicts are treated as having extra items of type object.
If the function being called does not have **kwargs, checkers may error if the TypedDict is open, and should error if the TypedDict declares non-Never extra items.
While it is unsafe to unpack an open TypedDict into a non-kwargs-containing function, type checkers are not required to error (note the “may” wording). This is because TypedDicts default to open, so requiring an error here would cause a lot of false positives in existing code. Pyrefly intends to implement this specific error as an optional error that’s enabled in strict mode.
The proposed changes are implemented in the attached PR. Comments/discussion welcome
Thanks Danny! I think it’s great that we clarify the semantics here before wider adoption of PEP 728 features.
I think this is the right compromise for the conformance suite for now. I would like to be able to specify fully consistent behavior (which would imply that implicitly-open TypedDicts can’t be unpacked into calls unless the callee can handle any kwargs of any type), but I do suspect that’s a bridge too far given existing use of implicitly-open TypedDicts and existing type-checker behavior. I think we will likely also implement this as an opt-in “strict mode” diagnostic for ty.
I think that ideally we would rewrite this section of the spec even more deeply, so that instead of listing out a bunch of TypedDict-specific cases, it instead specifies a general algorithm for normalizing a function with an Unpack[TypedDict] parameter into a “normal” set of parameters, and likewise for normalizing arguments when a TypedDict is unpacked into them, and then ~all behaviors should fall out correctly from normal call binding. But I think it’s fine to leave that as a separate future spec change.
How many false positives are we talking about? Because if this will only be an issue for a handful of projects, then we could just open a couple of PRs fixing it there, instead of having to compromise on type-safety.
FWIW, I agree that pyrefly is correct to have warned me about this issue. It was confusing that no other type checker complained.
The challenge from my perspective [1] is that doing the right thing requires either:
Defining each TypedDictonly inside a TYPE_CHECKING block or in stubs [2]
If a project is aiming to be zero-dependency and using the existing TypedDict at runtime - I did come up with an ugly hack - but I wouldn’t recommend it
Hmm yea I can see how it might be too much to ask of users to rely on backported TypedDicts for this.
So how about we keep it as “may” for Python <3.15, but actually require it on Python >=3.15, i.e. only enforce it for projects that require Python 3.15+?
We could certainly say this in the spec, and type checkers can implement it. It seems potentially a bit confusing for users, given that there is no runtime change in Python 3.15 that actually changes how dictionary unpacking in a call works.
Testing this in the conformance suite would require some infrastructure work; we currently don’t have any infrastructure for testing behavior across different Python versions in the conformance suite.
If there is strong sentiment in favor of strictness here, I’d probably favor the simpler approach of just specifying and conformance-suite-testing the strict version, and then type checkers can make their own decisions about the UX around it (is it a separate rule, is it enabled by default, is it only enabled by default on recent Python versions, etc). Even if the rule is not enabled by default, type checkers can still enable it and pass the conformance suite; effectively the conformance suite is saying that type checkers should offer this diagnostic, not requiring that it be on by default. I think we should be cautious about micro-managing too many details of type checker UX in the conformance suite.
I put up a draft ty PR that requires the callee to accept arbitrary **kwargs if an implicitly open TypedDict is unpacked into the call.
The mypy-primer ecosystem report on that PR shows 25 projects would be impacted (out of 162 total projects checked, so about 15% of projects), with a total of 117 diagnostics added (and 1 unused suppression diagnostic removed, so effectively 118 call-sites impacted.)
This doesn’t include any analysis of how difficult it would be for the affected projects to migrate to using a closed TypedDict instead.
This is significant ecosystem impact, but I don’t think it would have to block specifying the strict version, if we are in agreement that that’s the preferred interpretation.
Tbh I’d be totally fine with # E?/ the “may” wording. To me that says “this is a legitimate error but every type checker has freedom to choose whether/when to emit it,” and I generally don’t see a need to be more prescriptive. But if there’s a desire to be stricter, I also like Carl’s suggestion to specify the strict version and run type checkers with the error enabled.
I feel like a warning is probably good enough, most of the situations I use TypedDict unpacking are mostly internal or to keep my function arguments pretty generic when dealing with an external API that I’m trying to wrap.
I think for anything that’s unambiguously an error rooted in the type system, typecheckers should have a corresponding error. I don’t think “may” wording is useful here, because having an error for something is different from that error being shown to users under certain configurations.
I don’t think the specification should mandate anything about typecheckers allowing users to opt out (or not) of specific errors, so “may” is already (in my view) always the behavior users see for any error the specification provides, and “should” here seems to more describe that typecheckers should have an error for it, not whether or not the error needs to be on by default or not.
I think version-specific wording in the spec might be a bit unprecedented, given that there are no version-specific rules in the current spec + as I understand it typing constructs are supposed to be backwards compatible thru typing_extensions
I wouldn’t want to branch the behavior on Python version. We generally leverage typing-extensions so typing works the same across versions, and I think that’s a good thing. There is no real runtime reason here to vary the behavior by Python version.
My reasoning was that it might be a bit much to ask users that don’t have/want typing_extensions as a runtime dependency on older Python version to go through the whole if TYPE_CHECKING dance. In my experience, many aren’t even aware that typing_extensions can be used statically like that without having it installed as runtime dependency.
That said, if conditioning the conformance suite on Python versions would make things difficult, then I’d also be fine with going for the Python-version-agnostic “must” wording.
Thanks for the feedback everyone, I’ve updated the PR: the docs say “should error” instead of “may error” & tests assertions changed from E? to E
This means that type checkers are required to have the ability to detect & flag this error, but doesn’t say anything about whether the rule should be on by default.
