I think we have it in this thread already, and I don’t think it’s especially hard to elucidate.
I can’t believe I’m going to be the one advocating this, because I hate the idea of typing in general already (even more broadly, of trying to fight the type system of whatever language you’re using). But I think this thread is grossly unfair to the proposal. I don’t think it’s very much like Java’s checked exceptions at all.
Yes, an exception annotation that is actually used by a third-party type checker is in some sense a “checked exception”. But going from there to all the bad stuff associated with checked exceptions is a heinous fallacy. Having “checked exceptions” does not entail that the language dictates which exception types must be checked, and does not entail the absence of a compile-time workaround for the exception check - or even a compile-time means to un-check the exception.
(Rather like how “static typing” doesn’t entail manifest typing, nor an absence of implicit casts, nor the implementation of a notion of either covariance or contravariance, nor even basic type safety.)
I’m not surprised that multiple people seized on the same complaint, as it seems pretty central to what people don’t like about existing implementations of checked exceptions.
Fortunately, the fact that the typing system is optional and bolted-on means that there is not any actual issue here.
For example, add to typing.py:
def ignore_exceptions(expr, *excs):
"""Ignore exceptions that were raised in evaluating an expression.
This returns the value unchanged. To the type checker this signals
that exceptions declared raisable by the expression are
unimportant, but at runtime we intentionally don't check anything
(we want this to be as fast as possible).
"""
return val
By convention, the type checker could ignore all exceptions if the excs list is empty (since it would be useless otherwise). Exceptions could be allowed to propagate, without propagating annotations. This is a mature-adult decision: the choice to say “I acknowledge that I depend on code that claims it can raise FooError; I consider that the risk is minimal and that advertising it to my own callers is not reflective of the normal workings of the API I’m providing.”
Aside from that: static type checkers in Python aren’t part of the compiler, but they may very well be part of the IDE. In which case, rather than simply complaining about exception annotations not being properly propagated, they could just give you an option to fix the code to propagate them.
But it wouldn’t. It would also consider exception declarations attached to function calls.
You don’t always know what to test. And if the undocumented exceptions are in the piece that you’d normally mock out for testing, good luck.
This is the part where I actually get to an example. A few weeks ago, someone asked on Stack Overflow what exceptions “should” be considered when calling httpx.response.json. As noted, the documentation doesn’t give a proper indication. After reading the source, I concluded that it makes sense to check for JSONDecodeError and UnicodeDecodeError.
Do I need to be told that kind of thing? Arguably, yes. My conclusion from that investigation happened to line up with my common-sense assessment. But on a below-average day, I could easily have overlooked the possibility of UnicodeDecodeError, even though I’m normally the one playing the role of the old curmudgeon grumbling at people to Think about encoding and Stop Pretending that bytes are text because 2.7 is as old as Windows 7 and Unicode is older than either ASCII or EBCDIC were when Unicode was born.
Could I criticize the documentation for failing to cite those exception types? Yes, but. An annotation would equally well document the code, and it would potentially show up in someone’s IDE. Documentation could also be generated from it. It’s a little strange to see people wondering what the value is in having a form of documentation that’s also executable, in the same context where we’re talking about the importance of testing, which the XP folks have always marketed as being exactly that.
Speaking of which: could I determine those types through testing? Maybe. Kind of. If I’m lucky. I’d have to know how to mock just enough of the response to where the library hasn’t attempted either JSON or text decoding yet. Then I’d have to configure the mock in a way that caused those problems to occur. But if I knew how to do these things, I would almost certainly have already guessed the corresponding exception types just by the Feynman algorithm (not the one on Wikipedia).
Could it actually realistically raise either of those exceptions? Sure, pretty decent likelihood. Data retrieved from the Internet is invalid all the time.
Could it also raise TypeError or ValueError? Sure, but this is much more likely to indicate either that I gave it bad arguments, or that there’s an error in its own logic. Similarly, httpx.ResponseNotRead would seem to indicate an error in my own logic.
Could it also raise KeyboardInterrupt or MemoryError? Sure (notwithstanding that a KeyboardInterrupt would be more something that happens during the JSON parsing rather than because of it). But these are problems that I either care about regardless, or don’t care about regardless. They don’t meaningfully have anything to do with my decision to attempt to decode JSON from a web response.
Could it also raise any of countless other things? I’m sure it could, but. I maintain that JSONDecodeError and UnicodeDecodeError are qualitatively different from everything else in this context. They’re the ones that fundamentally are caused by something bad in the request data, rather than in the pre-existing program state.
Are those explicitly raised? JSONDecodeError is, but only indirectly. UnicodeDecodeError is not, unless you count the C source code for the bytes type.
As someone who is writing a function that calls httpx.response.json, is it likely that there’s something meaningful I can do about JSONDecodeError and/or UnicodeDecodeError? My claim is, yes, it’s much more likely for these two than for anything else that might be raised - because of the nature of the circumstances that are liable to cause them.
So this is something where I would think it makes sense for the library function - which is already using type annotations for its parameters and return value (which I personally would just ignore, as I am not using mypy etc.) to be able to declare that it notably raises those two things, in a way that mypy can be aware of. And then, in my own code, I have a choice:
-
I can just not use mypy (the choice I already committed to ahead of time).
-
I can add the appropriate try/except blocks around the call.
-
I can declare that my own function also raises those exception type, notably.
-
I can, perhaps, ask my IDE to automate either of those.
-
I can mix and match the above two.
-
I can politely ask mypy to shut up about it - perhaps with the typing.ignore_exceptions proposed above, or perhaps with some kind of pragma recognized by mypy (but I guess the latter is not favoured by type checker maintainers, or else typing.cast wouldn’t exist in the first place?).
And if I explictly raise something in the same function, I can ask mypy to hold me accountable for that, and document it - I probably wrote it for a good reason. On the other hand, if I decode bytes to string, maybe I don’t want mypy to nag me about that, because maybe I control the data and am confident about its encoding. If that’s not the case, I would still be able to volunteer such a declaration.
This is not at all like Java’s approach. We aren’t, for example, arbitrarily deciding that RuntimeError, TypeError, ValueError and SyntaxError (or some other set) are “free”, and everything else has to be checked. We aren’t preventing compilation if the checks aren’t satisfied, unless we give a static type-checker that authority explicitly. We’re allowing ourselves to ignore the red flags, with varying levels of control. We’re admitting the possibility that the library code itself ignored some red flag for a good reason (say, it knows about the ResponseNotRead possibility, but chooses not to propagate that to us from this part of the API).
Yes, you would lose the ability to advertise the database connection error though len. You would still have more information signalling than otherwise - in particular, the ability to advertise the possibility of TypeError from a non-iterable. (As an aside, I would not be very happy to find out that a library type was depending on a network connection for __len__. That’s normally something that’s supposed to be either cached or trivially calculable.)
Alternately: the subtype can perfectly well have an additional notation, and if the type can be statically proven (or cast), then the additional possibility is surfaced.
In principle, I’m sure. Sounds like it would be slow, and overkill. You do raise a good point about the C-Python boundary. But I don’t think I’d want the Python interfaces to, say, builtin types to report those exceptions automatically. Maybe I’m wrong. I wouldn’t use the feature anyway (I just think it’s an entirely reasonable idea now that the typing worm-can is open), so that’s for others to vote on.
I don’t see why. Will mypy complain about
def example() -> Optional[str]:
pass
? That declares that it could return a string instead of None, but it won’t. It seems bad for a type checker to complain about that. What if example is implementing a pre-existing interface?