A few thoughts which will hopefully be useful for the OP in particular:
Just use the docstrings. You can even put structured data in a docstring and parse it out later. Cheap and efficient.
You don’t want to use docstrings? Okay, write a decorator that stores exception classes in a function attribute. Now you have a place to store this data without asking for a typing change.
There are already enough ambiguities between type checkers about type correctness and safety. Anything which makes that more ambiguous (and annotations about exceptions would introduce many ambiguities, in case that’s not clear) should be viewed with extreme suspicion.
I share (1) and (2) as potentially productive lines of exploration for some concrete implementation.
I’ve put JSON, YAML, and INI data into docstrings for a variety of use cases, usually to positive effect.
(3) is more about trying to broaden the scope of what you’re looking at, from your own use-case (wherein I think we can safely presume the proposal makes sense) to the full community impact. Even innocuous looking features can have significant costs.
They are still things that a typical user of the code wants to know about and that a third-party tool can, in principle, use for linting. That is, after all, what third-party type checkers do in Python: linting. They do not prevent code from compiling, and they do not alter what bytecodes are produced. While return values are “inherently part of the API”, their types are not, in the same sense; Python does not enforce return types, and does not have a proper void-return concept.
As such, all the arguments for and against apply equally well to type annotations. The difference in how exceptions are propagated just isn’t relevant here - while it would mean that the linting rules should be different, that is still fundamentally an agreement between the third-party tool and its users, based on what the users think will work best for detecting potential issues in code.
What part of
was unclear?
Information in the docstring is only directly available to a) someone reading the code and b) someone using help at the REPL. To make it from there into actual documentation (i.e., the sort that an end user can discover with an Internet search) requires a third-party tool anyway, and for that third-party tool to treat the information as anything other than arbitrary text requires some combination of a) parsing and b) imposing further formats/protocols upon the docstring.
Stringly typed data is bad.
Again, this is the same argument as for the type annotations in the first place.
I didn’t answer it, because a) I considered that you drew a false dichotomy and b) there are multiple ways that people might want this to work.
Broadly: If the user chooses to use a third-party tool to analyze the signatures for correctness and also doesn’t use some form of signalling mechanism to indicate that the risk of the caller-raised exception has been considered, then it makes sense, to me, to expect propagated exceptions to be marked as well.
First off, the answer supposes that preventable/unpreventable and recoverable/unrecoverable are separate axes. But more importantly, a great many people seem to disagree with you on this point. While it is of course up to the caller to recover, there are reasonable expectations that can be set. If I offer you a library to make JSON web requests, it is reasonable for me to expect you to consider the possibility that the data at the URL you give me isn’t available, isn’t text, or isn’t JSON (because you supplied the URL and are now admitting data into your program from an external source that wasn’t there before). It is not reasonable for me to expect you to consider the possibility that there’s a logic error in the code I wrote (because it’s my responsibility) or that the entire program aborts abnormally (at least, no more than you were already considering it).
And, again, Java made a decision about which exception types were checked, and the rules they came up with are simple and straightforward, and explained in the documentation. They managed to figure this out many years ago and haven’t changed those rules. So it’s hard for me to fathom that the underlying concept is in any way incoherent or broken.
Well, this certainly is a bespoke criticism. I have no involvement with the Q&A from the previous post. I don’t care in the slightest about reputation on Stack Overflow, as I already have more than double the total required for all the privileges afforded to users who aren’t formally moderators. Aside from that, the evidence shows quite clearly that the real “power users” overwhelmingly gain their reputation by spamming answers to easy questions that should have been closed (either as duplicates, or because they are lazy debugging-help requests that either “need more focus”, “need debugging details” or “were caused by a typo”). Linking existing stuff is the point; Stack Overflow is intended to filter out the best information and then avoid the need to copy and paste it everywhere.
As far as advertising goes, I seem to be automatically avoiding it all - even while logged out - simply by having googlesyndication.com and googletagmanager.com blacklisted with Noscript. I don’t need an ad blocker to avoid seeing them; I can’t even recall how they are framed on the page by default.
They haven’t changed the rules because that would be a whopping backward compatibility breach. But if you want an indication of how broken the system is, show me other languages that have chosen to imitate Java in this. Surely, if it’s a useful feature, there would be dozens, perhaps hundreds, of languages with checked exceptions?
I’m hardly the only person to dislike SO for this. But if you really don’t care about rep, then why encourage the site’s gamification at all?
Yeah, that’s just a different form of adblocker, done slightly more manually. It’s no different from any other. The site is still ad-funded, and by linking to it, you are boosting their page views and ad impressions. Maybe you don’t personally see ads, but everyone who clicks on your links will. IMO that’s unethical.
Repeating the point from previous attempts: people don’t like Java’s implementation of checked exceptions because it is mandatory. Of all the various repeated attempts to propose some form of this feature, none have ever suggested that an incorrect annotation should render the code invalid or prevent the reference implementation of Python from compiling it. After all, that doesn’t happen with type annotations now, and won’t in the foreseeable future.
I don’t think I’m doing such a thing. I think the reputation system is very poorly designed (and have complained about it before within their own meta channels). What I think I’m doing is citing existing well-written information, just as I would from any other site, rather than expending considerable effort on rephrasing (or coming up with) the same ideas.
I don’t see how I can have a productive discussion about anything with someone who is opposed to me linking to technical content on web sites that serve ads. Or is it only sites where “gamification” is involved, like Reddit? Or is it because I happen to have an account on Stack Overflow?
For the amount of friction we’ve had on these and other topics before, I don’t think it will be worthwhile to try any further. I have better things to do with my time than go around in circles on a topic I’m not even that excited about.
Citing well-written information on other sites isn’t usually the main form of discussion; if anything, referencing someone’s blog is more about “this is what so-and-so says on this subject” rather than making the argument per se (that is, the point it’s making is that this person has said this). Linking to official documentation, of course, is quite different.
If it’s the official home of some first-party documentation, then it’s the best choice. Stack Overflow is not first-party documentation.
Agreed, there is no point debating this further. I originally posted in this thread in the (probably vain) hope that it would be a simple “link to previous discussion, end of story”.
Ignoring theoretical debates, there is a practical issue here. Every Python statement has the potential to raise at least KeyboardInterrupt and MemoryError. Also, stdlib and built in functions, as they do not have annotations, can raise anything. So the vast majority of functions a checker will encounter will ultimately resolve to “could raise anything”. Unless the checker allows assertions that it can demonstrate are false (such as “this function cannot raise an exception”) or users are forced to catch all unexpected exceptions (which is what Java did, and which is what made checked exceptions unusable).
Basically, because exceptions propagate, the exception equivalent of Any pollutes the whole call chain, making checking impossible.
Thanks for both agreeing to not continue that side-branch of discussion—its not really on-topic here In case someone does want to bring it up again, Discourse Feedback as the de-facto meta category would be the place to do it, thanks.
Just to note, I have no particular opinion on or stake in this proposal, but if this were to hypothetically be formally accepted and implemented, the stdlib exception information would presumably be added to typeshed just like the rest of the builtin/stdlib annotations.
The checker does not need to assert that “this function cannot raise an exception”. As you say it would almost never actually be true to say that so it would not be useful to have a checker to try to prove it. Whether exception annotations would be checked or not the idea of annotating (or documenting) exceptions can only be useful if it is to distinguish exceptional cases that are commonly expected and that calling code might typically need to handle. A checker’s job could never be to prove that no possible exception could be raised but it could be to check whether the expected and documented exceptional cases are being handled (if they are claimed to be handled).
For example int(string) might raise ValueError and it is not unreasonable to say that calling code should consider handling that case. It could also raise TypeError but in a type checking context it should be assumed that it was the type checkers job to ensure that string was already of correct type. It could raise MemoryError or KeyboardInterrupt but that can happen anywhere so it could only be useful to annotate those cases if there is some reason why they are more likely to be raised than anywhere else.
So int(string) should be considered to be code that potentially raises ValueError and it could be annotated as such. Calling code does not necessarily need to catch the ValueError but if it does not then it too can be considered to be code that potentially raises ValueError. This is not an exhaustive description of the possible exceptions that could be raised during the call but “raises ValueError” is a reasonable description of the public interface that int(string) exposes. The reason this discussion keeps coming up is because many functions do have an interface such that raising a particular kind of exception is as much a part of that interface as the types of the parameters and return value.
I dispute that, actually. In a LOT of situations, it’s best to just not catch the exception. What’s going to happen if you don’t? Your script terminates with a very useful traceback (or eg your web app returns a 500 and, hopefully, logs that equally useful traceback into the web server log). You should ONLY handle ValueError if you can actually do something about it. Maybe you decide that bad input should cause the entire line to be discarded (except ValueError: continue), or should be treated as zero (except ValueError: thing = 0), or something, but it should always be an active decision on your part - not “oh right, the editor’s warning me that I’m not catching ValueError, I should do that”.
That’s the bad mindset that checked exceptions give.
Note that I said consider handling that case. That does not mean that you have to catch the exception. It just means that if your function does not catch the exception then from the caller’s perspective your function potentially raises ValueError as well so the consideration moves up one level.
Also I didn’t intend for “handling that case” to be taken as just meaning “catching that exception”. There are other ways that you might handle it but the point is that if you write int(string) there is a clear failure case that should probably be considered: what if the string is not valid? If you don’t do anything to handle that case then there are probably situations in which your code would end up bombing out with ValueError. Sometimes that is acceptable but other times it is not.
Your examples of scripts or web apps are situations in which it is potentially okay to just say that every error should propagate to top-level handler for log or traceback. In that case you don’t need to care what the type of the exception is and a ValueError is just as good as an AttributeError or OSError or anything else. Not all code can be like that though and sometimes letting particular kinds of exceptions sneak out is bad (I’m sure you remember PEP 479).
In public API in library code you want to try to control what sort of error is propagated from a public API to the caller if there are predictable failure cases. The caller might be happy to let the exception bubble up or they might want to catch it. If the caller wants to catch it then they should be able to do that without using bare except. For that to work the raised exception types need to be predictable and documented. Ideally it would be possible to check those exception types at the API boundary because they are part of the API contract (e.g. changing them is not backwards compatible).
And that’s checked exceptions in a nutshell (although you’re basically describing doing it manually). If you don’t catch it, your caller does.
The alternative is: Ignore exceptions until you have the need to handle them.
Please go and reread previous threads on the subject. This has been dealt with. Every exception can be handled the same way: if you know how to recover from it, you do, and if you’re doing a “catch, log, and back to the loop” boundary, you do that, otherwise, you don’t. It’s that simple. There is no difference between one exception type and another.
That was a VERY special case due to the unintended interaction between a specific exception and the mechanism of the generator, and even so, it received massive pushback (for good reasons).
Why? A bug is a bug is a bug. I’d rather you just let the original exception propagate. I especially do NOT want you to absorb it and raise-from-none, which is what often happens.
Does your API contract say “there will never be any bugs in this code”? It can’t, because no code is bug-free. So if there’s a bug, what’s the most obvious way to report that? An exception. Letting the original exception bubble up.
First: because checkers are a) third-party and b) optional, it’s up to them to determine the semantics and to decide what level of precision is appropriate. It is not a problem if a checker allows declarations that it can demonstrate are false, because it is neither a theorem-prover nor a compiler. After all, Python functions can enter infinite loops; type-checkers are not required to solve the halting problem. For that matter, functions can have an unconditional raise at the start; type-checkers are not required to notice that the function will not actually return. I can use typing.cast to suggest obvious absurdities (like typing.cast(int, "hello" + "world")), and the type-checker is neither required to accept my proposed return type and reason from it, nor to reject it. Type-checkers are not required by Python to do anything, and they are only required by themselves to do what their own documentation claims.
Second: Java does not require people to catch or declare its equivalent of MemoryError. OutOfMemoryError does not derive from Exception (although it does derive from Error, and thus from Throwable), so it is in a category of exception that Java has recognized ahead of time as unpredictable and/or not reasonably recoverable. If Java used exceptions to handle Ctrl-C, it presumably wouldn’t require checking of the corresponding exception, either, since the cause is external to the actual program logic, and it is ordinarily not reasonable to recover from (as the purpose of Ctrl-C is to abort the program). Java’s approach to the problem that “some exceptions don’t make sense to check” is to filter them naively by exception type. I agree that this doesn’t make very much sense for Python. I disagree that there is no scheme that makes sense in principle.
What makes Java’s checked-exception system obnoxious isn’t that it requires the propagation of unreasonable types of errors; the issue is that it requires propagation of everything that qualifies a priori for propagation, and doesn’t build in an escape hatch to say “look, I can’t do anything if this library throws an exception that it thinks is important but I don’t; go ahead and crash the program if it happens; that’s better than either ignoring or logging it for my purposes”. But on a theoretical, language-agnostic level: just because situations like that can occur isn’t a reason to deny library writers a mechanism to say “hey, I think you should care about the possibility of this specific exception, because for typical clients it means something important!” in a way that is machine-readable. It just means that people will find that mechanism obnoxious if it doesn’t also come with an escape hatch.
And no, there is nothing wrong with escape hatches in general, in principle. (There are other reasons to have typing.cast besides “the system is not yet expressive enough”.) Yes, rigorous checking is impossible. Just as it is with types. (After all, there is a reason that proponents of languages like Haskell consider dynamic typing as not real typing, and refer to languages like Python as “untyped”.) That doesn’t make partial checks valueless, even if I personally would forego them in most places. What happened to “although practicality beats purity”?
See, Python HAS this escape hatch. It’s called “not having checked exceptions”.
So if the only reason that checked exceptions are obnoxious is that they are checked exceptions, and the correct solution is to have an escape hatch by not having them be checked exceptions, it kinda says bad things about the concept.
But we’re back, once again, to the fundamental problem: only the caller can know whether the caller can handle an exception. Not the callee. Not the type checker. Only the caller.
Perhaps a better example for where this sort of check would be useful is something like open(). In most cases, FileNotFoundError would be somewhat expected when opening a filename, and if you’re writing a robust program you probably should account for that in some way. I’m imagining the hypothetical linter for this would give 3 options to deal with the exception - handle it locally with try-except, forward the responsibility to the caller via your own “raises” annotation, or add a something like a comment to the line to indicate that the exception shouldn’t occur, so it crashing the program is fine.
Basically this would be a reminder tool, to ensure you’ve considered whether this exception can be raised here since it’s very common that it’ll happen. It’d be used in cases where the exception effectively is another return value from the function, about as important as the success case.
There is a fundamental misunderstanding here. You are right that only the caller can know whether it can handle the exception. A type checker’s job in this scenario would not be to know whether the caller can handle the exception but just to note whether or not the caller did catch the exception. If the caller did not then the exception would be propagated so a checker can infer that the caller potentially raises that exception as well.
Yes although the important part missing here is that it should not be necessary to add an explicit “raises” annotation to indicate that uncaught exception would be propagated. The checker could infer that an uncaught exception would be propagated because that is what uncaught exceptions do and it is easy for the checker to model that behaviour. See e.g.:
There very clearly is a difference between one exception type and another. Exceptions are caught based on their type so raising a different exception type than was documented and expected by the caller is incorrect behaviour that can lead to breakage. There will be code that catches the expected exception types e.g.:
while True:
string = input("Enter a number:")
try:
number = int(string)
except ValueError:
print("Invalid number. Try again...")
If there was a proposal to change the exception type that is raised by int(string) to a new exception like say ParseError then that proposal would rightly be rejected for being backwards incompatible. There are good reasons to catch the ValueError and the caller needs to know what exception type to catch. Changing that exception type would break downstream code.
What this shows is that the type of the exception that is raised is part of the interface of the function and that the type definitely does matter (this is exactly why exceptions have types in the first place). The fact that the type of the exception matters means that it is useful to be able to document what that exception type is and also that it would be useful to be able to check that the actual code matches what it claims to raise.
The focus seems to be mostly on annotating the throwing part and less on the catching part. Rust’s Result and leaving the feature as a linter-specific one have been mentioned and I would like to add an idea, even though the thread is pretty old now. I am thinking about asking something like mypy to check if my try/except block exhausts the possible exceptions, like Rust’s match.
Maybe it would be possible to annotate a block like this:
try: # lint-exceptions: <lint-exceptions-spec>
# some code
except FooException:
# some handling
Where the linting specification would include things like:
scope - how deep to search for exception sources:
this package,
other imported python code,
exceptions in stdlib - from python code or inferred from the interpreter’s version,
types of analyzed sources:
raise statements,
annotations if any form is ever popularized, eg. Annotated with linter-specific conventions,
types of checked exceptions:
exceptions defined by this package,
exceptions defined by imported Python code,
type trees, eg. Exception, OSError, ConnectionError, ConnectionRefusedError
out of thin air exceptions like KeyboardInterrupt or “protocol” exceptions (just for completeness really).
Maybe linters could have a (user-defined) profile configuration so one could specify a profile name instead of a long explicit specification at every block.
Finally, a linter wound give a report that:
all specified exceptions are handled,
some exceptions are not handled - list which ones,
check was incomplete, eg. linter was unable to check some parts of code (eg. missing type annotations), some parts of the linter are not implemented (eg. a call to dunder methods of an unsupported interpreter’s stdlib).
What it could solve for me:
check if one function of my module/package handles all my custom exceptions, especially after refactoring,
check if my code handles eg. all custom aiohttp exceptions,
check if my code using asyncio handles all subclasses of OSError it should.
You’re welcome to try to write such analysis yourself, or find a project where it would be appropriate to add it.
But the python typing community has more or less concluded that information about which exceptions to expect is not typing information.
In that light, such analysis would not be correct to include in mypy, pyright, etc.
For custom exceptions, a fun trick you can pull is to put enum members into a classvar on your exceptions. You can then catch your root exception type and match/case over the classvar. (I’d probably never bother with this, but if you have dozens of types and want exhaustiveness checking, it would be a tidy solve.)
If you just want to check a try-except against some known list of errors, that’s pretty easy to implement using ast. But it still shouldn’t be part of type checking.
The only good ways I’ve seen for having exception types be part of a type system are:
errors as values rather than as exceptions.
as algebraic effects (largely only currently available in research projects, but there’s ongoing work to bring this to more languages)
Outside of those, in terms of expectations:
documenting what you know your functions can raise, handling exceptions your code causes that are “your problem”, wrapping and re-raising if appropriate
There’s probably a way to indicate this in annotated for those who want some level of passing this info along, but it should be just as possible to parse that information out of doc strings, and you can write a linter plugin today that checks if you handle all exceptions that are documented in a function. Not everything needs new syntax.
In case someone does want to bring it up again,