PEP 601: Forbid return/break/continue breaking out of finally

The use of return, break and continue within a finally suite leads to behaviour which is not at all obvious. Consider the following function:

def bar():
    while True:
        try:
            1 / 0
        finally:
            break

This goes against the following parts of The Zen of Python:

  • Explicit is better than implicit - exceptions are implicitly silenced
  • Readability counts - the intention of the code is not obvious
  • Errors should never pass silently; Unless explicitly silenced - exceptions are implicitly silenced

Aim of the PEP 601 is forbid return, break and continue statements within a finally suite where they would break out of the finally.

2 Likes

It took me rather a long time to be surprised by the example :-/ I’m still not really surprised. The break statement explicitly quits the surrounding loop, the intention is (to me) obvious, and the “try” statement is all about doing strange things to exceptions. The documentation is even explicit: “If the finally clause executes a return or break statement, the saved exception is discarded”.

Using return, break or continue in a finally clause is probably inadvisable, but forbidding them seems unnecessary and unhelpful to me. I certainly don’t agree that your justification follows from your example at all; the results are explicit and obvious, just not often useful.

5 Likes

I agree with this change. For me there is a much simpler, more compelling example (maybe it’s worth including in the PEP):

def f() -> bool:
    try:
        return True
    finally:
        return False

What does f() return?

2 Likes

Currently, Python has a (mostly?) context-free LL(1) grammar. The 3.8 language reference
https://docs.python.org/3.8/reference/compound_stmts.html#the-try-statement
has this semantic interpretation sentence: “If the finally clause executes a return , break or continue statement, the saved exception is discarded:”

PEP 601 proposes to replace this semantic interpretation rule with a new context-dependent grammar rule: the finally suite may not contain any return or a continue/break not in a loop within the suite. In other words, replace "finally" ":" suite in the grammar with "finally" ":" escape_free_suite. where escape_free_suite likely cannot be defined by context-free rules and therefore has to be enforced by hand-written compiler code. At least so proposes the PEP, while falsely claiming that this added complexity make the grammar change not a grammar change. A strong minus one from me.

To me, this added complexity violates at least 3 of the Zen rules.

Anther issues: there is no consideration of what uses are or might be made of the current rules. To me, this proposal belongs on python_ideas, not in a PEP.

3 Likes

I also might suggest a slight improvement to the wording:

A return in any statement, at any level of nesting.

This seems to imply that the following code is illegal (I don’t believe it should be, or that it’s what the PEP intends):

def f():
    try:
        ...
    finally:
        def g():
            return

False, as stated in the doc, and verified in 3.9.

2 Likes

False, as stated in the doc, and verified in 3.9.

Okay, maybe that was too simplified. I think I’d have a tough time finding a typical Python programmer who correctly answers False on this, though:

def f() -> bool:
    while True:
        try:
            return True
        finally:
            break
    return False

I guess it’s more of a question of utility. Does this syntactical corner case really add enough to justify the mental gymnastics that are sometimes required to comprehend it?

1 Like

This seems to imply that the following code is illegal (I don’t believe it should be, or that it’s what the PEP intends):

… forbid return … within a finally suite where they would break out of the finally.

So yes, that is a valid syntax for 601.

Currently, Python has a (mostly?) context-free LL(1) grammar.

You are interpreting currently as 3.9a0, but for python community it means the latest stable version 3.7.4

replace this semantic interpretation rule with a new context-dependent grammar rule

I’d prefer call it extend operation instead of replace. We have a check at compiler for continue in finally. We can extend that to the return and break

No, Python has always, AFAIK, been defined with a (mostly) LL(1) grammar, with the parser and compiler auto-generated from the grammar (and any of the intentionally few non-LL(1) parts hand-patched). 3.7 is no exception. 3.7’s context check was temporary, to avoid crashes. It is no longer needed and is gone. Syntax change proposals have routinely been rejected when they could not be expresses in an LL(1) grammar. Guido has thought about loosening that restriction, but I am pretty sure that he is not contemplating jumping into the chaos of unrestricted context-dependence.

You both seem to be ignoring the tremendous utility of restricting the grammar to a well-studies class and avoiding context dependence.

As for the second ‘what does this do’ test, I correctly guessed that the final return False' would be executed. But so what? The crash reported in bpo-37830 was a real issue, apparently now fixed. In my two decades+ experience, Python programmers doing mental gymnastics to understandfinallywithreturn`, etc, without reading the ‘finally’ doc, is not a real issue. The toy examples no way justify adding context dependence to the grammar and adding the burden of hand-written compiler code to all implementations.

Maybe I’m atypical but I found no real difficulty determining that the answer was False. The docs say so (as has been mentioned) and the more examples you give, the more it becomes obvious(to me) that the behaviour is consistent and easy to reason about.

It’s useless, of course. Why would anyone write code like this? But to me that just says that there’s nothing like enough justification for adding a special case rule to forbid return as the PEP proposes. Particularly when the current behaviour is documented, consistent, and does no harm.

1 Like

The PEP seems like a fine idea to me. A finally block is supposed to pause execution, run a bit of code, and then resume what it was doing before. Jumping out somewhere else breaks this mental model completely. The rationale for keeping the grammar LL was always to keep things easier for human readers; using it try to preserve a confusing construct feels against the spirit of things.

I do have a technical question though. As we learned recently from the backlash escapes deprecation discussion, we don’t currently have an effective way to issue deprecation warnings at compile time. This PEP wants to deprecate some things at compile time. How do you propose to do it so that the right people see the warnings?

1 Like

I don’t have a strong opinion on either side of this debate, but the suggestion here is not to change the grammar. return and continue and break are also allowed outside of functions and loops (respectively) in the grammar, but the compiler then rejects it. This isn’t to do with the grammar, but with semantics of the language.

2 Likes

That’s the wrong question. The better question is, do these mental
gymnastics justify a special case in the language forbidding something
which is otherwise legal?

It might help if we write example code that was a little less pointless.
Here’s a sketch of something a person might actually write:

def func():
try:
resource = open(thing)
value = process(resource)
return value + 1
finally:
flag = close(resource)
if flag:
return “Failure :-(”

Is this great code? Is it code you or I would write? Perhaps not, but
by doing some actual (pretend) work, I would expect most coders would
grasp the intention and be able to predict what it would do without as
many mental gymnastics.

(Although the first time or two they might want to read the docs on
“finally” to be sure.)

You or I might not personally approve of that code, but does that
justify banning it outright?

Is that mental model of finally documented anywhere, or is it your
mental module? Because it isn’t mine.

According to that model, we ought to ban raising exceptions from a
finally block too.

One other point here. The motivation for the PEP is weak - it’s an argument that certain behaviour is “not obvious” and therefore should not be allowed. And furthermore, the survey of other languages (thank you by the way for adding that) seems to imply that most other languages handle this in style guides.

Is there any example in real world code (a bug report on a project on github, for example) where use of this construct caused an error in the project? That would be a better argument (although a single example would still be very weak - to be compelling you’d need to show that this mistake causing bugs was a relatively common occurrence).

Sorry, I’m still not really surprised. It’s a more convoluted example to work through, but it’s still obvious once you apply the principles the language definition gives us. It’s not like flow control statements are the only things in a finally clause that can cause you to have to go through your mental gymnastics. This PEP wouldn’t prevent this, for example:

    def f():
        result = True
        try:
            return result
        finally:
            result = False

It does state this in the doc, but it’s not at all obvious just glancing - you’d expect the return True to be successful, and thus the finally return statement not to execute because a return statement had already executed (as opposed to the finally releasing resources or printing to the console).

An aside - I believe this example is non-obvious from the docs. I suppose one could infer that “before leaving the try statement” means before a return statement in the try executes, but I would find this non-obvious. I’d expect it to be like code after a yield statement, the yield yields something, then the code after it runs, except if the code in the finally is a return, it doesn’t run because the function has already returned.

>>>def execution_order():
>>>    try:
>>>        print('first statement')
>>>    except:
>>>        print('oops')
>>>    finally:
>>>        print('last executed statement')
>>>        
>>>execution_order()
first statement
last executed statement

I believe the docs could be made clearer, specifying that the finally executes before a return statement in a try, as I on first read interpret “on the way out” as meaning after rather than before completing the final statement in flow changing/breaking cases’.

I’ve submitted a PR https://github.com/python/cpython/pull/15677 - if anyone has better wording than mine, feel free :smile:

Being surprised by what a piece of code does is hardly a reason to ban it from the language.

Are we going to ban raise from a finally clause too? There’s an intentional similarity between the semantics the various non-local control flow statements that this PEP breaks – they clear the pending exception from the finally block(s) and replace it with their own destination.

I could go on.

3 Likes