PEP 765: Disallow return/break/continue that exit a finally block

There are some unique issues related to implementation of the PEP 765 warnings. Other warnings are emitted either in the lexer/tokenizer or in the code generator. But the PEP 765 warnings are generated in a separate step which is executed twice when use ast.parse() for getting an AST and then compile() to compile the AST. Compilation emits multiple warnings in the finally block · Issue #131927 · python/cpython · GitHub

The fix for this issue was not correct – it introduced other bugs (Syntax warnings can be swallowed in import · Issue #139640 · python/cpython · GitHub), and is incompatible with solutions for other issue (Better module info for SyntaxWarnings during AST parsing · Issue #135801 · python/cpython · GitHub). Both of them affect ability to discover and fix the original PEP 765 issue. So it should be reverted, but then we should apply an alternative solution for warning duplication.

5 Likes

Thank you.
Yes, evidence regarding existing usage is undeniable.


So yes, the analysis and PEPs painted fairly grim picture.

And at the time I didn’t question it much and just took time to “learn what I should remember NOT to do”.

However, with this approach, by now, I could not remember how it works.


So I made an attempt to find out “how I CAN make use of it”.
And the result is that the picture I have now is much more balanced.

First of all, as I said, it is quite simple.

Secondly, github search:
/except BaseException\:\n\s+pass(\n\s+){1,5}return/ Language:Python

2.6K files where except BaseException could be replaced with finally. A lot of false positives (github doesn’t do back references on global search), but a lot of true positives as well.

(only couple of dozens of hits for continue/break though)


It is difficult to construct absolute beginner example which is useful as most of cases where BaseException is useful are interfaces/plugins where errors are being suppressed/ignored for the sake of not interrupting runtime. And either delayed to be treated later or default is returned as a valid alternative.

So alternative would be to acknowledge it as valid construct.
And those who wish can turn-off the linter rule.

Cons:

  1. Explicit is better than implicit - undeniable.
  2. If it doesn’t become more popular, education gap might not go away.

Pros:

  1. No need to condemn this construct. Effort has been made and it works well.
  2. Saves up to 10% of runtime. finally doesn’t have the same performance overhead as except.

After spending a bit of time with it, I quite like it and would likely use it - there is nothing too bad about it and makes my code a bit faster and a bit cleaner (for my taste).

However, with status quo I will naturally refrain.

One thing that I am quite confident to say is that the final result of issuing a warning is the same as SyntaxError - no one will ever use it and it will become a dark corner no one ever cares to look at again.


So I don’t have a (hard) set mind on this, but after spending a bit of time with it I would be a bit sad to see it go.

What I can say is that I largely support that it either should be a valid Python construct which can be used without any extra obstacles from Python itself or removed.

And to me this would hold regardless of whether there were issues with the warning or not.
I don’t think current half-solution is a good option for this specific instance.

def get_suppress_all1(mapping, key, default=None):
    try:
        default = mapping[key]
    finally:
        return default

def get_suppress_all2(mapping, key, default=None):
    try:
        return mapping[key]
    except BaseException:
        return default

The most interesting number is how many of these were known to cause an actual problem. By contrast the SyntaxWarning is known to cause problems and changing it to a SyntaxError would certainly cause more significant problems.

There are lots of things in the language that are often used incorrectly and that may or may not cause actual problems. I would say that bare next leaking StopIteration is one and yet we have linters that will proliferate bad code as an automatic fix (RUF015). I don’t think I have ever seen return-from-finally cause an actual problem as I have for a leaked StopIteration even though they both have a similar effect of swallowing exceptions that should really be visible as runtime errors. I think that the reason for this difference is really that return-from-finally just isn’t used much as shown in the survey while next(iter(x)) is very popular even though its semantics in the exceptional case are very rarely intended. Although return-from-finally isn’t used much it only needs to appear somewhere, however inconsequential, in millions of lines of code for the SyntaxWarning to show up.

6 Likes

You make a really good point. I’ve seen plenty of leaked StopIteration bugs cause real headaches in production, but I honestly can’t recall ever debugging an issue that traced back to return-in-finally swallowing an exception!

2 Likes

This is not an accurate description of what people have had an issue with here. It’s not possible to properly suppress only the warnings that have been checked in this case. This change broke the way people use warnings as errors. People use this to discover actual problems in CI, while suppressing the specific warnings that are known to be okay.

This has been explained multiple times now, and you still aren’t understanding why people use warnings as errors. It isn’t the new warning existing at all that is the problem, I use warnings as errors, and I expect I may have to add ignores not only every python version, but even on new dependency versions that emit their own warnings + open tracking issues for anything that actually requires changes. The problem in this case is that the new warning doesn’t play by the rules and interacts incorrectly with warning filters.

It’s that this change broke the actual existing standard provided by the language and that other tools interact with for interacting with warnings.

2 Likes

The point I am attempting to make (but apparently failing at) is that unlike runtime warnings, syntax warnings are also detected by code linters, making the syntax warnings redundant when a code linter is in use. They’re not redundant in general (since there are many cases where linters aren’t used), but they’re redundant in that case.

This means that turning off syntax warnings globally during dynamic CI creates a very different risk profile from turning off other kinds of warning: as long as doing that is paired with running a linter, you’re not going to miss anything.

However, we’re going in circles here, so I’m going to echo @iritkatriel’s suggestion: the way to get the Syntax Warning removed or altered at this point is to create a follow up PEP with the rationale for why the conclusion on this one was incorrect, and proposing to either revert to the previous status quo, or to do something other than what we did for 3.14.

3 Likes

The 3.14 release has already shipped with the change in it. People must assume this can happen in code and deal with it no matter what because code doesn’t get to choose which patch version of 3.14 it runs on, even if a later patch release were to remove it. Which makes bothering to do this in a patch release a lot less appealing unless there is some seriously compelling reason.

3 Likes

So, what you’re effectively saying is to turn off syntax warnings, but then lint 3rd party dependencies for those syntax warnings.

It’s clear you don’t get why people use warnings as errors. It’s to catch things before they become a problem later, in the entire dependency tree of the application.

I don’t think it’s reasonable to introduce a warning that doesn’t obey the warning filters and write it off as not a problem, but because it has already shipped, and people dug their heels in on this not being a problem and not caring to understand the impact people were telling them about, this effectively is the new normal.

I’ll be recommending at work that we stop trusting any sense of reasonable process from the python software foundation regarding warnings going forward, and replace all tooling that relies on CPython’s warning filters.

There isn’t any way to fix this now that it made it into a release, this is what we have to assume going forward.

2 Likes

And if that’s really an acceptable solution, I don’t understand how anyone can justify this warning existing. The guidance for detecting erroneous cases of this could have just as easily been “use a linter” since you’re fine suggesting turning off all syntax warnings as a category, but then using a linter rather than accepting that the current implementation caused problems.

2 Likes

I work at a university, developing software for researchers in various fields. Across different projects and training courses I’ve led, I’ve probably worked with a hundred researchers or PhD students who write Python code as part of their research. Most of them do not use linters and haven’t even heard of linters. (And I expect it’s very similar among many other Python users who are not professional software developers.)

“Use a linter” is good advice, yes. However, most of these people will never see that advice, because they don’t read DPO and they don’t follow Python-specific blogs/podcasts/YouTubers/etc.

The only way we can reach these people is to build a warning into CPython.

Now, we can absolutely argue about how to weigh these different perspectives[1] and whether we made the right trade-off in this case[2]. But I think it’s important to acknowledge that it is a trade-off; that there is a reasonable justification for this decision (even if you would’ve preferred a different outcome).

(And FWIW, I’m sorry you have to deal with a bunch of extra work because of this. It sucks to be on the receiving end of such a trade-off. :disappointed_face:)


  1. On one hand, how many bugs will this SyntaxWarning uncover to people like my researchers who don’t use linters? On the other hand, how much extra work does this cause for people like you who have powerful CI pipelines set up and can’t filter this particular warning in the usual way? ↩︎

  2. I don’t know. And I’m really glad I didn’t have to make that decision—the SC really has an unenviable job sometimes! ↩︎

7 Likes

Turning syntax warnings off globally when they’re redundant is just a workaround for the fact turning them off selectively doesn’t work properly. It’s not the ideal outcome, but the answer to that part is to fix the syntax warning filtering, not to decide never to add new syntax warnings.

The reason I consider that an acceptable workaround is because syntax warnings are essentially redundant when a linter is in use. They’re the only warning category for which that is systematically true, since they’re the only category that is necessarily detectable via static analysis.

I would also have considered it reasonable if the SC had made fixing the syntax warning filtering first a prerequisite for accepting PEP 765. That isn’t what happened though, so I’m appreciative to Serhiy for taking on the effort to fix the filtering now that folks have emphasised how dissatisfied they are with turning off runtime syntax warnings globally (even if it’s in favour of a static analysis pass)

Edit, as I realised I hadn’t addressed this entirely valid point:

Yes, I agree it would be preferable for folks to be able to easily audit their entire dependency tree for this issue without having to lint their entire site packages folder, as that is indeed part of the motivation for making it a syntax warning and not just a linter rule.

However, until the warnings filtering is fixed, that’s a pain, since there’s no way to flag the instances that have either been found to be OK, or already reported to the affected project. The “turn off runtime syntax warnings” suggestion is a way to effectively restore the 3.13 status quo until such time as they can be filtered correctly based on their location in the code. It does have the downside of turning off the other syntax warnings as well, but those currently can’t be common in the wild or we would have had more vocal complaints about the filtering issues in previous releases.

3 Likes

That’s not what @gpshead wrote.

You’d need to make your case in a new PEP rather than continue whatever it is that’s going on here.

1 Like

I wasn’t saying that because of what others said about the impact, but in what has been expressed as possible to change in combination with the existing outcome on CI/CD matrices.

Going forward, I have to assume the usage of warnings as errors is not going to be taken seriously or be understood by decision makers, and that they will happily add new warnings for things that are not actually incorrect, and with no intent for it to ever be a change, and without caring if the warning even correctly uses the warning system provided by the language. It’s harmed the previously exceptional signal to noise ratio for what we used to accept was the job of a linter.

It was pointed out that this broke existing uses and interacted poorly with existing methods, and the SC reaffirmed that it is here to stay with also no plan of promoting it to an error.

The only thing left to do in this thread is provide guidance to those effected, make sure they are aware they need to assume the same can and will happen again or find ways to ensure it doesn’t happen again.

This case of it has left the barn already. I can’t not run CI/CD on python 3.14.0, and it’s been clearly indicated that those in charge of making decisions are aware of this fact, but did not see it as an issue.

It’s behavior that now forever has to be worked around and assumed possible to happen again.

2 Likes

On this thread, all that’s left is to make noise. If you want to make an impact, you will need to write a PEP. The choice is yours.

1 Like

On one hand it makes sense.
The decision has been made.
Some issues appeared.
But it was all formal decision, so it is how it is.
If someone wants to change it, they need to go through another formal process.

On the other hand, someone has made some changes and it seems that the work is not complete to sufficient degree. For those who have worked on this already it is not only part of responsibility to finish what they started to sufficient degree, but is also simply easier to undo / lead further (as they know ins and outs of what has been done).

The former is a very standard corporate BAU.
The latter is more community friendly POV.

2 Likes

Again, the only impact left to have is to prevent it from repeating. There is no world in which we can undo the 3.14 release at this point. I thought it was fitting to continue this discussion until reaching some level of consensus from those who actually were effected, while figuring out what was immutable in the location most relevent, where people were discussing it.

I’ve opened a new thread in ideas for it, with the intent of turning it into a pep, but it’s really more about process than fixing this case of it, this case can’t be fixed now.

Guido suggested to make a change in 3.14.1. Scroll up.

1 Like

As they have repeatedly said, that doesn’t make 3.14(.0) go away. That version will always have to be dealt with in test matrices.

1 Like

Unless you’re suggesting I just tell users 3.14.0 is broken and unfit for use, and that you should block CI from ever even trying to run on it, undoing this in a later release isn’t the fix you think it is.

That sounds very bad indeed.