"with comprehensions"?

[tanj (itzmetanjim)]

Opening files, then reading/writing to it, then closing it is very common in Python:

# Open a file, then read it into a variable
with open("savefile") as f:
    raw_savefile = f.read()

# Open a JSON config file, then decode into a variable
import json
with open("config.json") as f:
    config = json.load(f) #or json.loads(f.read())

with open("config.json") as f:
    print(f.read())

etc. I think this kind of compressed alternative syntax will be helpful:

raw_savefile = f.read() with open("savefile") as f

import json
config = json.load(f) with open("config.json") as f

print(json.load(f) with open("config.json") as f)

This may already exist or may be already suggested or a bad idea in general.

[pf_moore (Paul Moore)]

Generally, just saving a line is not considered sufficient justification for a new language feature. And in any case, you can do this on one line already:

with open("savefile") as f: raw_savefile = f.read()

import json
with open("config.json") as f: config = json.load(f)

with open("config.json") as f: print(json.load(f))

You’ll need significantly more justification if you want this proposal to go anywhere (and even then, expect a lot of pushback).

[Tinche (Tin Tvrtković)]

from pathlib import Path

config = json.loads(Path("config.json").read_bytes())

Feels like we get suggestions to “optimize” context manager syntax relatively often. To me it seems folks bring up cases where context managers aren’t really necessary in the first place.

[layday]

read_bytes is a convenience function wrapping a context manager. I don’t think it’s quite accurate to say that context managers are unnecessary; rather, their verbosity might be giving rise to wrapper functions.

[adqm (adam hartz)]

@tanj

This may already exist or may be already suggested or a bad idea in general.

There was at least one recent thread proposing this same idea: A "with" variant for use in expressions . I feel like that thread kind of derailed, though, after not getting a lot of support.

---

@pf_moore

Generally, just saving a line is not considered sufficient justification for a new language feature. And in any case, you can do this on one line already:

with open("savefile") as f: raw_savefile = f.read()

import json
with open("config.json") as f: config = json.load(f)

with open("config.json") as f: print(json.load(f))

You’re certainly right that there’s not a big difference between those examples and the way they would be written with the proposed syntax, but I don’t think those examples illustrate places where this change would have its impact. In the thread I linked above, the proposal was similarly dismissed (by someone else) as being “code golf,” but to me this isn’t really about syntax, but rather about making context managers available in expressions.

I’m generally in favor of making more structures available in expressions, mostly because I tend to write code with a lot of comprehensions. If I’m writing something and decide that, for example, I want to do a json.load inside of a comprehension^[1], my natural tendency is going to be just to use open(filename) and assume that things will get cleaned up properly, rather than unrolling things into a loop so I can use the context manager, or adding an import so I can use the pathlib stuff. I think it would be nice to have the “safe” version available as an expression, in the same way that we have lambdas and if expressions and walruses and all that good stuff.

So my reaction hasn’t changed since the last thread: I’ve wanted this feature at multiple points in the past, and I’m pretty sure I would use it regularly if it were added.

---

1. Maybe this is a bad idea™, but it’s still something I do all the time, looping over structured files and extracting a small piece of information from each into some data structure or another. ↩︎

[dg-pb]

@adqm

I don’t think those examples illustrate places where this change would have its impact

Then I suppose what is lacking is examples / pointers to those places where this would be useful and nobody can come up with good workaround to achieve similar benefits.

If alternatives to achieve same benefit for all examples provided were given, then there is no footing left.

[JoBe]

Iirc I’ve seen this exact idea a while ago, however searching ‘context’ in the ideas category brings up too many matches for me to sift through.

[Nineteendo (Nice Zombies)]

If you pass a string or bytes, there’s no way to recover the original filename. So, if we ever decide to improve the error messages, you would need to pass it manually or use a context manager:

from pathlib import Path
import jsonyx as json

config = json.loads(Path("config.json").read_bytes())
#   File "<string>", line 1
#     [,]
#      ^
# jsonyx.JSONSyntaxError: Expecting value
path = Path("config.json")
config = json.loads(path.read_bytes(), filename=path)
#   File "/Users/wannes/Downloads/config.json", line 1
#     [,]
#      ^
# jsonyx.JSONSyntaxError: Expecting value
with open("config.json", "rb") as fp:
    config = json.load(fp)
#   File "/Users/wannes/Downloads/config.json", line 1
#     [,]
#      ^
# jsonyx.JSONSyntaxError: Expecting value

[pawamoy (Timothée Mazzucotelli)]

That’s also much less efficient. It’s loading the file contents in memory, and then parsing it, while json.load would read and parse the file at the same time IIRC.

But that’s just one example (and counter-arguments). I believe it would be useful to find actual uses of with blocks in the wild where this proposal would be helpful, and list these findings here, as suggested by @dg-pb.

[Tinche (Tin Tvrtković)]

@layday

read_bytes is a convenience function wrapping a context manager. I don’t think it’s quite accurate to say that context managers are unnecessary; rather, their verbosity might be giving rise to wrapper functions.

I mean, I agree. My point was that if you have a single statement in the body of a context manager it could/should be replaced with a more concise API, rather than adding a new feature to the language.

@pawamoy

That’s also much less efficient. It’s loading the file contents in memory, and then parsing it, while json.load would read and parse the file at the same time IIRC.

Not really: https://github.com/python/cpython/blob/9c6a1f847b648747414600f2cde18f3837505537/Lib/json/\__init_\_.py#L292

Besides, no one interested in efficiency is using the standard library JSON parser anyway.

[pf_moore (Paul Moore)]

@pawamoy

I believe it would be useful to find actual uses of with blocks in the wild where this proposal would be helpful, and list these findings here, as suggested by @dg-pb.

One other point is that I think it’ll be extremely difficult to find examples in application or library code where the proposed construct would be particularly beneficial. Conciseness comes a poor second to clarity in production code, and transforming to a normal with statement, or adding a helper function, are easy refactorings. So you hit the problem that it’s easy, and at least as good - if not better - to do with existing language constructs.

IMO, the only place this construct is likely to be useful is in interactive code - the REPL, or maybe in a Jupyter notebook. In those places, concise code and one-liners are far more compelling. Enough to justify new syntax? I don’t know - probably not in my view, but others may disagree.

[sirosen (Stephen Rosen)]

@adqm

I’m generally in favor of making more structures available in expressions, mostly because I tend to write code with a lot of comprehensions. If I’m writing something and decide that, for example, I want to do a json.load inside of a comprehension , my natural tendency is going to be just to use open(filename) and assume that things will get cleaned up properly, rather than unrolling things into a loop so I can use the context manager, or adding an import so I can use the pathlib stuff. I think it would be nice to have the “safe” version available as an expression, in the same way that we have lambdas and if expressions and walruses and all that good stuff.

This desire to have expression forms is something that I understand, and generally see as legitimate, but do not like for context managers in particular.

A context manager is, to me, an API which is about indentation. It says “there’s a resource to manage, and you could use try-finally to do it, but I want the indentation in Python to show that structure such that the dedent is analogous to the final cleanup.”
I say this because, to the point about pathlib APIs, there’s always a way to rephrase your API so that the context manager disappears from the layer at which you operate.

You mention comprehensions, which I think are a great example. The way I would personally restructure such code is typically to turn the comprehension body into a generator. e.g., iter_parsed_json_files(). That also means that the generator can have try-except, match-case, or any other syntactic form that I find useful as I iterate on the code.

I guess I would lean on try-except as my point of comparison here. Ruby has an expression form of it, rescue. Should Python add a rescue soft keyword, to allow an expression to contain try-except? More generally, are there any tools which we have in the language which should not go into expressions? What are they and why?
Please don’t take this as a slippery slope argument! I just mean to ask probing questions about “what’s special about context managers?”

I say all of this as a huge fan of the walrus. It has a relatively small number of extremely compelling use cases in which it greatly enhances clarity, and a more moderate number where it adds concision without harming clarity. The argument for “why allow assignments in expressions” starts (and pretty much ends) with “let me show you these use cases”.

[adqm (adam hartz)]

@pf_moore

IMO, the only place this construct is likely to be useful is in interactive code […]

This is the use-case I had in mind, quick loops involving file operations in the REPL, which I tend to use a lot.

---

@sirosen

The argument for “why allow assignments in expressions” starts (and pretty much ends) with “let me show you these use cases”.

Agree completely. And just to be clear, I’m not arguing that my use case is big enough to be worth the trouble; just saying that I would probably use this if it existed.

[NeilGirdhar]

@adqm

I’m generally in favor of making more structures available in expressions, mostly because I tend to write code with a lot of comprehensions. If I’m writing something and decide that, for example, I want to do a json.load inside of a comprehension , my natural tendency is going to be just to use open(filename) and assume that things will get cleaned up properly, rather than unrolling things into a loop so I can use the context manager

Yes, the tension between comprehensions and loops is unfortunate. If this is such a common thing for you, you may want to propose putting the with directly into the comprehension:

y = [json.load(f)
     for filename in filenames
     with open(filename) as f]

I have never needed this though.

[JoBe]

I think it’s pretty obvious that doing

y = []
for filename in filenames:
    with open(filename, "r") as file:
        y.append(json.load(file)

And as I’ve never even seen that being used (at least not very often), I don’t think we’ll need to create special syntax for this.

[tanj (itzmetanjim)]

@adqm

You’re certainly right that there’s not a big difference between those examples and the way they would be written with the proposed syntax, but I don’t think those examples illustrate places where this change would have its impact. In the thread I linked above, the proposal was similarly dismissed (by someone else) as being “code golf,” but to me this isn’t really about syntax, but rather about making context managers available in expressions.

(emphasis mine)
This expresses my idea better. That’s what I was meaning to say, saving a line was not my intention

[crj]

I’m aware of the “there should be one — and preferably only one — obvious way to do it” principle. Which does appear to argue against this proposal, because there is an obvious way to do it.

However, that principle offers less guidance when there’s a more obvious way to do it. At least in my code I suspect more than half the with statements would be more naturally written as an expression. It seems like a small quality of life improvement for many users in many circumstances weighs less heavily than making something very difficult much easier for a few people in uncommon circumstances?

I notice the walrus (:=) operator entered the language back in 2018, presumably with justifications very similar to those for what is being proposed here?

[JoBe]

I think talking about the walrus is a completely different topic, and it’s history is far from straight forward.

Just changing the syntax ‘cause the walrus changed syntax too’ is not really an argument. Most syntax sugars are for making long code be smaller. Here however I don’t really see how the current state is too long.

[tanj (itzmetanjim)]

Well, it’s correct that there are many statements that this could shorten. A simple GitHub code search suggests around 12.2% of Python with statements are simply storing f.read() into a variable, and ~7% are just storingjson.load(f) into a variable, for a total of around 19%. Adding with support in expression helps much more than just these examples, and these alone contribute to slightly less than one fifth of all with statements.

[peterc (peter)]

I’m -1.

To me a with expressions are a sign that you’re doing something complicated. And it’s more than fine that if you want to do something complicated, the code should look a little complicated too. To make hard things look simple is lying. (Or good programming. It depends )

There’s a reason .open() is traditionally a context instead of us just using expressions like json.read(f.open()). I can use it as json.read(f.open()), and besides my linter complaining nothing goes wrong. So far

At the same time, you can write (as long as f is a Path instance) f.read_text() or json.loads(f.read_text()) if you want a simple expression. You lose something in that conversion. (Some fancy optimisation tricks mostly.) That’s fine. I can’t think of any case where I’ve needed both the advantages of the context manager, and the advantages of the expression. If I ever do need both, I’ll write a small function.