PEP 841: Adding Frozen Syntax to Make Immutable Types Optimizable

PEPs
42

This is an interesting way of framing the motivation. Looking at the class constructors amongst the 3.15 built ins:

bool()
bytearray()
bytes()
complex()
dict()
float()
frozendict()
frozenset()
int()
list()
memoryview()
object()
property()
range()
set()
sentinel()
slice()
str()
super()
tuple()
type()

Of these: bool, bytes, complex, dict, float, int, list, set, str, and tuple have existing display syntax. slice also has dedicated syntax, albeit technically not a display. memoryview, object, property, sentinel, super, and type largely don’t make sense as displays (for varying reasons).

That leaves bytearray, frozendict, frozenset and range which could have a meaningful display, but don’t. bytearray is particularly interesting as an inversion from the situation posed here: a mutable variant of a more common immutable type. I don’t think anyone is clamoring for bytearray displays (except perhaps in an appeal to consistency).

So built in value types frequently, but not always, warrant dedicated syntax (usually a display). It’s a point in favor of making frozenset and frozendict displays, but certainly not enough on its own.

5 Likes
43

the fact you can iterate on a tuple is a design mistake IMO

Sorry if I’m missing something, but how can you unpack a tuple into variables in an assignment if it is not iterable?

1 Like
44

Thank you for pointing this out. We need to update this and preserve the current ordering guarantees.
cc @sobolevn

5 Likes
45

I’m +1 on the idea of frozendict and frozenset having display syntax, but am +0 on the PEP as is.

The motivation and framing (even the title) are too heavily focused on optimization and undersell other reasons to want this (such as making immutable collection types first-class equals with mutable ones). Steve landed a number of valid criticisms regarding how valuable such optimizations will be in practice; bringing in other motivations will strengthen the PEP overall.

I also dislike the syntax, for two reasons:

  1. It overloads the meaning of the “f”-prefix (and no, claiming that f-strings are also “frozen” doesn’t fly; all strings are frozen).
  2. Braces, particularly when used to define sets and dicts, are conceptually much closer to brackets and parentheses than to quotation marks. Yet letters “prefixing” parentheses and brackets have a very different meaning than what is proposed here.

While I have some thoughts on alternative syntax options, I’ll refrain from airing them unless the PEP authors are interested in revisiting that topic.

8 Likes
46

I think this is a valid criticism. If the PEP currently focuses too heavily on optimization, we are also open to revising its framing. We believe the display syntax is valuable in its own right, while the optimization opportunities are an additional benefit. (Well, I am too biased for this view)
We can revise the PEP accordingly and then continue the discussion in a second round.

We are open to considering alternative syntax options(then we may need 2nd round discussion for this)

3 Likes
47

Revised commit: PEP 841: Adding Frozen Syntax to Make Immutable Types Optimizable by corona10 · Pull Request #5049 · python/peps · GitHub

1 Like
48

I echo Jon Harding’s sentiments.

I see in the rejected ideas that the suggested syntax of {|'key': 1|} are rejected because:

  1. adding a single token f{ is easier than adding two tokens, and
  2. writing f{ is also easier than writing two different brackets.

To address both points, how about {|'key': 1} instead, so that it adds just one token, and requires writing only one different bracket? The mnemonic will be like {| looks like a side view of a shield that shields the content from being modified.

This notation has a side benefit of being possibly extensible to a tuple/frozen list comprehension in the future, e.g. (|... for _ in range(3)) or [|... for _ in range(3)].

`frozenset` and `frozendict` comprehensions
49

I just want to push back against this a bit. Things that get into Python — syntax, yes, but also builtins and even the stdlib — become part of what everyone using Python has to be somewhat aware of and conversant with, even if they themselves never use the functionality in questions, because they have to at least be able to read it in code they come across. Personally I most definitely do not want Python to take a “live and let live” attitude in this sense, which would lead to the language becoming a heterogeneous amalgamation of whatever appealed to some subset of users. It’s contrary to the “one obvious way to do it” philosophy.

Now, in the end it depends what appeals to the most people, and if stuff gets in that I don’t personally like I’ll survive[1]. But the bar has to be a lot higher than just “some people would find this useful”. The impact on the required level of “reading comprehension” for the community at large has to be considered.

Even if that is a goal, I still think it’s premature for the reason you gave in your parenthetical remark there. I made a similar comment in the other thread about methods to convert dict to frozendict.

Maybe contrary to some others here I’m not super in love with the idea of “oh this is just a small PEP that makes this one change”. If we want syntax for frozendicts without waiting to see how people use actually frozendicts, that should have been part of the frozendict PEP. We should not “salami slice” changes by introducing one change, and then piling a bunch of other related changes in separate PEPs right away, on the argument that now that Change X made it in, suddenly it justifies changes Y and Z.

Obviously, once a certain change is in place, the community may after some time realize that additional changes are a good idea to support it. But that’s not because the new features exist, it’s because we’ve learned through experience how we use them and what the rough edges are. If we know that adding frozendicts will mean we’ll want to add syntax for them, we should do both together; if not, we should wait and see. But in my view the mere fact that a PEP adding frozendicts was accepted has no bearing on whether we should add syntax for them; that change can only be justified after some period of experience with frozendicts.[2]

  1. fortunately for me, since it keeps happening over and over! ↩︎

  2. Even then we should be careful. To my mind, the amount of energy expended on various typing PEPs is an example of how things can go wrong once a “triggering” language change is introduced. ↩︎

7 Likes
50

IMHO adding two tokens is only marginally less easy than two. On the plus side, one can interpret {| and |} as two-char bracket which gives a nice symmetry, and may be more readable. Generally {|'key': 1|} and {|'key': 1} are valid candidates.

Maybe we need to first establish that frozen literals are worth it, and have a shortlist of reasonable spellings. That could decouple the discussion of “I don’t like the syntax” from “this is not necessary”. So

  • Step 1: Assuming we can find a reasonable spelling (and there a couple of good candidates to give ideas how this could look like), does adding frozenset and frozendict literals add enough benefit?
  • Step 2: Discuss the exact spelling. The candidates each have some advantages and disadvantages, which requires careful judgement. f{} is one of the good candidates, but locking in on it feels a bit premature.
    Note: The result of this can also be that there is no reasonable spelling, which is arguably the state of the empty set literal discussion.
4 Likes
`frozenset` and `frozendict` comprehensions
51

How this expected to be working?

This is a new syntax, which people will have to learn. Even assuming prior knowledge about frozenset/dict. While, with a same assumption, frozenset({1, 2, 3}) will be obvious for a reader.

Less characters to type? :slight_smile:

If this is all, I don’t buy it: “code is written once, but read many times” (c). Current state of the language makes using frozen sets/dicts more explicit. And that’s good.

BTW, I think the strongest argument in favor of the PEP — if it will allow some additional optimization, which are impossible (not just hard to do) currently.

2 Likes
52

I think this is not quite fair. There is a difference between “salami slice” and keeping the scope focussed. While I was not involved in the PEP-814 discussion, listing frozen literals under the “Deferred Ideas” section to me appears as a measure of focussing: “Adding frozen datatypes is a good idea”. Whether literals are justified as well can be discussed separately, and deferring it from the PEP is neither an argument for or against a follow-up on any timescale.

3 Likes
53

To me the benefit is both. For example, I’ve never written:

if item in frozenset({'foo', 'bar', 'baz'}):

but instead always write:

if item in {'foo', 'bar', 'baz'}:

precisely both because the former is many more characters to type and because it’s less efficient to run.

If a concise frozenset literal is made available I’d gladly switch to it from a set literal.

2 Likes
54

Yeah, this sounds better - something that sounds more in line with status quo of acceptable motivations that matter.

But reality is that performance is as important - especially to those who would like to do more and more things with Python and would have to resort to other languages due to performance reasons as little as possible.

Well, I don’t know what proportion of people are in this bracket, but there is definitely one. :slight_smile:

55

Performance can only be an added benefit. It’s unlikely that dicts/set are defined in hot loops, and if so and they are constant, they can be pulled out. So performance will aways be just a mirco optimization, not significantly influencing overall program performance.

The real question is: Do we want immutability as a first-class concept that is widely adopted, or is it just for some special cases. Only the intended wide-spread adoption would warrant new syntax with the cost of learning and adaptions in the ecosystem. And conversely, a literal syntax would foster adoption.

3 Likes
56

Weak argument for me. Perhaps, it does matter for interactive coding in some REPL. Though, even here auto-completion could help and I’m not sure that differences with immutable types are relevant here.

Entirely not relevant for usual coding. Use evil enough code editor :wink:

That’s a good argument. Assuming you can convince me, that former case can’t be optimized to avoid a temporary mutable dict, etc.

57

FWIW, frozenset is used already.

def in_set(x):
    return x in {'foo', 'bar', 'baz'}

"""
Disassembly of <code object in_set at 0x107c06410, file "f1.py", line 1>:
  1           RESUME                   0

  2           LOAD_FAST_BORROW         0 (x)
              LOAD_CONST               1 (frozenset({'bar', 'foo', 'baz'}))
              CONTAINS_OP              0 (in)
              RETURN_VALUE
"""

def in_list(x):
    return x in ['foo', 'bar', 'baz']

"""
Disassembly of <code object in_list at 0x107c05530, file "f1.py", line 4>:
  4           RESUME                   0

  5           LOAD_FAST_BORROW         0 (x)
              LOAD_CONST               1 (('foo', 'bar', 'baz'))
              CONTAINS_OP              0 (in)
              RETURN_VALUE
"""
11 Likes
58

Ah very cool. So that takes away some of the optimization argument from the proposal, although still if the set literal is assigned to a name first before performing the in operation the compiler won’t be able to make it a frozenset.

60

But with a frozenset literal it’s then foldable:

def do(a, b):
    allowed = f{'foo', 'bar', 'baz'}
    if a in allowed and b in allowed:
       ...

which isn’t possible if the above were written with a set literal.

1 Like
61

Sorry, I misread. That is a correct example where frozenset literals are useful. Without literals, you would need a global variable or a hidden keyword argument hack to avoid constructing the set every time.

We might be able to use the same compiler optimization by adding a .freeze_and_clear() method to set and dict. The advantage of using a method instead of display is that it can also be used when you want to create a frozendict from kwargs in O(1).

    allowed = {'foo', 'bar', 'baz'}.freeze_and_clear()  # compiled into LOAD_CONST frozenset
62

A downside of the double parentheses in frozenset((...)) is formatters such as Black and Ruff expand to add an extra level of indentation:

class _User(pydantic.BaseModel):
    __dmr_force_list__: ClassVar[frozenset[str]] = frozenset(('tags',))
    __dmr_cast_null__: ClassVar[frozenset[str]] = frozenset(
        (
            'promo_id',
            'tags',
        )
    )
2 Likes