I would go even further and say that’s an argument against the syntax. Do we want user’s cargo-culting this because one is currently better optimized? The use of a frozendict there isn’t obvious. The dict has no reason to be frozen other than the interpreter’s underlying optimizations.
I agree that users should not choose frozendict only because it is currently better optimized. But I think this is a separate issue from whether we should add syntax for it.
Python has already decided to add frozendict. We also already have syntax and compiler optimizations for tuples, sets, and dictionaries. So I am not sure why frozenset and frozendict should not get the same support.
I don’t think it is. Syntax pushes users to writing specific things. The question about syntax is not just what does it enable, but what does it encourage.
If you show up with an example of “I optimized this beyond the optimizations dict currently gets for a case that shouldn’t care about it being frozen or not”, it’s going to push people to use it for reasons that shouldn’t exist in the first place solely for that optimization. This shouldn’t be used as an example if you actually believe what you are saying about people not using it strictly for optimization.
Yes, encouraging people to use immutable containers more often is one of the goals, and I agree that the PEP should make this clearer.
However, the goal is not to encourage people to use them when immutability is not needed. The goal is to reduce the current difference in language support. Python provides display syntax for mutable sets and dictionaries, while their immutable counterparts must be created with function calls. Providing similar syntax would make these container types more consistent and symmetric.
I understand your concern. My intention was to show that adding syntax has benefits beyond providing a shorter notation, including compiler optimizations that already exist for other built-in types in CPython.
Because we have many built in types and most don’t rise to the level of getting dedicated syntax. “We made it built-in” isn’t a compelling case on its own, you need something to distinguish it from Decimal, complex, OrderedDict, slice, bytearray, memoryview, files, dataclasses, enums, and a bunch more stuff that comes from the stdlib but may not have been considered for syntax. (Personally, I’d find a lot more value in a prefix for a path that converts it into the sequence of lines read from that file, and would argue that one would improve all the criteria I mentioned in my earlier post.)
Something being “absent” from the standard library is not the same as inexpressible in the language. We add things to the standard library to provide the batteries needed for libraries to interact with each other, or to implement things impossible to provide from outside the runtime.
We add things to the language to improve expressibility - the ability of Python developers to write code that is fundamentally more powerful, more expressive, or more readable. It’s a totally different bar to qualify for. Adding the frozendict type is not the same discussion as adding something like decorators or if ... else, and doing one certainly doesn’t imply that the other should be done. Equating the two is a persuasion trick.
I might need a more complex example then, because this one doesn’t benefit from either the syntax or the function call to frozendict. Yes, it may benefit from the optimisation, but you’d get the same benefit from declaring the dict outside of the function, as well as the benefit of more readable code. At this point, the only reason to make it frozen is to prevent “people” from changing it, which is the bit that starts to go against Python idioms.
_STATUS_CODE_TO_REASON_NAMES = {
# 50 lines of codes that don't need to be read
}
def describe(status):
return _STATUS_CODE_TO_REASON_NAMES.get(status, str(status))
You also can’t blindly treat every frozendict as a constant, since it’s totally fine to put object references in there that are only defined at construction time. If you can deoptimise f{ like that, then we can equally apply the same optimisation to { and catch all the “only used for literal lookups” cases. I’m pretty sure we’ve also previously looked at making dict copy-on-write, which means you can even store the constant and have a very cheap way to create a new instance (and since setting an item could cause rebucketing anyway, we aren’t even breaking perf promises to guarantee that the first write will always reallocate the internal data).
So we can give most users the benefit of the optimisation idea you’ve got here for free without changing their code. That’s great, we should do it! It doesn’t actually require new syntax, and so the core of the proposal isn’t justified on the basis of the potential optimisation.
As I said earlier, the matmul PEP is the best example of this kind of proposal. We should want to see real code that has a need to be written, can’t already be equivalently written in a more Pythonic way, and would be considerably more readable under this proposal. Those are completely orthogonal to the performance characteristics, which don’t matter as much here as they did when proposing the data type itself.
But I do think there’s a hidden point here, to encourage people to use frozen sets and dicts more often - and I think if that’s genuinely a goal of the PEP, it should be made explicit.
One of the goals of this PEP is to increase the use of immutable containers
in CPython, preparing for a concurrent future in which free-threading and
subinterpreters become significantly more common.
Efficient creation of immutable data structures and convenient syntax would
encourage the use of frozendict and frozenset. This would reduce bugs
caused by accidental mutation of shared mutable containers while also
improving performance. For example, subinterpreters already share frozenset objects, and there are plans to share frozendict objects as
well.
We add things to the language to improve expressibility - the ability of Python developers to write code that is fundamentally more powerful, more expressive, or more readable. It’s a totally different bar to qualify for. Adding the frozendict type is not the same discussion as adding something like decorators or if ... else, and doing one certainly doesn’t imply that the other should be done. Equating the two is a persuasion trick.
I believe that writing a PEP is part of the process of making the case for why a change is needed. That does not mean everyone has to agree with its arguments, but I do not think presenting those arguments should be characterized as a persuasion trick.
I agree that, in this particular example, the same performance benefit could be achieved by moving the dictionary outside the function. We may be able to optimize it further, but that is somewhat beside the point. I included the optimization section to describe an additional benefit that dedicated syntax could enable. If the current title places too much emphasis on optimization, I am happy to change it.
I also do not think read-only mappings inherently go against Python idioms. Existing idioms developed before Python had a built-in immutable mapping, so there was no direct way to express that intent. Now that frozendict exists, usage may naturally evolve, especially if the syntax feels Pythonic.
As I said earlier, the matmul PEP is the best example of this kind of proposal.
I understand why you refer to the matmul PEP, but I do not think the two proposals should be evaluated in exactly the same way. The matmul PEP introduced a new operator and its associated protocol. This proposal instead extends the familiar dictionary display syntax with an f prefix while preserving its existing structure and readability.
class _QueryModel(msgspec.Struct):
__dmr_force_list__: ClassVar[frozenset[str]] = f{'query'}
# vs
__dmr_force_list__: ClassVar[frozenset[str]] = frozenset(('query',))
And:
class JobController(Controller[PydanticSerializer]):
# Disabling just to avoid fixing the real issue:
no_validate_http_spec = f{HttpSpec.empty_response_body}
# vs
no_validate_http_spec = frozenset((HttpSpec.empty_response_body,))
The last one:
class _User(pydantic.BaseModel):
__dmr_force_list__: ClassVar[frozenset[str]] = f{'tags'}
__dmr_cast_null__: ClassVar[frozenset[str]] = f{
'promo_id',
'tags',
}
username: str
promo_id: int | None
tags: list[str | None]
# vs
class _User(pydantic.BaseModel):
__dmr_force_list__: ClassVar[frozenset[str]] = frozenset(('tags',))
__dmr_cast_null__: ClassVar[frozenset[str]] = frozenset((
'promo_id',
'tags',
))
username: str
promo_id: int | None
tags: list[str | None]
class Example(msgspec.Struct):
items: frozendict[str, int] # requires Python3.15+ to be used
Example(items=frozendict({"pen": 1, "book": 2}))
# vs
Example(items=f{"pen": 1, "book": 2})
I would advocate that this new proposed syntax is much better.
To be clear, the “trick” I’m suggesting is “we added the type, therefore we’ve basically already decided to add the syntax” (exaggerated paraphrase to try and make it more obvious why it seems like a false equivalence).
The PEP process is absolutely the right process, but that process involves engaging with disagreements over the arguments, not simply dismissing them. This feels like they’re being dismissed - “we wrote a PEP, therefore we get to make the change in spite of opposition” (also exaggerated ) is also not something that’s assumed.
No, my point is that the dedicated syntax only makes it marginally easier to enable the optimisation, and it seems like the optimisation could be enabled anyway without the syntax.
The cost of new syntax also has to be factored in. That cost includes that it creates entirely incompatible code (you can’t even runtime detect within a source file on old versions), additional work for every parser or highlighter out there, additional teaching work, additional burden on people reading code, documentation, etc. A PEP proposing syntax without justifying all of these costs is an incomplete PEP.
Excellent! Thank you so much for bringing some real examples.
Ultimately, I disagree with you, based solely on reading the code to understand what’s going on.
In none of these cases is the fact that it’s a frozen collection critical to my understanding of what the code is doing. All but the last example are DSL-like definitions, and so while being defined as a frozen collection may be a microoptimisation, they could just as well be any sequence type without affecting my understanding of the code. So that to me fails to prove the “makes it easier to read the code” case, since there are other legitimate ways to provide the same value.
(To cut off a likely counter-argument, that you shouldn’t be changing those Pydantic fields after definition, it has always been okay in Python for changes to “read only” data to be merely ignored after they’ve been used. Changing it might not achieve what you want, and that’s why you shouldn’t be writing code that changes it, but it’s never been something the language has tried to enforce.)
As far as making it easier to write, there is indeed less typing. Not as much less as if any other data type were used, but I guess the design of ClassVar has painted us into a corner here. If it supports type inferencing by omitting the [frozenset[str]] part, then leaving that out would save more typing and make it more likely that I’d get it correct on the first try. Replacing frozenset() with f{} is not as big a benefit.
The msgspec example is the strongest one, since that is a legitimate case where you are passing a data structure into some code that may end up simultaneously accessing it from multiple threads. Allowing a mutable data structure here may well be dangerous, if the caller is still mutating it (which, IMHO, is a bug, but let’s assume they’ve got a bug and we don’t want to crash). Still, it should be entirely reasonable for Example to do the conversion on behalf of the caller, if it’s important to protect them from themselves, and this looks like a scenario unlikely to benefit from a stored constant, which means the optimisation has no real benefit. So this one doesn’t really win me over either.
Apologies for multiple long posts, I think I’ve probably said enough here now, but I wanted to “show my workings”, as it were, so I’m not unreasonably accused of opposing the proposal based solely on style or whether I’d use it myself or not. There are a lot of pros and cons to any language change, and I don’t believe they’ve been accounted for in the PEP text yet. My hope is that they will be accounted for, as that will make the PEP much stronger and more likely to be accepted, but my expectation is that the cons can’t be justified sufficiently against the pros when they are evaluated in full.
Rightly so, IMHO. Adding new syntax to the language has to be the highest bar for a proposal for several reasons. It can make the language harder to keep in your head, harder to teach, and harder to read. It also takes quite a while for new syntax to work its way into the ecosystem.
On the latter point, imagine f{} is accepted. Mostly libraries won’t be able to adopt it until after 3.16 is the minimum supported active version. And even then, people will continue to use unsupported versions of Python for which new syntax is a SyntaxError.
On the former point, something like frozenset({1, 2, 3}) is completely obvious. f{1, 2, 3} is less so. The counter argument is that eventually people will figure it out, just like with f-strings.
That all said, adding new syntax must continue to require a very high bar for acceptance. It has to noticeably and convincingly improve the UX of the language[1].
I strongly agree on the fact that one should always use immutability by default and mutability if needed. The biggest impact however would be encouraging tuples usage instead of lists, which are data structures considerably more used.
however, the biggest downside for me on this proposal is readability.
For example, I know how to write code in python and Rust. I can read some java, kotlin, typescript or elixir code and very quickly and inutitevely understand what is going on. Now with languages were special syntax instead of words is the norm, like C, C++, I find it absolutely unreadable. Words have meaning, operators much less if you don’t have context. I highly doubt that C++ code is inerenthly more complex than rust code.
My point is: python IS designed to be redabable, just like english.
The fact that many libraries have taken the IMO discutable choice of abusing operators overloading (e.g pandas, numpy) instead of explicit methods is not something I’m not a fan of.
now with new tools (polars…) there’s a shift in the other direction, thankfully, but the last thing I would like is python stdlib itself adding more and more special syntax just to save a few words. I really struggle to see how f{} is more convenient to write than frozenset(). Especially in python compared to other languages, since you don’t have to reach for curly brackets as often.
And about optimizations, python biggest runtime slower in my experience is function call overhead, not a few constants having to be created. This could provide some marginal improvements, but if that’s a concern, you have much higher priority to check at the start of your program: do you import slow modules (i.e dataclasses), do you use slots whenever possible? Etc..
IMHO adoption time is only a minor argument. Any language change must be strategic, so the key question is: “Does it make the language better in the long run while not hurting in the mean time?”.
Granted, new syntax is harder to adopt because you can’t use it conditionally (for frozendict() you can check the Python version and use a regular dict if it’s not supported). But that just means adoption takes longer. The decision today is whether we want to have the frozen literals systematically available in 5-8 years.
To me the strongest motivation is that immuatbility is beneficial to concurrency and thus will aid free threading.
I believe that the frozen literals would foster adoption. Personal opinion: If there is {} and f{}, I would perceive both as first-class data structures, and systematically choose between them depending on context. In contrast, if there’s only frozendict() it feels like one of the specialized variants like OrderedDict or defaultdict, which you only reach for in specific situations. The bar for using frozendict() is much higher. Yes, there is the difference that frozendict() is a builtin and the others are not. But in my subjective perception builtin is a weaker indicator for “first-class fundamental data structure” than “has a literal”.
Edit: Or maybe also the other way round: If a data type is important enough to be a builtin, it’s a strong candidate for also having a literal syntax.
(This is basically why I dislike the PEP itself – it feels like “thinking small” / not very imaginative about pushing the boundaries of python.)
Yes, this is exactly the point We are thinking with a very small scope (which is a single new token), so we can achive a very small goal (representing two frozen counterparts to existing builtin types), without changing any semantics / introducing new concepts / etc.
While I haven’t decided my personal view on the PEP, one thing it would make me want to have is proper typing support for frozendict like we have for dicts via TypedDict, e.g. some mapping type support in the type system. Otherwise we will end up with a type considered so critical that it has display syntax support and yet you can’t fully express it in the type system.
Tuples are not an immutable list; they are 2 different data structures. Tuples are for heterogeneous data and lists are for a sequence of homogeneous data (the fact you can iterate on a tuple is a design mistake IMO). If you wanted immutable lists it would require a “frozenlist” type.
The fact that this is narrowed down to minimal incision is IMO a positive.
However, I don’t agree that it is thinking in such a small scope.
When I wrote my initial suggestion, I was primarily thinking about how to cover all 6 base types in consistent manner in terms of convenience and performance.
So that people can use the most theoretically appropriate type instead of finding edge cases where some counterpart is faster / more convenient and using the wrong type.
Yes, it is not a massively pervasive innovation, but I think it has a good balance of impact and orthogonality.
Other improvements, such as optimizations on builtin function and method calls can be tackled independently.
I think the comment regarding small scope would be more applicable to len(string_literal) optimization. I think there might exist a generic way to trace whether called function/method is builtin at runtime and go via fastpath. is is blazing fast in C
Well, it would not be possible to reserve f[...] for a future frozenlist, which could be a reason to reconsider the proposed frozenset and frozenlistsyntax now.
That’s really just a convention, and one I see broken all the time. Yes, two different data structures (obviously), but other than mutability, their function is similar: iterable sequences of objects, and that’s about it. There’s no language or runtime level enforcement of the contents of those sequences. That snake done slithered.
I don’t think a ‘frozenlist’ would add much to the language or its expressibility.
Speaking of benefits, what contexts are there a benefit? It’s usually accepted that a small number of elements are cheaper to handle with linear search than doing a hash calculation, and I’m not sure where that line gets crossed but I’m fairly confident I don’t want to type out a big enough frozen set to make it worth it not just being a tuple.
in is the only possible operation that someone would do with a set/frozenset?
Anyway, even for the sake of just in there’s a clear performance difference on even a small tuple. This seems like a weak argument, even for a small amount of elements, doing a containment check with a set over a tuple, in a hot enough loop, can be noticeable.
In [1]: a = frozenset(range(1,5))
In [2]: b_tuple = tuple(range(10_000))
In [3]: b_set = frozenset(range(10_000))
In [4]: %timeit a.issubset(b_tuple)
58.8 ns ± 0.181 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [5]: %timeit a.issubset(b_set)
17.4 ns ± 0.0506 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [6]: %timeit a.symmetric_difference(b_tuple)
55.1 μs ± 329 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [7]: %timeit a.symmetric_difference(b_set)
18.8 μs ± 71.3 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [8]: %timeit a.symmetric_difference(tuple(range(1,6)))
115 ns ± 0.453 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [9]: %timeit a.symmetric_difference(frozenset(range(1,6)))
120 ns ± 0.42 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [10]: b_tuple_small = tuple(range(1,6))
In [11]: b_set_small = frozenset(range(1,6))
In [12]: %timeit a.symmetric_difference(b_tuple_small)
68.5 ns ± 0.215 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [13]: %timeit a.symmetric_difference(b_set_small)
56.5 ns ± 0.666 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
In [14]: %timeit 0 in b_tuple_small
18.1 ns ± 0.0669 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [15]: %timeit 3 in b_tuple_small
11.7 ns ± 0.0179 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [16]: %timeit 7 in b_tuple_small
18.2 ns ± 0.0595 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [17]: %timeit 0 in b_set_small
8.54 ns ± 0.0318 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [18]: %timeit 3 in b_set_small
8.63 ns ± 0.0918 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
In [19]: %timeit 7 in b_set_small
8.52 ns ± 0.0701 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)
how is a list for homogeneous data and a tuple not, if at runtime they both accept wathever you give them, with an iteration speed who’s more or less the same?
I haven’t really decided how I feel about the PEP as a whole yet (which means I’m somewhere between -0 and +0), but one specific “not OK” on a technical detail:
Constant deduplication. Frozen constants participate in co_consts deduplication: equal frozen displays within a code object share a single object (frozendict keys are compared insertion-order-independently, matching frozendict equality).
This would make the iteration order of frozen dict displays ambiguous, since their equality check doesn’t take iteration order into account.
That is, assert list(f{"a": 1, "b": 2}) == ["a", "b"] might be false if f{"b": 2, "a": 1} appears elsewhere in the same code block.
This is a problem due to the key ordering guarantees offered by the current function based spelling.
) is also not something that’s assumed.