There is also RustPython: GitHub - RustPython/RustPython: A Python Interpreter written in Rust
1 Like
I mostly agree with @timfelgentreff. From the PyPy point of view these small (and seemingly inconsequential) differences in behavior tend to be feasible, but often end up being somewhat annoying to follow because someone somewhere always ends up relying on every last precise detail. Therefore, the fewer behavioral quirks introduced by such a change, the better.
(Sidenote, GraalPy is pretty clearly the most well-resourced alternative Python from-scratch implementations at the moment.)
1 Like
FWIW, Cython’s FAQ is outdated. The latest Cython 3.0.0b1 basically supports Python 3.9 features. For those features that are not yet supported by Cython, please refer to: https://github.com/cython/cython/labels/Python Semantics.
Ah, thanks. I don’t really keep up with Cython, so all I know is what a few minutes of scanning the README and FAQ tell me. Thank you for the correction.
It’s not immediately obvious to me, what would happen with something like
def foo():
def bar():
print(x)
x = 42
bar()
_ = [bar() for x in [0]]
I am assuming that this case falls under the
Only comprehensions occurring inside functions, where fast-locals (LOAD_FAST/STORE_FAST) are used, will be inlined.
rule and so this comprehension won’t be inlined? Is that right?
So if I understand it correctly, the behaviour of locals, tracebacks and UnboundLocalError/NameError will change depending on whether bar has any references to x?
No, that’s a misunderstanding, but I see how the specific reference to LOAD_FAST/STORE_FAST lends itself to that misunderstanding.
Your sample case falls under the later paragraph that says:
In more complex cases, the comprehension iteration variable may be a global or cellvar or freevar in the outer function scope. In these cases, the compiler also internally pushes and pops the scope information for the variable when entering/leaving the comprehension, so that semantics are maintained.
So in your code example, the comprehension will be inlined regardless, whether x in the scope of foo is a cellvar, freevar, global, or local.
The paragraph you quoted is saying only that comprehensions at module scope (module top level, outside of a function) will not be inlined. That said, @markshannon has suggested offline an approach that could allow those to also be inlined for better consistency; I’m experimenting with that, and will either update the PEP and implementation to inline those also, or to clarify and strengthen the rationale for not doing it.
1 Like
I don’t see any mention of this: this will change the behavior of trace and profile functions installed with sys.settrace or sys.setprofile (or PyEval_SetTrace and PyEval_SetProfile).
This code:
def profile_func(frame, event, arg):
print(event, frame)
def func():
return [x for x in "12345"]
import sys
sys.setprofile(profile_func)
func()
sys.setprofile(None)
Currently gives this output:
call <frame at 0x7f26bbe91c60, file '<string>', line 4, code func>
call <frame at 0x7f26bbe08880, file '<string>', line 5, code <listcomp>>
return <frame at 0x7f26bbe08880, file '<string>', line 5, code <listcomp>>
return <frame at 0x7f26bbe91c60, file '<string>', line 5, code func>
c_call <frame at 0x7f26bbe91c60, file '<string>', line 10, code <module>>
If this PEP is implemented, the <listcomp> calls and returns will disappear.
1 Like
Yes, good point, the fact that the change will also be observable in tracing/profiling is worth mentioning in the PEP; I’ll add it.
I’ve updated both the reference implementation and the PEP to specify/implement the inlining of all list/dict/set comprehensions, not just those within functions. Thanks @markshannon for this suggestion.
I’ve also updated the PEP to mention that tracing/profiling will also be affected; thanks @godlygeek.
Appreciate everyone’s thoughtful comments; the PEP is better now than when I first posted it! I believe the PEP and implementation are now ready for SC consideration, but I’ll wait a couple days to see if there are any further comments before I submit.
2 Likes
Thanks. I think it looks great and I’m looking forward to seeing it hopefully in 3.12!
I do have one other question though which is how this works in debugging. Suppose I have this code:
def func(x):
result = [x for x in range(x)]
return result
print(func(10))
Then in pdb (python -m pdb script.py) after stepping forwards a few times I have:
> ./script.py(2)<listcomp>()
-> result = [x for x in range(x)]
(Pdb) l
1 def func(x):
2 -> result = [x for x in range(x)]
3 return result
4
5 print(func(10))
[EOF]
(Pdb) p x
3
(Pdb) u # up to func frane
> ./script.py(2)func()
-> result = [x for x in range(x)]
(Pdb) l
1 def func(x):
2 -> result = [x for x in range(x)]
3 return result
4
5 print(func(10))
[EOF]
(Pdb) p x
10
I often find that debugging is awkward in these cases because the listcomp function doesn’t see outer variables so inlining will help in that sense. My example here deliberately shadows an outer variable with a comprehension variable though because I wondered how the debugger would handle that case.
I think I know the answer but is it just that x will be 0, 1, 2, … while the comprehension runs and then become 10 again afterwards?
Is there a way to query the value of the outer x from the debugger during execution of the list comprehension (I don’t mind if the answer is no)?
1 Like
Yes, that should be the behavior.
No, I don’t think this is feasible. In your example, the outer value of x will be stored only on the (interpreter’s value) stack while the comprehension is executing, and it’s likely not reasonable to expect pdb to figure out where on the stack it is and go look for it (not to mention that inspecting arbitrary stack locations isn’t part of the frame API exposed to Python, and also that it’s not clear what pdb command would even make sense for this operation.)
Thanks all for the feedback and discussion! I’ve submitted PEP 709 to the Steering Council: PEP 709 -- Inlined comprehensions · Issue #179 · python/steering-council · GitHub
8 Likes
Does the implementation inline async comprehensions?
async doesn’t appear in the PEP.
I don’t have an opinion on this. I’m just curious.
It does. Async comprehensions are orthogonal to inlining, so it just falls out of the implementation without any additional handling. Some different instructions are used internally to the comprehension (GET_AITER instead of GET_ITER, etc), but this doesn’t impact inlining.
All existing tests for async comprehensions in test_coroutines.py pass unmodified, and I’ve verified with dis that they are inlined in those tests.
I can add a sentence to the PEP clarifying that async comprehensions are also inlined, thanks for pointing this out. I suspect the SC hasn’t gotten to the PEP yet anyway, and even if they have, it’s a minor clarification, not a substantive change.
There’s a bit of discussion of Cython above that I’ve only just seen.
That’s a reasonable summary of the current position. I don’t think the changes described here are going to make us lose any sleep if we don’t meet them. The frames to tracebacks and profiling probably just don’t apply. The changes to locals would be reasonable straightforward if anyone notices and complains.
Ultimately everyone who uses Cython is running it with a real Python runtime anyway, so they always have that option anyway.
Language compatibility target is mostly “as far forward as time to implement it permits”.
So to summarise, I don’t think this PEP is a real concern for Cython. It’s the sort of minor detail where we’re generally relaxed if we don’t quite hit.
1 Like
The Steering Council has decided to accept this PEP for 3.12, provided the implementation can be finished and submitted by May 22 (and preferably sooner :).
4 Likes
Will argumentless super() calls work inside list generators?
Here is a simple code snippet that does not work in Python 3.10:
class A:
def fnc(self):
return 3
class B(A):
def fnc(self):
return [super().fnc() for i in range(5)]
b = B()
print(b.fnc())
This finishes with:
TypeError: super(type, obj): obj must be an instance or subtype of type
Will it work as expected with PEP 709?
It looks like this does indeed work in a post-PEP-709 world:
On Python 3.11:
Python 3.11.2 (tags/v3.11.2:878ead1, Feb 7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
... def fnc(self): return 42
...
>>> class B(A):
... def fnc(self): return [super().fnc() for _ in range(5)]
...
>>> B().fnc()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in fnc
File "<stdin>", line 2, in <listcomp>
TypeError: super(type, obj): obj must be an instance or subtype of type
With a reasonably fresh build of CPython from the main branch:
Running Debug|x64 interpreter...
Python 3.13.0a0 (heads/main:6c60684bf5, Jun 28 2023, 12:52:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
... def fnc(self): return 42
...
>>> class B(A):
... def fnc(self): return [super().fnc() for _ in range(5)]
...
>>> B().fnc()
[42, 42, 42, 42, 42]
>>> exit()
I’m not sure whether this was intentional, however
I’m also not sure if the older behaviour, and the same behaviour with generator expressions, is intentional ;-). I understand why this raises an exception, but this feels more like an accident of the implementation than intentional.
5 Likes