There are actually a few steps involved here before we even get to compile.c.
- There’s first a tokenizer/lexer that converts a stream of characters into a stream of tokens. This figures out what’s an operator and what’s a variable name, etc.
- Then there’s a parser at Parser/parser.c that converts the stream of tokens into an “Abstract syntax tree” (AST), which figures out things like “this is a for loop inside of a function”. However, parser.c is a huge file that is literally not designed to be read by humans – it’s computer-generated c code, based off of the “grammar” at Grammar/python.gram.
- Then the AST gets fed into the compiler at Python/compile.c, which compiles into bytecodes and assembles all of those bytecodes into a bytecode “code object”.
- Then bytecode within code objects is interpreted by the main interpreter at Python/ceval.c, which actually carries out the actions that your original python code was supposed to do.
As to why list comprehensions are a bit faster than for-loop-and-append, we can check out the bytecode. Below is Python 3.10:
>>> def func1():
... my_list = []
... for i in range(10):
... my_list.append(i**2)
... return my_list
...
>>> def func2():
... return [i**2 for i in range(10)]
...
>>> import dis
>>> dis.dis(func1)
2 0 BUILD_LIST 0
2 STORE_FAST 0 (my_list)
3 4 LOAD_GLOBAL 0 (range)
6 LOAD_CONST 1 (10)
8 CALL_FUNCTION 1
10 GET_ITER
>> 12 FOR_ITER 9 (to 32)
14 STORE_FAST 1 (i)
4 16 LOAD_FAST 0 (my_list)
18 LOAD_METHOD 1 (append)
20 LOAD_FAST 1 (i)
22 LOAD_CONST 2 (2)
24 BINARY_POWER
26 CALL_METHOD 1
28 POP_TOP
30 JUMP_ABSOLUTE 6 (to 12)
5 >> 32 LOAD_FAST 0 (my_list)
34 RETURN_VALUE
>>> dis.dis(func2)
2 0 LOAD_CONST 1 (<code object <listcomp> at 0x0000025DA56DD580, file "<stdin>", line 2>)
2 LOAD_CONST 2 ('func2.<locals>.<listcomp>')
4 MAKE_FUNCTION 0
6 LOAD_GLOBAL 0 (range)
8 LOAD_CONST 3 (10)
10 CALL_FUNCTION 1
12 GET_ITER
14 CALL_FUNCTION 1
16 RETURN_VALUE
Disassembly of <code object <listcomp> at 0x0000025DA56DD580, file "<stdin>", line 2>:
2 0 BUILD_LIST 0
2 LOAD_FAST 0 (.0)
>> 4 FOR_ITER 6 (to 18)
6 STORE_FAST 1 (i)
8 LOAD_FAST 1 (i)
10 LOAD_CONST 0 (2)
12 BINARY_POWER
14 LIST_APPEND 2
16 JUMP_ABSOLUTE 2 (to 4)
>> 18 RETURN_VALUE
The inner loop of func1 has LOAD_METHOD, LOAD_FAST, LOAD_CONST, BINARY_POWER, CALL_METHOD, POP_TOP where func2 has just LOAD_CONST, BINARY_POWER, LIST_APPEND. The difference essentially boils down to the fact that Python can know that the list comprehension is dealing with just a list, and so it doesn’t have look up what append means over and over.