Abstract
This idea proposes adding the end line number, as well as the start and end column offsets, of the lines that define a function scope, to the line table entry of the RESUME bytecode that signals the start of the scope. This information will be used to improve the accuracy of the source code returned by inspect.getsource in order to reduce the complexity involved in runtime profiling and code transformations.
Motivation
The primary motivation of this idea is to improve the end user’s ability to obtain the precise defining source code of a given code object at runtime.
Python currently offers the co_firstlineno attribute to a code object so that runtime inspection tools such as inspect.getsource can extract the code block that defines a given code object by tracking indentations with a tokenizer.
However, for code objects not defined by a code block, such as ones defined by a lambda or a generator expression, having just the co_firstlineno attribute is not enough to determine where it is defined because as an expression it can be defined anywhere in a line, possibly with multiple other lambda and/or generator expressions on the same lines.
Consider the following code of a runtime call profiler:
import sys
from inspect import getsource
from itertools import takewhile
def profiler(frame, event, arg):
if event == 'call':
print(getsource(frame.f_code))
sys.setprofile(profiler)
for i in takewhile(lambda x: x < 2, filter(lambda x: x % 2, (
i for i in range(1)
))):
pass
It currently produces the following output:
for i in takewhile(lambda x: x < 2, filter(lambda x: x % 2, (
i for i in range(1)
))):
for i in takewhile(lambda x: x < 2, filter(lambda x: x % 2, (
i for i in range(1)
))):
for i in takewhile(lambda x: x < 2, filter(lambda x: x % 2, (
i for i in range(1)
))):
From the output with the same entire for statement being returned as the source code for all 3 calls that take place, it is impossible to determine which of the 3 reported calls corresponds to which of the three functions in the returned source code, including two lambda functions and one generator expression.
With the implementation of PEP-657, each bytecode is now accompanied with a fine-grained position, including both the start and end column offsets, exposed via a public code object method, co_positions, which in theory can be used to aid the accuracy of inspect.getsource’s output.
The problem is, however, that the start and end of a lambda function or a generator expression do not correspond to a bytecode with a meaningful position. For example, using co_positions on a lambda function’s code object:
import dis
a = lambda x: (
x
)
print(*a.__code__.co_positions(), sep='\n')
dis.dis(a)
produces the output:
(2, 2, 0, 0)
(3, 3, 4, 5)
(2, 2, 0, 0)
2 0 RESUME 0
3 2 LOAD_FAST 0 (x)
2 4 RETURN_VALUE
From the output, all that can be determined is that the definition of the code object starts somewhere on line 2, and that the portion of its body that produces bytecode starts at column 4 of line 3 and ends at column 5 of line 3. The end line number of 4, the start colum of 4 and the end column of 1 are all missing from the output. It therefore remains tricky for inspect.getsource to extract the more precise source code that defines the lambda function, in this case:
lambda x: (
x
)
Rationale
As the output of dis.dis in the last code example shows, the position information of the RESUME bytecode, except its start line number, is currently meaningless, where the end line number is always equal to the start line number, the start and end column offsets always 0.
It would therefore be reasonable to store the precise position of the source code that covers the definition of the code object, in the line table entry for the RESUME bytecode, a no-op instruction meant to signal the start of a function when its arg is 0. That RESUME is always there at or near the start of a scope also means that inspect.getsource can efficiently obtain the position information of the scope in constant time without having to iterate through all the positions generated from the co_positions method.
Since the additional position information is already available internally in the AST for the compiler core, all that needs to be done is for the compiler to pass the additional position information when entering a scope, and to replace the hard-coded position of (lineno, lineno, 0, 0) with (lineno, end_lineno, col_offset, end_col_offset) for the RESUME instruction.
Specification
With the proposed change, the line table entry for the RESUME bytecode when its arg is 0, will cover the entire source code that defines the scope, except when the scope is of a module, in which case the entry would remain (1, 1, 0, 0) as it is now.
The position covering the entire source code that defines the scope will come from lineno, end_lineno, col_offset and end_col_offset of the statement and expression AST structs, after applicable decorators are taken into account for the start line number as they are now.
Backwards Compatibility
Code analysis and coverage tools that currently presume a no-op bytecode to have zero width in the line table should be refactored to special-case the RESUME instruction to take its position information with the new meaning.
Reference Implementation
A reference implementation can be found in the implementation fork, where the supporting functions for inspect.getsource are refactored to make use of the new semantics.
Performance Implications
The compiler will spend minimally more time to pass the 3 integers end_lineno, col_offset and end_col_offset when entering a scope. The efficiency gain for inspect.getsource from not having to reparse the source code to find the code block given a start line number should be significant.