This is a proposal for the ast module in the standard library, to include a BidirectionalTransformer that extends NodeTransformer with leave_* dispatch after normal visit_* processing.
class BidirectionalTransformer(ast.NodeTransformer):
def visit(self, node: ast.AST) -> Any:
node = super().visit(node)
if isinstance(node, ast.AST):
return self._leave(node)
if isinstance(node, list):
# If visit_* returns a list, then leave_* is called on each element of that list
result = []
for item in node:
if isinstance(item, ast.AST):
item = self._leave(item)
if item is None:
continue
if isinstance(item, list):
result.extend(item)
else:
result.append(item)
return result
return node
def _leave(self, node: ast.AST) -> Any:
method: str = "leave_" + node.__class__.__name__
leaver: Callable[[ast.AST], Any] = getattr(self, method, self.generic_leave)
return leaver(node)
def generic_leave(self, node: ast.AST) -> Any:
return node
Another name I have considered is BottomUpTransformer, but BidirectionalTransformer feels more appropriate given that visit_* methods are still called on the way down, and therefore the transformer does not necessarily not solely post-order.
Post-order processing can already be achieved with the NodeTransformer by calling self.generic_visit(node) inside a visit_* method before transforming the node, but this can obscure the true transformation order since the method body must be observed in order to determine the traversal order of each node type. For example, for a NodeTransformer that processes variable assignment and loads, and collapses constant expressions:
def visit_Name(self, node):
# Transform loads on the way down, so that the substituted expression has a chance to be processed
return self.generic_visit(substitute_variable(node, self.locals_))
def visit_Assign(self, node):
# Transform assignments on the way up, so that the RHS is processed first
self.generic_visit(node)
bind_assignment(node, self.locals_)
return None # delete the assignment since it has been bound in a local variable dictionary for later substitution
def visit_BinOp(self, node):
self.generic_visit(node)
if isinstance(node.left, ast.Constant) and isinstance(node.right, ast.Constant):
return expr_eval(node)
return node
Especially in a more complicated transformer that may have dozens of both pre-processed and post-processed node types, it is difficult to tell at a glance what types of nodes are transformed on the way down and which are transformed on the way up. The BidirectionalTransformer makes this much more explicit:
def visit_Name(self, node):
return self.generic_visit(substitute_variable(node, self.locals_))
def leave_Assign(self, node):
bind_assignment(node, self.locals_)
def leave_BinOp(self, node):
if isinstance(node.left, ast.Constant) and isinstance(node.right, ast.Constant):
return expr_eval(node)
return node
It is now much more obvious which nodes are processed when, and also eliminates the need for boilerplate self.generic_visit(node) code solely needed to implement post-order hooks. leave_* methods also make explicit that those nodes are always processed after their children.
A node type may also have both a visit hook and a leave hook, though note that if a node has been transformed to a different type, each visit_Foo call may not necessarily be paralleled by a leave_Foo call. For example, an ast.Spam node transformed to an ast.Eggs node in the visit phase will have visit_Spam and leave_Eggs called on it.
def visit_FunctionDef(self, node):
process_arguments(node)
return self.generic_visit(node)
def leave_FunctionDef(self, node):
bind_function(node, self.locals_)
return None
The same mechanism could theoretically be applied to NodeVisitor, but I have not discovered a compelling use case at the moment.