New syntax `Trailing Block` for constructing objects with complex structure

Ideas
1

Dear friends: in this topic I am proposing a new syntax named Trailing Block or Trailing Compound Statement, for the purpose of constructing or instantiating objects with complex structure, especially tree structure. My vison for this proposal is that with this syntax sugar, Python can natively speek complex data structure as conveniently as XML or other declarative DSL does.

The new Trailing Block syntax consists of three parts:

  • new special type Trailer,
  • new dunder method __trailing__, named Trailing Method,
  • new block syntax & variable scope when constructing objects.

Let me take an example to explain my idea. With present python syntax, to construct a tree, usually you have to the following codes:

class Node:
    def __init__(self, *args, **kwargs):
        self.child_nodes = []
        ...

    def add_child(self, node):
        ...
    
root_node = Node(...)
child_node = Node(...)
grandchild_node0 = Node(...)
grandchild_node1 = Node(...)

child_node.add_child(grandchild_node0)
child_node.add_child(grandchild_node1)
root_node.add(child_node)

The add_child method is used repeatly for every level of the tree. With the proposed syntax, such codes are placed in __trailing__ method, as

class Node:
    def __init__(self, *args, **kwargs):
        self.child_nodes = []
        ...

    def add_child(self, node):
        ...

    def __trailing__(self, *args, **kwargs):
        """ trailing method """
        # codes that called whenever a node is constructed
        ...

Trailer refers to the type of object that with a __trailing__ dunder method. A Trailer object, right after constructed, can be followed by an indent code block, named trailing block, such as

child_node = Node(...):
    # trailing block
    Node(...)  # grand node 0
    Node(...)  # grand node 1

What connects __trailing__ method and Trailing Block is that local variables of the trailing block are passed to the arguments of __trailing__ method:

  • new Trailer instances, whether assigned to varibles or not, are passed as the positional arguments of __trailing__ method;
  • any assigned local variables are passed to the keyword arguments of __trailing__ method, just like passing the result of builtin method locals.

In the example above, the 2 newly constructed Node instances are passed to the __trailing__ method of Node class. And we complete the class definition as

class Node:
    def __init__(self, *args, **kwargs):
        self.child_nodes = []
        ...

    def add_child(self, node):
        ...

    def __trailing__(self, *args: Trailer, **kwargs):
        """ trailing method """
        for node in args:
            if isinstance(node, Node):  # it is OK to use the class itself here
                self.add_child(node)

And thus the tree constructing can be greatly simplified:

root = Node(...):
    Node(...):  # child node
        Node(...)  # grand node 0
        Node(...)  # grand node 1

Further more, you can set attributes for each node with clear syntax

root = Node(...):
    # attributes
    red_or_black = True

    Node(...):  # child node
        # function attributes
        def on_walk(...):
            ...

        Node(...)  # grand node 0
        Node(...)  # grand node 1

and define attribute-setting codes in __trailing__ method:

class NodeTrailingType(TypedDict):
    """ pep-0692: use `TypeDict` for kwargs typing """
    red_or_black: Bool
    on_walk: Callable


class Node:
    def __init__(self, *args, **kwargs):
        self.child_nodes = []
        self.red_or_black: Bool = True
        self.on_walk: Callable = None
        ...

    def add_child(self, node):
        ...

    def __trailing__(self, *args: Trailer, **kwargs: NodeTrailingType):
        """ trailing method """
        for node in args:
            if isinstance(node, Node):  # it is OK to use the class itself here
                self.add_child(node)

        # set attributes
        self.red_or_black = kwargs['red_or_black']
        self.on_walk = kwargs['on_walk']

Some clarification:

  • Maybe Trailing block syntax seems to the with statement without with keyword itself.
    However, the purpose and what focus on are significantly different. Trailing block has nothing to
    do with defer usage.

  • It does not conflict the Zen of Python: flat is better than nested. The new syntax is designed specially to handle nested object.

I am eager to hear your comments about this idea :blush:

3 Likes
2

Not sure what you mean by ‘defer’ here, but you’re right that this does seem very similar to a context manager. Another thing to consider, though, is a class statement. This is actually extremely similar to what can be done that way, for example:

class root_node(Node): # abuse of subclass notation here, for the moment
    class child_node(Node):
        gc1 = Node(...)
        gc2 = Node(...)

While I wouldn’t call this good design, it’s certainly possible, and might lead to some further inspirations. I have certainly made some use of class blocks for this sort of “nested namespace” in the past.

1 Like
3

Very grateful for you reply! This is my first topic on the discuss.python.org, I feel honer to get comments from you :blush:

The defer I used here meanings to release resources, just like the defer keywords in golang language. I looked up the with statement PEP 343 and it says:

This PEP adds a new statement “with” to the Python language to make it possible to factor out standard uses of try/finally statements.

I interprete this words as the same meaning of defer. Maybe it is not quite exact.

The ambition of this idea, is to write inline xml just as easy as native python codes :blush: . The first point is use less keywords as less as possible. Using nested class or with-statement may accomplish the same goal as well, but it does add many syntax noise.

When I thought out this idea, I got some inspiration from the library dominate · PyPI . It does implement part of my goal with with statement. But it use too many with and seems less pythonic; besides, the implementation involves classmethod hack, which I thought is also not so elegant.

4

Ehh, I’m just a talkative loon :slight_smile:

Ahh, thanks for the reference there. I just looked up what golang’s meaning of defer is, and according to this article it’s often used as a sort of “at-exit” cleanup. So in that sense, yes, it is similar to Python’s with statement.

That’s not really a common use-case, but let’s say “to define complex data structures”, which is much broader in goal :slight_smile:

5

Yes, that is exactly what I think!

Once python owns the same capabilities of xml (Let’s not say that) on data declaration, it can make an omni-potent language of UI design & other complex works!

6

Initially I thought out these scenarios for Trailing Block to make code more simple and easy-to-read:

  • UI design
  • ORM object creating, with many referenced objects
  • Design workflow with higher functions, such as Celery task or Asyncio task
  • Create various useful DSL
1 Like
7

Why would I ever use this for something that isn’t a tree? If it’s only for trees, does that really justify the complexity? (Two-phase construction is inherently more complex, and any time I see it, I expect it to be justified by a reference cycle.)

How often do people really need to build a tree? (And if that were a real problem, and building trees could be meaningfully simplified - wouldn’t it make more sense to just have a standard library tree implementation?)

And even then - what’s wrong with just passing the child nodes to the constructor?

class Node:
    def __init__(self, children=(), *args, **kwargs):
        self.child_nodes = []
        ...
        for child in children:
            self.add_child(node)

    def add_child(self, node):
        ...

Why is having red_or_black = True in an indented block any better than having root.red_or_black = True normally? And how often do you really need to assign function attributes?

8

I came up with this idea when I was doing full-stack development in my company :grin:. I developed the backend with django, which is cool and relax; when I turned to front end, the template rendering is so inconvnient and tedious. Then I learn the Javascript React Framework & JSX syntax, I finally figured out that: to have one lanuage in both data declaration & scripting is such a great, great adventage.

This idea trailing block is just inspired from that. I am hoping python to be as omini-potent as it can be, both backend & frontend, data & view, declarative & progressive.

I can understand your divergence, that is just what I want to put them together :blush:

***red_or_black = True in an indented block *** is declarative, *** root.red_or_black = True *** is progressive. Isn’t it great to have both programming style for one python? In some domains, especially UI development, declarative programming is dominant.

9

Passing children is usually recursively.

You have to construct the nodes at the last level and then pass them reversely level by level; it may not be an elegant way.

10

This reminds me a little of Result Builders in Swift, which are used in (amongst others) in SwiftUI a Swift native GUI framework on Apple platforms.

Some questions that your proposal doesn’t answer:

  • How does this interact with temporary variables that aren’t supposed to be used by the __trailing methods, E.g.:

    child_node = Node(...):
    name = "foo"
    Node(name, ...)

    Will “name” be passed to __trailing__? If yes, how are you supposed to perform intermediate calculations?

  • How are loops handled, e.g.

    
child_node = Node(...):
    for idx in range(10):
         Node(...)

    Will __trailing__ get 10 instance of Node in *args or does something else happen?

To be honest this feels a bit too magic to my taste. That said, I have at times wished there’s a better way to constructs UIs than large blocks of serial code, and dislike the class statement trick that @Rosuav mentioned earlier.

1 Like
11

Sometimes I’ve wanted a general construct that behaves kinda like a REPL, capturing unassigned results. It could be used here for this purpose, too. Using your example of a UI, imagine something like:

class Window:
    def __init__(self):
        self.widgets = []
    def __value__(self, val):
        self.widgets.append(val)

gather Window():
    MenuBar(...)
    for lbl in field_names:
        Label(lbl)
        Input(name=lbl)
    Button("Save")

where each of the objects (presumably all subclasses of some Widget type) would be passed in turn to the Window’s __value__ method. This is just spitballing, but it’s the kind of situation where I have generally ended up going with a class block, and you’re right, it isn’t really the correct tool for the job.

This would require some compiler support: Any bare expression would be implicitly “and call the method with this result”. Ideally this should be able to be nested arbitrarily, so (eg) you could have a Box widget that gets attached to its parent, but which itself contains several children using the same syntax.

Would that suit the purpose of XML design?

1 Like
12

I am so excited that core developer sent me a reply! :blush:

I am surprised by your sharp insight, cause the name trailing method is just after swift’s trailing closure syntax by me.

Regarding these questions:

temporary variables are passed to the kwargs of __trailing__. In the example above, it will pass

{'name': 'foo'}

as the kwargs of __trailing__. In my design, it is like that at the end line of tailing block, you call the builtin locals method, and pass the result as kwargs of __trailing.

In this case, a length of 10 tuple of Node will be passed as *args. In my design, any instance with a __trailing__ method will be passed to *args.

I also recognize this purposal may bring less-explicit code. That is the cons. The pros is that most UI DSL use these pattern, so developer may be feel familiar to it.

13

I’ve thought over this, and it also reminded me about REPL and underscore variable too. In my design: In the trailing block, among unassigned results, only those of Trailer type, are captured and passed to __trailing__; those of non-Trailer type are thrown out. This way prevents too many values passing to __trailing__.

14

I can rewrite these codes with trailing block syntax as:

class Widget:
    def __init__(self):
        self.widgets = []

    def __trailing__(self, *args: Self, **kwargs):
        for widget in args:
            if isinstance(widget, Widget):
                self.widgets.append(widget)


class Window(Widget):
    ...

class MenuBar(Widget):
    ...

class Label(Widget):
    ...

class Input(Widget):
    ...

class Button(Widget):
    ...


win = Window():
    MenuBar(...)
    for lbl in field_names:
        Label(lbl)
        Input(name=lbl)
    Button("Save")
1 Like
15

Sorry I’ve ignore this question. Indeed, all local variables passed to kwargs may be too ugly. In my design, maybe following the class scope convention, any variable with leading double underscores is marked as “no need to capture”, like

child_node = Node(...):
    __name = "foo"
    Node(__name, ...)

then the intermediate variables will not be passed to __trailing__.

16

With no easy way to create nested scopes creating tree-like structures might be inconvenient. Here’s another take on the problem:

class Node:
  def __init__(self, val):
    self.val = val
    self.children = []
  def __str__(self):
    children = ', '.join(map(str,self.children))
    return f'({self.val}, {children})'
  def add(self, child):
    self.children.append(child)

class TreeBuilder:
  def __init__(self, gen, attach):
    self.gen = gen
    self.attach = attach
    self.stack = None
  def __enter__(self):
    self.stack.append(None)
  def __exit__(self, *_):
    self.stack.pop()
  def build(self):
    iter = self.gen(self)
    self.stack = [next(iter)]
    for child in iter:
      self.stack[-1] = child
      self.attach(self.stack[-2], child)
    return self.stack.pop()

def blueprint(node):
  yield Node(1)
  with node:
    yield Node(12)
    with node:
      yield Node(123)
      with node:
        yield Node(1234)
    yield Node(15)
    with node:
      yield Node(156)
      yield Node(157)

print(TreeBuilder(blueprint, Node.add).build())
3 Likes
17

To me, the proposal is too complicated to be usable by people who do not have the background that Alfred (and Ronald) have. Turning assignment statements into compound statement headers is rather radical and magical. Special methods usually implement operators and functions calls, and their arguments are those passed explicitly by the user.

The proposal is only useful when at least some of a nested structure can be statically defined. It ignores or wrongly dismisses what can be and is done in Python today to make nested structures. But it prompted me to think about and classify some alternatives.

  • Pass children to a node constructor (Karl Knechtel), or left and right child to a binary tree node. The objection ‘recursive’ is hardly a real objection when building a recursive structure. Anyway, the proposal is also recursive and can be seen as an alternate syntax for passing children to a constructor.

  • Pass the parent to a node constructor, as done by tk and hence tkinter. The child calls parent.add_child(self). The proposal could also be seen as a disguised way of doing this, with indents indicating parents. For the example:

   root = Node(None, ...)
   child1 = Node(root, ...)
   grandchild1 = Node(child1, ...)
   grandchild2 = Node(child1, ...)  # The tree construction is done without explicit 'add' calls.
  • Use Python’s collection displays, with indents. IDLE’s two-level window menu is defined as a list of tuples, with each tuple comprising a top-level name and a list of tuples, with each tuple comprising a drop-down menu name (with hotkey indicator) and event to invoke when the name is clicked. A separate function, currently in idlelib.editor, traverses the Python structure to produce a tk UI widget structure.
4 Likes
18

Sorry for the late reply. It is the timezone 


I agree with you on that: it is much helpful to have a easy way to create nested scopes & tree-like structures.

Like your implementation, I inspected the 3rd library dominate · PyPI, which is a toolkit for making up HTML with native python codes. Here is its example:

doc = dominate.document(title='Dominate your HTML')

with doc.head:
    link(rel='stylesheet', href='style.css')
    script(type='text/javascript', src='script.js')

with doc:
    with div(id='header').add(ol()):
        for i in ['home', 'about', 'contact']:
            li(a(i.title(), href='/%s.html' % i))

    with div():
        attr(cls='body')
        p('Lorem ipsum..')

What i think can be improved for both yours & dominate is that: there are too many key words like with and yield, which interrupt reader from reading smoothly. Hence, I think that introducing a special syntax may be a better way.

19

Sorry for the late reply. It is the timezone


And, it is my pleasure to receive reply from core developer ! :blush:

There is already precedent for special method uages other than operators or function calls . With statement uses __enter__ & __exit__ special methods for composing compound statement. Regarding the signature of __exit__ method:

object.__exit__(self, exc_type, exc_value, traceback)¶

some of the arguments are caught from the compound statement block, not explicity passed.

I agree that the design is kinda magical; however it is not imaginary-magical but virtual-magical. Some 3rd libraries already implemented similar syntax. I got inspiration from popular, many-stars python UI framework kivy (https://kivy.org/), which is a modern UI tookit compared to Tkinter. kivy introduce a DSL that describe complex structure, like

<MyWidget>:
    label_widget: label_widget
    Button:
        text: 'Add Button'
        on_press: root.add_widget(label_widget)
    Button:
        text: 'Remove Button'
        on_press: root.remove_widget(label_widget)
    Label:
        id: label_widget
        text: 'widget'

It is clear & simple for UI design. The main differences between kivy DSL and trailing block introduces here is: kivy developed a parser & translater for translating this DSL to python code, while with trailing block proposal, this can be done with native python.

The first example that I took is a static tree structure. But I am looking forward to much bold usage. For exmple, like Ronald’s comment, involving loops:

furthermore, there can be more aggressive ways, like using methods of functools:

child_node = Node(...):
    map(lambda i: Node(name=str(i)), range(10))
    Node('OK') if flag else Node('Cancel')
    filter(lambda n: n.name == '...', nodes)
20

As you said your proposal is very implicit which makes it very hard to control and see what goes and where.
Frankly, I’m really not sure why JSX exists at all. It is a very thin syntactic sugar over function calls and it heavily depends on anonymous functions, so it wont fly in python.
That being said jinja and jinja-like templates is pure cancer. Weird semantics, kind of python, but not really, obscene syntax. To me it’s basically unwritable without code snippets. So something like this proposal would be really nice to have, but it would have to support jinja-style inheritance to be truly worth the effort. Looks like gargantuan task to me.

2 Likes