Allowing <<= and such operators to work if LHS doesn't exist

Python Help
1

Hello!

I’m new to this forum, so please let me know if I’m asking in the wrong place.

TDLR: implement __rilshift__(self, *args) and such functions for cases when LHS doesn’t exist or doesn’t define __ilshift__. Allow introduction of new variables as the return value of __rilshift__.

Justification (usage example):
I’m working on a Python library that allows building and executing dataflow graphs. These graphs have nodes and edges (big surprise). To create edges between nodes, I’m using the “<<=” operator. So, if you have X and Y as two existing objects, you could say:

X <<= Y

to crate a connection between the two. For more complex scenarios, nodes have named ‘ports’ which can also be connected:

X.input42 <<= Y.output12

Sometimes it’s convenient to give ‘names’ to these edges by assigning temporary variables to them. I call these ‘wires’. The syntax would be:

my_wire = Wire()
my_wire <<= Y.output12
...
X.input42 <<= my_wire

Notice, how I have to declare my_wire up front before the first ‘connection’ of it to output12 of Y. This is not very “pythonic” and in general it would be nice if I didn’t have to do that.

Implementation idea:
My idea would be to introduce a new dunder function:

__rilshift__(self, *args)

This would be called if the LHS of the ‘<<=’ operator doesn’t exist (in which case *args would contain no arguments) or doesn’t implement __ilshift__ (in which case *args would contain a reference to the LHS). Either way, the return value from this function would be bound to the named variable whether it existed prior the call or not.

I don’t believe this change would break any existing code.

Obviously, if this idea is implemented, it would be done for all in-place operators, not just ‘<<=’.

Thank you for reading all the way to the end and please provide feedback!

Thanks,
Andras Tantos

2

Does this not work?

my_wire = Wire() << Y.output12

That is, there’s already a great solution for cases when the left-hand side doesn’t exist: assign the value to it! Making the in-place operators no longer do in-place stuff is confusing and appears totally unnecessary to me. Is there some reason the above wouldn’t work instead?

3 Likes
3

If the LHS doesn’t exist, then what is self?

2 Likes
4

James, thanks for the suggestion. I don’t think it will work (in my case at least). The reason is that the << operator has a different meaning (right shift) where as <<= is redefined as to mean ‘connect’. So, what you’ve written means (again, in my case):

  1. Create a ‘node’ object that does right-shift
  2. Connect one of it’s inputs to Y.output12
  3. Connect the other input to my_wire
  4. Don’t connect the output to anything

I do agree with you: under regular circumstances in-place operations on non-existent objects are not meaningful. However in cases (and I don’t think I’m the only one) where they are (ab)used to mean something different, they can be.

One final node: when constructing data-flow diagrams, in-place operations are inherently meaningless: every value is inherently immutable. So (again in my case) there can’t be any confusion.

5

Matthew,

Good question. The suggested __rilshift__ method would be defined on the RHS of the operator, which does exists. So self is set to the RHS and *args would be an empty list.

Thanks,
Andras

6

I think it’s unlikely that you will get support for uses that are this far from the normal semantics of the operator. For as long as I can remember (and that’s a pretty long time :wink:) Python has had a policy of not encouraging this sort of writing a domain specific language in Python.

From your original post:

You have a mistaken view of what is “pythonic” if you think that. I understand that you’re thinking in terms of “declaring things is not normally required in Python”, but the line my_wire = Wire() is not a declaration, it’s creating an object explicitly before we operate on it. And “explicit is better than implicit” is one of the principles in the Zen of Python, which in this case I would interpret as meaning that your proposal, to implicitly create a new object via the <<= operator, is less “pythonic” than the explicit creation patternt that you currently use.

4 Likes
7

That reason seems pretty flimsy to me. <<= has a different meaning too: it means “in-place right shift”, but you made it do something else instead. You can make << mean something else just as easily. Is the whole motivation “I like the way <<= looks”?

8

Pretty much. To be more specific, I need a ‘connect’ operator. ‘=’ is not overridable (for very good reasons), so I need to find something else. Something else, that doesn’t make sense as a node object in a data-flow graph. Pretty much any unary or binary operator does, but none of the in-place operators do.

Of the in-place operators, the best looking one was ‘<<=’. IIRC other libraries chose ‘@=’ in similar situations, but that suffers from the same problems.

Andras

9

Moving this to the Help category as it seems more about finding a good design for the library you’re building.

1 Like
10

So… you’re trying to force Python to be a DSL, which it isn’t good at. Unfortunately that’s never really going to work out well. But on the plus side, creating a real DSL actually isn’t that hard! You could even have it be a Python-derived DSL and you write a program that compiles it to actual Python code, letting you take advantage of all the rest of what Python does.

11

Yes, you are right. But how does one go about creating a ‘Python based DSL’? It very much sounds like a fork of CPython with my custom (never up-streamed) changes. Or is there a better way?

Thanks,
Andras

12

Normally, a DSL is its own entire language. You design a grammar, build a parser, and make a way to execute it. A lot of work, but in my opinion, a lot of fun.

As an alternative, though, you can build your DSL on top of an existing language. You write portions of the file using your special syntax, but for the rest, it is literally just Python code. Then you have a program that transforms your wire design file into a runnable Python program. This is far simpler than building a full language (since you can just say “that part is plain Python” and copy it in verbatim), while still having an extremely rich set of features available (ie all of Python).

That’s what I mean by a “Python-based DSL”. You still write a program (in Python - no need to fork CPython or write any C code) that parses your file, but for large slabs of that code, it lets Python do the work.

13

For example, GitHub - pyparsing/pyparsing: Python library for creating PEG parsers would be a good library for writing such a DSL.

14

Got it, thanks. I’ll give it a read. I’ve also found ply which might work. It still seems difficult to recognize ‘just Python’ and send it over unaltered, especially when constructs are mixed in. Anyway, a lot of reading ahead…

Andras

15

You wouldn’t write a parser for Python, you’d write a parser for your dataflow graph DSL. So you’d write a file that describes your dataflow, parse it, and then use that parsed structure and do something with it in Python.

That was just someone’s suggestion though. If you want a Python-based API for this, I’m sure there is one that’s still nice and doesn’t require fundamental changes to the language. Which is why I moved this to Help, so people might help you figure out what that might look like.