hmmm maybe. Maybe I’m missing something here and I know this was brought up in the context of generator pipelines. But a generator pipeline is just a use case of function composition. Other than reversing associativity and using different symbols for the operator I don’t see it as much different. But maybe you’re thinking about something I’m not or have some perspective I don’t. 
It’s the order in which the functions are listed. (Maybe that’s "reversing the order of associativity?)
With @ for function composition you write functions the same order as ‘normal’, with the last applied function to the left.
With a pipeline, (or pseudo-universal function call syntax as I like to think of it,) you write the last applied function to the right. (Or at the bottom, as I prefer.)
Either way you get rid of brackets that are very far separated from their closing bracket, which is nice.
But changing the order in which functions are listed isn’t unimportant (to me).
yeah that’s what I meant by reversing the order. I could totally be using that wrong. I meant it in the sense that @ is an operator and g and f are the left and right operands respectively.
Yeah I don’t have too strong of an opinion on the order really. I could see pros and cons for both. I’d be happy if it was added either way. It’s still just composition or pseudo-universal function call, if you prefer.
Not really a serious suggestion but this talk of wring a series of function calls in the order they’re called makes me wonder how putting the arguments before the function name would go down… 
((4, ((1, 2)do_something, 3)then_do_something_else)and_another_thing)print
((1)__add__.(1)int)__mul__.(2)int
1 Like
This is the Reverse Polish Notation I used on my HP calculator during the last century!
Please give it a try:
https://sadaszewski.github.io/python-pipeline-operator/dist/console.html
and let me know what you think. Is it PEP-able?
3 Likes
>>> lst = [1, 2, 3]
>>> for index, item in lst |> enumerate():
... print(index, item)
...
0 1
1 2
2 3
Could you provide some concrete examples? I’m familiar with this thread, but others might not be, and examples could help clarify the discussion.
2 Likes
Thank you for your reply! The syntax is the following:
[1, 2, 3] |> map(str) |> ", ".join() |> _.split()
The _ plays again its role of a soft keyword here and acts as a placeholder to indicate where to insert the results of the LHS into the RHS. If no placeholder is placed, the LHS is injected as the LAST argument. This plays better with existing Python functions. The RHS must be a call.
Can pass the placeholder by position:
5 |> range(1, _) |> list()
1 |> range(_, 5) |> list()
Or keyword:
5 |> sum([1, 2, 3, 4, 5], start=_)
Only one placeholder is allowed, since everything happens on the stack!
As a notorious user of dplyr’s/magrittr’s %>% and R’s |> I am more than happy to provide more examples here, in the repository and hopefully in the PEP in the near future.
2 Likes
Added support for star arguments:
[[1,2,3],[4,5,6]] |> zip(*_) |> list()
Support for keyword arguments:
{'a': 1, 'b': 2, 'c': 3} |> "{a} {b} {c}".format(**_)
2 Likes
Functionality and usage of this looks great!
Implementation, however, I imagine is purely at parser level.
To eliminate strain on parser and make it more modular, I would suggest the following:
class Callable:
def __rcall__(self, positional_arg):
return self(arg)
And implement the following at parser level:
function($) -> function
function($, 1) -> partial(function, Placeholder, 1)
Unfortunately, _ can not be used and other symbol would have to be found.
This way:
- There is convenience for
partial, which doesn’t have to be used in conjunction with new funnel operator - There is a new operator, which follows usual standard of operators with its magic method and this opens it to be used in new innovative ways without being tied to specific application.
Restriction on partial has been merged (see: gh-125028: Prohibit placeholders in partial keywords by dg-pb · Pull Request #126062 · python/cpython · GitHub) which was done exactly to pre-empt such possibility for extending it for keyword arguments. I could extend partial with this if this approach was taken.
The only missing piece of the puzzle would be:
function(*$) -> ?
function(**$) -> ?
I have played with implementing various objects that can do this as well.
So I am sure it is possible to both:
- (a) Come up with syntax for such
- (b) Make an appropriate transformation object[s]?
Given partial lives in functools, maybe a new object could be made specifically for this at lower level - the one which isn’t exposed to needing both C and Python versions and other existing predicaments.
I have done some experimentation with indexed placeholders as well. See: Future of `functools.partial`
If you were to take this path, I could help out.
We could polish the concept, identify needed components and I could take up partial-like object implementation / extension while you could cover operator and parser level shorthands.
I am not 100% sure, but my guess is that if this wasn’t a monolithic implementation but rather a neat combination of modular extensions the chance of it being accepted is much higher.
But don’t get me wrong. I see pros and cons in both directions. Implementation at parser level would be faster (not having to construct intermediate objects along the way) and most likely much easier to achieve.
Thanks for the feedback! To be exact it’s implemented almost exclusively in the compiler, which actually let’s it stay extremely lightweight. The changes to the parser are almost non-existent - it’s just introducing the new token |>, a new BinOp - Pipeline and defining how to parse it. That’s it. Everything else is compiled into as efficient bytecode as anything else. It’s not using any AST manipulation either, not converting to partials, etc. In terms of performance it would be hard to beat with other approaches. The existing approach can even be further optimized here and there. And the compiler did not get that much more complicated with this change IMHO. Thanks for starting the discussion.
diff --git a/Grammar/python.gram b/Grammar/python.gram
index 5a181dc1578..a59f606872e 100644
--- a/Grammar/python.gram
+++ b/Grammar/python.gram
@@ -810,6 +810,11 @@ factor[expr_ty] (memo):
| '+' a=factor { _PyAST_UnaryOp(UAdd, a, EXTRA) }
| '-' a=factor { _PyAST_UnaryOp(USub, a, EXTRA) }
| '~' a=factor { _PyAST_UnaryOp(Invert, a, EXTRA) }
+ | pipeline
+
+pipeline[expr_ty]:
+ | a=pipeline '|>' b=primary { _PyAST_BinOp(a, Pipeline, b, EXTRA) }
+ | a=power '|>' b=primary { _PyAST_BinOp(a, Pipeline, b, EXTRA) }
| power
power[expr_ty]:
1 Like
Ok I see.
I would definitely make use of such in appropriate places. The fact that it is highly performant would allow using it places where otherwise slowdown could be highly undesirable. Also, its functionality is close to complete in a given the scope in mind. And learning such is easy given it is contained in a closed design.
However, as a bit more experienced user that likes to take things apart and combine them in new ways I would have a bit of troubles. I would definitely want to override operator in custom objects and wouldn’t be able to. At the same time being able to twist RHS into a new form of partial in this specific construct but not having the same functionality that I can apply in other places would also feel inconvenient.
From design perspective, the operator is reverse call with 1 argument, and while __call__ exists the reverse of it not having magic method is not consistent placing this alongside assignment, or, and and similar which are on slightly different page. And the fact that RHS has to be a call is a strict coupling and I can not think of any similar existing design in Python. The closest one that resonates with this is match cases. Which is arguably much more appropriate construct to have its own DSL and also, users sometimes want to re-use some of appropriate functionality in different places and can not. E.g. dict expansions.
I think the intrinsic components of this structure would be very useful to have separately. Additional operator requests have happened occasionally in the past and having one more override available would satisfy a part of those without needing to resort to infix and other less desirable approaches. At the same time, function transformations that can be done on the RHS are useful in different contexts, furthermore the shorthand for these transformations would also be very useful in various places independent of this specific pattern - partial is quite commonly used object.
So to sum up -0.5. Immediate satisfaction of this would very likely be cancelled out to a large degree in the long run as I encounter cases where I want to make use of individual parts but can not.
It might be possible to get the best of both worlds:
- Implement this as independent components
- Add optimizations in compiler that implement fast path when these components are used in conjunction - most of what you have done would be applicable here.
This IMO would be ideal.
It would be along the following lines:
1. partial extension and syntactic convenience
from functools import partial, Placeholder, ArgsPlaceholder, KwdsPlaceholder
foo($) -> foo
foo(*$) -> partial(foo, ArgsPlaceholder)
foo(**$) -> partial(foo, KwdsPlaceholder)
2. implementation of operator (default for all callables or even object)
class object:
def __rcall__(self, arg):
return self(arg)
class A:
def __call__(self, *args, **kwds):
print(args, kwds)
1 |> A() # ((1,), {})
3. compiler optimisation
obj |> foo($)
obj |> foo(1, $)
obj |> foo(*$)
obj |> foo(**$)
Would not construct partial, but instead use your current implementation
As partial object would never have __rcall__ overriden, this would be unambiguous
(just not sure about the symbol…)
1 |> str
Would use standard track.
1 |> foo($, $)
Would also use standard track as this would be applicable strictly for 1 Placeholder.
1 Like
I like this choice of syntax, and this is something I have long wanted to do in Python.
In particular using |> map for generator pipelines allows the |> to stay nice and general, and the _ looks cleaner than for example $.
Based on the fact you have this implemented it is actually well-defined?
It looks to me like
[1, 2, 3] |> map(str) |> ", ".join() |> _.split()
would be equivalent to
[1, 2, 3] |> map(str, _) |> ", ".join().split()
so then
5 |> range(1, _) |> list()
would be equivalent to
5 |> range(1) |> list()
?
Am I understanding correctly that the thing before the pipe gets put after the last positional argument, so that for example
"hi hi hi" |> print(sep="-")
works intuitively?
That does work really nicely in most cases, and for the cases where it doesn’t quite work due to positional/keyword-only arguments, I expect one could use the explicit _ construction.
I’ve been reviewing some of the posts on this thread - I think, the most remarkable thing here is that to program like this in Python, there is no need for any syntax or other language change at all.
All one need is to encapsulate the first element, or iterable, of the Pipeline in a specilalized class (which could be named Pipe - and them, voilá:
from functools import pastial
from pipes import Pipe, render
def filter1(param):
# do stuff
...
return result
def filter2(param, operator):
...
return result
Pipe( messages) | filter1 | partial(specialized_filter(operator=3)| filter3| render
One can even implement different semantics for | and >> operators (I wouldn’t recomend re-using > due to chained comparisons - it woudn’t work if one had more than one in the pipe).
I actually remember even building some toys using this more than 10 years ago (let me see if I can find them around)
Indeed- here it is:
Please note that is really a toy - the class that shows in the README is a weird version where I use the “.” operator to make the data travel from left to right - the “Pipe” - although working for a single point a time, is implemented in the sole Python file in that project. (which lacks even a stub setup.py (not to mention pyproject) to be installable ).
here is the (raw) experience after cloning and CDing into that project:
Python 3.13.2+ experimental free-threading build (heads/3.13:fc1c9f8, Feb 17 2025, 17:58:08) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from chillicurry import Pipe
>>>
>>> Pipe(5)| lambda n: n* 5| lambda m: m + 5 # Oopss..lambdas are greedy with the `|`
File "<python-input-2>", line 1
Pipe(5)| lambda n: n* 5| lambda m: m + 5
^^^^^^
SyntaxError: invalid syntax
>>> Pipe(5)| (lambda n: n* 5)|( lambda m: m + 5)
<chillicurry.Pipe object at 0x5d2a0503790> # meh? where is my result? Maybe in __call__ ?
>>> _()
30
3 Likes
Nice! I can see that there are some concepts above that would be difficult to replicate but I could image that we could ship perhaps a “pipe” concept (maybe it’s a subclass as generator ? ) as part of functools and perhaps provide some less than ideal functions to help wrap functions that need you to control where to put the placeholder
It would be clunky but I guess this might be a better first step and then as we find use cases that need language level support we can add it at that point ?
I’m sure there are concepts in python introduced that way.
This is one that didn’t go through. Addition: functools.pipe · Issue #127029 · python/cpython · GitHub for a composition pipe.
I think what is needed is wider consideration of how everything could potentially fit together few steps further down the line for functional programming in Python. I.e. pipes, partialization, predicate construction, operators, optimizations, etc…
Without higher level view, things as such will unlikely be accepted, because they bare risk that they will not fit well with the things of the future, while anyone can implement them for themselves with few lines of code or use 3rd party package.
E.g. For type of pipe that @jsbueno showed I use:
class rpipe:
def __init__(self, obj):
self.obj = obj
def __iter__(self):
return self.obj
def __or__(self, func):
return type(self)(func(self.obj))
2 Likes
Note that historically, Python hasn’t been in favour of functional programming (that’s why reduce got moved to functools). That may have changed with the rise in popularity of languages like Rust, but be prepared for a certain amount of pushback. You’ll need good arguments beyond intangibles like “readability”, because to many established Python programmers, the functional style isn’t particularly readable.
8 Likes