`functools.pipe` - Function Composition Utility

Ideas
53

There are 2 main types of “pipes”:

1. Feed Pipe

# Custom object
# Standard syntax
result = rpipe(obj)(func1)(func2).obj
# Custom Operators
result = rpipe(obj) | func1 | func2 | UNWRAP
# Or:
RPIPE = rpipe()
result = obj | RPIPE | func1 | func2 | UNWRAP

The issue of adding this to standard library is that there exist some variations.
Also, there are some things that people would like it to do that are a bit tricky.

E.g.:

  • “feed pipe” can be eager or “lazy”
  • Some want it to support “inplace” evaluation
# Potential stdlib syntax
result = obj |> func1 |> func2

The issue with custom syntax is that addressing the above becomes even harder. E.g. rpipe class can have argument lazy, but where would the above syntax be parameterised?

So my current take is that “feed pipe” is tricky and costly and I would prefer and advise not to rush with things in this direction.

2. Function Composition Pipe

# Standard syntax
pipeline = pipe(func1, func2)
result = pipeline(obj)
# Custom operators
PIPE = pipe()
result = obj >> PIPE | func1 | func2

In comparison to (1) this is much more straight forward.
There are no big nuances in it.
The only variation that I see is whether to allow full signature of the first function.

Also, this is much more general object which is useful not only for “piping”, but also general function composition. E.g. map(pipe(int, str), iterable).

Also, having efficient implementation in stdlib, user could easily implement operators that would provide piping syntax which is fairly good.

So after going through another round of discussions in Introduce funnel operator i,e '|>' to allow for generator pipelines - #181 by elis.byberi I am back to my initial position.

  1. implement simple efficient functools.pipe
  2. further extend functools.partial

With (1) and (2) above user can do:

from functools import pipe, partial, Placeholder as _
class mypipe(functools.pipe):
    def __or__(...)...
    def __rrshift__(...)...
PIPE = mypipe()

result = obj >> (PIPE
    | func2
    | partial(zip, *_)          # Needs partial extension!
    | chain.from_iterable
    | sum
    | partial(operator.add, _, 1)    # Already available
)

So without any syntax changes:

  1. Functionality needs are fully covered (after partial allows keyword, *args and **kwds placeholders)
  2. Reasonable performance of all components
  3. Easily implementable piping DSL via operators, which is pretty much as convenient as the one of the “feed pipe”.
4 Likes