I’m not sure if ignoring the incomplete batches is intentional, since there’s no assertion on the obj’s length. However, I think this version makes it clearer what’s happening:
obj = [*range(1, 7)]
size = 3
sums = [0] * size
for i in range(len(obj)):
sums[i % size] += obj[i]
result = max(-s for s in sums)
print(result)
Well, I think what makes it clear to me is keeping in mind that this proposed functools.pipe is function composition in the first place, and applicability for instant piping, syntactic conveniences, etc, are secondary considerations.
You can think of pipe:
new_func = pipe(func1, func2)
# Is the same as:
new_func = lambda x: func2(func1(x))
# Or with full signature support of 1st callable:
new_func = lambda *args, **kwds: func2(func1(*args, **kwds))
Seems like a perfectly normal pattern. E.g.:
flat_array = [0, 1, 2, 3, 4, 5] # known shape - (3, 2)
array2d = batched(obj, 3)
transposed = list(zip(*array2d)
Similar would be completely acceptable in some iterator recipe.
Yup. I will collate some real life use cases if it comes to that, but the base real life use case is simply Function composition - Wikipedia
And potential use cases can be found in many places.
E.g. map first argument lambda composing 2 functions: Code search results · GitHub
And there are many different cases that function composition applies to. Of course, I don’t suggest that it should replace all of them, but it would definitely find its niches.
Having that said I would replace all of my lambda with it whenever possible as it has no disadvantages and some advantages:
lambda in some casesOther reasons why others might want to use it:
I understand that you consider composing functions more important than “one shot” pipes, that makes sense as once composed you can apply it just once or n times, but I don’t understand why not to implement the operators directly in the main pipe class (off the shelf) and have to do it in a derived class not included in functools.
Of course it is. But I find f(g(x)) easier to understand than pipe(f, g)(x). I know it’s all a matter of taste, and functional programming is becoming more mainstream, but I still think the former is easier to understand. And I’m not new to functional programming - I learned Haskell when people first started trying to explain monads… 
I don’t think I’ve ever seen it in real-world code. I’m not saying it’s unusable, just that it’s obscure. The zip(*seq) idiom is not obvious, even though it’s well known (in some circles). I’d always add a comment saying what it’s doing if I wanted my code to be maintainable.
I suspect that if you do, you’ll simply get into arguments about whether the original version or the rewritten one is “better”. This is all a matter of preference at some level. What I’d be more interested in is objective data on which form is more maintainable in a large code base. But that’s going to be impossible to get (without more resources than I expect either you or I have access to) so it’s going to end up being a judgement call for some core developer.
But I don’t think this sub-discussion is going anywhere. You seem to think that a functional style is self-evidently worth supporting in Python. I think that a functional style can be great, but in the context of Python it needs to be judged in terms of how well it fits with the broader style of the language[1]. We’re not going to agree, so let’s stop trying to convince each other that we’re right.
Each individual functional construct needs to demonstrate that it’s “Pythonic” on its own merits, if you like. ↩︎
Because I am not sure which operator configuration is “the best”.
There are many possible valid configurations that I have been using with such class.
And opinionated DSLs is not standard practice for standard library.
In short, implementing operators would need a separate discussion.
This does not fall into category of use cases that IMO justify this extension.
The main one is to make a new function:
func3 = pipe(func1, func2)
, which can then be used as an argument to another function:
map(func3, iterable)
some_function(..., callback=func3)
# etc...
Or pre-stored for repeated usage.
OK, that makes sense now, thanks.
For one shot cases having to pass the parameter at the end breaks the left to right flow, so based on what someone wrote some messages ago I would favor having 2 functions (both in the stdlib)
Im often terrified of proposals like this one as they try to translate an ideal that’s absolutely amazing in languages like ocaml/haskell into python
as python has a very different flow and without the surrounding fp Infrastruktur its just very foreign
A bit like trying to sell a swan as a duck
I would just like to point out that I am not proficient in any functional programming languages - just did some for fun in the past, but never got into it too much.
This proposal came about spontaneously from concrete investigations within Python space.
Research about other languages came later as part of research before making proposal.
Could you elaborate on this?
To me it does not seem like it at all. For example, 2 functions map and filter are in builtins module and are in extensive use. Furthermore, partial is probably the most used function in functools module.
So I am not trying to bring in random stuff, but instead I think this could be the next component which would be in resonance with already available and widely used components above.
E.g. it integrates into existing iterator utils for more performant predicate making:
map(pipe(sorted, reversed), batches)
# or
filter(pipe(sum, partial(add, 1)), batches)
And also closely follows current patterns of functools:
class A:
def add(self, a, b):
return a + b
add1 = partial(add, _, 1)
add1_minus_1 = pipe(add1, partial(sub, 1))
All examples you just outlined are code id never expect to see in python
Unless there’s a practical real world example consider me out
Some of those are there `functools.pipe` - Function Composition Utility - #36 by dg-pb
At first glance alle those examples read orders of magnitude better as generators or list comprehensions
I want real examples not toy copies of functional beginner tutorials
Is this assumption that these map()/filter()/operator/functools.partial() are faster than comprehension loops based on anything? I can’t find any example that backs this claim up. e.g.
sum(i + 1 for i in range(1000000))
and
from functools import partial
from operator import add
add_1 = sum(map(partial(add, 1), range(1000000)))
both run in 0.061 seconds for me.
“Extensive” is a stretch, IMO. Comprehensions are usually a much better choice than map/filter.
And I dispute your comment on partial - in my experience, lru_cache is far more commonly used, and wraps being very common among people writing decorators. Also, in general I’d say that functools is a pretty rarely used module, so being the “most used function in functools” isn’t that big of a deal anyway…
Ultimately, this is all a matter of coding style preferences, though. Some people like functional styles, some don’t. And in my experience, the majority of people frankly don’t care, they just want something that solves their problem.
Proposing that an existing, commonly used 3rd party implementation[1] from PyPI gets added to the stdlib seems like a much better idea here. There’s as yet no real evidence that people will suddenly start using functional idioms and piping just because it gets added to the stdlib. So a 3rd party library should fulfil most of the need, and provide objective evidence that there’s sufficient demand for the functionality to justify it being in the stdlib.
If none of the existing ones is suitable, then write one! ↩︎
Whatever do you mean?! I’m constantly using cached_property these days!
partial? Never.
So let’s not rely on domain-specific popularity of a specific usage too much. I don’t particularly care that a given builtin is “popular”. By that logic, let’s add statistics.avg to builtins![1]
This thread doesn’t seem to have shaken loose from the fact that function composition can be phrased multiple ways and that the entire feature proposal rests on a particular stylistic preference.
I’ll use the type system to raise a question.
The following two examples are more or less equivalent under this proposal:
# with a pipe builder
pipeline = pipe(f, g, h)
# with current language features
def pipeline(x: T) -> R:
return f(g(h(x)))
Can the type of pipe be expressed? I believe that it requires higher order types which cannot be expressed today. By contrast, the explicit pipeline function definition can be type checked today.
Not all language features (e.g. metaclasses) can be expressed in the type system. But if a proposal creates a new gap, I think that’s a mark against it. And unless there is a non-stylistic difference between these, I’m content to say “not every three line function needs to be in the stdlib” and let anyone who wants a composed function builder write one.
For why not, see the many threads which have covered this topic. ↩︎
To reiterate, those were there to demonstrate that this doesn’t need to be in the standard library. This is such a simple thing that anyone could write it, the fact that people don’t frequently says that it isn’t important enough to put in the standard lib as a standardized form of it.
There are so many things that typecheckers model imprecisely, incorrectly, and incompletely that I think this argument should never be made.
It’s not even just advanced things like metaclasses, it’s issues like iteration and unpacking not being accurately handled in the typeshed or typecheckers.
If it’s important that these and other existing things be typed, the type system should evolve. That was the deal with making the type system and type checkers optional and separate from the language specification.
I do actually agree that the lack of typing support might contribute to why people don’t do it this way in python, but that shouldn’t prevent anyone from trying to argue inclusion on other merits and then have the typesystem evolve to understand it whenever someone who cares about it being typed works on that.
Your point is fair. But interplay with the type system is part of several proposals now, so we’re giving some weight to “how will it type check” in ideas.
We should know when a proposal is going to create a new hard-to-type scenario or put an existing problematic case into the stdlib. It may still be a good idea which makes it into the language, but the decision to accept should be made with that knowledge.
I stand by my preference not to add more such things, everything else being equal. There has to be a compelling value-add to offset that, as far as I’m concerned.
You are right. I am sure I had cases where map was faster, but now they seem very similar.
Then the remaining case is lambda replacement for cases where appropriate.
E.g.:
callback argument. I have laid out advantages in `functools.pipe` - Function Composition Utility - #62 by dg-pbiterator recipes.Github
/\bpartial\(/ Language:Python 881K/\blru_cache\(/ Language:Python 195K/\bcached_property\(/ Language:Python 9KI think a lot of opposition that is happening here is simply by the people who are not big users of such things and it might not reflect wider user base.
I don’t use typing at all so I could use a bit of help here. Is it really hard to type scenario? E.g. is functools.partial problematic case? It seems to be handled in typeshed/stdlib/functools.pyi at main · python/typeshed · GitHub. Is it causing issues?
I look at typing once a year, so don’t judge.
P = ParamSpec('P')
class pipe[**P]:
def __new__(cls, func: Callable[P, Any], *funcs: Callable[[Any], Any]) -> Self: ...
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> Any
No judgement at all. Typing is optional (for users; I don’t think that consideration of typing is optional for language design). I’m far from an expert, but I am a pretty heavy user of typing, so I’ll share my current understanding and hopefully won’t make any major errors.
I believe that it’s still difficult, but typing grows new capabilities every year, so I may be behind the times.
I think that functools.partial is special cased by type checkers – the current content in typeshed doesn’t look sufficient to me to describe it (unless there’s a subtle trick to it that I’ve missed).
As far as this case is concerned, we have a function whose input is a series of functions, each of whose outputs must be valid inputs to the next. The result of this function should take the first function’s input and return the last function’s output.
Basically, if we have
def f(x: T1) -> T2: ...
def g(x: T2) -> T3: ...
def h(x: T3) -> T4: ...
then the type of pipe(f, g, h) should be T1 -> T4.
This much is intuitive. And it’s pretty easy to write down what the type of any given pipe ought to be.
But as far as I’m aware, there isn’t a typing construct which allows you to capture and refer to the types of all of your heterogeneous inputs, and then apply a constraint on them based on their adjacency in the input.
To be properly typed, I think it needs to be able to catch that order matters. If pipe(f, g) is valid, pipe(g, f) might not be.
There are additionally tradeoffs in any type system that actually can express this that may not be obvious at first. Without currying as syntax, we can’t use things like continuation passing representations to get simplified forms, and the same things that allow expressing this for variadic inputs instead of actual syntax chains that directly indicate the relationship at the language operation level very quickly turn into “the type system is itself turing complete” and “It’s impossible for inference to be fully decidable”, the latter of which already shows up in some places.