Introduce funnel operator i,e '|>' to allow for generator pipelines

Ideas
269

I abstained from one of these as it presupposes an option which I am very strongly against. It’s also hard to express preference in polls – for the first question I find two options to be about equal and a third option to be unacceptable.
So just a reminder that polls skew in various weird ways, but thanks for writing these all down!

I like the use of an assignment expression at the start of the pipeline. I think that’s clever and demonstrates a good way of leveraging an existing language feature.
I believe that it would be a requirement that the assignment expression’s LHS consists of a single, unadorned identifier.

x |var> func(var) is not a viable option. Those are already operators, meaning x | var > func(var). I believe that’s the only one which “just won’t work”.

3 Likes
270

I agree with all these points. I don’t think the poll will actually be very informative as a result. However, I did vote on at least some of the questions.

On (2) I only have a mild preference between the two “explicit placeholder” cases, but I strongly dislike the “implicit placeholder” cases.

On (3) I don’t care much, so I abstained. But it’s complicated - I prefer _ somewhat, but I’m uncomfortable with the clash with internationalisation.

For (4), all of them look pretty ugly. I don’t think there’s a good answer for how to configure a placeholder yet, it’s something that still needs work.

I abstained on (5) because there was no “I don’t want this in any form” option.

3 Likes
271

Sorry for the delay in responding, contrary to popular belief it’s not because of disinterest but because I was on holiday and because I’m not an expert in language design

I wanted the experts to throw in their opinion so I could allow things to stew

I’ve been following the thread and can distill my thoughts follows

It seems like people reasonably don’t like the implicit arguments.

I understand that other languages this is a common pattern but I also understand that Python language design is usually about making things explicit (although there are exceptions such as generators looking identical to function calls)

Python already handles such cases by highlighting it typically with a keyword for example with the keyword match making clear that in that context “|” has different meaning

A keyword such as “pipe” could do same thing

images = pipe glob(“*.*”) | map(PIL.Image.load)

This would have advantage of not needing to introduce a new operator

Perhaps ability for library’s such as open CV to detect when they are part of a pipeline and apply optimizations such as in place memory mutation would require a different method now we can’t use Dunders

The second thing I’ve noticed is the arguments around placeholder syntax.

I agree many samples have been obtuse or not very pythonic or not explicit

In this cases I would actually say the way import, with or except works could be a good compromise.

images = pipe glob(“*.*”) as paths | map(PIL.Image.load, paths)

Of course this could break if the user reorders the statements

But if they follow the convention of always using “as _” and then the statement after the “as” only applying to the next statement after the |”

Then they can have the same _ placeholder everywhere whilst being explicit

Sorry if this is very layman-ish

Seems most people are working at a higher level than me :grinning_face:

1 Like
272

For the _ vs PIPE vs [something user defined] debate, how about using it (borrowed from kotlin’s equivalent of lambdas)? It’s only one more character than _ and it’s sufficiently useless as a variable name that it’s unlikely to clash with anything else.

3 Likes
273

Is it actually? I think there’s some very bad misunderstanding by some thread participants about what it means to pass pipeline stages to an uncalled callable vs injecting them into a syntactically ordinary looking function call.

Maybe some languages do this but I honestly don’t know of any language which does.

If map(range(10)) produced a callable as its result, that would be dramatically different. But that’s not what it does.

And the disadvantage that you then cannot use a valid binary operator (|) inside of a pipeline.

I don’t think we should want to co-opt an existing operator. Better to establish a new operator which “does the right thing”.

No need to apologize! Venturing into this sort of discussion if you’re new to this sort of debate is a great opportunity to learn.

But we are all going to have some strong opinions because language design is a kind of aesthetic endeavor. There are some “wrong” answers but no fixed “right” answers. As long as those opinions are backed by some knowledge or experience, there’s a ton to learn from one another.

4 Likes
274

Just a reminder, it’s always about making code easier: easier to write, easier to read, easier to follow, and easier to explain aloud in a conversation. The key word is easier. It’s not about whether ‘language X has it’ or whether a particular paradigm is missing.

For example, this simple sample introduces too much cognitive overhead by requiring the reader to mentally parse the one-liner and go back and forth. Also, could you write out how it could be read aloud so that a listener could clearly understand it?
images = pipe glob(“*.*”) as paths | map(PIL.Image.load, paths)

This is the simplified code after removing the redundant syntax pipe, as, and |:

paths = glob(“*.*”)
images = map(PIL.Image.load, paths)
6 Likes
275

Thanks for the great examples. It seems like it still requires a lot of work but I will try. For completeness, this is where the injected placeholder would go in each of your cases.

x |> (f(_), g())
x |> f(g(), _)
x |> f(g, _)()
x |> f(_)()
x |> (f := g(_))()
x |> [f(_) for f in funcs][0]()
x |> y[f(_)](g)()
x |> (f(_) * g(),)
x |> f(_) if g() else h()

Great examples for testing the correctness of this temporary implementation - I haven’t tried it yet but this is where the injections should happen.

276

I think you’ve missed the point - it’s not a question of whether there’s a clear rule that can be used to determine where the placeholder would be injected, it’s the fact that the rule isn’t intuitive. You may think it is - as the implementer, you’re so familiar with the design that it won’t be easy to see it from a newcomer’s perspective - but the feedback you’re getting is that it’s not. Explaining what will happen doesn’t help, anyone can go and read the documentation. What matters is that people won’t be able to infer (or remember!) the correct behaviour in these edge cases.

Again, you seem too focused on explaining how the existing implementation behaves than on accepting feedback that says people want the behaviour to change :slightly_frowning_face:

8 Likes
277

no no it’s clear let’s do as the poll indicates, propose an explicit placeholder instead. i just provided info for the record.

1 Like
278

I like the idea of it if _ were to be a blocker to proceeding. Shall we repeat this part of the poll?

  1. The preferred hardcoded or default placeholder if you had to choose only among the three:
  • _
  • PIPE
  • it
0 voters
Code samples with the 3 placeholders 
df_mc_break = read_csv(
    'data/src/sample_pandas_normal.csv',
    index_col=0
) |> assign(
    it,
    point_ratio=_ / 100
) |> drop(
    it,
    columns='state'
) |> sort_values(
    it,
    'age'
) |> head(
    it,
    3
)

df_mc_break = read_csv(
    'data/src/sample_pandas_normal.csv',
    index_col=0
) |> assign(
    _,
    point_ratio=_ / 100
) |> drop(
    _,
    columns='state'
) |> sort_values(
    _,
    'age'
) |> head(
    _,
    3
)

df_mc_break = read_csv(
    'data/src/sample_pandas_normal.csv',
    index_col=0
) |> assign(
    PIPE,
    point_ratio=_ / 100
) |> drop(
    PIPE,
    columns='state'
) |> sort_values(
    PIPE,
    'age'
) |> head(
    PIPE,
    3
)

It kinda even looks better in the vertically spread code. I will abstain from the vote not to push it in either direction.

279

I didn’t vote, because I don’t think voting on details like this is productive. What matters is the higher level “feel” of the feature, and fiddling with details does nothing to alter that. The proposal still feels over-engineered and with insufficient real-world use cases for me, and altering details won’t change that. We’re nowhere near the point where bikeshedding over the name of the placeholder is a worthwhile use of people’s time.

Please, go back and look at the big picture, and address some of the higher level comments you’ve received. Otherwise, you’re going to find less and less people interested in continuing the discussion, until it simply dies from lack of support.

11 Likes
280

As long as we agree what the pipeline is best for:

  1. passing results without naming intermediate variables (shunning even the _= statement approach and the even more awkward _:=) - due to expressivity, readability and brevity
  2. building nested comprehensions and generators - same as above
  3. more flexibility arranging processing between expressions and statements to reflect the natural organization of a problem
  4. the most basic ability to capture processing in the natural left-to-right order without resorting to ugly workarounds
  5. irreplaceable for interactive environments, e.g. in exploratory data analyses where going back and forth for each added processing step is beyond silly,

I will be able to provide dozens of real world examples easily. OTOH if there is a very strong bias for even an ugly syntax as long as it “does the same thing” without introducing new elements to the language, then we will keep facing difficulties. A heads up here though, you can do everything possible in Python in any other language including punch hole cards :wink: The entire premise, the origins and the raison d’être of Python was/is to make things nicer.

281

Okay, in that case I don’t think I’m going to be able to help you.

Not only do I not agree with your list of benefits for pipelines, I do not agree with your way of running this discussion. This last post, especially in it’s tone, convinces me that you aren’t, at least right now, prepared for the work of authoring a PEP, as regards steering a discussion and listening to feedback.

It’s a shame, since you’re clearly smart, interested in the topic, and even have a working prototype. But you aren’t charting a course here for success.

5 Likes
282

Thank you for your comment. You are clearly meticulous about scrutinizing my ideas and methods which is great. However, I’m just trying to be productive here. If I’m supposed to incorporate the feedback from the community it cannot only be a destructive one (in the sense of rejecting, perhaps sometimes rightly so, my ideas without coming up with new ones). You don’t agree with the list of benefits? Please kindly share which benefits you see or mention that you already did so and I will look for your post. You don’t see any? Please mention what makes you think that pipelines are cool like you said. We need to pre-align on some axioms, otherwise the discussion will become circular again and I will spend time looking for real life examples which will convince no-one but myself. I hope you are not trying to set me up for such a wild goose chase to make a point.

1 Like
283

We had some, the original proposal before __pipe__ was added. Much of the recent confusion and frustration has been a result of that expansion in scope, which appears to have mostly been prompted by a feeling that it would be “nice to have”, rather than following naturally from the original scope of an operator that simply allowed chaining of function calls.

Having said that, I agree with @sirosen here. It’s your job to manage the discussion here, and you’re not really achieving that. Instead, you’re treating the discussion as a defense of your proposal against criticisms and objections. That’s not how to develop a PEP. You need to adapt your proposal based on feedback, not defend it. And you shouldn’t be asking others to do the work of presenting an alternative proposal - if they do, then great, but it’s not a requirement for offering feedback, and the lack of an alternative is not “destructive”, it’s simply feedback that you need to incorporate.

But I (and a number of others) have said all of this before. Regrettably, I don’t think this proposal is going anywhere useful at this point.

1 Like
284

I don’t think so. By axioms, I mean identified situations/scenarios where the pipeline operator is agreed upon to be a superior alternative to the existing syntax which misses only real life examples of actually occurring in code repositories in order to be a basis for a PEP. We never quite got to that stage as far as I can tell.

I am happy to go back to the early proposal. What we have right now is the result of incorporating the community feedback in the best way I thought of at the time. __pipe__ - dg-pb. Being able to “intuitively” chain with the fluent syntax (which prompted passing to arbitrary expressions on the RHS) - yourself. Maybe I should have done a better job about notifying you but I thought it would be clear enough.

Well that’s because you’re refusing to collaborate. When I ask for a particular type of feedback and people refuse and say it’s my job to figure it out based on their original feedback that’s not just childish. It’s unprofessional, inefficient and quite honestly a bit impolite.

You keep coming back with this argument but this is patently false and I have no clue what could make you think that. I organized the polls and keep probing the community because I am fully willing to adapt the proposal. I just refuse to do it in an uninformed or ill-informed fashion which will just further waste everyone’s time.

I’m not asking anyone to present an alternative proposal, just to express themselves in a coherent, logical and as helpful as possible fashion. If one says that the pipeline idea is cool and that they would like to see it advance then they should be able to say why. It’s not rocket science.

Sure it’s going to be incorporated once we vote on the use cases.

Please feel free to tune out at any point. I’m sure we can wrap it up without using more of your time. Thank you for your help so far.

285

OK, you’re now going beyond simple disagreement and frustrating discussions into outright name calling and abuse. Comments like this are in clear violation of the code of conduct. While I don’t plan on reporting you for a first offense, I could, and others might well not be so forgiving.

If you think this is an acceptable escalation for any discussion in the Python forums, you need to seriously rethink your behaviour, or you will end up getting banned.

I don’t know what to say to this. Did you at any time before now ask for clarification of why people (not just me, @sirosen has made similar comments, if I recall correctly) were saying you weren’t incorporating feedback in your design? You’ve posted polls, but multiple people have said that they aren’t able to respond to those polls - did you try to determine why? Did you ask? Did you consider my post saying that bikeshedding over names was premature?

I think you massively underestimate the level to which many people react instinctively to proposals like this. Understanding why people dislike something is not as simple as “please tell me, in clear logical terms, what you dislike and how I can address your concerns”. It’s a discussion, where you offer alternatives and see what people think (as a conversation, not as a poll), and gradually come to an understaning of what’s behind the feedback you’re getting. That’s a lot of work for you. Not doing that work won’t mean that the feedback you’re getting will improve, it’ll simply mean that people feel ignored and will stop interacting with you, resulting in your proposal failing.

Which is what is happening right now.

You’re so insistent that voting is the way to reach a design. I guess I can’t persuade you otherwise. But your latest vote has three participants. That’s meaningless. How do you plan on getting a reasonably representative number of people to participate in your polls?

At some point I will, no doubt. But I’m stubborn, and a bit of an optimist. I still hope that you’ll change your approach and we’ll get a viable proposal. Because as I’ve said on a few occasions, I like the idea of a pipeline operator in principle. So I keep trying to help you improve your management of this discussion and of the proposal. I suspect you no longer welcome my help, though, so my comments are likely falling on deaf ears :slightly_frowning_face:

4 Likes
286

These polls create some polling related biases in how you have created them, to the point where even the ones where I have a clear stance, I’ve abstained from voting.

When trying to pick between multiple options, it’s generally better to provide non-exclusive approval voting to avoid issues with strategic voting and allow expressing things like “any of these, but definitely not this”

That’s before getting into things like “the assumption that one of these options is workable, and we’ve exhausted alternatives not listed” which I dont think is the case here. You dont have a “split consensus” where there’s agreement that something should happen and it’s merely choosing what color(s) to paint it in, there are still more fundamental disagreements that you either need to accomodate or decide the proposal is fine despite that, and explicitly create it with that known edge. This is one of the responsibilities you have when proposing something, it’s okay to decide that a proposal explicitly doesn’t address certain things, those things may act as a barrier, but they arent always showstoppers. It’s your responsibility to figure out which ones to work on and which ones to place out of scope.

7 Likes
287

I don’t think it’s a violation of the code of conduct to define an undesirable behavior. You have no reservations about pointing out my job and I don’t see why I should have any more reservation pointing out every productive community member’s job which I believe IS most definitely NOT to partake in childish, unprofessional, inefficient and impolite behaviors. If those are protected behaviors in this community, I feel sorry for the community.

Please feel free to report me but I seriously doubt it would be a fair process being pitched against someone from the core team.

BTW you mentioned at some point that achieving stakeholder alignment takes a very long time in these forums. I don’t think this should be carried as some badge of honor but rather as a signal that something could perhaps be improved in the process or in the people. Holding “the community” up to a certain standard of behavior towards the potential PEP author could be a promising first step. So could be at least trying to provide the feedback he’s asking for.

I did what I thought would most be most efficient to get a quick look at the “community” opinion. Quite a few people did participate in the polls. I assumed you would elaborate without being pulled by the tongue. If you were some kind of a newbie here like myself I would have probed you more. Honestly, I don’t know why you wouldn’t provide everything you’d like to share upfront. If I haven’t asked other veterans it is for the same reason or because I assumed it’s because of the missing/weak use cases. Was I wrong? Did I assume too much? Please let me know.

I am happy to help and guide as long as it takes but whenever I try to understand and ask for more info, you chime in and claim people are not required to provide it. That’s sabotage.

I am going to ask the core team to kindly stop discouraging people from taking part in the meaningful and extremely helpful polls, and also to stop stating as soon as a new poll comes out that they themselves don’t vote, due to a disproportionate influence this kind of statements coming from the core team have on the community members. The first polls went very well and we had 10+ answers for most questions.

Thanks, all help is welcome. I’m open to change but not without good reason and a clear direction. If you don’t want to share everything that’s on your mind upfront, fine I can keep asking. Could you please kindly share why you feel that I am not incorporating your feedback into the proposal? Could you please specify as well why you think the “bikeshedding” (totally not name calling) is premature?

288

Thanks, I’ll take this into account in any potential future polling.