Because he doesn’t argue with that, he just tries to clarify what he has said, because you instead of saying “ok, I understand, but there is another issue”, go arguing that he was wrong in his simple and clear statement - “This would not break existing code as it is”.
I said above that I can see how this could be problematic.
You are still arguing with me about a point that you think I am making rather than the point that I actually made: this proposal does not break existing unchanged code.
1 Like
It makes all existing code with that pattern unsafe breaking the pattern itself rather than the code directly right this moment. It makes a new place where people have to watch out an be surprised by overriding what was previously a language level guarantee. Can you not see how removing a language level guarantee about short-circuiting is breaking? Not just “problematic”, but that it actually breaks a promise people rely on?
I suspect that I agree with you about the idea but I am not going to discuss it further. It should be possible for you to understand the clear point that I made (and reexplained several times) without needing to argue with me against a proposal that I have not actually expressed any support for.
1 Like
I don’t think your point was as clearly made as you think it was, and I specifically asked a clarifying question about whether or not you think changing language guarantees should qualify as breaking since you insisted that the code was not in fact broken.
It’s possible we have some agreement here, but I don’t think your insistence that this isn’t breaking somehow is accurate to what I perceive as reality, so I was trying to figure out if it’s just you being pedantic about what broke, or if you disagree with me about if this breaks.
To address the specific question asked in the title of the thread:
- reviving PEP 335 directly has essentially zero chance of success, since the rationale for its rejection is still valid
- reviving PEP 532 and PEP 535 is potentially plausible, since they’re specifically designed to sidestep the problems that got PEP 335 rejected by largely leaving the existing logical operators alone
What’s missing from the latter two PEPs is compelling motivating examples that would take advantage of the short circuiting logical operator customisation in order to justify the additional complexity general purpose short circuiting operators bring to the language definition.
The PEPs also need updating to account for other changes since they were written (such as assignment expressions and the switch to PEG as the reference grammar format).
6 Likes
Just my two cents: I remember various discussions about this in the past. I don’t recall the exact differences between PEP 335 and other related proposals, but ultimately I’d like to see something along these lines. I think being able to override and and or would be useful. I don’t see it as essential to maintain the implicit guarantee of “and and or always short-circuit no matter way”; it’s essential to ensure that it short-circuits with the same operands it short-circuited for before, but if people want to write new code that defines new kinds of operands for which it doesn’t short-circuit, that’s fine. I don’t see this as fundamentally different from any other kind of operator overloading. We all assume that a + b will not delete your hard drive but someone could write a class that makes it do that. If we didn’t have operator overloading, people could be relying on a + b raising an error unless, say, both objects were numeric types and then adding __add__ would “break” this, but I wouldn’t consider that a blocker.
3 Likes
As the author of PEP 335, perhaps I can offer some perspective on why I didn’t pursue it any further.
It’s something of an XY problem. The main reason I wanted to be able to override logical operators in the first place was for implementing DSLs such as SQL where logical operators are used a lot. Being able to override and and or would make things like that read much more smoothly.
However, there’s a fundamental flaw with that approach – it relies on enough of the leaf nodes of your expression being of the appropriate special types, which can be laborious to achieve. And in the case of logical operators, it only works if the left operand is one of your special types, otherwise normal short-circuiting semantics will be applied.
These are not insurmountable problems, but they somewhat take the shine off the idea of being able to just write your DSL expression in Python. You need to be aware of various pitfalls and take care to avoid them.
So I no longer think it’s worth going to extroardinary lengths to make logical operators overridable, just to provide what will only ever be a half-baked solution to the DSL problem.
I would rather take a step back and look for a different solution altogether. I have some ideas about that, but they belong in another thread.
8 Likes
It’s also worth noting that PEP 532/535 don’t fix the problem @gcewing mentioned (if anything, they make it worse, since it’s only for short-circuiting else that the left operand has to be special, while for short-circuiting if it’s the right operand that needs to be a circuit breaker. And if anyone slips up and uses the native and and or operators, the custom short circuiting semantics get bypassed entirely).
I don’t think those problems are bad enough to outright withdraw PEP 532 (it’s still the nicest idea I’ve come up with for offering a way to override logical operations, assuming that is something we actually want to do), but they’re certainly part of why I don’t feel any motivation to start actively working on it again.
1 Like
→ True.
(Related :Linked Booleans Logics (rethinking PEP 505))
The only way out of this would be to create a “deferred” context, with a root-based approach, rather than leaf-based, and where the global operations would be locally overriden within a context, e.g. overriding the __getattr__, __getitem__, __or__, __and__ of roots a, b, c in :
WithDeferredContext{ a | b & c[d].e } # pseudo-code
Not sure it is pythonic (thus viable) though, because it is a mini-language.
But it would cover PEP 505 issues and possibly more…
Totally legit. I think if you want to do such specialized stuff, you better use explicitly specialized operations, because you can specialize yourself in understanding the specialized ops, and leave the general operators alone.
1 Like
Thanks for this. And when you say “PEP 335”, does it apply to this specific PEP or “any form of logical operator overloading”?
This is one of the reasons I took a step back to PEP 335.
PEPs 532/535 looked fairly complex in comparison while achieve pretty much the same thing, but from certain POV even less.
Maybe, maybe not.
If it is possible to achieve something without adding to syntax it is a plus.
Also, if extension to achieve new things makes use of existing syntax while retaining consistency in meaning - it is a big plus.
So in no way I see this argument as “winning by default”.
Yes, from POV of DSL only, leaf problem makes it not very attractive, but there are 3 underlying things behind this:
numpy.array() or numpy.array()1 < numpy.array() < 2- Simple DSLs:
maybe(a).attr or maybe(b).attr
To me, all of these are of interest to address if possible.
If there is an extension that can address all of these without introduction of new syntax, then the leaf-issue could be a reasonable cost compared to alternatives. And for cases where this issue is a deal breaker, then one should indeed needs to look for alternative solutions, but it doesn’t necessarily invalidate benefits of this.
Although PEP 335 is heavy on complications to deal with, compared to alternatives that I have seen, it is very light in terms of syntax and high level implementation design. Furthermore, as I said, it has a very big advantage - nothing else will be able to address (1) apart from itself.
Would be interested to hear. Maybe Linked Booleans Logics (rethinking PEP 505) could be a good place for it?
I mean changing the runtime semantics of and and or. It’s just fraught with too many backwards compatibility risks for a highly arguable gain in expressiveness.
New binary logical operators (such as if and else) would provide comparable expressiveness, but without the same level of compatibility risk. It’s still highly debatable as to whether they offer enough additional expressiveness to justify offering two different ways to express logical conjunction and disjunction in the core language, though.
2 Likes
Ok, I can see and appreciate that.
So or/and are not going to be adapted to serve things such as numpy.
Then, another option could be to introduce actual logical operators (|| and &&) that are not short-circuited and used by a < b < c if implemented, s.t.:
_expr = EXPR
_lhs_result = LEFT_BOUND LEFT_OP _expr
if hasattr(type(_lhs_result), "__logical_and__"):
_expr_result = _lhs_result || (_expr RIGHT_OP RIGHT_BOUND)
else:
_expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)
This would cover (1) and (2) from my previous post and then DSL stuff could be addressed separately.
Or if logical operators isn’t worth the effort, then to address (2), all that is needed is one extra method:
if hasattr(type(_lhs_result), "__chained_cmp_and__"):
_expr_result = _lhs_result.__chained_cmp_and__((_expr RIGHT_OP RIGHT_BOUND))
else:
_expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)
In either case, I think it is worth considering separating these 2:
- Overloadable logical operators and
a < b < c - Custom DSL with short-circuiting
In this case short-circuited DSL solutions are not constrained with needing to solve other (directly unrelated) problems and could instead leverage the freedom to achieve good degree of flexibility and customisation.
Was using new operators ever been suggested? like lor land or something along these lines.
These operators would allow overloading of the comparison but could have the same precedence as other operators. This would avoid requiring the use of () in operations like (arr > 5) & (arr < 10) that’s a common cause of issues since the binary and/or have higher precedence and and/or cannot be overloaded.
Just a though
But those are currently valid names for other things, so that’s a different breaking change. Or is that a numeral 1?
If they can be made soft keywords that should not be an issue. Not in support of the idea however.
Those were just examples, not suggesting them as the best options.
In any case I think they could behave like match or case that are soft keywords, avoiding a braking change.
Yes, making if and else usable as binary operators is the key surface syntax change differentiating PEP 532 from PEP 335 (there are underlying semantic differences too, but it’s the choice of different keywords that helps mitigate the compatibility risks)
I’ve started a new thread for my current ideas on DSLs:
https://discuss.python.org/t/dsl-operator-a-different-approach-to-dsls/79874
3 Likes