Any chance for reconsideration of PEP 335?

Ideas
1

So PEP 335 was rejected.

Then a series of related PEPs followed: PEP505, PEP531, PEP532, PEP535.

However, all what they are trying to achieve (which are several different things) would be possible to do with PEP 335:

  1. PEP 505 - pymaybe library could just make use of it
  2. PEP 531 - would be unnecessary due to (1)
  3. PEP 532 - probably would not be able to do all of that, but would cover the main presented use case (this one looks very heavy in additional design)
  4. PEP 535 - would be covered by this if 1 < a < 1 was to make use of it.

PEP 335 from a design perspective seems a pretty solid solution:

  1. It seamlessly integrates into the way things are - from user’s perspective it doesn’t bring in anything completely new - just one unary and 2 binary operators that one implements in the same manner as others
  2. Doesn’t seem to have any backward compatibility issues

Rejection reasons. 2 Sources:

  1. [Python-Dev] PEP 335 officially rejected
  2. PEP 532 – A circuit breaking protocol and binary operators | peps.python.org

So I am not sure about performance, but from my experience, there are always new ways to optimise if very much needed.

I don’t think the fact that this is a non trivial extension can be refuted. Learning materials would need to be updated and a lot of work to unwrap all that was built on top of the status quo would need to be done.

The benefits are well known, but to me the biggest sell point is that it is extremely unlikely that any alternative is going to appear that addresses the underlying need.

And 20 years later the need is still here. (At least 3 threads on PEP505…)

And I doubt that any idea will look “attractive/simple/pythonic enough” given the underlying need is simply “overloading logical operators”.

Anything else in comparison will look like A Rube Goldberg machine.

And to me it seems that any further endeavours are destined to fail as they are “trying to fill a circular hole without using circular shapes”.

So my questions:

  1. If this decision was to be made at the very beginning of Python, would this be a good idea?
  2. Is there any chance for reconsideration?
2 Likes
2

(I suggest mods move this thread to the Ideas category. PEPs is for things actively being considered).

personal non-SC hat:

At a high level, “3 threads in 20 years” does not sound like a need being expressed to me. To demonstrate a need, bring everyone up to date with modern practical examples where it would be important to do this and why the alternatives are not great.

1 Like
3

(done, moved) I put it into the general category since it isn’t really an idea either, but it can be moved again if others feel it’s better there.

4

3 threads on PEP505 only, which is only one specialisation of this. And PEP505 was created 9 years ago.

If a genuine willingness to engage in this was to follow I could make an effort to gather evidence, but I think the situation to those who are not new to community is more or less clear - numpy, proxy objects, short-circuiting, many failed PEPs, no one got what they wanted, numpy.logical_or, more PEPs, import pymaybe - no one uses it as it doesn’t short circuit.

I have made a fair amount of statements and guesses to start a discussion - happy to discuss them one by one. But my hope is not to “prove something to get my way”, but rather to see if there are others who see things the same way.

I don’t want to repeat what is already well known.
Instead I can depict the current situation as follows:

  1. I want to write object with logcal operations
class Proxy:
    def __init__(self, val):
        self.val = val
  1. I can not overload logical operators, so I think “maybe I am not going to need bitwise so I can use them instead”
class Proxy:
    def __init__(self, val):
        self.val = val

    def __or__(self, other):
        return self.val or other.val
  1. Then I realise that this is not a very good idea:
    a) It is inconsistent with other applications. It is bitwise operator afterall
    b) I actually need bitwise operator after all…

So I think maybe I can implement a method:

class Proxy:
    def __init__(self, val):
        self.val = val

    def __or__(self, other):
        return self.val | other.val

    def logical_or(self, other):
        return self.val or other.val
  1. Oh, but I need __getattr__ as well. What do I do?
class Proxy:
    def __init__(self, val):
        self.val = val

    def __or__(self, other):
        return self.val | other.val

    def logical_or(self, other):
        return self.val or other.val

    def __getattr__(self, name):
        return getattr(self.val, name)
  1. Oh but now my __getattr__ will not work if name="logical_or". Maybe I will just use external function instead:
def logical_or(self, other):
    return self.val or other.val


class Proxy:
    def __init__(self, val):
        self.val = val

    def __or__(self, other):
        return self.val | other.val

    def __getattr__(self, name):
        return getattr(self.val, name)

So this is final solution of numpy. And the situation is as follows:

  1. This does not work in case Proxy is written in C and val is not exposed. So there are additional issues to be addressed. Either
    a) exposing val, but then __getattr__ will not work for val
    b) Write everything in C, but what if I am cython compiling…? There are workarounds for this too…
  2. My DSL looks like:
logical_or(a, b)
# Compared to
a or b
  1. There is no (and never has been) short-curcuiting in equation. Just need to make peace that it can not be emulated.

So I have (arguably) found the best possible solution, but I can not shake the feeling of how all the ugliness (in design, result and the process I had to go through) above could be solved with PEP335.

5

If you want a or b to continue to short-circuit, the ONLY part that the value of a is allowed to override is a simple yes-no question: “are you a thing, or should I go figure out what b is?”. This is not overriding the logical Or operator; this is overriding its truthiness, ie __bool__().

If you DON’T want it to short-circuit, why do you want it to be written as a or b rather than a | b ? If your DSL is fundamentally built on top of Python, it should continue to act like Python, including short-circuiting behaviour.

Maybe you would do better to build a full (mini-)language from scratch, rather than trying to do everything with operator overloading. It isn’t as hard as you might think.

2 Likes
6

If you don’t gather evidence and present a case for resubmitting PEP 335, how do you expect this to go anywhere?

Remember, ultimately you don’t need community engagement, all you actually need is

  • The interest of at least one core dev to sponsor a new iteration of the PEP.
  • Arguments sufficiently convincing to sway the SC.
  • Evidence that the reasons for rejecting PEP 335 are either no longer applicable, or are outweighed by new benefits that weren’t part of the arguments in that PEP.

Community support is at best a proxy to indicate whether your arguments are convincing. The SC will almost certainly take a more positive view of proposals that have strong community support, but it’s no substitute for a solid set of arguments in the PEP. And community interest (or “willingness to engage” as you put it) isn’t any real use on its own.

So I suggest that if you want to take this forward, focus on building a good set of arguments, and a compelling proposal. You’ll get the engagement when you have them.

1 Like
7

This is obviously false, check PEP 335 for a suggestion for how this could be implemented.

The point is that you sometimes want a or b to return an object that is neither exactly a nor exactly b, but instead a combination of the two. This is tricky because short circuiting still has to be supported, but this is not an unsolvable issue - see the PEP for a possible solution.

| is bitwise or. or is logical or. Both of these operations make sense for e.g. integers with different results. Integers can be packed into a list. On this list we can define operations elementwise. So a | b does elementwise bitwise OR and a or b does elementwise logical OR. Sadly, the latter is impossible to implement.

Which of these is a fundamentally incompatible with design goals of python? That operations can be elementwise?

1 Like
8

While it could be possible to create a way to not short-circuit or (and you are right that the original pep actually provides one such way), I don’t think any amount of improvement in specific cases will be worth every python developer then having to question “is this use of or lazy/short-circuiting.?” when reading code.

5 Likes
9

Why would you need to think about this all the time? 99% of the time it doesn’t matter at all. And unless people start abusing this feature, it will be obvious most of the time: Is this a numpy array or similar “proxy” object? Yes? Then this doesn’t short circuits. Otherwise, it short circuits. People already have to learn if + means addition or concatenation, if if a: is a valid operation or if a | b is bitwise or logical.

Yes, people will have to learn that it is possible for both sides of or to be evaluated unconditionally. But I don’t believe it’s a huge adjustment.

1 Like
10

That’s a pretty large change when talking about the behavior of operators and that these have been short-circuiting forever. There are plenty of real-world examples of people relying on the short-circuiting behavior of or and and to guard access. I don’t think any amount of uncertainty on this, however slim you think it will be, is going to be worth not just calling an appropriate method.

There’s also the other downside that pep 335 mentioned, but didn’t really address, it breaks things like refactoring and optimizations based on De Morgan’s laws. You’d also need to accept this as a cost, or provide an alternative method which does not have this drawback.

11

Conversely, I do think this is a huge adjustment. Having to worry about the type of the LHS of an and/or expression would be a significant bug magnet.

5 Likes
12

But you already need to do this. You need to worry about:

  • Can you call bool on it at all? (e.g. numpy arrays)
  • When is it truthy? When is it not truthy? What does this mean for what is valid on the right side?

I genuinely don’t understand where you are seeing an increased potential for bugs here; Can you provide an example that isn’t already breakable by providing an unexpected data type?

1 Like
13

You have that backwards. We don’t need to provide examples to show how broken it would be to massively change core semantics. But here’s one example. In documentation, it is VERY common to write semi-equivalencies such as:

a < b < c
# is the same as
a < b and b < c
# except that b is only evaluated once

Similarly:

a and b
# is the same as
a if a else b
# except that a is only evaluated once

It would be EXTREMELY surprising and error-prone if these were no longer equivalent. Imagine seeing the second example in code review. Is it important? Aside from figuring out whether it’s significant that a might be evaluated twice, now you have to consider whether a has overridden boolean or.

This proposal has been dead for years, with only a very small amount of interest in it, much less actual support. In order to revive it, you’ll need a lot more than just “why not”. You’ll need some compelling arguments in favour of breaking these equivalencies.

1 Like
14

And yet I am getting told “It’s obviously a bug magnet”. If this is the case, then surely you can show me examples? I am asking for examples because I don’t understand the push back. I was told “No argument could convince me because the change in semantics is terrible”. So the change should have obvious negative consequences that you can show me, right?

This should continue to be equivalent. Otherwise the proposal would be far less useful.

Fair, this would change behavior, although I continue to be of the opinion that the only usecase where the new overloading should change behavior is if this would previously raise an exception.

15

Anything that relies on shortcutting. How about short_calculation() or long_calculation()?

1 Like
16

So if a < b returns something that overrides the and operator, this override would be implicitly called by chained comparisons? Is that how the equivalence is kept?

Really? The existing behaviour is valid with pretty much any objects as a and b. The only way it would raise is if bool(a) raises. But you’re proposing letting any object redefine this behaviour. On what basis do you say it will only change behaviour if it would have raised?

17

Ok, I am bowing out of this discussion. At this point I believe that @Rosuav Is deliberately misunderstanding me (which is typical behavior) and I am not going continue engaging.

@pf_moore Sorry, you were less at fault, although Is till believe you could have a bit more of an open mind in trying to understand what I said. (i.e. try to understand why short_calculation() or long_calculation() might already have surprising semantics based on the returned type of short_calculation())

18

Either that, or I’m genuinely misunderstanding you, and you don’t want to explain yourself better.

19

I think you’re misunderstanding my role here. Whether I support this proposal or not is completely irrelevant. I don’t intend to sponsor a PEP on this, and I’m not in a position to approve one. But I do approve packaging PEPs, so I have a good idea of what makes a PEP successful. I’m trying to help you (or anyone who wants to take this forward) to create a PEP with a reasonable chance of success.

And for that to happen, the PEP needs to take the problem of breaking existing code very seriously. It’s not something that can be casually assumed to not be an issue.

The example I gave was intended to make you think about what code this proposal would break. It’s the type of code that might appear in a caching application - short_calculation() looks a value up in the cache, and if it can’t find a cached value, we do the expensive computation to produce the value we need. This proposal breaks that if the values we are caching ever include a type that overrides or - it’s critical that the operation short circuits as the whole point is to avoid doing the expensive calculation when we don’t need to.

And before you say it, whether there are better ways of writing this is irrelevant. We can’t break code just because we disapprove of it. The language spec is a set of promises we make to our users. If we change those promises, we have to accept the responsibility and ensure that the benefit justifies the cost.

But fine. If you’re not interested in my advice, that’s OK. I think you’ll end up wasting your time on an unsuccessful proposal, but it’s your time to use as you want, and it’s entirely possible that I could be mistaken about the SC’s views. I’ve said what I wanted to, and if it’s of any use, no matter how small, then that’s great.

2 Likes
20

It is not exactly clear what is being proposed here but as I understand it no existing code would be broken unless something is changed to use the proposed new features.

1 Like