Is using `is` with arbitrary objects safe?

Python Help
1

I am implementing a dataclass that allows adding dataclasses of the same type, like this:

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class Example:
    main_data: tuple[Any, ...] = ()
    extra_data: object = None

    def __add__(self, other: Example | object) -> Example:
        if isinstance(other, Example) and self.extra_data == other.extra_data:
            return Example(self.main_data + other.main_data, self.extra_data)
        else:
            return NotImplemented

The problem with this approach is that in addition to being slow, if you do Example((), []) + Example((), []), then it silently discards the second list. The only solution I see it to do self.extra_data is other.extra_data.

However, because extra_data can be anything, it might be an immutable constant, making the expression similar to 0 is 0. Obviously, a = 0; b = 0; a is b is undefined and may change in minor versions without warning. But what about a = 0; b = a; a is b? This seems equivalent to 0 is 0, which causes a SyntaxWarning. If this could be False, then my object could break if I repeatedly added it to itself.

I wasn’t able to find a definitive answer in the specification. The Python Language Reference 3.1. Data model § Objects, values and types states:

Types affect almost all aspects of object behavior. Even the importance of object identity is affected in some sense: for immutable types, operations that compute new values may actually return a reference to any existing object with the same type and value, while for mutable objects this is not allowed. For example, after a = 1; b = 1, a and b may or may not refer to the same object with the value one, depending on the implementation. This is because int is an immutable type, so the reference to 1 can be reused. This behaviour depends on the implementation used, so should not be relied upon, but is something to be aware of when making use of object identity tests.

It doesn’t define exactly what “operations that compute new values” are. This might actually create a problem for some implementations’ optimizations. For example, say func(a) is called in a loop from 0-255, and JIT optimizes it to use a register that holds the value of a instead of the actual object (assuming the function only performs arithmetic operations on a). If you call id(a) inside the function, is it now required to be the same ID as the other one, or can the interpreter create a new integer of the same value?

2

From this small example I can’t see why that would be a problem. What would you want it to do instead?

That is guaranteed to be true, since you set b to whatever object a is pointing to.

I’m not sure what “the other one” means here. For the a=1; b=1 case you cannot in general rely on is returning true. This an implementation detail. But as I said above, it’s not clear to me from your example why it would matter for what you’re doing.

3

It’s fine as a shortcut (if is messes up, that’s on the user in my opinion), but doesn’t the std lib use that ‘short circuit’ in its equality checks already, so there may be no need for extra code at all?

4

The problem is that the second list disappears, so among other problems it breaks the communicative property of addition:

>>> a = []
>>> b = []
>>> example_a = Example(a)
>>> example_b = Example(b)
>>> sum_a = example_a + example_b
>>> sum_b = example_b + example_a
>>> a.append(123)
>>> sum_a.extra_data
[123]
>>> sum_b.extra_data
[]
5

I didn’t know that, where does it use it?

6

There’s a very important difference here. When you set b to be the same thing as a, it is really truly the same object and the identity check is guaranteed to be True. The reason there’s a SyntaxWarning for 0 is 0 is that literals are not guaranteed to create the same object every time. For an obvious example, take lists - [] is [] will always be False, because every time you create a list, it is guaranteed to be a new list.

So, if you want to see if they have the same extra_data, you can indeed check if they are identical.

7

Assumming I’m correct, in the generated dataclass __eq__ or maybe even in the implementation of object.__eq__

8

There’s a very important difference here. When you set b to be the same thing as a, it is really truly the same object and the identity check is guaranteed to be True. The reason there’s a SyntaxWarning for 0 is 0 is that literals are not guaranteed to create the same object every time. For an obvious example, take lists - [] is [] will always be False, because every time you create a list, it is guaranteed to be a new list.
So, if you want to see if they have the same extra_data, you can indeed check if they are identical.

That makes sense, the issue is I can’t find anything in the specification stating this has to be true for all implementations of Python. The section on assignments only guarantees that a = b = 0 makes them the same object.

9

I am not sure what exactly you want a reference for? That a = b makes sure that a and b point to the same object? That the same object always means is returns true? That they have the same id?

10

The same guarantee can be spelled like this:

a = 0
b = a

The syntax a = b = 0 is a shorthand for the same thing. Either way, it’s guaranteed to set both a and b to the same object.

11

Looking at that example I fear there may be other ways in which various suggested solutions might not satisfy you. I wouldn’t characterize that example as “not satisfying the commutative property” but as sharing data between objects in a way that might have unexpected consequences. My first inclination is to say that you would want to have __add__ store a copy of self.extra_data, but I think you’ll need to say more about your requirements for this class. (Another reason I think this is that your original example does not specify any behavior for when the two added objects do not have equal extra_data, but different suggested implementations may have different consequences for that case, and presumably those differences would also be relevant.)

12

Depends on the context.

This for example prints False:

class D(dict):
    def __getitem__(self, key):
        return {'a': 1, 'b': 2}[key]

exec("a = 0; b = a; print(a is b)", locals=D())

Attempt This Online!

13

The code is either buggy or ‘consenting adult’ code in that it subverts a language guarantee.

class D(dict):
   def __getitem__(self, key):
       return 42
d = D()
d['b'] = 2; print(d['b'] == 2)

This simplified code prints False, so D is not properly a mapping as required by exec.

14

IMO it’s the latter, and it’s an example of a proof that any claim about “this does that” in Python can be subverted if you try hard enough. Fortunately, only the most utterly pedantic of people would claim this sort of thing.

Unfortunately, “the most utterly pedantic of people” happens to include the majority of the world’s computer programmers. Or maybe that’s “fortunately”. I’m not sure.

15

My gripe is that multiple people said it’s guaranteed without showing where to find that guarantee. Given that the OP talked about searching in the specification, and given that it’s not always true, I think it would be good if someone showed where it is.

16

That may be but I’m not sure a contrived example showing how to violate it in extreme circumstances helps that. Even if python didn’t have all its easy ways of subverting behavior there is always the option of buggy implementation, memory corruption, cosmic rays.

There is some level where you just have to assume the underlying objects are behaving sensibly and it doesn’t really help the clarity and usefulness of a specification to state them explicitly. That local variables are backed by a mapping that returns the same object for a key that was stored I would say is one of them.

For a definition I would say that it is implied by the requirement that the execution model here

Names refer to objects. Names are introduced by name binding operations.

17

Yeah, which is why IMO these kinds of “haha check this out” examples are distinctly unhelpful in threads of this nature.

Stefan, please ignore the pathological[1] case with a deliberately misbehaving namespace dictionary. It is not relevant. People CAN depend on the fundamentals.

  1. but definitely entertaining ↩︎

18

a = b implying a is b is an emergent property of three parts of the language spec:

Yes, it’s possible to pass ill-behaved namespaces to exec, but at that point you’re executing a language that shares Python’s syntax while providing different semantics rather than executing regular Python code (metaclasses sometimes even do that without it being a problem!).

19

Thank you, I missed this part in particular in assignment statements:

Otherwise: the name is bound to the object in the global namespace or the outer namespace determined by nonlocal, respectively.

I agree that a broken namespace messes things up so much that virtually any code won’t work correctly. I was thinking more about interpreters in the future (including, potentially but highly unlikely, CPython) changing this between minor versions to implement an optimization, and not violating the spec by doing so.

20

That isn’t actually the code I showed, it’s that code wrapped in an exec. As others said, this isn’t really the same thing. The dynamism of Python means that virtually nothing is fully guaranteed. I seem to recall seeing examples where people use ctypes hacking to make 2 + 2 = 5.