~bool deprecation

of course; I was just addressing the point that people trying to do ~bool don’t necessarily have “deeper conceptual gaps”, because it’s entirely natural in large amounts of existing code.

1 Like

Then they’re conceptually confused, seduced by a footgun popularized by an extension. It’s not true, e.g., that “it only works for arrays”, but the conceptual model is baffling:

>>> import numpy
>>> ~numpy.True_ # works fine for a scalar of numpy's bool type
np.False_
>>> ~numpy.False_ # ditto
np.True_
>>> -numpy.True_ # but unary minus blows up
Traceback (most recent call last):
    ...
TypeError: The numpy boolean negative, the `-` operator, is not supported, use the `~` operator or the logical_not function instead.
>>> numpy.True_ * 2 # you can multiply by an int
np.int64(2)
>>> numpy.True_ + numpy.True_ # but a different result than adding to itself!
np.True_
>>> t = numpy.True_
>>> t+t+t+t+t+t # bug magnet
np.True_
>>> numpy.True_ - numpy.True_# but you can't subtract them
Traceback (most recent call last):
    ...
TypeError: numpy boolean subtract, the `-` operator, is not supported, use the bitwise_xor, the `^` operator, or the logical_xor function instead.
>>> 0  - numpy.True_ # but you can subtract from an int
np.int64(-1)
>>> numpy.True_ * numpy.True_ # or multiply with a surprising result
np.True_
>>> numpy.True_ / numpy.True_ # or divide
np.float64(1.0)
>>> numpy.True_ // numpy.True_ # getting a float64 (as above) or an 8-bit int
np.int8(1)

Everything Python does follows from that bool is a subclass of int. That’s all you have to remember. numpy’s bool stands as unique in its type system, and is not even “a numeric type” there - although various operations’ special cases make it act like one in various ad hoc ways.

It’s simply incoherent, a grab-bag of special cases. The core language shouldn’t budge the width of an electron to try to cater to any such stuff.

9 Likes

Could a lot of these operations that make no sense for boolean types be reasonably turned into TypeErrors – absolutely[1][2]. But the surface area of numpy’s boolean type has nothing to do per se with the fact that a single-character unary operator for negation (other than the untypeable ¬) fills an obvious and natural demand.

CPython followed C, which at the time had no boolean type itself; a hole that has been filled in the meantime by introducing a proper type bool that only has values true and false[3]. In fact, C at the time didn’t have a “logical not” operator either, but does have ! for that purpose now[4].

So in short, a lot of things (CPython, numpy, C itself) grew mostly organically and in parallel. Now that C99 has both ~ and !, perhaps the best long-term solution would be to “rebase” onto the newer C standard and introduce ! as a synonym for not? Notwithstanding the migration pain, the next major version of numpy could then follow suit, at least in principle.


  1. as always, there’s a lot of historical cruft involved; e.g. numpy predating the broad availability of a boolean type in C, and the wished-for deprecation of other operations never happening. ↩︎

  2. what you call a bug magnet is arguably more reasonable than returning 5. From a quick search on the issue tracker, it seems to be a deliberate choice. ↩︎

  3. still an integer type, but with only 1 bit. ↩︎

  4. I went spelunking a bit in the paper log of the committee, it’s not present in a draft from 1995 [section 6.3.3], but appears from 1998 [section 6.5.3] onwards; however, I didn’t manage to pinpoint the paper that added it. Not much changed since then, though obviously the most recent draft is more readable. ↩︎

2 Likes

I definitely used ! in 1990 or like. And I am sure it was in books long before that.

1 Like

Sure; the standard self-assembled over time from existing practice, incl. common extensions. I don’t have older copies of the (paid) ISO text, but AFAICT ! wasn’t actually standardized before C99 (but will happily admit that I’m wrong if someone has a good reference).

I’m highly dubious. GCC is perfectly happy to compile !0 in C90 standard version [1], and like Serhiy, I was using that operator WAY earlier, and using obscure compilers to do so.

In one of your footnotes, you cited https://www.open-std.org/jtc1/sc22/wg14/www/docs/n457.pdf and claimed that the logical negation operator wasn’t present. Section 6.3.3.3 defines !E as equivalent to (0==E). I also found this copy of K&R’s original book which has similar wording in section A.7.4.7 [2], though I can’t vouch for its authenticity.

So that leads me to conclude that this operator was in the very first definition of the language, and every ISO standardization since. I’m going to flip this around: I’ll believe your claim if you have a good reference.


  1. I compiled int main() {return !0;} //This is a comment and got an error, but it’s fine without the comment ↩︎

  2. which, for a section literally named “A 747”, is hardly large enough to count as a Jumbo Jet ↩︎

3 Likes

No need, I’ll concede immediately. I had dug up the oldest standard available (which isn’t text-searchable), and though I had found the right section, overlooked the definition of !. Apologies.

I’m trying to reconstruct some history here way after the fact; early numpy developers would know what I can only guess at. Presumably the fact that CPython hadn’t made ! legal syntax-wise (preferring the more readable not), meant that numpy’s choice of characters for boolean negation was very constrained, and fell on ~; that had a different meaning in CPython (which was evidently less important for numpy), but at least was legal/overrideable.

Overall I still arrive back at the same point though: The rift underlying this discussion is that large swathes of the numpy-powered side of the Python ecosystem are used to ~, while CPython assigns totally different meaning to that operator. So perhaps the least bad compromise would be to make ! legal in CPython, and gradually adopt that in numpy[1].


  1. while phasing out ~, and removing it with at a major version boundary ↩︎

3 Likes

No probs, an understandable oversight. I myself was looking in a completely wrong section at first too, but in hindsight, it does make sense for ! to be next to unary minus, separate from && and || which are specifically short-circuiting operators.

2 Likes

This is the original sin. It should not be a subclass of int.

It isn’t just numpy that uses ~ for logical not e.g.:

>>> from sympy import *
>>> x = Symbol('x')
>>> p = Piecewise((conjugate(x), ~Contains(x, Reals)), (x, True))
>>> pprint(p)
⎧_            
⎪x  for ¬x ∈ ℝ
⎨             
⎪x  otherwise 
⎩  

The not operator cannot be used for this:

>>> class A:
...     def __bool__(self):
...         return 'foo'
...         
>>> not A()
Traceback (most recent call last):
  File "<python-input-25>", line 1, in <module>
    not A()
TypeError: __bool__ should return bool, returned str
1 Like

So it seems like an underlying issue many libraries are overloading ~ because they can’t overload not - potentially confusing users about the difference between then.

Maybe with a __not__ overload (see PEP 335 or PEP 532), there’d be less confusion?

6 Likes

Not facetious. There is a difference between bools and ints in Python they have their own operators that work as expected on their own types but the crossover needs caution. The bitwise operator works on “all” bits of an int; the bool as int and int as bool only really works for one, least significant bit. You might need judicious use of a bit mask when mixing types/operators if the intcan have more bits.

If I want to calculate the same boolean logic on 2**10 bits at a time then I might want to use bitwise operators, but when I need the individual boolean bits I treat the resultant int as a (boolean), bit array via a moving one-bit mask.

There isn’t a difference and they don’t have their own operators. People’s mental model is that there is and that behavior is wrong, but it’s actually the other way around.

That mental model itself is what’s wrong, not the behavior

Here is a simple script demonstration with all of the unary and binary operators

import operator

unary_ops = [
    operator.not_, operator.truth, operator.abs,
    operator.index, operator.invert, operator.neg,
    operator.pos
]

binary_ops = [
    operator.lt, operator.le, operator.eq,
    operator.ne, operator.ge, operator.gt,
    operator.add, operator.and_, operator.floordiv,
    operator.lshift, operator.mod, operator.mul,
    operator.or_, operator.pow, operator.rshift,
    operator.sub, operator.truediv, operator.xor
]

falsy = (0, False)
truthy = (1, True)

for int_value, bool_value in [falsy, truthy]:
    for op in unary_ops:
        int_result = op(int_value)
        bool_result = op(bool_value)
        if not isinstance(bool_result, type(int_result)) or bool_result != int_result:
            print(f"{op}: {int_value!r} = {int_result!r} | {bool_value!r} = {bool_result!r}")


for lhs_int, lhs_bool in [falsy, truthy]:
    for rhs_int, rhs_bool in [falsy, truthy]:
        for op in binary_ops:
            try:
                int_result = op(lhs_int, rhs_int)
            except ZeroDivisionError:
                int_result = ZeroDivisionError
            for lhs, rhs in [(lhs_int, rhs_bool), (lhs_bool, rhs_int), (lhs_bool, rhs_bool)]:
                try:
                    result = op(lhs, rhs)
                except ZeroDivisionError:
                    result = ZeroDivisionError
                if not isinstance(result, type(int_result)) or result != int_result:
                    print(f"{op}: {lhs_int!r}, {rhs_int!r} = {int_result!r} | {lhs!r}, {rhs!r} = {result!r}")

If you run that script with Python 3.12+, the only thing that gets printed is the deprecation warning

1 Like

And here’s a script highlighting how the bitwise invertion of a val (with respect to a mask of bit_count bits), is unlikely to give a resultant bool interpretation of False; (It would need a call using a one bit mask)

# %%
def strict_bitwise_invert(val, bit_count):
    # Calculate the maximum value allowed for this bit-width
    mask = (1 << bit_count) - 1
    
    # 1. Refuse negative values
    if val < 0:
        print(f"Refused: Value {val} is negative (inappropriate for bit-flip context).")
        return

    # 2. Refuse values larger than the bit-count (mask)
    if val > mask:
        print(f"Refused: Value {val} exceeds the {bit_count}-bit limit (max: {mask}).")
        return

    # 3. Perform the Bitwise Inversion
    # We use ~val, then mask the result to keep it within the bit_count
    inverted_full = ~val
    inverted_masked = inverted_full & mask
    
    print(f"--- {bit_count}-Bit Inversion Logic ---")
    print(f"Original: {val:4} | Bits: {val:0{bit_count}b} | Boolean: {bool(val)}")
    print(f"Inverted: {inverted_masked:4} | Bits: {inverted_masked:0{bit_count}b} | Boolean: {bool(inverted_masked)}")
    
    # Notice: In a multi-bit flip, bool(val) and bool(inverted_masked) 
    # are almost always both True because bits remain non-zero.
    print("-" * 35)

# Example: 5 (0101) becomes 10 (1010) in 4 bits
strict_bitwise_invert(5, 4)

# Example: 255 (11111111) in 12 bits
strict_bitwise_invert(255, 12)

# Example: 255 (11111111) in 4 bits (Refused)
strict_bitwise_invert(255, 4)

# Mask restricting to one bit gives bool inversions too.
strict_bitwise_invert(0, 1)
strict_bitwise_invert(1, 1)

Output:

--- 4-Bit Inversion Logic ---
Original:    5 | Bits: 0101 | Boolean: True
Inverted:   10 | Bits: 1010 | Boolean: True
-----------------------------------
--- 12-Bit Inversion Logic ---
Original:  255 | Bits: 000011111111 | Boolean: True
Inverted: 3840 | Bits: 111100000000 | Boolean: True
-----------------------------------
Refused: Value 255 exceeds the 4-bit limit (max: 15).
--- 1-Bit Inversion Logic ---
Original:    0 | Bits: 0 | Boolean: False
Inverted:    1 | Bits: 1 | Boolean: True
-----------------------------------
--- 1-Bit Inversion Logic ---
Original:    1 | Bits: 1 | Boolean: True
Inverted:    0 | Bits: 0 | Boolean: False
-----------------------------------

If you use & or | on bools, you get a bool result:

>>> True & True
True
>>> False | False
False

Only when you mix in an int do you get an int:

>>> True & 1
1
>>> False | 0
0

I can see no reason why using ~ with a bool shouldn’t return a bool and complete the set, other than we already have not, except for the fact that there’s no dunder method for not.

numpy uses ~, & and | for arrays of bools because there are no dunders for not, and or or.

1 Like

Could be useful for new API. But you won’t get rid of 20 years (in case of numpy) established use of the ~ pattern.

1 Like

That was discussed in depth in the original issue bool(~True) == True · Issue #82012 · python/cpython · GitHub. Overall, the behavior change was considered too risky. - The deprecation is already the safer version (prohibit and error out clearly). Every use case that now speaks against the deprecation would be much worse when changing the result.

There’s reasoning behind the LSP, and it’s to avoid questions like the following.

With ~bool throwing an error, in this code, where is the bug?

def f(x: bool):
   ...
   # Perhaps unusual in reality, but
   # expressions like g(a == b) do occur.
   g(x)
   ...

def g(x: int):
   ...
   ~x
   ...

f is passing an int as g asks for. (It is passing a bool, which is an int.) g is calling bitwise not on an int. Both of these operations are supposed to be OK.

These functions may actually be in different packages, of course.

I can see a few answers, only one of which attracts me.

  1. f. Functions should not pass bools where a ints are expected. The best choice for a new language, but a huge change to Python.
  2. g. Functions should ensure that ints they don’t control are not bools before they use bitwise not on them. Bitwise not is pretty rare, but this still seems rather annoying.
  3. g. g should have documented that x must not be a bool in this case. This leaks implementation details of g and makes applying ~ to ints that are passed in a breaking API change.
  4. f. f should not call g with a bool unless it’s checked it doesn’t ~ it. This is a sufficient burden to put on every bool passed as an int argument and every (even non‐breaking) library update that it’s better just to shove int(x) in there, making this more or less equivalent to 1.
  5. Doesn’t matter – this never comes up in practice. Functions like g certainly come up in practice, I’d imagine as a pretty high proportion of all functions using bitwise not, so this doesn’t really rid us of the question. If the potential bug isn’t g’s fault, then it implicitly blames potential future fs (which it assumes don’t exist).
  6. Both functions are OK, and the bug is in bool. :wink:
1 Like

You are missing the point here: bools do already behave like the corresponding int. There is absolutely no reason to convert them to ints first or do any checks:

>>> ~False, ~int(False)
(-1, -1)
>>> ~True, ~int(True)
(-2, -2)
1 Like

The reason why is that there is no set to complete

In Python, a bool is an int. That is clearly specified in the language reference and is not an implementation specific detail.

There are two types of integers:

Integers (int)
These represent numbers in an unlimited range, subject to available (virtual) memory only. For the purpose of shift and mask operations, a binary representation is assumed, and negative numbers are represented in a variant of 2’s complement which gives the illusion of an infinite string of sign bits extending to the left.

Booleans (bool)
These represent the truth values False and True. The two objects representing the values False and True are the only Boolean objects. The Boolean type is a subtype of the integer type, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings "False" or "True" are returned, respectively.

False is an integer with a value of 0 and True is an integer with a value of 1.

The & operator yields the bitwise AND of its arguments, which must be integers or one of them must be a custom object overriding __and__() or __rand__() special methods.

The | operator yields the bitwise (inclusive) OR of its arguments, which must be integers or one of them must be a custom object overriding or() or ror() special methods.

By those rules, True | False must return an integer with a value of 1 (which True is) and True & False must return an integer with a value of 0 (which False is).

The unary ~ (invert) operator yields the bitwise inversion of its integer argument. The bitwise inversion of x is defined as -(x+1).

~False must return an integer with a value of -1 and ~True must return an integer with a value of -2.

Anything other than that is a breaking change to the Python language itself.

4 Likes

What error?
If you gave an example that produced an error, we could better reason about it.Currently I think that remembering that:
~ works on, and gives easily relatable answers in the int domain, and not works on, and gives easily interpretable answers in purely bool domains, and that the mixture of domains with these operators needs further thought
That should be enough to find the issue. A function needing an int, like your g, if used with a bool should be expected to work correctly with zero or one. If there were a function h(...) -> bool: that returned an integer other than zero or one, then that should be treated as an error - it returns an int not the bool stated.