Deprecate str % format operator

Ideas
61

operator.rmod(a, b) might be acceptable.

1 Like
62

That is the entire reason. If %-formatting were absent from Python today, we would have no reason to add it to the language. However, since it has been present from the outset and tons of code depends on it, removing it would simply cause massive pain to users with no offsetting benefit.

Is there another discussion forum where this has already been hashed out?

There was a python-dev discussion on this as a subtopic for the Python 3 migration. IIRC, the decisive point what that 2-to-3 could not make an automated transformation of code, so users would have to convert everything manually. There were some other points such as %-formatting being friendly to people coming from other languages that use %-formatting, its heavy use in logging and templating, and the convenience of having two styles available during multistep transformations. In short, deprecation was a recipe for unnecessary user pain.

5 Likes
63

3. Data model — Python 3.13.3 documentation is my reference for how operators work in Python.

But even that page doesn’t tell you exactly what happens when x % y is evaluated, it just describes the two kinds of numeric-operation methods.

So my understanding of current behavior is:

  1. if x.__mod__ exists, call it, and if it doesn’t return NotImplemented, we’re done
  2. if y.__rmod__ exists, call it, and if it doesn’t return NotImplemented, we’re done
  3. raise TypeError

The way I see the decorator for __rmod__ working is: the decorator just sets some attribute (let’s call it __takes_precedence__) on the function to mark it as taking precedence. Then, the evaluation changes to:

  1. if y.__rmod__ exists and y.__rmod__.__takes_precedence__ exists, call y.__rmod__, and if it doesn’t return NotImplemented, we’re done
  2. if x.__mod__ exists, call it, and if it doesn’t return NotImplemented, we’re done
  3. if y.__rmod__ exists, call it, and if it doesn’t return NotImplemented, we’re done
  4. raise TypeError
64

Ehh, ok.

  1. That check would happen on all x % y. Why should everyone pay for this edge case?
  2. Why old code should change behavior? For example, if x is str literal, and y is an input.
  3. Why discourage new code from using an operator? The luck of knowledge will force some of the new code to prefer directly calling x.__mod__(y), just to be safe.
  4. What if type(x) want’s to be stronger? Should __mod__ also accept this decorator? If yes, U can follow this logic to infinity.
1 Like
65

Yeah, that’s what I realised after mulling it over more. It would easily provide for the use case here, but wouldn’t likely be accepted for these exact reasons.

Not sure what you mean by this one, though.

Maybe a better solution would be to modify str.__mod__(self, other) slightly?

Currently, it will raise TypeError if the combination of self and other isn’t suitable:

>>> 'a' % 5
TypeError: not all arguments converted during string formatting
>>> '%d%d' % 5
TypeError: not enough arguments for format string
>>> class A:
...  pass
... 
>>> '%d' % A()
TypeError: %d format: a real number is required, not A

The implementation of str.__mod__ could, after determining that it should raise TypeError but before actually raising it, check if other.__rmod__ exists, and if so, call it; if it raises anything then discard the new exception and raise the original TypeError; if it returns NotImplemented then again raise the original TypeError, but otherwise, return the result of other.__rmod__.

(It’d be simpler for str.__mod__ to return NotImplemented itself, but then if other.__rmod__ didn’t exist or wasn’t suitable, only a generic TypeError would be produced, without the detail shown above that currently exists.)

That would allow for this use case while not imposing a performance penalty on anyone. It’s still a rather specific solution, though, and it’d have to be documented as such, so I imagine this wouldn’t quite meet the bar either…

Come to think of it, it’s kind of a shame that we can’t just monkey-patch str.__mod__ ourselves:

TypeError: cannot set '__mod__' attribute of immutable type 'str'

If we could do that, we could customise the functionality as needed.

1 Like
66

There’s a bit more to it than this. In particular one part you missed is that there is a special case that checks if the rhs is a strict subclass of the lhs. That provides one way to make it work if you must:

class A(str):
    def __rmod__(self, lhs):
        if isinstance(lhs, str):
            return 1j
        else:
            return NotImplemented

Then:

>>> 'asd' % A()
1j
4 Likes
67

I feel I vaguely recall reading about that special case long ago, but I couldn’t tell you where it’s documented…

Knowing that we already do have a special-case check (which must presumably come first) certainly weakens the argument of “we can’t add this because it’d add a check that almost never happens and it’d be bad for performance”.

I still dislike my own proposal on the grounds that it’s “ugly” that you only get one additional level of precedence. I haven’t been able to think up a better solution, though.

68

That check happens any time two objects interact with ANY binary operator. If one of them is a subclass of the other, the subclass gets to choose how they interact.

69

There are problems with the dynamic dispatch that goes in on binary operators. It is not worth changing anything for this particular issue though.

Probably it was a mistake to define % for strings rather than using a method. There are also problems with the way it works by special casing tuple and dict.

70

I found it in the already linked Data Model section; it’s there as a note (gray background).

1 Like
71

I feel like the only problem with them is that, people have different mental model of operators. It’s not really natural that symmetric operators are asymmetric. Or that they are polymorphic, and that the left side type is more important than the right. Maybe right side importance support is a good idea?

72

Right side dispatch does usually work as noted in the OP. The issue is that str.__mod__ is greedy about types. It is basically:

class str:
    def __mod__(self, other):
        if isinstance(other, tuple):
            return self.sprintf_tuple(other)
        elif isinstance(other, dict):
            return self.sprintf_dict(other)
        else:
            return self.sprintf_one(other)

Since it never returns NotImplemented it doesn’t allow the rhs to overload.

The way cooperative dispatch works in Python is supposed to be that you have a type U that is upstream and a type D that is downstream. Then U should only handle types that it knows about and otherwise return NotImplemented. Then D can overload the operator from the rhs. Here str is not being cooperative because it consumes any type.

This dispatch mostly works for simple cases but falls down when it is not clear which type is the “upstream” type. If you add a gmpy2.mpz with an int then it is clear that mpz is downstream. If you add a flint.fmpz with int then fmpz is downstream. If you add an mpz to an fmpz then it isn’t clear which type is downstream. So should the result be mpz or fmpz? In practice you will get an incoherent mixture of results when you mix up the types from libraries that don’t explicitly depend on each other and there is no clear way to resolve it so that all things work nicely together.

2 Likes
73

No. The biggest problem is that in the presence of subclasses, there is no natural ordering of overloads, which makes precedence extremely difficult to even define, much less implement.

Given op(p1, p2) where p1 is of type T1 and p2 is of type T2, suppose T1 has a base class B1 and T2 has a base class B2. And suppose we have definitions for op(B1, T2) and op(T1, B2). Both definitions match the call op(p1, p2). Which should we use? You’ve said that it’s not natural for the operator to be asymmetric, and a rule that prefers either is “not natural” on that basis. The only remaining option is to reject the call, saying there’s no valid overload - that is not likely to be acceptable either.

Basically, every other problem of multiple dispatch comes down to people not being able to agree on how to handle this fundamental choice.

5 Likes
74

This is an example of what I meant about not knowing which type is the “upstream” type. The resolution would have to be multiple dispatch e.g. this is the multipledispatch library:

from multipledispatch import dispatch

class T1: pass
class T2: pass

class B1(T1): pass
class B2(T2): pass

@dispatch(B1, T2)
def add(a, b):
    return 'B1, T2'

@dispatch(T1, B2)
def add(a, b):
    return 'T1, B2'

print(add(B1(), B2()))

If you run that then you get:

$ python t.py
/.../.venv/lib/python3.13/site-packages/multipledispatch/dispatcher.py:27: AmbiguityWarning:
Ambiguities exist in dispatched function add

The following signatures may result in ambiguous behavior:
	[B1, T2], [T1, B2]

Consider making the following additions:

@dispatch(B1, B2)
def add(...)
  warn(warning_text(dispatcher.name, ambiguities), AmbiguityWarning)
T1, B2

You get warned about the ambiguity and because it is multiple dispatch rather than methods on individual types you have the opportunity to add the missing overload that would resolve the ambiguity. No type needs to be upstream or downstream from another and anyone downstream can add whatever overloads they like as well without modifying the upstream classes.

It would be nice to have multiple dispatch in general in Python but I don’t think it is going to happen because it would be even slower than cooperative dispatch and it can’t work with static types like list[int] which makes it considerably less useful. It is very useful in Julia but there the overloads are resolved statically at “compile time” which makes it all work better.

2 Likes
75

Why is that true?

I think it can work with such types if you give a TypeIs function instead of just a type. This is exactly what the sequential approach would currently do.

76

This doesn’t have much to do with the string "% operator, really. The binary operator methods are inherently asymmetric with respect to types:

These functions (the __ r"op" __ ones) are only called if the left operand does not support the corresponding operation [3] and the operands are of different types. [4] For instance, to evaluate the expression x - y , where y is an instance of a class that has an __rsub__()method, type(y).__rsub__(y, x) is called if type(x).__sub__(x, y) returns NotImplemented.

So a class MyClass can’t be guaranteed that

“unknown type” % MyClass

will ever call a MyClass method.

1 Like
77

Just ou to f curiosity - you are aware there are billions of lines of code in projects using these- right? You expect everyone to just go changing those?

As for " n. It’s a math operator." from my POV here, it is you that seems to be trying to use str objects to do Math. Maybe you could just think around it - one idea could Just import whatever fancy object you need to express as a string and do math on as a single-letter name:

from mylib import SuperMathClass as V

V("lit") % x

OTOH - maybe a viable, non-breaking change to Python could be possible here, which would allow one to override % to use with string literals:

Instead of asking for a breaking change for a feature used in millions of LoCs, which is still the preferred way to interpolate a string, what would be possible is: if a class str where allowed to return NotImplemented, currently we get a runtime TypeError - we could change the string implementation of __mod__ to catch this TypeError, check if it was due to NotImplemented being returned, and then call __rmod__ on the other object.

This wouldn’t break any existing code, but making the objects that do implement that a bit awkward to use (as they won’t be able to be directly converted to __str__ in other contexts as well) - but this is work-aroundable, if needed. (Maybe just making print , f and r strings, str.format aware of NotImplemented and defaulting to __repr__ in the same change could overcome this - still no code would be broken - the changes are just so that NEW classes implementing an __rmod__ intended to work with strings can be printed)

(BTW, by proceeding with the deprecation idea, you’d still have to wait until the % formatting get actually turned off by default in any Python versions you’d intend to run your code - you can give or take a 10-12 years timeframe for that)

78

How did you determine that?

79

I intended to mean, the preferred way in those millions of lines.
Ayway, that is far from the mai point in this post.

I had not read the whole thread - and I saw later that what you intend is actually possible if the class implementing the special operations just inherits from str itself, and them __rmod__ will just work. So, I don’t think any language change is due at all, not even the non-breaking change I’ve suggested.

1 Like
80

Yes, this is basically how it should be done, semantically…
Then when it comes to having a code running some specifical algebra extensively, you would get these instanciations everywhere, with a lot of parentheses. Of course you can propagate your wrappers (here V) by overriding the operators (which is what OP done, and realized it was not possible for % with strings, also it is not possible for the and operator generally, because of short-circuit behavior), but this impure coding and somewhat hacky. At the end the real clean way of doing this would be semantically equivalent to meta-programming and some ideas are materialized here : https://discuss.python.org/t/dsl-operator-a-different-approach-to-dsls/

1 Like