Replace non-evaluable expressions by a placeholder in string formatting

Python Help
1

Consider this statement inside an error handling code:

print(f"function failed, DEBUG INFO: X={expr1}, Y={expr2}, Z={expr3}"

If the evaluation of any expression inside the f-string fails, the whole print fails and all debug info is lost. In an error handling code there is plenty of reasons for that, because the data might be incomplete or in an inconsistent state due to the error being handled.

The idea is to allow for a “best effort evaluation” and replace failed expressions with a placeholder. I leave open how the placeholder should look like, that’s a secondary question now.

In the case this proposal wouId find supporters, I think the Python developers should specify how to achieve this. Just to start the discussion, here is an example of a new conversion specifier 'e' (in the case of an error / exception) that could be combined with existing 'a', 'r' and 's'.

foo = None
f"state={foo.state}"     # attribute error!
f"state={foo.state!e}"   # "state=<N/A>"
2

“Best effort evaluation” sounds like PHP’s style, not Python’s. Can you give a realistic example of where you need this sort of thing?

3

You can do something like this using the __format__ method on a class that makes an effort to eval:

class Attempt:
     def __format__(self, evalstr):
        try:
            return str(eval(evalstr))
        except:
            return "<N/A>"

class A: ...

attempt = Attempt()
foo = A()

print(f"{attempt:1+2}")  # "3"
print(f"{attempt:foo.state}")  # "<N/A>"

foo.state = "valid"
print(f"{attempt:foo.state}")  # "valid"

Of course, this is only readable to someone who understands what attempt is. If this is just for attribute lookup, you could demand less of the reader by using the builtin getattr:

f"state={getattr(foo, 'state', '<N/A>')}"

This says clearly that the attribute may not exist, and if it doesn’t, <N/A> will be printed in place of its value.

1 Like
4

  1. In my understanding “best effort” means if the code asks for a printable string, the Python will do its best to produce a printable string - either the correct one (if possible) or the placeholder.

  2. The example in my original post is unfortunately too realistic, at least for me.

5

Yes, and in my understanding, that’s a really really bad idea. Leads to far worse problems.

You mean trying to get an attribute off something that might not have it? You’ve been given getattr as a good way of doing that. Your “expr1”, “expr2”, “expr3” one isn’t realistic so I assume you must mean “foo.state”.

Remember, too, that you can always do something like “foo and foo.state” if you’re expecting the possibility of None.

6

This is not about a missing attribute, that was an example of a conversion specifier. I could also wrote:

f"freq = {1/t}" # when t was not computed yet or is 0 by mistake

Let me repeat, in the case of an error, debug data are valuable. It’s bad to lose them all just because some part of it was not initialized or computed in the moment the error has occured.

Yes, and in my understanding, that’s a really really bad idea. Leads to far worse problems.

What problems do you mean? Could you please give an example.

Its an explicit instruction from the programmer. A shortcut for:

try:
    freq = 1/t
except Exception:
    freq = "<N/A>"
print(f"{freq=})

It saves four extra lines for every expression when trying to make a log/print statement fail-safe.

7

Yes, and if you do, you’ll get an exception, which can itself be logged.

Absorb ANY exception and just print <N/A>? This seems like a really good way to lose useful information.

Once again, can you show a real-world example please? How often does this happen, how frequently do you really need to use print (rather than a dedicated logging subsystem), how many times do you need multiple of these expressions, and do you actually need to have multiple expressions that can succeed or fail independently? I can’t tell you what I would do in this situation because your examples are still entirely artificial. Yes, they’re vaguely plausible, but that’s not enough to make concrete recommendations.

1 Like
8

You shouldn’t really be writing debug data so that everything is lost of there’s a problem with one of the values you want to display. Error handling is hard, and yes, it’s nice to make it as easy as possible, but at some point you have to be responsible for thinking about what could go wrong and being prepared for it.

In your example

you’ll get a NameError or a ZeroDivisionError with a traceback that points you at the problem. That doesn’t sound like you’ve “lost all your debug data” - rather the opposite, actually, you have exactly what you want. Or were you hoping to debug a different problem? Fix one problem at a time - fix this one, then reproduce the issue and fix the other problem. If you can’t reproduce the issue, you’re always going to have problems, no matter how much debug data you write (you’ll always, in my experience, want the precise thing you forgot to log…)

2 Likes
9

While a traceback is helpful, sometimes it does not point to the problem.

try:
    # some long non-trivial code controlling
    # an external system while maintaining a complex state
    # dealing with not 100% reliable sensors and not very well
    # documented hardware
except Exception as err:
    # we encountered an unexpected situation, let's log as many
    # values as possible for further analysis
10

If you catch the exception and then have an error in your exception handler, fix that first (as I said).

But also as I said, I concede that error handling is hard. I don’t think this particular issue is what makes it difficult, though.

11

A log with 10 values of which 3 are “<N/A>” is much better than just an exception saying that a log attempt of 10 values failed, isn’t it?

12

A log with 10 values might be better written as 10 logs of one value each, if there’s any risk of the value calculation failing.

13

IMO it is not an error in the handler, it is a problem with the data. An unknown problem.

To succesfully print everyhing helpful in a human readable form either a LOTS OF try-except or if-else constructs would be required cluttering the error handling code or a simple feature I dared to propose today would do the same in one line.

1 Like
14

Sure would it work, but it is 40 lines versus 1 line, albeit a longer one.

15

No, no it is not. The log attempt would fail with an exception. Not just with the text “<N/A>” which tells you nothing about WHY it failed.

When you’re working with unknown/unexpected failures, get all the information. Throwing that away in favour of a generic “<N/A>” is about the worst thing I can think of for a debug message.

No actually, that’s not true; a worse thing is when attempting to log a non-scalar value segfaults the entire program. That is definitely worse, and extremely annoying to debug. But bland non-information isn’t far behind.

16

At least for numeric errors, it might be better to use numpy, which already has mechanisms for specifying ‘ignore’, ‘warn’, ‘raise’, ‘call’, ‘print’, ‘log’ for different kinds of error: numpy.seterr — NumPy v1.25 Manual

And if you’re tabulating data, consider pandas:

In [1]: import pandas as pd
In [2]: import numpy as np
In [6]: df = pd.DataFrame(dict(vals=np.arange(-5, 5)))

In [7]: df.apply(np.log10)
Out[7]: 
       vals
0       NaN
1       NaN
2       NaN
3       NaN
4       NaN
5      -inf
6  0.000000
7  0.301030
8  0.477121
9  0.602060
17

You’re aware, I hope, that it isn’t necessary to have a newline after each print output, that it can be suppressed using the end keyword argument?

18

Yes, I know. I’m sorry if it is still not clear, but I really don’t care about why it failed, I simply don’t want that failure to affect the logging of other data. If you don’t have an use-case for it, that’s fine. But I do have. In my work there were situations when I would really appreciate the proposed feature. It does not change any existing code or workflows, you don’t have to use this option it when it does not suit your needs. As I have shown, It can be achieved by standard Python, but at the cost of many additional lines violating the DRY principle.

19

It does change existing code if it changes what should be an exception into something that keeps running. Especially if you are applying this to any string, not just in logging statements [1].

You are proposing a fairly major change to the entire language as an alternative to you debugging your code differently.

  1. given that you’re using print instead of logging, this must be the case ↩︎

2 Likes
20

I use logs to debug large and complex async code all the time.
It is my day-job to do this and I add debug logs all the time.

I have never hit the issue you want to fix.
Usually, always(?), it is obvious what variables are in scope and defined.
For the case of attributes of an object that may not be set yet getattr() works.