Find root cause of flaky test by tracing

Python Help
1

I have a flaky test which failure is not reproducible for me.

def test_bar():
    ...
    foo = bar(blue)
    assert foo == 'sunshine'

Up to now I only see a meaningless error message of the test fails.

like moonlight != sunshine

I would like to trace bar() every time it is called in the test.

If the assertion fails, then I want to see the trace as text.

The trace should contain all lines which get executed by bar() (excluding Python stdlib).

Maybe something like this:

def test_bar():
    ...
    tracer = SomeMagicTracer()
    foo = tracer.call(bar, blue)
    assert foo == 'sunshine', tracer.trace_to_text()

How to get SomeMagicTracer and trace_to_text()?

1 Like
2

If you want to see an error message when an assertion fails, you can
provide one:

>>> assert 1 == 2  # no error message
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

>>> assert 1 == 2, "1 isn't equal to 2"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: 1 isn't equal to 2

The error message doesn’t have to be a string. It can be any expression,
which is only evaluated if the assertion fails.

Remember that assertions are only enabled in Python’s debug mode, so
they can be disabled if you run under -O.

1 Like
3

Sorry, I was not explicit enough. I updated the question:

If the assertion fails, then I want to see the trace as text.
The trace should contain all lines which get executed by bar() (excluding Python stdlib).

1 Like
4

Ronny Pfannschmidt gave me a hint to sys.settrace

Here: pytest discussions

1 Like
5

I found a nice solution at Stackoverflow: unit testing - Hunt for root cause of flaky test in Python - Stack Overflow