Unittest module puzzle..... two unit tests pass when run *separately* but not *together*?!

Python Help
1

I have two unit tests in a file called tests.py. If I run them one at a time
then they pass. If I try to run both at once, the same one always fails!?

Why would the behavior be different? I assumed maybe I was accidently
passing some state between unit tests but I can’t find it.

Here is the proof both tests passes one at a time…

% ./tests.py Tester.test_min_py_grammar
.

Ran 1 test in 1.153s

OK

% ./tests.py Tester.test_min_c_grammar
.

Ran 1 test in 0.190s

OK

But now look if I try to do both…

% ./tests.py
.F

FAIL: test_min_py_grammar (main.Tester)

Traceback (most recent call last):
File “/home/cs/Ws/Wb/foo/./tests.py”, line 915, in test_min_py_grammar
self.assertEqual(output, answer)
AssertionError: Tuples differ: (‘pro[203 chars]exp_prefix’, (‘exp_inv_ind’, (‘exp_base’, ('VA[412 chars]))))) != (‘pro[203 chars]exp_pdbn’, (‘exp_pow’, (‘exp_inv_elems’, ('exp[425 chars])))))

First differing element 1:
(‘sta[191 chars]exp_prefix’, (‘exp_inv_ind’, (‘exp_base’, (‘VA[411 chars]’))))
(‘sta[191 chars]exp_pdbn’, (‘exp_pow’, (‘exp_inv_elems’, (‘exp[424 chars]’))))

Diff is 1390 characters long. Set self.maxDiff to None to see it.

Ran 2 tests in 0.210s

FAILED (failures=1)

Each tests builds a parser for a tiny toy language using parser_gen.py…in one case for a C like language. In another for a Pythonic language. The beginning of each unit test makes the parser and a related function called p_and_tok. Can I define different versions in different unit tests?

Here is a link to my tests.py file…

Here is a toy repo with all the code…

2

At a glance, your cache decorator doesn’t consider whether parser_ changed, so you likely have bad cache sharing between calls with different parser_s.

3

In any such case, one test is changing something somewhere. Unittest recognizes a setUp method (check the manual) to properly initialize the ‘something’. Or add a tearDown method, or do it in the methods.

4

Wow that was brilliant! I suspected some state was sneaking in but I couldn’t find it.
The cache didn’t occur to me but that totally makes sense.

Reloading the relevant modules fixed the problem.

Thanks so much! As I told Terry, I can’t believe how fast this forum caught this one.
I thought it would take a long time.

5

Yes thank you. I love setUp and tearDown. They avoid having
to copy boilerplate to every unit test. Reloading the relevant modules
in setUp gave every unit test cleaned state and solved all.

Thanks so much. I can’t believe how fast this forum caught this one.
I thought it would take a long time.

cs

6

If you switch to pytest, fixtures are a more modern way of doing this.

7

Interesting. I’ve also heard of nose in addition to pytest. The only reason I can see to ever sticking with unittest is because it’s in the standard library.

1 Like
8

You wrote a clear title presenting a problem that many of us have had.

9

nose is no longer developed - it’s last release is almost 10 years old.

pytest is the de-facto standard for unit tests in python. Basically all major packages use that. It’s much more capable and convenient compared to the standard library’s unittest. Unless you have very strong resticitions to not have any additional dev dependencies, go with pytest.

1 Like
10

Thanks :+1::folded_hands: pytest looks great.