`unittest`: Add `addCleanup` to `subTest`?

Ideas
1

unittest’s addCleanup schedules a function to run at the end of a test, so it doesn’t work for a subTest that needs to clean up after itself, before another subtest runs.

Parametrizing tests using subTest involves two levels of indentation; cleanup using try/finally would add another. Also, finally won’t mark its contents as clean-up (i.e. use testPartExecutor), unless it calls doCleanups (which runs all of the tests cleanups and so doesn’t work well if there are multiple subTest blocks, or code outside subTest).

It’s too late to change the scope of existing cleanups, but, would it make sense to add a addCleanup method to the subTest context, like this?

for param in 'a', 'b', 'c':
    with self.subTest() as sub:
        tempfile = make_tempfile()
        sub.addCleanup(os.unlink, tempfile)
        do_actual_test(tempfile, param)

If so, adding enterContext and doCleanups would also make sense.

2 Likes
2

Makes sense to me. +1

This looks like a typo, which probably should’ve been:

        sub.addCleanup(os.unlink, tempfile)
2 Likes
3

Looks reasonably. Don’t forget about sub.enterContext().

It is not directly related, but I thought about adding a decorator, so that:

def test_something(self):
    for param in 'a', 'b', 'c':
        with self.subTest(param=param) as sub:
            ...

could be written as

@unittes.subTests('param', 'a', 'b', 'c')
def test_something(self, param):
    ...

This would save two levels of indentation. More if you have several parameters.

I wrote a code long time ago, but not published it, because I was not sure that such minor limited feature would be useful. It has limitations:

  • setUp()/tearDown() are called only once, for all parameters.
  • It does not allow to parametrise the whole class.

This is why I named it subTests instead of more general parametrize.

Your feature allows to mititigate the first problem. But we need to pass also sub in addition to self and parameters in the wrapped method. This complicates use of multiple subTests decorators. Or make it a full proxy (with all assert* methods etc) and pass it instead of self. Alternatively, we can add addSubTestCleanup() in TestCase.

4

This is like pytest.parametrize, which we actually already have: cpython/Lib/test/test_zipfile/_path/_test_params.py at 319acf3d6cb05f5429422fda5c15bbd64fe9bd28 · python/cpython · GitHub. Maybe we should have a more reliable parametrize decorator?

2 Likes
5

pytest.parametrize is one of the best parts of pytest, it would be great to have a matching unittest.parametrize.

1 Like
6

You can do it with one level of indentation with ExitStack:

for param in 'a', 'b', 'c':
    with self.subTest() as sub, ExitStack() as stack:
        tempfile = make_tempfile()
        stack.callback(os.unlink, tempfile)
        do_actual_test(tempfile, param)

So it’s not too bad

7

If this gets added I’d also like to see an async version for IsolatedAsyncIOTestCase

async with self.subTest() as sub:
    sub.enterAsyncContext(asyncio.timeout(1))
    ...
1 Like
8

OK, I filed an issue for addCleanup, enterContext & doCleanups. These should be quite straightforward.

A parametrization decorator is a slightly bigger can of worms, e.g. what exactly happens when you apply more of them? Should they be based on subTest or generate new test cases? Can you parametrize dynamically, with a generator?
IMO, that could be prototyped in a third-party project, (or if you need it for CPython’s tests, in CPython-specific test.support), where it’s OK to start with a incomplete implementation or dead-end design.

You can, but then it’s treated as part of the test, not as a cleanup.

2 Likes
9

I think we shoud just do exactly what pytest does. There is a good reason why people use pytest and there is a good reason why pytest.parametrize works the way it works. So I think it should really work similarly, or at least, have similar semantics.

IIRC, pytest.parametrize() creates new functions; this also means that every setUp/tearDown logic is duplicated, so we should be aware of side-effects.

So, we should create new methods. OTOH, we could also make the choice of having a subtest() method which simply creates subtests instead of new tests.

what exactly happens when you apply more of them

It becomes a product, namely

@parametrize('a', [1,2,3])
@parametrize('b', [4,5,6])
def test_foo(self, a, b)

should be equivalent to call test_foo with all possible (a, b) pairs.

Can you parametrize dynamically, with a generator?

Well, you could dynamically parametrize but this means that the generator must be consumed when executing the class body:

class A:
    @parametrize('a', iter('1234'))
    def test_a(self, a): ...

should be valid. I don’t really see what you meant by “dynamically” otherwise. Are you suggesting that a test function can be executed and more cases would be dynamically added to it? where would you call this parametrization method? something like:

def test_foo(self):
    for param in self.parametrize_more(iter([1,2,3,4])):
        do_some_test_with_param(...)

I had a similar idea where doing:

def test_foo(self):
    for a, b in self.parametrize_more("what", a=[1,2,3], b=[4,5,6]):
        do_some_test_with_param(a, b)

would be equivalent to:

def test_foo(self):
    for a, b in product([1,2,3], [4,5,6]):
        with self.subTest("what", a=a, b=b):
            do_some_test_with_param(a, b)

The product-approach usually saves a lot of indentation at the cost of some readbility.

1 Like
10

Well… the mechanism pytest uses for setup/teardown “contexts” is fixtures, and I don’t think those, with the name-based lookups, should go in unittest.

As in, separate attributes of the TestCase class? Sounds like we’re leaving decorator-land…

11

Arf, right, I missed the fact that unittest is 1-pass and not multi-pass like pytest. Ok, dynamic test creation may be a bad idea. WDYT about a helper that does product+subTests? like Serhiy/mine’s suggestion?

Definitely not, so maybe, to accomate for future improvements, let’s not name it parametrize. OTOH, we could solve the lack of fixture as follows:

Given:

@parametrize('a', [1,2,3])
@parametrize('b', [4,5,6])
def test_foo(self, a, b):
      bar(a, b)
  • We could generate methods that are decorated with @parametrize based on the fact that @parametrize adds an additional attribute to the test function.
  • In TestCase.__init_subclass_, we would then generate methods that are known to be decorated with @parmetrize.
  • The generated methods will actually be partializations of the decorated function with bound arguments.

While I don’t know exactly how it would be implemented, I can more or less assume that it could be quite slow (__init_subclass__ + parsing of the entire class hierarchy + handling overridden methods etc) and quite complex, so I wouldn’t recommend this approach as a first iteration and would first add some helper for product+subTests which are the most common cases in the test suite IMO.

12

Depends on what self is, and when the cleanups added by addCleanup are called. I don’t see a straightforward answer here…
I guess a addSubTestCleanup method would be keeping with the existing unittest style, which duplicates these things for the different scopes (module, class, method). That doesn’t handle all the use cases for nested subtests, but that’s consistent with the “product” approach which doesn’t let you run code outside the “inner” subTest.

13

This is a great idea! I opened #135120.

It contains also an optional feature to call doCleanups(), because your implementation in test_ntpath needs it. This is a temporary feature, until you implement subtest-level cleanup.

While working on this, a thought came to my mind, that we can implement a general fencing mechanism for cleanups. When you leave a fence, all cleanups added in a fence are called in the LILO order.

It can be used as a context manager:

def test_foo(self):
    for filename in filename():
        with self.fence():
            f = open(filename, 'w')
            self.addCleanup(os.unlink, filename)
            self.addCleanup(f.close)
            ...

or as a decorator in combination with test generating decorators:

@subTests('filename', get_filenames())
@unittest.fence
def test_foo(self, filename):
    f = open(filename, 'w')
    self.addCleanup(os.unlink, filename)
    self.addCleanup(f.close)
    ...

doCleanups() will only call cleanups in the current fence, so there will be benefit to use the decorator even without @subTests – it leave cleanups added in setUp to be called after tearDown. It may help to solve other issues like unittest: tearDown is not called if asyncSetUp fails · Issue #96139 · python/cpython · GitHub.

It’s disadvantage is that when used as a context manager, it adds an indenation level (although you can combine it with subTest() in the same with).

So, there are three alternative solutions of the same problem:

  • Add addCleanup() etc for subtest objects.
  • Add addSubTestCleanup() etc for TestCase.
  • Implement fencing.

They have their own advantages and disadvantages. Their applicability partially overlaps but does not coincide.

1 Like
14

Is there any situation where you would want fence without subTest, or subTest without fence?

It seems that subTest could open a fence automatically, but so could the higher-level contexts (module, class, method). And a unittest.addCleanup function (not method) could add a cleanup to the current fence. Then the user could call addCleanup right after any kind of initialization code.