Unittest: ignore test cases that are also ABC

Ideas
1

I often had to use mixins to group test methods that can run on different implementations of the same interface, for example:

class BackendTestsMixin:
    backend: BackendInterface
    def test_read(self):
       self.assertEqual(self.backend.read(), "42")

class MemoryBackendTests(BackendTestsMixin, TestCase):
    def setUp(self):
        self.backend = MemoryBackend()

class DiskBackendTests(BackendTestsMixin, TestCase):
    def setUp(self):
        self.backend = DiskBackend()

Now, mypy rightfully complains on the example above because assertEqual is not a member of the class; however, if I make BackendTestsMixin a subclass of TestCase, unittest will try to run its test methods.

This seems to be a common pattern/problem (see various workarounds proposed at Python unit test with base and sub class - Stack Overflow ).

A rather elegant solution could be to change loadTestsFromModule from:

              if isinstance(obj, type) and issubclass(obj, case.TestCase):

to:

              if isinstance(obj, type) and issubclass(obj, case.TestCase) and not abc.ABC in obj.__bases__:

That would follow the common pattern intentions beautifully:

class BackendTestsMixin(TestCase, abc.ABC):
    backend: BackendInterface

    @abc.abstractmethod
    def make_backend(self) -> Backend:
        ...

    def setUp(self):
        self.backend = self.make_backend()

    def test_read(self):
       self.assertEqual(self.backend.read(), "42")
2

It strikes me as overkill to run mypy on test code, but there could be good reason.

There’s an argument as well, when it comes to simple test code, to not worry about fancy design patterns, nor about making things as DRY as possible. Some kind of duplication is the whole point of the exercise - to independently verify an implementation, and keep the two separate. I.e. minimise, and ideally remove, the dependencies of the test code, on the code under test.

Anyway, an alternative methodology instead of mix-ins, is to use factory functions to generate the test methods. I use this a lot for parametric testing schemes, that work with both unittest and pytest, without requiring pytests’s decorator.

3

This can be solved be restructuring your classes as follows:

class MemoryBackendTestsMixin(TestCase):
    def setUp(self):
        self.backend = MemoryBackend()

class DiskBackendTestsMixin(TestCase):
    def setUp(self):
        self.backend = DiskBackend()

class BackendTestsMixin:
    backend: BackendInterface
    def test_read(self):
       self.assertEqual(self.backend.read(), "42")

class MemoryBackendTests(BackendTestsMixin, MemoryBackendTestsMixin):
    pass

class DiskBackendTests(BackendTestsMixin, DiskBackendTestsMixin):
    pass

In fact, this is how tests for many CPython modules with both C and Python implementations are set up:

1 Like
4

I do not see significant differences between the CPython example and the one I sketched, and unfortunately BackendTestsMixin.test_read would fail type checking in your example the same as it fails in mine

5

There’re lots of ways to do it with existing tools. The essence in Pytest terminology, is running the same test twice using two different fixtures.

E.g. if you want to keep the mix-ins, TestSuite.addTest could be used with a class factory:

def make_backends_test_class(Mixin: BackendInterface):
    class MyTestCase(TestCase, Mixin):
      def test_read(self):
         self.assertEqual(self.backend.read(), "42")
    return MyTestCase

backends_test_suite = unittest.TestSuite()

for Mixin in BackEndsMixins:
    backends_test_suite.AddTest(make_backends_test_class(Mixin))
6

If I understand your proposal correctly, hiding test classes inside a factory method would hide them from the unittest default loader and allow me to add only the classes that I need.

It is indeed, technically, a way to do it with existing tools. It would involve quite a lot of overhaul of how unit tests are usually written, though.

I didn’t mean to propose a way to do something that isn’t currently possible at all: I mean to propose a straightforward, intuitive way to do something that currently involves significant nontrivial runes.

7

I’m not sure how it works with Pytest, but if Unittest doesn’t automatically run instances of TestSuite already (kind of the point of them), it’s a simple matter of calling the TestSuite’s run method.

Even if none of these solutions are to your liking, that is scant justification to make a breaking change to Python, that will suddenly alter the behaviour of everyone else’s tests, by making loadTestsFromModule ignore Class(TestCase, ABC).

If you really want that it’s fine. Implement your own loader.

8

I think your problem is here:

If mypy is correct to flag this as an error, then you shouldn’t be writing your code like that. And any of the alternatives proposed are good ways of avoiding the error.

If, on the other hand, you consider your code to be correct, then mypy is incorrect to flag it - you can’t have it both ways. That’s not surprising - mypy by design doesn’t handle the more dynamic runtime behaviours of Python, as it’s a static type checker, and by choice it doesn’t (as far as I know) do global analysis so it may miss “long distance” interactions. So it may be that the mypy developers don’t consider this as a bug they’d be willing to fix. In which case, you have the # type: ignore directive which is explicitly designed to override mypy and tell it you know what you’re doing.

So there are lots of ways of handling this within the existing capabilities of Python and mypy. I don’t think this issue warrants making a special case change to unittest (especiallly as your proposed fix is incomplete - it doesn’t work if BackendTestsMixin is dynamically registered as an ABC via ABC.register).

1 Like
9

I don’t understand why you consider it a breaking change: ABC classes are not supposed to be instantiated, and trying to run Class(TestCase, ABC) in a test suite should already fail?

10

Oops I was indeed mistaken about the issue here.

Like @JamesParrott suggested, a class factory is indeed the better solution, but with the dynamically created classes returned and assigned to global names instead so that the default loader can find them at the module level.

A good example can be found in CPython’s test_abc.py:

11

I appreciate all your efforts trying to give me alternatives, and indeed I learnt a trick or two in the process.

I still do not see any attempt to evaluate the proposal, besides @pf_moore’s mention of ABC.register, which I find relevant.

I have no idea if there can be a general way to detect if a class is registered as abstract that would work well with ABC.register, though without that I still see value in using abc.ABC as an intuitive and effective marker for test cases not to be run.

Alternatively, nose seems to support a __test__ = False attribute to skip test cases, which could be incorprated into unittest for a more general solution, though that would be a different idea to address similar user stories

12

I would be happy to evaluate your proposal once it’s apparent that no existing solution can elegantly satisfy your needs.

It would help if you can explain why the solution demonstrated in test_abc.py doesn’t do that.

13
  • It requires an extra level of indentation
  • I don’t think it plays well with type checkers
  • It doesn’t play amazingly with IDE hints either
  • If you have tests you only want to perform on some of the situations, you need another class that potentially does the same setup again (e.g. if setting up DiskBackend is a decent amount of work), or your factory template classes get even more messy.

Clearly, the intend of OP is to use static typing features in their test files. Whether this is a good idea is something that might be worth discussing, but just suggesting pretty dynamic class creation patterns is missing the point entirely.

1 Like
14

You’re somewhat missing the point of the replies. One of the most important things needed with a proposal is an explanation of why it’s needed. That involves exploring what the problem is, and how or if it can be solved without changing the language. Remember, it’s you who needs to persuade us that your proposal is worthwhile.

To that end, I tried your example, and I couldn’t get the error you claimed. I needed to create dummy classes for BackendInterface, MemoryBackend and DiskBackend, but having done so, mypy didn’t give any errors. So the first thing you need to do is give an easier to reproduce example of the problem you’re trying to solve.

Then you need to be prepared to give a fair response to the various alternatives proposed. You’ve not responded to my suggestion of # type: ignore yet, and @JamesParrott suggested a class factory, which you dismissed with (essentially) the reponse that you don’t like that approach. But class factories or subtests are the standard way of writing parametrised tests in unittest, so simply dismissing those isn’t really a fair response.

For reference, using subtests would look something like this (warning: untested code):

backend_types: list[BackendInterface] = [MemoryBackend, DiskBackend]
class BackendTests(TestCase):
    def test_read(self):
        for backend_type in backend_types:
            with self.subTest(backend_type=backend_type)
                backend = backend_type()
                self.assertEqual(backend.read(), "42")
15

I’m curious if anyone has a reason why we shouldn’t change unittest to understand abc classes? Currently this example fails:

import abc
import unittest

class Mixin(unittest.TestCase, abc.ABC):
    @abc.abstractmethod
    def make_backend(self):
        ...

    def test_it(self):
        assert 1 == 2

unittest tries to instantiate the TestCase class, which it cannot do:

... etc ...
  File "/usr/local/pyenv/pyenv/versions/3.12.4/lib/python3.12/unittest/suite.py", line 24, in __init__
    self.addTests(tests)
  File "/usr/local/pyenv/pyenv/versions/3.12.4/lib/python3.12/unittest/suite.py", line 57, in addTests
    for test in tests:
TypeError: Can't instantiate abstract class Mixin without an implementation for abstract method 'make_backend'

Since the code as presented is now an error, wouldn’t it be better to update unittest to not attempt to instantiate classes that can’t be instantiated?

3 Likes
16

FWIW the following code passes both mypy and pyright just fine and produces good type hints in VSCode with PyLance:

import unittest

class BackendInterface:
    def read(self) -> str: ...

class MemoryBackend(BackendInterface):
    def read(self):
        return '42'

class DiskBackend(BackendInterface):
    def read(self):
        return 'fail'

def make_test(backend: type[BackendInterface]):
    class Test(unittest.TestCase):
        def setUp(self):
            self.backend = backend()

        def test_read(self):
             self.assertEqual(self.backend.read(), "42")
    return Test

MemoryBackendTest = make_test(MemoryBackend)
DiskBackendTest = make_test(DiskBackend)

unittest.main()

And I personally don’t see an extra level of indentation to be an issue.

Conditional test suites can be enclosed in if statements, and conditional tests can be decorated with the skipIf decorator. It’s a common practice.

17

That’s a good point (and if it’s what the OP was trying to suggest, I’m sorry I didn’t get that from the way they presented their argument). I agree that fixing this seems like a reasonable thing to do.

I’m still not 100% sure if it’s as trivial as checking for a base class of ABC, though. At a minimum, it would need to check the full MRO, and it probably also needs to look for the ABCMeta metaclass rather than the specific class ABC. And dynamic registration may be relevant as well, although I’ve not been able to consistently trigger the “Can’t instantiate” error in my tests, so I don’t fully understand the complexities there.

18

We can use inspect.isabstract for this purpose:

import inspect
from abc import ABC, abstractmethod

class A(ABC):
    @abstractmethod
    def foo(self): ...

class B(A): ...

class C(A):
    def foo(self): ...

print(inspect.isabstract(B)) # outputs True
print(inspect.isabstract(C)) # outputs False
19

See also Add means to mark unittest.TestCases as "do not load". · Issue #58739 · python/cpython · GitHub.

20

I don’t understand why you consider it a breaking change: ABC classes are not supposed to be instantiated

Because there’s a non-zero probability someone else has written class MyTest(TestCase, ABC): for their own reasons, and wants to run that test case with loadTestsFromModule. There’s an awful lot of Python code in the world. Yes indeed ABCs probably should be intended not to be instantiated, even without abstract methods. But sometimes weird hacks are necessary, and people do a lot of things they shouldn’t do. That doesn’t mean we should break their tests.