Best practice to do the same tests in pytest with different classes

Python Help
1

I have many tests for a class with a C extension. I redo the same tests for:

  • the c implementation
  • the pure py implementation
  • the subclass of the c implementation
  • the subclass of the py implementation

Since not all tests are common for the 4 cases, I created 4 files:

  • test.py
  • test_c.py
  • test_subclass.py
  • test_subclass_c.py

and a common.py file filled with pytest.fixtures and test_* functions.

To call the same tests for the different classes, I exec the common.py. This is an example with test.py

import pytest
import pippo as pippo_module
from pippo.purepy import pippo as pippo_class
from pathlib import Path

c_ext = False

curr_path = Path(__file__)
curr_dir = curr_path.parent

with open(curr_dir / "common.py") as f:
    common_code = f.read()

exec(common_code)

with open(curr_dir / "pippo_only.py") as f:
    pippo_only_code = f.read()

exec(pippo_only_code)

pippo_only.py contains tests for only the pure py implementation.

It works, but I feel it really hacky. Do you know if there’s a more simple and elegant way to achive this?

2

I’m not sure I quite follow your example, but yes, using exec is pretty hacky :slight_smile:

I think you can use pytest parametrize to do this, something like (untested):

@pytest.mark.parametrize('class_to_test', (c_class, py_class))
def test_something(class_to_test):
    instance = class_to_test(some_args)
    result = class_to_test.method()
   ...

So this will create two tests, exactly the same except which class to use.

3

Just use class inheritance.

common.py:

 class CommonTests:
     ... all the common test methods ...

pippo_tests.py:

 from unittest import TestCase
 from .common import CommonTests

 from class PippoTests(TestCase, CommonTests):
     ... the pure py specific tests ...

and so forth, suitably structured.

Cheers,
Cameron Simpson cs@cskk.id.au

4

I could, but I have many tests. They are ~100.

What surprised me a lot is that import does not work with pytest. I suppose pytest do some magic.

(Digression: I think it’s the same problem with functions vs objects. Sometimes it’s more handy to have a single function that accept many object types instead of the same method in all the types).

5

Import works just fine with pytest. Can you show us a failing example?

6

So you’re suggesting to use class-oriented tests? I always found them very verbose, but they should do the trick. I have to try!

7

Sorry, I deleted my reply because I realized you was talking about importing class-oriented tests, and not function-oriented tests. Importing the second type in different test_*.py scripts does not work.

8

The verbosity is reduced the more test methods you put in the class.

9

I think I still need to see an example to understand what you mean about the imports.

10

I completely agree. I find function-based tests very much cleaner for simple tests, but my case is not so simple.

11

Simple example:

common.py:

def test_1():
    assert pippo_class(a=1) == {"a": 1}

test_a.py

from pippo import pippo as pippo_class
from common import *

test_b.py

from pippo.core import pippo as pippo_class
from common import *

Result:

(venv_pytest) marco@buzz:~/sources/venv$ pytest
============================= test session starts ==============================
platform linux -- Python 3.10.0, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/marco/sources/venv
collected 2 items                                                              

test_a.py F                                                              [ 50%]
test_b.py F                                                              [100%]

=================================== FAILURES ===================================
____________________________________ test_1 ____________________________________

    def test_1():
>       assert pippo_class(a=1) == {"a": 1}
E       NameError: name 'pippo_class' is not defined

common.py:2: NameError
____________________________________ test_1 ____________________________________

    def test_1():
>       assert pippo_class(a=1) == {"a": 1}
E       NameError: name 'pippo_class' is not defined

common.py:2: NameError
=========================== short test summary info ============================
FAILED test_a.py::test_1 - NameError: name 'pippo_class' is not defined
FAILED test_b.py::test_1 - NameError: name 'pippo_class' is not defined
============================== 2 failed in 0.02s ===============================
(venv_pytest) marco@buzz:~/sources/venv$
12

Simple example:

common.py:

def test_1():
   assert pippo_class(a=1) == {"a": 1}

This won’t work anyway, regardless of pytest. This is wrong in a generic
Python sense: you haven’t imported the name pippo_class. Like any
Python function, test_1 runs in the namespace in which it was
defined: common.py’s namespace.

Is there a reason this file can’t:

 from pippo import pippo as pippo_class

This is why you’ve been getting away with it with your
read-file-and-exec()-code approach - effectively you’ve reciting
common.py’s code inside your other file: a C-like #include instead
of a Python-like import.

test_a.py

from pippo import pippo as pippo_class
from common import *

Just a remark here: as a matter of good practice, if you use import *,
it should always be the first import. That way you know what’s from
where.

What I’m getting at here is: suppose common.py also defined a
(different) name pippo_class. The import * would have discarded the
one from pippo. Putting the * first avoids this possibility.

Result:

FAILED test_a.py::test_1 - NameError: name 'pippo_class' is not defined
FAILED test_b.py::test_1 - NameError: name 'pippo_class' is not defined

These are both from the missing import in common.py. This is normal
Python behaviour and nothing to do with pytest.

Have you considered the viewpoint that class inheritance of some
hypothetical CommonTests class is essentially a more structured form
of your from common import * module-level gathering of names?

Cheers,
Cameron Simpson cs@cskk.id.au

13

Why not use parametrize? You can define it once, use it multiple times.

You can also use a parameterised fixture:

@pytest.fixture(params=[ClassA, ClassB])
def tested_class(request):
    return request.param

def test_repr_nonempty(tested_class):
    instance = tested_class()
    assert repr(instance)
1 Like
14

So? some copy and paste and you are done – but you can parametrize test classes, and even modules to save some typing.

15

Note that you can use test classes in Pytest without unittest – then only. a touch more verbose (extra self all around) It can be handy when a bunch of tests need to share info.

-CHB

16

You’re very right. I think I’ve found the solution:

common.py

pippo_class = None

def test_1():
    assert pippo_class(a=1) == {"a": 1}

test_a.py

import common
from common import *
from pippo import pippo as pippo_class
common.pippo_class = pippo_class

I omit test_b.py

I think this can be done in the same way with classes, adding an attribute pippo_class.

PS: for non-italian people, “Pippo” is Goofy. pippo, pluto and paperino (Goofy, Pluto and Donald Duck) is the italian way to say foo, bar, baz

17

This isn’t very good; you’re setting a global. Supposing you ran
test_a.py and test_b.py in the same Python run, eg via something
which finds them both and runs them by importing them and then running
each test suite? Then the actually class used in test_1 would
depend on the import order because each file overwrites
common.pippo_class.

Personally, I would much rather go:

 def test_1(pippo_class):
     assert pippo_class(a=1) == {"a": 1}

passing the class as a parameter, and remove the global entirely.

“But wait!” I hear you cry. “I’ve got 100s of tests!”

Maybe, and this is where a class is of use:

 class CommonTestsMixin:

     def test_1(self):
         assert self.test_class(a=1) == {"a": 1}

     etc etc ...

then in tast_a.py:

 from common import CommonTestsMixin
 from pippo import pippo as pippo_class

 class PippoTests(unittest.TestCase, CommonTestsMixin):

     test_class = pippo_class

     ... maybe some pippo sepecific additional test methods ...

That’s the hardwired version, where PippoTests has a hardwired test_class = pippo_class.

You could also parametrise things, depending on your use case.

This approach lets you, for example, have a test_all.py:

 from test_a import PippoTests
 from test_b import PippoDifferentTests

or just have pytest or something find them all. No import order
dependencies.

Cheers,
Cameron Simpson cs@cskk.id.au

18

or pytest’s parallel test running features – unit tests should be isolated, and depend as little as possible on when and how they are run.

I don’t see why you are so resistant to subclassing or parameterize, but if you REALLY don’t like those 'cause it feels like too much repeated code, have a simple code generator auto-duplicate all the tests :slight_smile:

19

That’s a good observation.

Playing devil’s advocate, I do not run tests concurrently, since the tests are really fast. Furthermore, sometimes is good to have sequential tests (for example, for integration tests).

Anyway, I find that the class-oriented approach suggested by you is more clean and robust. I’ll change my tests (well, when I have time…)

PS: I have a curiosity, but I thought it was better to open another thread: