I find that pytest is really handy and powerful test tool. But it’s not in the py stdlib.
Is it because it changes the behavior of assert?
There are many possible reasons, for pytest in particular I’d think a combination of the following:
- the unittest module exists, and the stdlib doesn’t need two “competing” libraries.
- If a library gets added to the stdlib it essentially gets “frozen” in a state where each addition and change and their consequences must be carefully considered. I haven’t followed the development of pytest that much but I reckon many of the features wouldn’t be there if it beeded to be more stable.
- Someone correct me if I’m wrong, but I assume that none of the maintainers of pytest have offered to put it in the stdlib. Compare this to the addition of the tomllib module, where the maintainer was asked and was willing to put the effort of porting the project to the stdlib. This includes (co-)authoring a pep and participate in discussions, which can be mentally draining.
Well, there’s a precedent: pathlib and os.path
Indeed I surfed on the pytest release notes page and it’s full of “breaking changes” announcements.
I really don’t know if Python asked pytest team to add the module in the stdlib.
In addition to the points Jacob already made:
-
In today’s virtual environment world, you probably only need pytest during development, only in your build time virtual environments. It will just get downloaded and used when and where it’s needed.
-
I know it’s been more-or-less separated from the original py package, but are there still dependencies on that? If so, it might well bring along a lot of extra baggage that isn’t necessarily suitable for the standard library.
-
Some core dev(s) has to take ownership of it (or a pytest maintainer has to become a core dev).
Many of the points made in this thread would apply when considering other popular packages for inclusion in the standard library.
1 Like
Well, there’s a precedent: pathlib and os.path
Pathlib is very recent, and represents a significant change to pocessing
filesystem paths, at least in terms of notation.
More to the point, there are several “more than unittest” test
finders/runners. pytest does a lot, and so to the others. Differently.
Putting eg pytest in the stdlib preferences that test kit over the
others in a sense. unittest is there as a basis, and most others build
off it or replicate its fucntionality. unittest does grow more
assertion types of time.
Indeed I surfed on the pytest release notes page and it’s full of “breaking changes” announcements.
Yeah. We don’t do that in the stdlib very much. Nearly never.
Modules in the stdlib need to be very forward compatible. People
need the stdlib to just continue working with newer python releases.
Also, stdlib module updates only happen with Python releases. And the
flip side is: they happen coercisvely with a Python release. Using a
newer Python? You’re using the newer stdlib modules. With a third party
library you can pin to a particular release, which can be important if
you haven’t updated for some breaking change. It’s also important for
reproducability.
Also, everything in the stdlib is a maintenance burden for a limited set
of people. The third party libraries have their own authors/teams.
Broadly, it’s usually a good thing for stuff beyond the basics to be
third party libraries.
Cheers,
Cameron Simpson cs@cskk.id.au
Yes, a lot…
I understood pytest can’t join the stdlib for too much reasons you all pointed out.
I hope someday pytest will overcome these problems, or unittest will incorporate some of the pytest features.
The replies after this point is only for the sake of arguing, not to encourage to put pytest in the stdlib.
Not sure to understand. Also unittest is used for development.
I’m not sure you can define Python 3.4 “recent”. Furthermore, a new module hardly is mature enough to be a stdlib candidate.
Well, of course. If you choose to integrate a module in the stdlib, you clearly prefer it over the others.
They are not “problems”, they are features.
Third-party libraries can innovate as much as they want, they can change their own APIs whenever they want, they can release whenever they feel like it. These aren’t bad things to be overcome, they are positively good things for third-party libraries (especially when they are young and experimental).
unittest’s fundamental design is very different from pytest’s design. They work in very different ways. What sort of features do you think it can copy from pytest?
Good point.
Not exhaustive list:
- simple tests can be functions
- simple fixture decorator
- parametrize
- simplified discovery with
test_prefix - more detailed and knowable output
And I like also assert x == 10 vs self.assertEqual(x, 10) etc, but I don’t think it will be ever in the language core.
Disclaimer: my pytest experience is microscopic.
Suggested features:
In unittest simple tests are methods. In the simplest case of all, that just adds one line and one indent to your test function:
class MyTest(unittest.TestCase):
def test(self):
self.assertTrue("this test passes")
unittest already has test discovery.
I don’t know what “more detailed and knowable output” means to you.
As far as parametized tests go, I have found the easiest way is to put the test in a separate method and the parameter handling in the test method:
class MyTest(unittest.TestCase):
def check_something(self, arg):
self.testEqual(arg, "something")
def test(self):
for param in things:
self.check_something(myfunc(param))
@steven.daprano Will your parametrization show, in the output, as one test per parameter, or one test for all parameters? Pytest shows one test per parameter and that’s what I assume the question about parametrization was about.
Don’t worry, not an expert here X-D
in pytest, this is:
def test_1():
assert "this test passes"
Yes, but with pytest is quite more simple. You have just to start your tests files and methods with test_.
Mh, I think you should simply run a pytest example and see the difference.
I add that it’s quite more simple to write, since it uses a decorator.
My personal take:
unitest is an old and klunky and not very “pythonic” test framework. It was ported from Java, so that’s not a surprise. One of these days, I’ll write a more exhaustive review, but I’ll leave it at that for now.
So I don’t recommend it to folks looking to set up tests for their own new projects. And if you look at large third party projects – a huge number use pytest (and nose before that), or maybe some others, I’m not sure.
I don’t mean to be critical or second-guess its original inclusion in the stdlib – at the time Unit Testing was pretty new, and it was a very good idea to get a test framework into the stdlib. junit was popular and successful – much better to bring in something that was known to work than to start from scratch.
And it’s in the stdlib because it’s used to test the stdlib itself – that’s why it’s not simply a third party library.
Should pytest be included in the stdlib?
I think not – it’s a large and powerful test framework, and I think everyone should use it (at least in preference to unittest). But you really can’t get far with Python without using SOME third party package. Particularly for testing, it’s a very small lift to require pytest to run your tests – so no need to put it in the stdlib.
But as unittest is in the stdlib, it is often seen as the “standard” or “official” (and implicitly the best) way to write tests, which is too bad. And unitest is the only option for testing the stdlib itself.
My (half baked) Proposal:
-
Soft-deprecate
unittest: While it should never be removed, we should stop recommending it as the “standard” way to write tests – even implicitly. -
Add a “pytest light” to the stdlib: pytest has a LOT of features and complexity – and it continues to evolve - moving too fast for the stdlib. But it would help a lot to have a simpler, more “pythonic” test framework in the stdlib – for use by the stdlib itself, and as a starting point for folks to write their own tests.
A “pytest light” may or may not use any pytest code, but it would be a pytest compatible subset of python. So any tests written for pytest light would run under pytest – that way folks could easily move to pytest to get more features as their testing needs got more complex. Much like you can now use pytest to run your unittest tests out of the box.
So what would a pytest light include? Here’s my off the top of my head personal list, based on what I commonly use:
1 implicit test failure messages
This is THE BIG ONE – pytest uses fancy introspection tricks to tell you why a test failed – it is incredibly useful, and completely removes the need for “special asserts” – just write:
assert result == expected
and you are done. You can even embed a function call and get a good error message:
assert math.isclose(result, expected)
So no need for special asserts
2 No need for testcase classes
Any function can be a test:
def test_this():
[some setup]
assert something
and you have a test.
( you can still make test classes if you like)
3 Parametrized tests
Very easy to write tests with a bunch of variied input – and they show up as separate tests.
4 function and generator based fixtures.
This goes with the no test classes:
@pytest.fixture
def my_fixture():
[do some setup]
yield the_setup_thing
[do some teardown]
def test_something_with_a_fixture(my_fixture):
...
result = something(my_fixture)
...
assert result == expected
Simple and very flexible.
5 contex tmanager for testing exceptions
with pytest.raises(ZeroDivisionError):
3 / 0
That’s my 5 feature list – I’m sure others have other opinions about what’s most important, and once we included everyone’s ideas, we may end up with all of pytest – but I think a simple version would be very helpful – even only the first two on that list would be great.
1 Like
This was new to me, so I went looking for how to use the feature.
And, I thought you’d misspelled “parameterized,” but when I checked the pytest docs I saw it’s misspelled there, quite consistently, both in code and prose. Is there something I’m missing? If pytest is so fluid, can’t they fix their spelling errors? In this case they could provide a correctly spelled attribute then downplay the incorrect spelling, but leave it in the code to avoid breakage.
Did you mean to add a winky-face?
Maybe putting a pytest-light into the std lib would be a way to enshrine the “correct” spelling 
– best argument for it yet!