Test & Code

I started using pytest primarily for the fixture system. pytest fixtures are kind of like setup and teardown in other test systems, but way more modular and flexible and usable across a test suite, and you can share them with other projects through the plugin system. The pytest plugin system allows plugins to share fixtures, but also allows them to share hook functions. So what's a hook function?

A hook function is a way to modify part of the pytest behavior. A couple of common examples are, for instance, adding a command line flag to be able to pass in information to the test run. You do that with a hook function. Another example would be to deselect or skip some tests based on some runtime factor. You would do that in a hook function as well.

This plugin system that allows sharing of fixtures and hook functions is so powerful and popular that there are at least 1,487 pytest plugins on PyPI. Where did I get that number? If you go to the pytest.org, there is a site, there is a pytest plugin list that lists 1,487 plugins. This is done by searching PyPI for pytest. actually projects that begin with pytest dash or pytest underscore, since that is a common convention. If you go directly to PyPI, you can just search for pytest and it comes up with 10,000 plus projects. That's a little overwhelming. So while I appreciate the restriction down to 1,487, that's still kind of a lot. So what should we do? An alternative list is on pythontest.com. There's a list I put together called top pytest plugins. And that is based on the counts for the number of downloads over the past 30 days-ish. And the list in there has currently 133 plugins.

Although popularity isn't everything, I was curious about the most downloaded pytest plugins, and that's why I put this list together. I actually used another dataset from a website from hugovk called Top PyPI Packages. And this is regularly updated about once a month to dump the 8,000 most downloaded packages from PyPI. And I grabbed all 8,000 and searched for pytest in that list. And I do some exclusions. So there are some packages that are not supported anymore, so there's no point in looking at them. And...

There's also pytest itself, which isn't a plugin. So those are removed from the list. And that's why right now there's 133 items, but that could change over time. So this list and pytest plugins in general is what we're going to focus on for this season. We're not going to go through all 133, but we're going to go through quite a few. And we're also not going to restrict ourselves to just this. There are other plugins that are super cool. That might not be on this list i'd like you to share your favorite plugins whether on this list or not that you would like to or plugins that you'd like to have me take a look at and we can talk about it on the show if you're a plugin author and you really think that your plugin should you'd want to talk about either being an author of pytest plugins or this a particular plugin or two or three, let me know. I'd be happy to try to get some people on the show.

So I've got this list of the top plugins from one to 133, and it's not going to be ordered by top to bottom, but I'm, you know, I might start with then ordering and then jump around a bit, depending on popularity of my, of myself, the things that I use, I can talk about easily. Some things I might I have to do a little bit of research with. And some, I just am not interested. Also, I really don't want to do 133 episodes in this season, but we'll see. Who knows? Might be longer, but probably not. For instance, let's take a look at the top. The number one downloaded plugin is pytest-cov. And that is a pytest plugin for measuring coverage. It actually uses coverage.py and it provides some conveniences. and I won't get too much more into it because I want to talk about that in the next episode. So we'll use pytest-cov in the next episode. But there's a whole bunch of great ones in here. Let's take a look down the list.

Currently, number four is pytest-asyncio. That sounds cool. I don't write a lot of async code, but that'd be one that I'd love to talk to somebody about, either the plugin author or anybody that uses it on a regular

basis and wants to talk to me that'd be great. pytest metadata I use all the time and that's along with pytest HTML but metadata is usable by itself and those are great. Those are number five and six. pytest timeout is fun. Skim down a bit. pytest Ordering, there's various ones for ordering that I'd like to kind of like talk about those together that might make sense. There is pytest Repeat for repeating tests. It's one that I have helped with in maintenance before. pytest Check is my plugin. That's number 24. That's one of mine. It allows multiple features, multiple failures per test case. pytest Playwright. this is on my list of things I want to try because I'm working on a web project so I do definitely want to try pytest Playwright but it'd be fun to talk to somebody else about that what else? Let's go down what's near the bottom.

There is pytest MD report at 133. Now that's not the least downloaded one, that's just the least downloaded in the in my list but it still gets looks like 112 12,000 downloads per month. So pytest plugins are downloaded a lot. So this one is a pytest plugin to generate list test outcome reports with markdown table format. Cool. So reporting the outcomes in a markdown format, this would be cool if you're piping the output to somewhere where you can read it better or that can, yeah, maybe to a website or something. That's cool. I better, I'll definitely check this one out. And one of the things that if I, so I'm looking at this list and what I've done with this list is it's got the number, the order number, the name of the plugin, plus a link to the plugin in the name and the number of downloads and, and the, so I'm getting that all right from the top PyPI packages. And then I also wanted to have like the description. So this one says a PyTist plugin to generate test outcome reports with Markdown table format. And I'm getting that directly from PyPI. So there's an API to PyPI to pull that out. So all of those descriptions and the name and links, those are all on the top PyTist plugin list.

And yeah, that's partly where I'm getting this. Like I said, I will cover other stuff too.

Popularity isn't everything. The top of the list is going to be not too surprising. Let's actually talk about some of the other, the top. Let's go through the top nine, because these are all ones that I'm definitely going to talk about. Probably the top, yeah, more than that. But top nine are pytest-cov, pytest-mock, pytest-xdist, pytest-asyncio, pytest-metadata, pytest-html, pytest-timeout, pytest-rerunfailures and number nine is pytest-django and I definitely will talk about that because I'm working on a django project so uh these are these will all get covered but I'm not gonna and then I i don't know if I will do all right away or jump around who knows what I will think about doing so that's covering pre-existing plugins which is I want there's a few reasons why I want to do this. One, I want to let people know about these without having to go find them themselves.

Two, I want to plug my top pytest plugin list, but I'm not making any money from it, so I don't know why. Three, it's just good to like sort of hear about some other things. And the episodes might be short, so I might have some short episodes because I don't know much about something, but I don't want to like lump things together too much. If there's like, there's a whole bunch of random ones we'll probably do those together we might do those together and then there's a different ordering ones that might do those together because it makes sense when we're thinking about if you're going to reorder stuff which which ordering plugin to use that might make sense to lump those into one episode but a lot of them might be just individual plugins that we'll talk take a look at now there's a lot of plugins that you should take a look at even if you're not going to use because they either will show you some cool use of fixtures or some cool use of the hook functions. So I find that learning how to use the hook functions from examples using the hook functions is the best way. Now there's the pytest documentation and it's really good but around hook functions and how to use hook functions I think it might still be lacking and is this something that like I think how to write pytest plugins that would be.

Actually a decent course or a book. I do have a section of the Python testing with pytest and the complete pytest course, so both in book form and in video course form. I do cover building plugins, but I don't go through all of the hook functions, just a few. So opportunities there for people to jump on board. Anyway, this introduction is getting long. I'm going to talk about not just some plugins that you need to know about but plugins that you might want to look at for their source code and also also might talk about some of these these hook functions and and how to utilize those and what they do that might be a fun thing to do in this series and I'm not sure where this series will go but I thought it'd be fun to do a season of pytest plugins so welcome thanks.