Test & Code

pytest-html has got to be one of my all-time favorite plugins. pytest-html is a plugin for pytest that generates an HTML report for test results. This episode digs into some of the super coolness of this plugin. Getting started with pytest-html is easy enough. You just run pytest with an --html flag and give it a file name, like something like report.html or something, and it will generate a report. By default, it actually puts a bunch of the extra stuff, like the CSS and everything, into a directory. But you can pass in a flag, --self-contained-html, and it will stick all of that into one file.

So you just have an HTML file as output. That's what I usually do. That approach also works great if you're going to move the report somewhere else, like email it to somebody or whatever.

The report has some environmental info at the top and a summary of the test run, things like number of tests, time it took, and the numbers of all the result types like passed, skipped, failed, errors, etc. And then you've got a results table. This is a table with result status and those are in color too. So the failures are in red and passes are in green and stuff. And then you've got the test name and duration. And I actually don't remember what the default columns are because I often customize this table. The table columns are all sortable too. So you can sort by result or by test name or duration or whatever and starting by result means like you click at the top and you'll get like all the fails at the top or something but you can also use the table to find what if you've got the durations you can find what the slower tests are or the faster tests for anything with extra information like errors or failures and stuff you can click on the result and it expands into a block that shows the traceback or any information around there and if you haven't seen this before you got to try this out it's super helpful even if you're just debugging a big suite locally but it also works great to link to in a ci server or something like that there's also a screenshot of the report in the repo readme so you can check it out there if you don't really want to give it a try right now.

But I highly recommend trying it. One of the cool things about pytest-html is the customization. The docs are really clear and helpful regarding customization. And I'll have a link in the show notes to the docs as well. You can customize the CSS, and you can add stuff to the environment section. For this, there's another plugin called pytest Metadata that makes that super easy. This extra data gets passed in on the command line or even at runtime with pytest hooks. For the table, for the results table, you can add columns, you can customize existing columns, you can hide columns, you can add stuff like links, extra data, and even pictures. I've seen the pictures used where somebody will take a screenshot during the test, and if the test fails, they'll attach that to the report. That's super cool, super helpful.

Remember when I said there was an environment and summary section at the top of the results table? Let's go back up there now. The environment section, by default, has Python version, platform, which operating system you're running on, pytest, and plug-in version, and which pytest plugins are present. But you can add to that section. The easiest way is to use the pytest metadata plugin. That's a different plugin, which is cool enough on its own to deserve its own episode. but it gets installed also when you install pytest-html and you can use that to extend the environment report from the command line or from hook functions or from fixtures. You can even remove some of the default stuff if you don't want it there. I like to add things like code versions, hardware options, device configurations of the thing I'm testing. You can put whatever you want there that's helpful to have at the top of the report and you can do it very easily.

Now let's look at just under that at the summary section. You can also add to the summary via a pytest_html_results_summary hook function. That hook function gets past three parameters, a prefix, summary, and postfix, which are just lists that you can add to with HTML, and that shows up in the report. For me, I don't want to clutter the default view too much, so I usually stick the extra summary info into a details HTML block with a summary line, and it just shows up as a single line that someone can click on to see more info. I just used this feature recently to include some aggregate test data collected at runtime into the report. Having it collapsed assures that only people that care about it need to look at it, but also being able to control the HTML makes it super easy to present it exactly how you want it.

Anyway, I love pytest-html. Check it out if you haven't already.