Test & Code

Brian: 00:38

Welcome to Test and Code. Welcome to Test and Code. I'm thrilled to have James Falcon on the show today. We're gonna talk about, pipest and testing and auto use and mocking and all sorts of stuff. But first, I'd like to let everyone meet James.

Brian: 01:02

So, James, can you tell us who you are?

James: 01:04

Yeah. Hi. I'm James Falcon, and I'm currently a software engineer at Canonical working on Cloud Init. And so for people unfamiliar with Cloud Init, it is early boot instance initialization on the cloud, which, for most people that would mean when you go launch something out on any of the major public clouds, any of your compute instances, there's some stuff that happens there to, do things like create a default user, lay down SSH keys, set up networking, those sorts of things. And so, yeah, I've been doing that at Canonical for about the past 3 years now, and have been a Python developer for the past 15 years or so.

James: 01:44

And so, various number of development and testing jobs.

Brian: 01:49

Okay. So you've been using, you've been using Python for a long time.

James: 01:53

Mhmm.

Brian: 01:54

And is it so right now, you're kind of in a economical in a lot of the networking stuff. Is the networking stuff kinda something you've been in for a long time also, or is that more recent?

James: 02:07

Yeah. Somewhat. Yeah. Here, it's it's mainly about, taking a user provided networking config and making it so that it can work on any particular distro that is supported by Cloud in it. But, I've I've done some some bits of networking in the past, but I don't know.

James: 02:26

All all kind of in different contexts. So I guess it's hard to to go into detail about the specifics specifics of those.

Brian: 02:32

No. That's fine.

James: 02:34

Cool.

Brian: 02:35

And, I guess let's jump into Pytest. Do you you Sure. We brought you on to talk about Pytest. What did do you use it in your in your role at work, or do you use it just personal project? Or

James: 02:48

Yeah. So we use it at work. I've used it at, I mean, I've been using it for years in in most of my work roles. And so, yeah, I've been using it for a while, you know, discovered all the fun parts of Pytest, like, you know, fixtures and parameterization and all that kind of stuff years ago. And so, have been putting those to use as well.

Brian: 03:06

And what so one of the reasons why actually, the reason why I asked you on the show is, on a recent on a recent Python bytes, we talked I talked about auto use fixture. Oh, we talked about a Pytest article, and one of the things was a good use for oh, there was a mistaken, auto use fixture example. And, and I admitted that I don't know very many good examples for why you would wanna use auto use fixtures. I have a couple use cases, but they're they're really niche. So, and then you brought up that you have some auto use cool use cases.

Brian: 03:44

So, I guess let's before we jump into, that too much, I It's possible that somebody listening to this ups this podcast might not know what a fixture is yet, but, do you wanna try to take a stab at, describing what a fixture is?

James: 04:02

Sure. Yeah. So, fixtures and pytest are a way of basically performing setup and and tear down actions, And it's done in a way that's, more considered dependency injection where, you know, you you define it somewhere and then just list it as a parameter in your test function, and, pytest is smart enough to go find it and go grab it and use it and kind of insert it into your test. So allows for easy setup and teardown, in kind of a, yeah, a dependency injection way.

Brian: 04:34

And so it is setup and teardown around the test. And, normally, like you said, normally, you list it, it the the name of the fixture is listed as a parameter to the test. Now that there's a couple cool things with that. One is it's really easy to tell which tests are using which fixtures. The other thing is that the, that way, if the the fixture returns any data or returns an object or returns anything, the the test can use that that, that parameter to read to read that data.

Brian: 05:06

But, but there are case there's a a one of the things we're trying to talk about is auto use. If you want the fixture to run, but you don't need the you don't need to list it, but you want it to run for everything. That's where auto use comes in into play, and it's just a way to when you when you're defining a fixture, you just say auto use equals true, and then it just it runs for anything in that scope of of fixtures. Right. Now for my the use case that I, I am familiar with is from for that I use a lot is if I'm using a hardware instrument and I'm testing that and I'm running a use case against that or a test many tests against it, but I want to, like, interrogate, the the the data logs on the the error logs stored on the instrument.

Brian: 06:02

And I wanna make sure that nobody forgets to do that, and making sure those are clean. I've used auto use fixtures to make sure that the air there's no air any error logs. And then the but there's a prob there's a little bit of an issue with that because if you if you notice it after during teardown at the end of the that after the test run, you can throw an exception there, and it results in a test being an error, but it doesn't fail it. And that's the only downside. But, anyway Mhmm.

Brian: 06:32

Aside from that caveat, those are that's the use case that I have used. And, but, you've got

James: 06:40

a couple more or at least one more. So what are some cool auto use uses? Yeah. I mean, our big one is around, is around mocking. And so I know, you know, not not everyone does mocking.

James: 06:53

Not everyone is, super familiar with mocking because there's not always a lot of use cases around mocking. So, maybe I'll go back and kind of explain our our use cases around mocking first and then, talk about the auto use pieces. So, for us, you know, since since we're doing things to actually set up the operating system, a lot of times there's there's things happening that you don't want happening in unit tests, you know, reading and writing to your password file or creating users or those sorts of things. Obviously, you don't want that happening. And also the fact that Cloud Init runs, on a number of different types of operating systems, you know, Linux and and BSDs.

James: 07:31

They're you know, when when you read certain information off the device, you might be getting different things for for different systems that you're on. And so, those are the main use cases for us for mocking and that, you know, I just I don't wanna run these things on a local system. And, sure. You know, we could we can make integration tests where, you know, you actually go and run it on a full separate system, and we do do that. But, if you think about it, you know, when you're spinning up a whole cloud instance, that's that's gonna take a long time.

James: 07:57

And so, spinning up new ones for each individual test is actually just not gonna work for, for all of the things you might wanna be able to use a unit test. And so, for us, yeah, we'll we'll mock out the things that, you know, if if we're trying to run some external process that, might be doing something destructive on your hard drive, we'll we'll mock that out and, run the rest of it with it. And so So, yeah,

Brian: 08:23

can we can we deep into like, dig into a a little detail of, like, at least a use case? Like, what what would, like, what piece of that could you mark and how would how could that be useful?

James: 08:36

Yeah. So for example, let's say we're just testing we we have all these various modules in Cloud in it, and each module can just kinda do its own individual thing. And so, for example, one of them is creating users and groups, just within a Linux system. And so, you know, obviously, I don't want various users sitting around on my system that aren't actually, you know, what I want on my system. And so, we'll we'll wanna test all the code around calling the call that creates the user, but we don't actually wanna test that call because that call is just, you know, running the, you know, using, like, subprocess and running the create user command that's on the operating system.

Brian: 09:16

Okay.

James: 09:16

And so we don't we don't need to test that. We we assume that, you know, the the Linux implementation of that is right and that it works, But, we have a bunch of code surrounding it that lets you, set up exactly what that call might look like, what to do based on, if it fails or if something's not set up the way you're expecting to. We still wanna be able to test all of that code, and so we basically just mock out the piece that calls out to the operating system and, but test all the rest of the code around it.

Brian: 09:42

Oh, cool. That totally makes sense. So you're mocking it. You got somehow, you've wrapped the you probably are wrapping the system call with some other, method that you call instead, and then you can mock that wrapper. Right?

Brian: 09:58

Right.

James: 09:58

Mhmm.

Brian: 09:59

And then well and then, also, it kinda brings up another way reason that mocking is awesome in some cases is because you can have failure conditions. Right. And it's often hard to generate failure conditions, especially with operating system things. Like, what happens if, the use like, I don't know how why it would fail, but creating a user, I assume, can fail in some cases. Yeah.

James: 10:25

I mean, you know, if if user already exists, for example, or, I mean, the other would probably be some more esoteric use cases. But, yeah, in in the case of mocks, mocks lets you, set up side effects. So, a side effect could be an exception. It could be you could have various types of return values. So if I call the same mock multiple times, I could get, you know, maybe I created this user here and this user there and pretend that I've done it multiple different times.

James: 10:49

So, the mocking library the mocking library in Python is really powerful and kinda lets you simulate pretty much anything that you would wanna be able to do otherwise.

Brian: 11:01

Yeah. So I guess I've got another some more questions about mock. The Sure. Yeah. Are you do you're using straight, like, unit test mock, or are you using, the pipest mock wrappers for that?

Brian: 11:14

Or

James: 11:14

Yeah. We use unit test mock. I'm I'm aware there was an independent mock library, but I think that was just before it got, packaged in with unit test itself. Yeah. I'm I'm not aware actually of a particular Pytest specific one.

James: 11:28

Maybe there is one, but, that we haven't had any need for. The one that comes from unit test, works works fine for us.

Brian: 11:35

Yeah. The, and I think the pytestdashmock is a plug in that does, it does make sure that your mock is cleaned up after the test. But, but mock has a, when I'm using I usually use it with a straight unit test mock because I I guess once I by the point where I wanted to use the unit test mock, it already had, context managers. So, I I usually use mock with a contact manager so that it doesn't so that it cleans up after itself afterwards.

James: 12:09

Right. Yeah. So you can use, you can use it a couple different ways. One is with the context context manager. So, you know, you just have your with statement, and then within there, the mock is applied.

James: 12:19

And as soon as you're out of the with statement, the the mock is, basically cleaned up. Because, yeah, if you you can just you can just create a mock patch, and if you do that and you you don't clean it up, then it will still be live after your test ends, and you obviously don't want that. One other way too that's pretty common that we use is, using it as a decorator, and it's the same kind of thing as, if you if you put it as a decorator on a test, by the time the test ends or when the test ends, it'll automatically clean up there too. And so that's useful. You you you kinda patch it up there in the decorator, and then you get a reference to it in your actual test definition as, like, a test parameter.

James: 12:56

Okay. Just specify it in there, and then you can you can set things up that way too.

Brian: 13:00

Can you now, thinking about using mock with fixtures, can you use the decorator with fixtures? Do you know? A decorative fixture, or at that point, are you using, either cleaning up specifically in the the fixture or, using it with a width block.

James: 13:18

Yeah. You are are you saying, like, applying mocks within a fixture and then just applying that fixture to the test? Is that what you mean?

Brian: 13:24

No. What I mean is, like, if if I were to apply the mock in a in a fixture in the setup part and then wanna tear it down in the tear down part, can I use the decorator thing just to decorate my fixture, or does that work? I I

James: 13:37

don't think I believe it should. I don't know off the top of my head. But, yeah, I I think we've done that sort of thing in our fixtures too.

Brian: 13:42

Okay. Or it like, the the in the case where I've I'm usually using them with a width block because I just like that model. Within a fixture, you can yield from the inside the width block, and, then it should be fine. Clean up afterward.

James: 13:59

Yeah. So we we've set them up. We've done a few that way too where, it it's it's easier, especially if there's multiple mocks that you need to apply to kind of a group of tests. You can just have a common set of fixture, you know, apply these 3, 4, or 5 mocks, and then use that one throughout all the various tests, rather than having to define everything multiple times across the test. Because that's that's one issue, and this this will kind of get into the some of the auto piece auto use pieces too.

James: 14:26

But, you know, things can get a little ugly if you have to apply multiple mocks because if you think of a context manager and now you're, you know, 5 levels deep or something or, you know, you have 5 decorators and then multiple, function parameters, and that gets really tall vertically, all just for one test. And so, yeah, I we we found a few different ways of making it nice, but, I don't know. It it just depends on the on the context on the use case, I guess.

Brian: 14:51

Okay. So how do we pull in auto use to this? What do you

James: 14:54

Yeah. So so for us, our our biggest use case is at the top level. We'll, if so, again, for people that aren't unfamiliar, if you have a top level conf test dotpy file, you can set up certain py test things in there, and you can put an auto use fixture in there that will apply to all of the tests that are defined anywhere further down. And so we have one of those, and, within there, we have a mock actually of subprocess. So, I mean, we we have a subprocess wrapper, but it it's kind of a mock of that wrapper to say, you know, if if you try to use subprocess in any of your tests, we're automatically going to throw an exception and say, no.

James: 15:33

You can't be doing that. And so that's for our own benefit, as we're writing tests, making sure that we're not accidentally calling up to something that's going to do something bad on our machine that we don't want to be doing. And, you know, it's not common that you're going to accidentally do that, but, you know, every once in a while, you'll call something that's, you know, can call multiple other things that and then a few layers deep, you forgot, oh, hey. Within this call chain, we are actually calling something that I don't wanna be called locally. And so, it prevents us from being able to do that, and automatically lets us know that, you know, you've you've you messed something up here.

James: 16:06

You gotta add you gotta add the MAC locally. Because the nice thing is too is that if you have a a top layer MAC like that, you can then mock it locally, and that local one applies instead of the global one. And so that's that's useful. And so, yeah, we we also set it up so that, you can you can mark the test. If you've ever used by test marks, you can mark the test and say, actually, we do wanna allow a sub process call here just in case we need it in any of our tests.

James: 16:32

But, I don't there's there's a couple rare occasions where we actually do, allow calling subprocess, but most tests, we we don't want that to happen.

Brian: 16:41

Okay. Now is it is this so these these are these might be really interesting use cases for other people. Is this something that's, in an open source project that people can look at, or is it, closed source?

James: 16:55

Yeah. I mean, we you could go look at our, cloud init, test code. I I wouldn't necessarily look at all of the test code there because we do have some, some some old code in there that I wouldn't recommend as being good examples. But if you look particularly at the, top level com test of our test directory, you can see what I'm talking about in there and and kind of how we apply that. Okay.

James: 17:17

We do also have a few other auto use fixtures. One I know is for making sure we're not reaching out to do, like, DNS type queries out to the Internet, because there's a couple things, again, within cloud in it where, just for various networking type reasons, we might want to try to resolve a host name. But we also wanna be able to run Cloud in it in a or run the unit test, excuse me, in, you know, in on Jenkins nodes that don't have access to the Internet. So, you know, it's something that might pass locally, but then once it got onto Jenkins, it would fail. And another thing to just let you know, like, hey.

James: 17:51

You're trying to to reach out to the Internet in a way that's not gonna work when you try to run it on Jenkins.

Brian: 17:57

Okay. Yeah. Actually, this so, it from what I've heard so far, you're using mocks and, and auto use around mocks, but mocks in general, to, to isolate the testing away from so that you're not testing bits that really aren't your code. They're either system system calls or networks, services that you're, like a DNS. That that's a service that you don't that's not your part of your code.

Brian: 18:28

So you're you're you're stubbing out, mocking or stubbing things that are external to your software. So do you have any, the things that I I particularly I know it's just just a testing style, but I shy away from mocking out my own like, there is there is a model of, of unit test where you mock out different subsystems, and you're just or even different functions within our within your system. You doesn't sound like you're using that sort of a model.

James: 19:00

No. Yeah. I also I I don't usually think that would be a very good idea, unless, you know, there's there's something that's gonna take a really long time or, you know, have some complicated setup, that sort of thing. I I could see maybe doing it in those cases. But, yeah, in general, I think I think the better, you know, places for Macs is where, like you said, you're calling out to something external or something that you don't think that you need to be testing in spite of another system that should have been tested separately.

Brian: 19:30

Yeah. And it like, in a in a general sense, one of the, like, examples that I've heard before are, or things like if you're gonna charge a credit card or send an email, you don't really wanna do that all the time in your testing. Right. But the, but the those are obvious. Don't you'd wanna mock that case.

Brian: 19:54

Mhmm. And then there's, like, the less obvious where you get into, like, do you wanna touch the file system or not? And that's around probably test speed. And I I generally used to, like, avoid do it mocking the file system or even even using, but I do. I guess I gotta admit, I use temp files, and temp files are kind of like, you know, pretend file system part.

Brian: 20:20

And then you can also within the temp temp file system, you can, have it not actually be a file, and you can have it be a in memory file. And then that's kind of the same as mocking the, file system. But

James: 20:34

Right. Yeah. And we we we'll do something similar where, this isn't an auto use thing, but, we we have a whole series of functions that we kind of retarget. So let's say I need to write to Etsy password. That's not something I want to do locally, but I do wanna test that we can actually write Etsy password.

James: 20:53

And so we have a series of mocks that will go and retarget it into /temp. So instead of writing etc password, it writes /temp/etsypassword. And then now within temp, we can kind of do all these things safely without worrying about opening anything on our own systems.

Brian: 21:10

No. That those are great use cases. So, and that yeah. I I think these are these are awesome examples for complete reasonable you don't want your tests to muck up your system. So you don't wanna, like, change services or file system stuff on your own computer and also stubbing out or mocking network services.

Brian: 21:30

So Mhmm. Great great example.

James: 21:33

Yeah. Yeah. There's one that might be a little more controversial that that I'll do is that, sometimes even at the top of a module, talking about, you know, mocking multiple things, sometimes I will make an auto use fixture that works for, kind of everything within the file. And so, again, at the test level, you can't see what's necessarily being mocked, but sometimes, for me, it's a trade off of, like, you know, readability and style and that kind of thing. And I think we have enough modules that do that that we know to kind of look at the top of our modules to to see, you know, what we've mocked before, you know, wondering if there's if there's something some weird behavior that's not making any sense to us.

Brian: 22:15

Okay. So any that I think those are those are great rules of thumb. So if you're having an auto use on a in a global scale on your all your whole test suite for things you really, really don't want to have happen, like, don't call subprocess unless Right. Or unless you specifically know what you're doing and you've marked it to say, yeah. I wanna use it, but, I think that's a cool way.

Brian: 22:38

Like, have a global by default, don't, but have an escape hatch if you need. That's that's kinda cool. It's a cool cool way to do it. The other thing, like you said, is it other other than that, try to keep auto use fixtures as close to the dust as possible because why? It's because they're they're hard to they're it's it's weird when your test behaves the way, you know, in a weird

James: 23:07

way. I guess Yeah. And I I have been places where we haven't done that, and you zoom out to an outer level and there's, you know, some auto use fixture that's in the middle of, you know, it's like a you have multiple layers of com test files, and in the middle of there, there's an auto use fixture somewhere. And, it's really hard to know what's going on in your test because, you know, like you said, it's not it's not close to the test, and there's real no no way to see where is this being applied, how do I use this. And so, yeah, that's a great rule

Brian: 23:37

of thumb. I I have, in the past, thought it was really awesome that that you could have, test files at every level of your test, hierarchy.

James: 23:45

Mhmm.

Brian: 23:46

But I've kind of gone back to but it's bitten me a lot of times to have especially if I have an auto use fixture sitting in a comp test file in a direct in mid level directory, I will forget about it. Right. And and even a fixture that's there, I it it doesn't it make it made sense to me at first, and I it still kinda makes sense logically to say I wanna keep a, like, a fixture even if it's not if it's not an auto use fixture, just a normal fixture. Keep it as close to the test as possible. So if it's just used by 1 module, throw it in there.

Brian: 24:21

If it's used by a few modules in a directory, that made sense for me at one point to put it in a comp test file in that directory.

James: 24:31

Mhmm.

Brian: 24:32

Now I'm leaning towards, is there put it at the top level unless there's a real good reason to not. Like, for instance, if I've got guess I had I've got I've got projects where I've got, similar tests that, access either that test the same behavior, but test it through the user interface or through the, API. And in those cases, we got little stubs that, like, you know, some fixtures that are the identical names and but they they act either on the CLI or they act or the the user interface or the or they act on the API. Those make sense to split those up. But Mhmm.

Brian: 25:15

Normally, now I'm thinking, well, who am I to say if it might be useful for other tests? Maybe I'll throw it at the top level. And maybe it doesn't make sense at the top level on your whole test suite, but unless there's a harm done, it's to to me, it's handy to just have those findable. They're either in the module or they're in the one conf well,

James: 25:36

Right. Yeah. We we do something similar. You know, we'll have a a few in the actual modules, and then, we do have a top level comp test, but then we also split unit tests and integration tests under that. And so each set of those will have their own.

James: 25:48

But beyond that, yeah, I I don't think we have additional comp test files beyond that.

Brian: 25:53

Also also, I do have, I have seen people, like, stick all their, fixtures in a separate fixtures file, and then and then just have that import import star from your, an a conf test file. It's a interesting design, like, the hierarchy design, but, yeah, I've never done that because I felt the need.

James: 26:15

But Yeah. I I could see, you know, if you have a lot of fixtures or your your comp test is really huge, you know, sure. Makes sense to split it out that way. But, if if your comp test is, you know, a few 100 lines or something, I I don't see a a big need to to need to import anything separate.

Brian: 26:32

Yeah. Anyway cool. Any other, things you wanna touch on with this? How long you've been using Pytest for this? Is it forever, or did you switch at one point?

James: 26:44

I mean, I've been I've been using Pytest a long time, you know, several years. But yeah. I mean, early days, I was just using unit test things, but it was, you know, in the early days of Pytest, it's just people were using unit test before that. So

Brian: 26:57

that's Are there are there like, in this Cloud init test suite, for instance, is it using both, the unit test and Pytest, or just

James: 27:06

Yeah. There there is a mix in there. It it started off as just a unit test, unit test project. And so since then we have adopted pytest. And so as we go along, we do try to migrate things when we can to use pytest.

James: 27:20

And, there's there's a couple of base classes that we used in the unit test thing to do a lot of things that we can do with fixtures now. And so migrating away from that big base class into using various fixtures and parameterization and and that sort of thing. And, yeah, so it's it's an ongoing project. There's there's still a lot of pieces that need to be migrated. It's it's it's not something that will probably take a big chunk of time and just ever do it all at once.

James: 27:46

It's just, you know, as we come across things that we need to modify, if if we got to sign, go ahead and and do the, the migration first.

Brian: 27:53

Yeah. As actually, I think it was a brilliant decision on the Pytest people. I wasn't around at the time, but the the the idea of having Pytest to be able to run unit tests also so that you can you can switch to a Pytest as a test runner and, and run both your unit test and your Pytest test with one invocation. I think it's brilliant. But, but it it and it allows you to gradually do this.

Brian: 28:19

You you can just start using Pytest. You don't have to convert other stuff if you don't want to.

James: 28:24

So Yeah. It's it's a really nice thing about Pytest because previously we were using nose, and I don't think nose is even, supported anymore, right, or or developed at this point? So

Brian: 28:35

I don't think so. I think there was a nose too, but I don't know where that went. But, and then Pytest did used to support nose, the nose nose test, but it doesn't I don't think it does anymore. Not sure.

James: 28:50

Mhmm. Yeah. We we were just using it as a runner. So everything that was actually defined as test was just unit test compatible.

Brian: 28:58

Okay. Nice. Cool. Well, thanks so much for, sharing your time with us today, James, and I wish you well.

James: 29:05

Thanks for having me on. It was great being here.