Supercharge Your Testing with inline-snapshot

Speaker 1 [00:07]

Thank you. A little bit about myself. My name is Frank Hoffmann. I'm working on open source projects since four years. And I'm doing Python tooling related stuff, PySource Minimize or CodeGen, where I find also bugs in CPython, for example. But this talk is actually not about finding bugs in CPython, but about Inlet Snapshot and how you can use it. I want to start with a short introduction of what snapshot testing in general is. You all know normal testing, assertions where you have equal operator and you test stuff. The difference to snapshot testing is that you do not write your expected value in a snapshot testing but only the thing you want to test, in this example 1 plus 1. The Snapshot testing library records the given value the first time the code is executed and stores it somewhere. For example, in an external file or inside your source code or in a database, this really depends on the Snapshot library you are using, but the concept is always the same. The Snapshot testing also allows you to record the test later again when the value has changed And I now want to show you how I have implemented this in Inline Snapshot. Inline Snapshot provides you one function. And this function is called Snapshot. And you can use it instead of your expected value. And when you run PyTest the first time, the expected value is put in as the first argument to the Snapshot function. and it's stored there. The code is generated basically by calling rapport on your object and this allows you to create snapshots. What you can do later is you can, for example, if the behavior of the application has changed because the requirements have changed, in this example I have replaced the one with five. Inline Snapshot tells me hey your snapshot value has changed here's a diff do you want to approve these changes and apply it to your source file. When you say yes the value is changed in the source file and you can continue the test because you pass again.

Speaker 2 [02:42]

Um, and

Speaker 1 [02:44]

In my following slides, I will use simple examples like this because I want to focus on the snapshot testing. But the real value of snapshot testing comes when you have more complex examples. I have a quote from Sebastian Ramirez. Inline snapshot is also everywhere in the tests for FastAPI and my other projects. It has enabled real complex migrations and internal defectors. Inline snapshots plus and equals are an amazing combination for complex tests. And to give you an example how larger snapshots could look like, I have this for you. Here a request is done to this endpoint and the response is converted to JSON and compared with the snapshot. You can see that you can store a large list, dictionaries or other things inside the snapshot. But these examples are really complicated and big for normal slides, so I will stick with simple examples. There are some rules which are the core of Inline Snapshot. The main one is that actual snapshot creation and snapshot comparison are two different things. You can create your snapshot for example also in the global scope and assign it to a variable. The snapshot function takes a value and puts a thin web around it which has an equal operator which is customized. This equal operator is later used when you compare your snapshot with the actual values. You can also compare it multiple times. you can also do is you can pass the snapshot value inside other functions for example here and to check call. This enables you to write custom functions which you can use to test specific things in the application and to abstract away different testing code and call it multiple times if you want. What follows from this rules is you can use inline snapshot for example

Speaker 2 [05:02]

Um...

Speaker 1 [05:05]

PyTest parameter-wise, you can just put it there as a parameter, in this case as a result parameter, and you can use this result later on in your test, and every time the snapshot is compared, the value is stored inside the snapshot argument, and it just works. There's no special code for PyTest to make this work, it just comes from the design of Inline Snapshot. You may ask what you can store inside Inline Snapshot. Inline Snapshot supports all custom data types from Python, like for example lists, dictionaries, sets, tuples and enums. It also supports data classes, pedantic models and others. And basically every Python object has a representation of the object is valid Python code and it also has a way to customize the representation in case your object is part of an external library where you have no influence how the representation looks like. You can use the customization to generate valid Python code or the code you want. And this example below shows how a data class is stored in cssnapshot here. Well, the code before talked about .equals. If you don't know .equals, .equals is a library which is really useful for cases like this. The problem here is that we have a data function, and this data function returns us a user Bob and an ID. The problem is this is a UUID and it will change every time this test is executed and will make it basically fail every time. So the way you can solve it is by using .equals and you can put this .equals directly inside the snapshot. And this allows you to customize the snapshot and make it less strict. And what's also important here is that Inline Snapshot supports it that when the user Bob, for example, changes name, Inline Snapshots will also change the name of Bob, but it will not change your SUU ID which you have added here. What's also possible is to use Snapshots inside Snapshots. In this example, I have the product and the product is converted with is upper to an uppercase string, but the problem here is that I have a really long header.

Speaker 2 [07:50]

Um, um,

Speaker 1 [07:51]

And you maybe have similar cases where you want to test if the header is actually provided, but you don't want to repeat it everywhere. An Inline Snapshot by default would create a snapshot this way and repeat this header everywhere. But what you can do is you can create an extra snapshot in a global scope. It can be empty by default initially and you can use a snapshot everywhere this header is used. is used. An Inline Snapshot will then, when the equal operator is called for this header, Snapshot records the correct header and this allows you to store the header only once in your test and make sure that every time you compare this header, it's actually the same header and it will also allow you that the header is updated when it's changed for some reason. Another use case for Inline Snapshot is when you have tests where you have an assertion like this. The problem here is that this assertion evaluates every time to another value. When you would use a normal snapshot, the problem is that you cannot use the same value, the same snapshot, because the test will fail the second time this assertion is called. What you can do instead is to use index operator to create sub-snapshots. These These sub-snapshots are represented as a dictionary, like this. In this case, the indexes are converted to the keys of the dictionary and the values become your new snapshots. You can use this to generate snapshots on the fly. amount if you want and Interesting thing is that the sub snapshots have same properties like normal snapshots You can use it for comparison but you can also use to pass to different functions or you can actually use them and create another subset sub snapshots from the snapshots and Create sub snapshots inside sub snapshots if you want I guess that I would this talk would take much longer well I have much more time but I have

Speaker 2 [10:41]

I have

Speaker 1 [10:42]

I skipped some things about Inline Snapshot, which I want to talk here shortly, more or less, a little bit longer maybe. Inline Snapshot has three more comparison operators defined.

Speaker 2 [10:54]

Uh, Lord.

Speaker 1 [10:55]

lower equal and greater equal at n

Speaker 2 [10:59]

Um

Speaker 1 [11:00]

You can use some if you want for example lower equal and greater equal if you have some code You want to optimize and it takes different steps and you want to minimize the amount of steps which are needed You can say that the amount of steps is should be lower equal to a snapshot and your snapshot is able to minimize or shrink to the given value and cause an assertion if a value is launchers in your snapshot. It basically creates a list for you where you can say this warning for example has to be in this list which is useful if you have multiple warnings or error messages which you want to check. You have also the ability to write custom code with this. This allows you you to replace parts of your snapshot with custom expressions and let's not until and you tell them that's not this is

Speaker 2 [12:06]

Um, is that?

Speaker 1 [12:08]

That it should not touch your written code also not when you and the value actually does not match There's another function external external allows you

Speaker 2 [12:21]

Um, to

Speaker 1 [12:22]

to write a file to disk. And this function is currently under development again because I want to... make it better for the future and add some features and that's the reason why it's really not part of this talk because it will probably change in the next weeks and I have also several utility functions like prints, raises and warns. They are context managers and they can be used to check if your application prints something to stand out or raises an error or warns about something.

Speaker 2 [13:01]

Um

Speaker 1 [13:02]

All of these context managers accept snapshots as arguments and can be used this way. If you want to know more about these detailed features of inline snapshots, you can check out the documentation. There is also a quote. Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. You might know this code.

Speaker 2 [13:35]

Um

Speaker 1 [13:38]

But you ask yourself, I have talked the whole talk about one function, what can he take away? And there is actually one thing which I was able to take away, and this feature is available as an insider feature for my GitHub sponsors since two weeks. And the thing which I took away is exactly the snapshot. This insert feature allows you to use ellipsis object of Python when you want to create a value and it allows you to fix normal assertions. These normal assertions have to be of a specific format. For example, the value has to match a specific heuristic which was detected as a value for a line snapshot. But when this is the case, and this is usually the case for most of your tests you have written out there, you're actually able to use Inline Snapshot with any existing codebase out there and have the ability to fix your assertions. This is really cool because this requires no extra API, AI, which allows you to use this feature with existing libraries without adding Inline Snapshot as a test dependency. This means if you want to sponsor me, for example, you can use this feature with existing libraries without enforcing that other people also have to use Inline Snapshot. And that's it. I would like to thank my sponsors and also everyone who uses Inline Snapshot. This talk was a little bit shorter than expected, so we have a lot of time for questions.

Speaker 2 [15:39]

Um

Speaker 1 [15:42]

Thank you.

Speaker 3 [15:50]

Yeah, thank you for the talk. And now we can ask the questions. So the first question would be, can you repeat why are snapshots useful? Which problems do they mitigate?

Speaker 1 [16:05]

That's true The main use case for my snapshots are useful is because it

Speaker 2 [16:15]

Um

Speaker 1 [16:17]

change a bit the game of the player here. Usually you have to invest a lot of time to write your test and to change your test when you change the behavior of your application. But what Inline Snapshot offers you is the ability to change a lot of tests in one go when you have, for example, added a new property to your API or when you did slight changes to something or changed wording for warnings or something like this you can in one go fix a lot of tests without going through the manually and this is a big time saver basically as a main use case is saving time

Speaker 3 [17:00]

And are data frames fully compatible with Snapshot?

Speaker 1 [17:06]

I have...

Speaker 2 [17:08]

problem

Speaker 1 [17:09]

project inline snapshot pandas

Speaker 2 [17:12]

Thank you.

Speaker 1 [17:13]

where I try to integrate DataFrames into InlineSnapshot, but the problem there is, which is solvable, but the problem is that DataFrames, when you compare them, return something which is not a Boolean. What I did there is I customized the AssertFramesEqual function and the way that you can use a Snapshot as a second parameter which then will work for your data frames and the data frames are represented as list with dictionaries or something like this.

Speaker 3 [17:58]

Do you think it's a downside and how can you prevent snapshots from being abused to not think about the tests but to generate them automatically, the expected values of the tests?

Speaker 1 [18:16]

downside how can you

Speaker 2 [18:20]

Um

Speaker 1 [18:26]

You have basically two views there. It could be a downside, yes, but it could also be upside because usually when you write a test, you only think about one specific thing you want to test and you maybe forgot a lot of other things because they are more complicated, more to test, more to write and you say, oh, I have tested this and good, fair enough. But when you are able to, what Invalid Receptual enables you is to write larger tests to cover more of your code and to cover more of the expected behavior. And you can always make this snapshot more precise or more general by using dot equals, for example, and customize it there.

Speaker 3 [19:15]

So how do snapshots differ from writing fixture files for the expected values of a test?

Speaker 2 [19:25]

um

Speaker 1 [19:27]

Fixture files. Pytus fixtures or custom snapshots.

Speaker 2 [19:33]

um

Speaker 1 [19:41]

Big benefit is that you have a general general API which you can use and for all your tests and do you don't have to why it's all the serialization and storing of something and files by hand if you want to otherwise test and write on the expected values to files, for example, you can skip this part and use an SNAPSHOT there and And, well, the mentioned external implementation.

Speaker 2 [20:12]

um

Speaker 1 [20:13]

We will solve a lot of these use cases because it allows you to store binary data and larger PNGs, for example, in external files, if you need this, for example. The external implementation is more designed for this not representable data, which cannot be really put into source code.

Speaker 3 [20:36]

So how does the inline snapshot play with version control? Something like it.

Speaker 1 [20:41]

The real benefit is that your snapshots are really written inside your code and what you maybe saw is that the snapshot function is only a thin wrapper around your value, in this case your 2. This allows you to...

Speaker 2 [21:06]

So, uh, season.

Speaker 1 [21:06]

see the snapshot function as a no operation and want your value and in fact inline snapshot has ability to disable the snapshot testing

Speaker 2 [21:16]

Um...

Speaker 1 [21:17]

which means that the snapshot function will return the user's original value. And this means also that you can, for example, if you really really want, remove all the snapshot calls from your codebase and you have basically the same code which you would write by hand otherwise. I would not recommend you to delete all the snapshot calls because they're really useful. But if you really want to, you can do this. And there's basically no vendor logging or something like this.

Speaker 3 [21:46]

So one question that also goes into this is how is the code modified to update the snapshot value?

Speaker 1 [21:55]

The internal implementation uses executing, executing is a library by Alex Hall.

Speaker 2 [22:00]

Um, uh,

Speaker 1 [22:03]

can be used to map a specific bytecode instruction back to the AST node of the abstract syntax tree and I use this information to find the call of the snapshot function and put the code.

Speaker 3 [22:22]

One question was also from your point of view, which way is better, using raw values like strings inside the snapshot body or comparing directly to variables from test setup stage?

Speaker 2 [22:37]

Um, I...

Speaker 1 [22:42]

What I see sometimes is that users convert their snapshot into strings and use this for comparison. I would not recommend this. It is better to use your list of dictionaries directly because it enables you the use case of 30 equals, for example.

Speaker 2 [23:06]

Um

Speaker 1 [23:08]

I hope this answers the question.

Speaker 3 [23:15]

Can these value snapshots be applied in other use cases other than PyTest?

Speaker 1 [23:23]

I hope so.

Speaker 2 [23:24]

Okay.

Speaker 1 [23:26]

I have a vision in mind and I hope that I am able to make it also work with unit tests.

Speaker 2 [23:33]

Um...

Speaker 1 [23:35]

And there are also other testing libraries out there, but they are not very widely used.

Speaker 3 [23:46]

And can you give some more example cases, maybe a bit more complex cases where snapshots excel in usage?

Speaker 1 [23:55]

We are snapshot with

Speaker 3 [23:56]

Snapshots are very good.

Speaker 1 [23:58]

This is the documentation. I have internet here, sorry.

Speaker 2 [24:23]

Um

Speaker 1 [24:26]

I have no other examples here, but if you want, you can look at the code basis of LogFire. In FastAPI, Patentik is using a lot of snapshot testing, there are large examples and big snapshots with a lot of dead-equals usage and you will find your complex use cases there.

Speaker 3 [24:54]

And what is the difference between snapshots and what Python regressions does?

Speaker 1 [25:12]

Next question.

Speaker 3 [25:13]

Okay, and next question is how easily can you write functions similar to isUUID that dirty equals provides?

Speaker 2 [25:22]

Um

Speaker 1 [25:24]

Actually really easy. The only thing you have to do is to write a class which overloads the equal operator.

Speaker 2 [25:35]

Um

Speaker 1 [25:37]

And that basically is everything you have to do. Inline Snapshot by default will not treat it like DirtyEquals, but the basic code is there and I can expose the functionality for you to write DirtyEquals similar objects.

Speaker 3 [26:05]

And what is the advantage of inline snapshots versus writing and updating a snapshot file?

Speaker 2 [26:13]

Um

Speaker 1 [26:14]

It's easier

Speaker 2 [26:15]

Thank you.

Speaker 1 [26:17]

Well, if you again if you want to write an update snapshot file by hand you If you write it by hand you have Serialize your code and use it use it that way or you can use or maybe basic question is snapshot files with other libraries The problem with snapshot files in general is that you have always in direction Every time you want to review your code or to read your code, you have to make the connection between your test and the snapshot file. And this is really time-consuming when you just want to read your test or approve the changes. And Inline Snapshot is really useful here because it really just eliminates this interaction And it has also nothing to learn if you want to read a new test because everyone who reads this you can just tell them ignore the snapshot function, adjust the value and you will be able to understand it and it does not have to learn any new format where the snapshot files are saved or something like this.

Speaker 3 [27:30]

How and where exactly are the snapshot values saved and what considerations should we have regarding this, especially also with potential merge conflicts in version control?

Speaker 1 [27:43]

Is the snapshot, is the snapshot faster saved?

Speaker 3 [27:48]

The snapshot values.

Speaker 1 [27:49]

The snapshot values are saved inside your source code exactly where the snapshot function is as a first parameter.

Speaker 2 [27:58]

Um

Speaker 1 [28:00]

I think I Well here for example the two and this is a code is generated by Inline Snapshot and put exactly there and yes, that is the version control can save it and

Speaker 3 [28:17]

And what can someone do if they want to load a snapshot from this because it is very large, like a large numpy array?

Speaker 1 [28:29]

This external implementation is exactly for this use case.

Speaker 2 [28:35]

um

Speaker 1 [28:37]

It will allow you to read files from disk and use them like normal snapshots. It will have a de-serialization and serialization mechanism, which means you can also read data frames, for example, as pocket objects or so, and compare them or use them. Or pictures, for example, or other binary data or whatever you want. So it will be a customizable option. you can define the encoding or format.

Speaker 3 [29:09]

Thank you. There are still a few more questions left, but our time is already up. Maybe you can come to the front and ask the speaker.

Speaker 1 [29:16]

Yeah, after the talk.

Speaker 3 [29:18]

So thank you, Frank, for the talk. Yes.