Prototyping to tested code

Speaker 1 [00:03]

Yeah, thank you very much for the introduction, and thank you for being here and I hope I won't keep you from lunch, so I will try to stick it to the time and Just quickly actually I nobody seen doing this, but just officially everything. I'm talking about is my own opinion have nothing to do with my employer Maybe if it's good, please attribute it to my employer otherwise. It's all my own opinion Maybe quickly about me. I'm a physicist by training actually I studied here in card school My first job was also at Blue Yonder, which you may have seen by now at a couple of slides. And currently I'm working as a data scientist in Volkswagen in the data lab in Munich. So we have a small part there that's doing data science for the whole group, essentially. And ever since I started in this scientific computing stack, I have been a huge fan of Jupyter notebooks and use them essentially for everything. So in particular, when I start something up, usually the first thing I do is open a notebook. But I have to admit I'm also somewhat conflicted about notebooks. I don't know whether you ever had any issues with it, but I think there are tons of them. So it's like a love-hate relationship. I think I'm not the only one. So, for example, there was this, on Twitter at least, controversial talk by Joel Grues, who said, I don't like notebooks, gave it at JupyterCon, and I really recommend to check it out. He actually has a lot of valid feedback, valid criticism, maybe also alternatives to the notebook workflow. So please check it out. And there are two things in particular that will come up in the following minutes. One is like notebooks have tons and tons of hidden state that is easy to screw up and difficult to reason about. So, for example, you have these global variables that you maybe mutate, then you go back to cell, and then you no longer understand what you did and how you ended up in the state you are in now. And the other thing is what he's missing is some kind of first-class support to refactor code out of notebooks into modules. And so I can't help you with the first part, but I want to show you how I personally work with the second part and how you can combine testing in this regard. So my take on it is essentially notebooks are hard, so I'm regularly confused by global state. Sounds like an admission, essentially. So, for example, when I refactor a function or code into a function, I typically forget somehow that some parameter actually has become a parameter, no longer a global variable, and suddenly I want to rerun it again. I don't understand where the results come from. I have the feeling that Git and notebooks do not mesh well, so diffing, merging can be hard. I think there are tools for that. But then, for example, your web UI is not configured to show notebooks, and then you have the issue there. With, of course, GitHub it's different, but with internal tools it could be the case. And also personally speaking, my notebooks always become messy over time. So you have this giant file that essentially contains everything, and you add to it, you rearrange code, you maybe only want to execute part of it to do a quick experiment, and suddenly it's this giant mess of spaghetti. At the same time, notebooks increase my productivity quite a lot. So you have this interactive environment where you can play around with your code, you can explore it, and you get this rapid feedback cycle. And what I also really like is that you have this documentation feature built in, so you can write these self-contained narratives where you go maybe from a paper that you want to implement to the data and then to the implementation itself and show the results. And also math comes really in handy if you have a way to typeset math in those systems. so and now there's this question actually notebooks versus modules is this actually even a question i try to use what is most efficient i think both have their valid use cases and i think you can also combine the best of both worlds i think it's nothing new probably many people do this already by moving code progressively out of notebooks so you start in a notebook but then like move code to external files into packages into modules however for this to work you really need to use the same libraries and tools inside and outside of a notebook so if you need to rewrite everything because for example your library doesn't support or only works in a notebook or doesn't support a notebook then suddenly it becomes very very hard to do this workflow and for me there's this notebook module continuum how i called it if i make kind of a new term and so for example plotting there are different libraries these are The ones I use, probably the others that you use. And for me, this is pretty much always a notebook task. So I start in a notebook and stay in a notebook. I hardly ever write modules or scripts that generate plots, personally speaking. But then, for example, how about distributed computing? Maybe we have a large data set that you want to analyze or maybe need to preprocess. And even then, typically, I start in a notebook. And, for example, Dask and PySpark are now very well integrated with notebooks. you can just define your code locally and it will just run also when you ship it for example to worker nodes or you use this in a single on a single machine and it works as well and so i would maybe prototype in a notebook and then you can move it out and somewhere you're in the middle right you have parts of your code that is maybe still interactive you want to use live and you have parts of your code that you want to maybe have in modules that's part of a data engineering workflow there's maybe you want to run machine learning experiments there's this library called MLflow. They also talk a lot about how to use it in notebooks, for example, in the documentation. And you can use it to start the experiment in your notebook, and then when you're satisfied, you move it outside, put it on a cluster, run hyperparameter search on it, or just try out different variants of the same model, and then you get this nice dashboard where you can analyze itself. And what I'm personally quite excited about, I haven't tried it yet, is this library panel, which should allow you the same workflow for dashboards. So you start prototyping a dashboard in a notebook and then you can just move it outside and it's running as a standalone service and i guess there are many many more examples of these libraries that are both designed to be worked used inside a notebook and outside a notebook and but what i want to talk about in the following a bit is like how this looks in practice how you can set up your environment so such that you can copy code from a to b and it still works everything and maybe also do it progressively. And I also want to specifically talk about testing and how you can write tests that you can start in a notebook and then move outside. And I've written a small package called IPyTest that allows you to do this in PyTest. I will also go a bit into the implementation detail, but it's honestly a very, very small part of code. So to get started with this testing thing in notebooks, you need to install PyTest and IPyTest. um in the beginning it was actually very much separate so ipy test doesn't come from py test but from ipython so uh there was no py test support in the beginning so this is why it's still not a dependency um but if you install them separately it will just pick it up and work as expected and there's a small amount of configuration involved um so first i would really recommend you to use uh magics um that i supply probably maybe you also have your own opinion on that, but personally it's quite effective for me. And to enable them, you can import this Magix module that I shipped. Then second, I would really recommend to use assertion rewriting of PyTest. I think it's one of the major setting points. I will go into this in a second and show an example. But you also need to enable this manually, because this will become the default, but for backwards compatibility, even though there's probably five users of this package, it's still hidden behind the configuration option. And finally, you need to make the notebook file name available to IPyTest. The reason is that the kernel knows nothing about the document you're writing. So due to the setup it's built, essentially you can't know where you're running in, so you need to supply this manually. And then if you want to write your first test, you use this PyTest convention of defining a function that starts with test underscore. in this case I assert that the numbers 1 to 3 are equal to 1 to 3 I add this decorator on top of it and suddenly you have your test that is now running inside the notebook so everything you see is actually a notebook that I executed before generating the slides and this is now using pytest inside the notebook to execute your test you've written the main API that I defined with this ipytest package is this magic command it has three parts I guess the first part, run PyTest, is quite self-explanatory. So we will use PyTest to execute our tests. And now already the issue with global state creeps in. So typically, I refactor my tests. So I start with this very general test example thingy, and then I rename it, become more specific. And you are left with the old test and the new test, and everything is in this global state, in this global namespace. And PyTest will find any test you ever defined in this notebook. And this is why I always delete any tests I defined, just define them in new, and then run only these tests. And IPyTest ships with a screen command that does exactly that. So it will delete any tests you ever defined, and then you can define your new test in your cell, then execute the cell, and only test the cell essentially itself. And finally, you can pass arbitrary arguments to PyTest. PyTest has a very rich command line interface. I typically use this minus QQ to quiet down the messaging, messages, such that when everything works, it's just a single dot or this 100% you've seen before, but you can also go the other way and essentially increase verbosity and do other things. I will talk a bit about it in a second also. There's also a more like classical function interface to this whole thing. If you want to know more, check out the GitHub page. It contains everything in the README, and is hopefully somewhat self-explanatory. If not, please raise an issue and i can improve the documentation so what is working of pi test this is somewhat of a tricky slide i have to say because most of the stuff is actually done by pi test i haven't done most of the stuff but it's just nice that it works um so everything i tried so far is is working however i should also say probably my workflow is a bit limited in particular when it comes to plugins maybe there could be issues that they are in there so if you encounter anything please let me know what i typically use is marking features for example to skip tests or to parameterize tests to mark known failures you can use the fixture system both built in like the temp directory or your own fixtures you can attach a debugger i will demonstrate this in a second you can list local variables there are many other things you can do and most importantly i guess this is assertion rewriting that pi test chips with so how does this look like when you have a function this will be somewhat of a running example in the following this keep odds which should take an iterable and keep all the odd elements in this case i made this huge error of only returning the even element so this is like wrongly implemented such that the test will fail and if i now write a test and want to assert that the odd elements from one to four are one and three, that would be correct. I get this test error due to this function error, implementation error. And what you see in the following is what PyTest gives you is the return value of the function, which is very nice because typically you only get that you called this function but you never know what came out of it. You get also the right-hand side in case this is another function call, and then you get this detailed diffing that tells you where the issue is. And PyTest is doing this by rewriting your code behind the scenes. But this also works in the notebook, like I said. You can also attach a debugger. This was probably the biggest surprise to me. So what I would really recommend is to use this minus X command line flag over here. Otherwise, it will attach the debugger to every test that fails. And typically, if one test is failing, many tests are failing. and then you have like a lot of queues to press and so i would add this minus x and then minus minus pdb which will then attach inside the notebook uh the pdb uh debugger and then you can debug your test inside uh inside ipython inside the jupyter notebook in jupyter labs there's a strange bug the way it's displayed but it should also work there in principle and uh yeah and then with queue you get out of here but in principle you could have used up down jumped into definitions or printed variables or expressions. So I kind of wanted to quickly talk about IPyTest. So honestly, it's a very, very small package. Most of the code is actually to support unit tests, no longer about PyTest. And it has, I would say, it's somewhat of a creative use of extension APIs of both IPyTest and Jupyter. So both have a very rich plug-in system. and if you kind of like use it creatively you kind of can fill it together so for pi test you can define these plugins and essentially all it does is it like fakes that the module you're running in the notebook i don't know whether you're aware but everything you do is reflected in the underscore underscore main underscore underscore module and what this plugin is doing is essentially it's faking that the import or the test of the notebook runs against this main module so this is why you also have this global state issue that i talked before so it will not be redefined but it will run against the current state of your notebook and this plugin is essentially faking that part and now you also need now to integrate pytest into into ipython and like i said ipython has this very rich api for extensions and what is actually really cool and really scary is you can define arbitrary AST transformations so you can essentially arbitrarily redefine your code before it's run so you have what you see on the interface then it's passed and then you can completely re-transform it and then execute that code and in this case I'm hooking in the PyTest part that is essentially rewriting to add this assertion mechanism but if you ever want to bother your colleagues I guess there are plenty of options of completely freaking them out by changing anything in the background but now to the topic at hand maybe a bit and how uh i do this notebook module uh thingy and how i navigate this continuum how i i coined it um there's some setup required so typically my directory layout i try to isolate the notebooks as i said they become messier over time and then i want to have like this bad bank of exploration code maybe um i highly recommend that you define your requirements. I recently found this tool, pipenv, that also Patrick talked about yesterday, I think, that I really like, and it gives you two files, one with the concrete and one with the abstract requirements. And then you will need to have somewhere where you want to move your code to, so you need to set up a package that you will then use later on. It's actually quite simple. I like the source directory convention, but I guess there are also other conventions, and you need to add It's a setup.py file. And then, of course, we want to move our tests also outside the notebook. The pip file itself, the main trick here is essentially that you install your local package in an editable way so you can reload it live so you don't need to always restart the kernel, which then takes quite a long time. And the setup.py file essentially only defines the name, the version. a shout out to Patrick again who told me that you should use PyModules if you have a single file which I wasn't aware of before and then for this package dear thingy due to this source directory convention and now you can install your local package and it will be found by the Python interpreter so now I want to quickly walk you through how you can move start in a notebook develop prototype in a notebook and then move everything outside so we again have this keep odds functionality this case defined correctly and then i typically play around with it in a notebook so if i have code written i try different input output pairs input input arguments and then you see the output directly and in this case it's easy to verify just by eye and logic it's yes this is what i would expect so how do we turn this into a test now in the same interface you're still in the notebook you have this into exploration pair you just wrap a function around it you add the decorator the magic command to to execute the test and suddenly you have a test running against what you just essentially generated as an input output pair if you now want to start moving code outside you can essentially copy the code the definition of this keyboard into a file so it's now sitting in source keyboard dot pi and now you need to slightly modify your test so first of all maybe the first obvious or maybe more easier part is you need to import now from the module where you moved your code to and now suddenly the import system of python is another global state so it will cache your imports and what you actually need to do is essentially is to invalidate this cache um i think there's this auto reload option of ipython i never really used it lots of people however i told about this told me i should use it so maybe this is an option for you i'm a bit uh i didn't want to change too much for this talk so i'm using this simple reload function where i can pass a list of strings and can manually reload but probably both ways uh kind of work and and would mesh together but now you reload your module you import your code from the module and run your tests for example still in the in the notebook which is also nice because you can also test for example existing code and maybe add new functionality if you have errors for example and you want to explain these with a test and finally you can also of course move out the test and the important part here is that we didn't change that much right we removed the reloading part if you use auto import you probably don't even need to do that and remove the magic line to execute the code otherwise it's the same exact code as before and this happens of course because we use the same tooling inside and outside the notebook and if you run pytest it will just pick up the test in principle you could also use the magic command to execute against against the directory so how well does everything fit together and does it work, everything. Personally speaking, for me, this workflow of moving stuff out of the notebook works quite well. I think the developing package and reloading part is a bit of a manual setup, but overall I'm quite happy how it works. What I also really like is that more and more libraries ship out of the gate with support for notebooks. They're tightly integrated into the display system. They take care that you, for example, ship local code to a remote executor like PySpark and Dask are doing, which really simplifies a lot. I think where, personally for me, where things start to break down a bit is development inside a notebook. And maybe a quick caveat here, I know this stuff is really, really hard and I don't want to say the guy should work harder or something. And if I would know how to fix this, I would also contribute stuff. It's just an observation, so it's not blaming anybody. But I would really like to have more support to reason about global state. So, for example, maybe something like highlighting global variables would be really helpful maybe. I think also there are still a lot of holdouts which make it really, really hard to execute your code in a notebook. So I would like for this to change. And also the tooling inside the notebook. It starts with type checking, completion, refactoring. It's mostly manual, I think, nowadays. So this is a bit tricky. and the last point maybe is uh keeping everything and things and suddenly becomes an issue so this is like where i struggle the most currently so i moved out all the code into no into modules but now maybe i want to move it back because i want to have everything in one place my experiments my code and this involves this huge like hunting for parts that need to go together and copy them manually um regress and checking for notebooks so i'm thinking about maybe running something on my own but you now changed your modules are your notebooks still functioning with your changed modules this is also somewhat unsolved and in general this notebook and package aware tool is that you can more easily integrate everything into one cohesive workflow so to conclude i think notebooks offers this very effective environment for for rapid iteration what's actually nice about it is you can also generate these input output tasks for pairs for tests quite quickly because you can inspect everything manually and by eye and by hand, essentially, which is manually. Notebooks can become cumbersome for large codebases. I think some people were nodding when I was saying this in the beginning, so I think I'm not the only one. I would really recommend to move code out progressively. So as you start with a notebook, then move out code to packages. For this to work, you should really take care that you use the same tooling inside and outside of notebooks. I hope I convinced you that IPyTest could play a role, at least for testing in this regard. And there's still some hidden state that is lurking around and that you need to be aware of. So it's not really very simple to use. I think it's very simple to use, but it can go wrong in very surprising ways if you're not aware of all this global state that is in the background. And finally, I would really like if some of you would give it a try. If there are any issues, any feedback, either contact me on Twitter, open with a GitHub issue. I will try to fix it or address the issue. So thank you very much. Of course, I'm also happy to answer any questions.

Speaker 2 [22:50]

Thank you.

Speaker 1 [22:52]

Just in case we get out of this

Speaker 3 [22:55]

And yesterday I just learned in another talk about a thing called NB-VAL, which...

Speaker 1 [23:03]

has a similar

Speaker 3 [23:04]

similar goal, but has a different approach, because you are starting to write your tests in the notebook and then moving them out. They seem to use the notebook as the test, because you already have the information that's the input, and the notebook also knows the output and stores it, and so they use this as the test. So for your problem keeping your modules in sync with your notebooks, if the notebook is like your documentation of how to use it, maybe this is a thing to look at. So just as a comment, but thank you for your time.

Speaker 1 [23:37]

Thank you very much for the comment, by the way, before I quickly re-comment, so to say. I'm aware, and I think it's a very nice solution. At the same time, I really try to keep the notebooks. In the end, it should be like a hollow shell that only contains maybe some explanatory plots and maybe explain the results. So this is why it doesn't really fit my use case, but I think if you want to keep more stuff in notebooks and test the notebook itself, I think it's a very valuable solution. Thank you very much.

Speaker 2 [24:02]

Actually, I have a comment about that. My personal opinion would definitely prefer your approach to NB-VAL, but that's my personal opinion. That's because if you check out that talk, which is very interesting, the one he mentioned about Joel Bruce, he basically showed a very nice example of having a notebook in which you have the output, then you change the text, You saved the notebook, but you didn't execute the cell again. So if you go that way, basically you have an output of a cell that will fail, so a failing test, which is fine. But in this case, you have assertion to control that. So that's my personal feeling on that.

Speaker 3 [24:52]

I just want to come back to the nbval discussion. I'm one of the authors of that package. It's a different use case. So with nbval, we're not aiming for creating unit tests. It's more like a free value you get from having the cells. And this is actually better to export your code and create audios because you're writing properties. So it's a different thing. I think they are complementary.

Speaker 1 [25:13]

At the same time, I think this regression part is probably covered by this to some extent, like checking that your notebooks still work.

Speaker 3 [25:23]

Yeah, thanks. I tend to notice that sometimes someone like Code in your notebook sticks there and uses some parts of your code which is then outdated. So I'd like someone like Notebooks to be integrated into my build chain. So tests in the notebook to run automatically in the build chain. Is that possible with the tool?

Speaker 1 [25:42]

and so you still would need something to run your code so it's a bit hacky and currently also exceptions are swallowed but i'm working on surfacing them so i would really not recommend it but i would rather check out something like mbval for example that gives you exactly this use case where you want to ensure that your notebook still runs and gives the same results as before but in principle like i'm working on it that also for example a failing test will raise an exception and then you can just check that your code for example with paper mill still runs and if there's an exception also paper mill would stop and that you can then detect in your build pipeline but currently as of now it's not the case but it will be in a couple of days

Speaker 3 [26:23]

days. Okay, thank you.

Speaker 1 [26:23]

okay

Speaker 2 [26:27]

Is there any other question?

Speaker 3 [26:32]

Just a quick comment on the same kind of line.

Speaker 1 [26:33]

kind of line so in a few days I'll

Speaker 3 [26:35]

In a few days, I'll be able to

Speaker 1 [26:36]

I'll be able to run PyTest from the command line and it'll pick up notebooks as well as test modules? No. Okay, because that would be awesome. I think there's actually some plug-ins that do this in PyTest. Like I said, I guess it's more of a different use case. I don't try to use my essentially notebooks as an output, like an artifact that stays there forever or also describes code, but rather as a documentation of what I did and in this case I would still like to verify that it still runs through if I re-execute it but I don't want to make it part of my testing chain for example. But I guess there is some example but potentially also this is again solved by mbvar to some extent. Okay, thank you.

Speaker 2 [27:25]

Another question? All right. So thank you very much for a fantastic talk.

Speaker 1 [27:32]

Thank you very much.