Documenting R&D Progress using jupyter-book - and feel safe for the next performance audit

Speaker 1 [00:05]

So thank you for that wonderful introduction to the talk. It's now just me standing in between you and the lightning talks that we're all here for, so I better be quick. You might have seen that new sponsor name there and probably already asked yourself, how the heck is that pronounced? And that's RosenNext. And even RosenNext might bell whistle a bit, I think, because it sounds like Rosen and there's a reason for that because all we at Rosen Next have been Rosen before. So there is a new part. Rosen itself is the market leader in the pipeline inspection business and Hermann Rosen has... Could we have the loudness a bit up, I think? Is that fine? Can everyone hear me? not so much may I yell or something so they are leader in the pipeline business the pipeline inspection business and Herman Rosen the founder had developed it to that point and there are a few ventures in that company that have new ideas that don't really fit that much into the old business and Herman Rosen at his age of 77 years said well let's just found another company and drive these things there so here we are with Rose Next we are already more than 400 people worldwide as you can see a carve out from the old company you can find us right everywhere on the world and we have some interesting fields that are not that very similar not very obvious like renewables or integrity solutions that is probably what which is the old business the most. And industrial wood is in front of that, like before the pipelines are built. There are some processes where we are involved already. And subsea and oil field, and also special products. And if you're interested in where I'm working, come and talk to me after the talk. I would be keen to see where you would put me in. And we have the usual buzzwords that you might expect these days, like artificial intelligence, of course, and autonomous robotics and smart electronics. There's probably a bit of big data as well in the business. It's quite interesting what we're doing. And that's already it for the company, but because we are here for something very different, and that is documenting R&D progress. We've chosen a special way for that. We are very motivated in trying out new things and make rapid progress with our ideas. So we can't wait to do the next steps and see the results and do quick evaluations and then move on from there. What stays behind is the documentation that you would need to be safe for the next audit that is sure to come. and we found probably we found a good way to integrate these fun parts and also documenting things and that's what I'm trying to show today so why all this we've just been founded working on cutting edge ideas we work to make progress fast and documentation is important and the performance order will be there at some point what we decide for is using the Python ecosystem as a base for that because we're generating our data from sensor readings we're driving machines with Python on Linux systems and the ecosystem is there the data science stack to show you the data that you just measured it's right at your hands and you want to have these things in your documentation as well and doing that screenshot thing and put it into your Word document that's not what we like it's not for the scientists It's not for the software developers. We like to have our hands on the keyboards and not pushing mouses around. So what is probably better is the idea to stay in the ecosystem. And the Jupyter book project is a quite nice solution for that. As it offers you to just use markdown language, very easy markup language to write your documents. And with some extensions like the column fence extension, you can do wonderful things, very, very easily and create documentations that just look very, very nice. So all we needed to do is to have the Jupyter notebooks around in a structure. It should be repeatable in some sense. And have a file system structure that supports our workflows. We have some new files that are supplementing that work that help us to describe what the thing is about that we are investigating and even write a change log so that everyone knows what has changed in that setup from time to time so that you can cross-check if things behave not normally anymore. And you have that set of obvious things like photos and videos and other documents like data sheets and stuff like that that you want to have at hand if you have the complete documentation for what you are doing. And our R&D is around qualifying components that we're using, like sensors, for example, if they really perform like the vendor says. And for that, we're doing possibly simulations for different scenarios, and we do testing, and we do studies. And all of these things have to be put in one central place and be well organized to build up that book. so for the competences we all use rigorous testing before they qualify many tests have to be passed we do simulations, we do studies everything is documented each and every simulation result is there every test result is there every study is there and that brings us to your next development step that you've performed and that means in whatever direction it might be a step forward, it might be a step back that's how R&D works. All of that can be documented in the changelogs and one central aspect is that the whole thing, the whole processing, the workflow behind creating the book is sort of non-destructive. You can repeat it again, it's like the idea of developing raw photos that you did, that you never touch, you never modify the original image, the original document but develop it to your result that you would like to see. And that's a key thing we're using as well. The raw data that we are measuring, that the sensors deliver, is something that has never ever been touched again. It's always the source for a processing step for intermediate results up until any final results that may end in a database of something and you can always repeat the whole thing. It's like rebuilding the book will reprocess all of the data to each and every development step of each and every study, each and every test that has been done before, again, and it will have a consistent result. So how to do this? One key thing is a well-chosen file system structure. And it's just that. It's as simple as that, not very easy to read, so let's have a look at the details. So there's a root directory for each and every component that we're investigating, and there are some files that you need to make that logbook, make that Jupyter book and that's the config.yaml file that describes your book and you have some introduction as markdown and the table of contents and that's really it, that's where you place your content and it grows and you can recreate the Jupyter book and your book will grow as well and there is a documentation part it's the supplemental documents that we provide like the changelog that I mentioned earlier, the readme file that describes the component that is under investigation. And there are some default folders like for photos and videos and other documentants where you place everything that describes your project. And then there is a file system structure for simulations with that entry point that is just called simulations. And then there is some pattern available that you probably already noticed. It's like simulation 01 and then some date string. And the reason for what is currently simulated, that gives you a number that you can reference on later on in your study maybe to find the raw data that you'll be using. And in it is just some files that describe the simulation results that might or might not have been done with Python, might be some FEM software that generated the data. and we bring it into a form that we can reuse. And, well, yeah, Excel documents are quite handy to read as a Pandas data frame, really. So that's really simple. And there's the part for the test. It follows a similar pattern. There's the root entry for tests. And then you have a subdirectory, again, with a number and a date string and something that describes the location where the test is done and what the reason was for doing the tests. and then in that specific thing that we are looking at that will be a sensor that has been tested for pressure for water pressure and we are interested in some speed dependence for if that sensor is pulled with a velocity of say 5 or 10 or 15 kilometers an hour below water at some level will it really measure the depth or will there be something else that we need to have an eye on and so we have repeated tests with the parameter set that we are investigating and they all end up in these test run directories and each and every test run directory has a pandas data frame in it simply coded in JSON so it can be easily read and easily processed and some Jupyter notebook that is in standard format that is always the same in each and every directory that is just reading in the Pandas data frame, processing it, creating some results, and all of them will be ending in the Jupyter Notebook later on. And then there is this last folder for the studies. Again, following that system, it's a bit easier because it references on data that is already stored somewhere else. So to prepare using the Jupyter Notebook, All you need to do is creating some Python environment for that and install the Jupyter book software in it with the pip install and also install everything in it that is necessary to process each and every notebook that should be part of your book later on. Each and every evaluation, in our case notebook, should be there. And probably some Sphinx extensions because Jupyter book uses Sphinx underneath. So if you, for example, like to add videos to your Jupyter notebooks as well or to your Jupyter book as well, then you need to have that specific extension for that. So to create that book, you simply can use Jupyter book create and then the name of that book. It will create a folder and in that folder some sample files like these minimum necessary set of files, the talk, the intro, and the config, and some examples that you can delete afterwards if they have no meaning for you. Or you can simply, of course, just create everything yourself and be ready to go. And then you fill these files with meaning, like the introduction markdown document with some specific text that describes your book, like the first page that you're seeing that will be part of the book, that will be what you were seeing. And then you configure it like with the config YAML file and that gives you just the title of the book and the author, some copyrights and how everything should be processed like the Jupyter notebooks that are available. If you're just building with a default configuration, then each and every Jupyter notebook that is found below that entry point directory will be executed. That might or might not be a good idea. and to have a bit more control on that there are two options one is a blacklist where you simply state patterns to skip these Jupyter novels that you don't want to have or say only build TOC files true so then you have to reference these things and they have to be part of your table of contents and that is processed in that case and then there is the table of contents itself where you just state the format that should be used. There is the book and there is the article format with pros and cons on your liking. You define where the intro document is and then you start with the paths. In our case, we call it the main path and it's just the two supplemental files that we provide like the readme and the changelog. And afterwards, there needs to be something that recursively scans your directory structure to create that table of contents with all the files in it. That is what you run on a regular base whenever a simulation has been completed or when a new test has been completed or a study is done so that it rebuilds your table of contents and will then update your book with the next call. and this may for example then end in something that looks like this with the main section that we've seen before and a simulation section and a test section and a study section with the Jupyter notebooks and all the markdown documents that supplement your book make that enjoyable in the end and then you simply build that thing and what you then can see is for example for something that looks like this. That's an excerpt of a real Jupiter book that we're using for this very water depth sensor. And that has that main section with the intro file where you have links for the README and the change log and the directories with photos and videos and the documentations and entry points for simulations, tests and studies. And in the README there is something that describes that water depth sensor. There are links to the website of the vendor, the digital data sheet, and a Python driver to use this on a Raspberry Pi, for example, using the I2C bus. Then there is this changelog document where I stated that I have assembled this thing and then attached it to that Raspberry Pi so that it is ready to use at some date. And then we're already in the test section, and there are hints when this has been done and where this has been done and what the aim was. We can describe that aim in more detail, what we were up for in this. We conduct a whole series of combinations of tests at different velocities and different water depths just to see what the outcome would be. And then there is the summary, but also there are all these evaluation documents done. And that's the Jupiter notebook behind it, where all the cells have some meta-information on it that decide on if they are shown in this book or not, or suppressed, and in which way they are shown, so that you don't have all the source code in the Jupyter notebook format available, but just the outcome, just all these things that you need to see in that document. And then there are all sorts of data displays. There's all this Bokeh-based, for example, with the tabs, with the raw data, for the GPS information, for the speed, for the different sources. They are interactive because the bokeh has all the data in the plot, so I can zoom into it afterwards and see the details. And for every component that is interested for us, we see these results and some summaries and some quality checks and some tabular overview on these very specific results. and these results are all just for one parameter in our whole parameter room. Just one combination of velocity and depth. And there are quite a lot of these things that you can see that you have to be doing and it's good to have an automated workflow behind it, right? Imagine having this in a Word document. And in the end there might be a study that is built upon this which just loads all the data in and shows the raw results. And looking at it with the actual probe depth, for example, plotted over the carrier speed, what we like to see, of course, is some horizontal line behind it, some horizontal model that would be ideal. And that is not the case. Physics is bad in that case. But what you can try is to have a correction model, maybe, some simple thing, some linear regression thing. And if you try that, you might be successful and correct the data, and then that's probably one of the outcomes that there might be a very, very simple solution for that problem to qualify that sensor for that very purpose. And with that, we're already at the end of the talk. What we've seen is, hopefully, that there is a good way to use our tooling that we're using, like Jupyter Notebooks and Markdown Text, to create something very visually attractive to help documenting the whole R&D progress that will be helpful in an audit when someone in the end asks us well show us all the data where you have derived your performance from then that's the thing that I will be showing and it's everything there in each and every detail and it is very much fitting into our standard evaluation workflow it's all the same documents it's just a different view on that same thing so i think it's it's a very very good idea and promising way at least for us forward to to work like that and that's a wrap so thank you for your time um by the way we are hiring look at the QR code to find a nice job description where you can be part of our team, especially work with me actually, and with Tobias sitting there in the audience. We will be working together on these wonderful things and all the sources behind it. It will be just great fun. And if there are questions, then now is the time that I may have answers for them.

Speaker 2 [20:23]

Yes, we have some questions for you. It actually looks very promising and very interesting. I personally will give it a try 100% after your presentation. How well does it work together with Git and version control systems? Do you Git ignore some parts of the book, for example, larger files? how you keep track on intermediate results.

Speaker 1 [20:54]

Yeah, you should ignore everything that hurts you. It's not that simple, actually. And still to date, just having all the Jupyter notebooks in your GitHub repository is a challenge. There are solutions for pre-commits that help you strip out all the content to make sure that it's just source code in it. And with a bit of carrying in it and git ignore magic, there will be ways. I would say put the engine behind it separate and then have that in a GitHub repository I'm not aiming for seeing this all in a GitHub repository so that all the measurement data that we're doing all the evaluation would be also in a GitHub repository might be a bit too much and there are different solutions for that if the data has to be under control as well then there are existing solutions for that. That's just for getting things done for the documentation.

Speaker 2 [22:00]

For example git LFS

Speaker 1 [22:02]

Right, that would be a thing, yeah.

Speaker 2 [22:04]

Okay, good. A few words about how much it's reproducible. So can multiple team members work on one notebook at the same time?

Speaker 1 [22:18]

No, that would currently not be possible, and it's not really a thing for us. The most notebooks that are there are all the same. They are prepared once, and they are also developed, and they are replacing older versions of them. What we're looking for is that the evaluation notebook that we will be using works with every data set that we created beforehand, so that I have always this one reference notebook and I can copy it over and execute it again, and that's an automated process, there's no interactive work, it's prepared once on a sub-data set, qualified, and then used throughout all of the testing. And there are maybe studies that might be a thing where more than one is really interactively working on and usually for us it's that one person that is doing that and he will save that work in that directory structure afterwards and then have a chat with me to see these styles should have that meta information and then it will look eventually good afterwards and that's it for us.

Speaker 2 [23:31]

So probably even if a big group of people is working on the same topic, one of them might be responsible.

Speaker 1 [23:39]

was for...

Speaker 2 [23:40]

for loggings as

Speaker 1 [23:41]

logging this book that will probably our go-to solution to start with

Speaker 2 [23:46]

Okay, good. You've shown us something about pip install, and we missed, do this configuration have some place where all these libraries and dependencies which should be installed to reproduce?

Speaker 1 [24:05]

No, it's basically the files that make the Jupyter book are in the root directory of your book. And this will be created, for example, by the Jupyter book command that you pip install beforehand. So the idea is to create a Python environment. It's always a good idea to isolate what you're doing in a Python environment. Use the tool set that you like and install everything in it that is necessary for the Jupyter book. it's pip install jupyterbook and if you're using notebooks in it then make sure you have the notebook engine installed or JupyterLab for example and have everything that is necessary to execute the notebook like for me, Bokeh must be installed otherwise there will be no graphs, just error messages so the JupyterBook is probably a combination of your daily work environment and that JupyterBook engine And you'll probably be fine by just putting the Jupiter book in your execution environment anyways. It's just a few extra packages. It will not hurt your dependency hell at all.

Speaker 2 [25:16]

Okay, thank you. And maybe the last question. Is the example you are shown available somehow?

Speaker 1 [25:23]

If there is some interest in it, then I will probably find ways to make them available. Currently, it's not.

Speaker 2 [25:32]

Or maybe you can give a recommendation for good examples.

Speaker 1 [25:38]

I have not seen so many examples towards that thing and that way to doing things in the Internet. It's probably, I hope, it's a new way of doing these things. So let me know by Discord or find me here, talk to me, and I'll find ways to give an example in a GitHub somewhere so that you can start off with that. In the course of the next days to weeks, don't expect hours or something.

Speaker 2 [26:06]

Okay, good. Thanks to Jans for your presentation. It was a pleasure.

Speaker 1 [26:10]

It was a pleasure. Thank you.