From notebook to pipeline in no time with LineaPy

Speaker 1 [00:03]

I'm really happy to be here. It's my first time at PyCon, and I was already told that it's, well, a big conference in the sense that we have just listened. There are 1,700 attendees, but I was told that it's still familiar, so I'm quite excited what's expecting me the next three days. and but I definitely know what is expecting you in the next 40 minutes so the title of this talk doesn't leave space for speculations they're just coming in and in. Cool. Wow. It's getting full. No, never start on time. Wow. It's getting really full. Yeah. Wow. So welcome again to everybody who joined us. I'm happy that this year of today's talk fascinates you because you've chosen the way to this room. And well, the title of the talk is From Notebook to Pipeline in No Time with LinearPipe. So I I present you a way to convert a notebook to a pipeline, like Apache Airflow, Ray, Argo, DVC, but I'll not do it by hands, not by fair means, no. I'll use a very new cool and open source cool, LinearPy, because it saves you time and nerves. For me, this is a relevant job, because I have to do it a lot of times. At least you're sitting here, so it seems to be also relevant for you, But in case you're not convinced why you should convert a notebook into a pipeline, let's just draw the big picture. So let us assume we're all data scientists, or at least we get a project where we do something with data and AI. And such a project normally starts with doing analyzes on the data, doing some prototyping. We are looking for a model, for example, machine learning model, which works, which does the trick and we got the goal that in the end, we provide a service, a microservice, so we have to integrate it in a product. And from going from the left to the right, you cross a line and the line separates the field of development from the field of operations. So it separates that from ops or when you do machine learning, MLOps. So what's the characteristics of these fields? In development, you're looking for iterations. You're looking for feedback. We don't write code, well, in a complete sense, no. We are implementing some lines. We are writing a test. We are starting the compiler. get feedback does it work does not work so in the development side we just try to get things working and on the ML upside or upside the focus switches so we want to keep things working we don't want that something changes without our intention and the last three years tools you've emerged for those two sides so a So a classic choice for the development size are Jupyter notebooks. Jupyter notebooks are great for doing experiments. We get immediate feedbacks. We can do a lot of things without doing other things we did before, because we're just using variables in the memory. And Jupyter notebooks are great, but they are also bad, because they're not reproducible. So when you've ever tried to share a heap in a notebook, you know what I mean. So on the other side, there are pipelines. Pipelines like Apache Airflow, DVC, Argo, and Kubeflow. And it makes sense to use pipelines because pipelines are reproducible. You're fixing things there. You can check them in and you track the code. And when you track it, you can do continuous machine learning, because you get metrics out of your CI-CD, you can do continuous improvements. Well, when there are so many benefits in pipelines, so let's transform our notebook into a pipeline. Can't be so hard, right? Well, when you think about your last experimentations in a Jupyter notebook, it's not. a Jupyter notebook is normally a mess. So it all starts, for example, with package management. Who does fix the version when he does a pip install? Nobody. Just do pip install package. So that's where it starts. You're completely dependent on what the Jupyter notebook at the time finds in your package management system and you rely on the underlying Jupyter kernel you probably don't know about it first problem in reproduction so and the second problem is that you are playing with variables so you just use variables you reuse them you transform them you just do a lot of things you delete cells you edit cells you depend on variables which were generated from deleted cells so and and that's only the first thing so the second thing is that your order of your cells does not reflect the order of execution and there's no history about what you did in the past so when you found a model you rely on your brain what you did until the model worked and perhaps you're lost in reproducing these steps but there's a solution you can use linear pi for all this and it does a lot of stuff for you and saves your lives in case okay and to see how this works we We look on a simple example. But before, we prepare our notebook. So if you want to give superpowers to your notebook, you have to do these three slides before. First, you do a pip install LinearPy. And you don't start fixing the version because you never did. Then you load LinearPy as extension. And that's where the superpower begins, because now LinearPy watches the memory for you. It looks how you arrive to the variable. What did you do in the memory? It looks about the ancestors of the memory, of the variables. That's where the word comes from, lineage. And last but not least, you import LinearPy, because we want to use some methods from the package okay so far so good fast new seatbelts a simple example so we say variable a equals one variable B equals two I'm allowed to use these variables because I studied math so I get the license to use such variables and in the front line we sum up those two variables C equals a plus B C equals three just to point. And the first thing LinearPy gives us is a method called save. We save the variable c as artifact name c. And now comes a magic part. We can now say, okay, let me just print the code what happened. How did I came to this artifact? And LinearPy now connects all of things together and it says okay in order to reach C you've done the definition of a the definition of B and afterwards you summed it up and this even worked or works when you've deleted the cell before you did the artifact safe because linear pi doesn't watch the cells it works the memory what happens there inside. That's pretty cool, I think. So because when you get to the point your model worked, you can just say, okay, I saved the code which delivered me this model. Wow. So let's push the limits a bit. We now do a small import, we import numpy, but just to do a simple multiplication as complex as possible. So we define a parameter, my parameter equals 3, and then we multiply c by the parameter using numpy. The result is that d equals 9. So far so good. And we do the same thing again. We define an artifact and print out the code. So I've missed to copy the print command there, but I just want to show you the result. I hope you trust me that this is the result in fact. And LinearPy also tracks the imports. So you also get the information on what imports you rely on. And this will be also important later to build the pipeline. And we are just one step before building a pipeline. So that's a feature we are all waiting for. Let's see. LinearPy has another method. only the save method, but also the toPipeline method. And we are now saying LinearPy, okay, take the two artifacts, C and D, and build a pipeline for me. So the framework should be, in my case, DVC, because I like DVC pipelines, we'll see later on why. But it could also be a Kubeflow pipeline, Apache Airflow pipeline, Argo, or even Ray, I just looked inside the code yesterday and I saw that there's also a way plug-in inside Okay, and to make things a bit more complicated we say that an input parameter of the pipeline should be my parameter That's perhaps important when you do machine learning and you find a good solution for a model But you want to do hyper parameter optimization then you can use the pipeline in order to just play around with the parameters Okay, and then we just say there's a pipeline name, my pipeline, output directory, cool, and let's see what's happening. Now LinearPy does a lot of work for us. So it generates multiple files. First of all, a module file, and we'll dive into these files later on. The module file contains the source code, which we also extracted before using the print command and the get code method. And we get a requirements file, we'll see that what's inside there, but LinearPy tracks in fact all the packages which were used in the kernel. And then I get four DHE files related to the DVC pipeline. And last but not least, a Dockerfile. I like Dockerfiles, and we'll see later on why. Okay, but let's just go into the source code file. So this is an extraction of the my-pipeline-module file. And first of all, we see that he also imports numpy there, because he's now aggregating all the artifact code. And he's generating code how you get to the artifact C. This reflects the output we saw before for artifact. And now, the next code, getD, is slightly different from the thing we saw before, because linearpy just analyzed the two artifacts and realized, oh, D relies on C, so we don't have to do here the recomputation of C, no, to see by fair means from the beginning, no, we will use C to return. And the user said myParameter is an input parameter. So before, the myParameter variable was defined here as myParameter equals 3. But now it's an input parameter of the method. And we see later on in the DHG files why. And last but not least, it returns as d. So it's not only the code we got before when we printed it with the getCode method, LinearPy just put the things together in order to build a multistage pipeline as DVC supports. Who's in common with DVC pipelines? One person? It's a pity, you should use it. use DVC pipelines. Now one thing I like at DVC pipelines is that DVC keeps track on things you've already computated. So and to get this trick you have to define in this YAML how things are connected. So LinearPy generated a multistage DVC pipeline for us because LinearPy I realized that D relies on C. So, and here in this YAML, DVC YAML, we also have this multi-stages and the first stage is to compute the variable C. And doing so, he calls a command. So every DVC YAML provides in every stage a command which has to be executed and you tell DVC okay this the stage relies on source code of course so for here for example the my pipeline module where all the code is inside and well it relies on of course the source code file you're executing and on the other hand you say oh okay when you execute this there's an output and in our case linear pi saves the artifacts between the stages in pickle files so we come back to this later why this can be a problem but it's it's a common choice and the next stage on stage D you also have a little helper file task D we also refer on the dependencies on the source code file pipeline module and task D and now we We say also that it depends on this pickle file, which is the output of the first stage. So what is DVC doing now for you? DVC is tracking the checksums of the files. And when the checksum of these two files didn't change, DVC says, okay, then this output will also not change. And if the output didn't change, well, the stage D depends on, then I don't have to do the computation either, only when these two files are changed. Just think about machine learning. You get several stages, so perhaps loading the data, preprocessing the data, training the model, and doing some evaluation. When you change the evaluation, you don't want the first three stages to be changed or recomputed. You're saving a lot of energy because GPUs are using energy. Every computation needs energy. So your carbon footprint is reduced by using DVC. Just remember it. I think it's important to say because we have to take care of our Earth. So that's why I like DVC pipelines. And just to come back, what LinearPy is doing for us, it also will add a params section in the stageD. Why I say will, because I've seen that I left a bug, so I did this contribution which generates this code and I have to fix this. But I'll do it when I drive from Berlin to Munich in the train. It's not a big deal. It's also perhaps a good point to mention how are these files generated. So LinearPy uses ginger templates. And you are passing variables to these ginger templates. And these ginger templates are generating these files in a deterministic way. Okay. So this was the DVC YAML and let's dive into the other DHG files. So this is the first helper file, the task C, Python file. We see we import pickle just to execute the code getC in order to get the variable C and to dump it in a pickle object and that's the first thing which can break your solution who tried to pickle a tensorflow model in this room yeah did it work yeah it didn't work yeah that's That's it. So you cannot pickle every object. For TensorFlow object, it's not, TensorFlow models, it's not possible. There are other things you cannot pickle. And there are also things you shouldn't pickle because there is more efficient way to save it. So if you want to store a Pandas data frame, you want to perhaps use an HDF format or something like this. This pickle attempt is a first shot, but it has to be improved in the future. But it's open source. If you have good ideas, so also the LinearPy community is discussing about this, how this can be done. Will there be an attribute in the notebook telling them, oh, don't use the pickle, use something else. So that's something we're thinking about, how to change it in the future in order to reflect these edge cases which are not edge cases I think so it's really important to fix this. Okay but we got an artifact and the artifact is loaded again in the next step so this helper file this these task C task D helper files are doing the job for you in storing objects and loading them back again and last but not least we see that linear pie also now takes care about the input parameter so it passes the input parameter to the method and the parameters in fact stored in the parents Yama and you can say DVC okay here's my power on Sam when the parameter gets changed it's connected to the stage and you have to re-compute the stage. And that's how things are glued together. Okay. Any questions so far in the room? Yeah? How was the pilot at the end part B and part B? Why was it made in one part? Oh. Yeah, yeah, okay. read this because okay so your name is already a real ask why did we separate it the computations in different stages that's that's a good point yeah that's a good point so in fact you can there's another argument in the to pipeline a method you can say okay I want everything in one stage so for other frameworks also so there are possible to to say okay I want not everything in one stage but in multiple stage so it depends on the plug-in so there's some DVC plug-in framework plug-in there's an airflow framework packing ray cube focus that and they all have different arguments on what to choose as a solution so it's it's completely well and this was also the first thing implemented by the linear pi team to just put all together in a single stage it's a fair choice because when you say when something changes you have to do most probably everything it's completely fine so it was my choice to do this a multi-stage attempt and that's why I contributed to this multi-stage attempt to the library so it's both possible yeah Can you separate the scripts from the data, so for example, the data output is done with the subdirectory data and... Yeah. Your name is? Mark. Mark. Mark just said two questions. The first was related to type hints. So I think it should be possible. It will make the ginger table aid more complex, but it should be possible. I think. It's a bit work. The second thing is, could you also choose a directory, not single objects? Would also be some sort of work. So I don't know whether it's straightforward to implement, but that's also a big plus point of the LinearPy team. So they're really community-driven. So when I just said, oh, it's really a pity that you don't have DVC pipelines and multistage DVC pipelines, and they said, oh, okay, we don't have so much practice in doing DVC. Do you want to do the contribution? And I said, well, why not? And they really helped me a lot. So Andrew helped me a lot to implement the code. And I think just join the Slack community of LinearPy. They're really nice developers, and they're happy for feedback. And perhaps you're also doing the next contribution to LinearPy. Yeah? Yeah, for instance, when you have a database, Mm-hm. Mm-hm. Um, so you mean you leave the notebook using somehow an individual, um, individual implementations or individual tools outside, or are you calling a command, bash command or something like this? No, for this, I, I want to make it computerized, okay? Yeah. So you have a large, a large database, okay, and what you need to train the model is that you're loading parts of the database and calling them with some randomness and that's the model. Okay. So I see but then Yeah, oh, oh, okay. Okay, so I Because I hear you So just repeat the question. So When you're doing computer version You're loading the model and you're doing some random stuff on it or random stuff on the data you how is this randomness reflected in in such a pipeline, so I I think when you really want to do a pipeline out of it, you have to fix the randomness, but on the level of what randomness you're doing. So for example, when I do a splitting, a pandas with sklearn and test data splitting, I set a seed. I think you have to set the seed in the code, because arbitrary randomness cannot be reflected by transforming it into a pipeline. We have to see, I think. Let's make a pair programming session afterwards. Oh, we also have some slides left, okay? Yeah? Is there some considerable number of Oh, overhead. You mean when executing the cells? a good point I didn't look for it but I think it's really ah it's the session chair really does a good job so I have to repeat the question so is there an overhead computation overhead in doing so so do I slow down my computations um I don't know but I don't think much but I have to admit I don't know but trying to select community ask a question because the Doris and the other developers they have really good insights there okay I see we are we're really having good questions but I got some slides rest and I'll just go forward to the slides so we are now through that the chief us so we also mentioned what other problems what is future ways of contributing for solutions and we are now coming to the docker file and the package management So when you look inside the Docker file, on the first line, there's a from Python 3.8, which is a very generic choice. So a linear PyChast looked, what's your Python kernel? So what Python version did you choose? Okay, and then it says, okay, then I load the Python image. Then it copies the requirements file to the building process in the Docker, the image. and it doesn't pip install and when we look inside the requirements file LinnyPy does not only say oh, install NumPy, no it also looks inside the kernel what version is there. And that's cool I think because you don't have to do this. And there are now two things I think to say about the choice. I already said that I like Dockerfiles, because I only trust Python interpreters in Dockerfiles, because they are really robust in the sense of reproducible. And that's the pro, but there's a small drawback in this Dockerfile. So if you want to do machine learning, you want to use GPUs. And when you do the generic choice from Python 3.0, you don't have GPU support. Even if you installed your PyTorch CUDA libraries or something like this. But that's fine for me. Because I did a lot of image building to get CUDA support on my hardware. And it's not easy. So at some point you still fail. For example, when there was the switch from CUDA version 10 to CUDA version 11, you had to install libraries from the 10th version and the 11th version just to get it running. And I think GPU is, at the moment, still too specific as to automate the thing. So if you would like to automate it, You have to say, oh, LinearPy, I got this hardware in A100. Oh, no, I got an AMD hardware. Please look for proper images in combination with the packages I'm using. And, oh, no, I don't think that this... So if you're using GPUs, you know what's waiting for you. And I think it's the best to fix it yourself. Yeah. So from my point of view, this choice is really completely fine. And then there are some lines of code doing this DVC thing. Whoever tried to dockerize a DVC, so we only got one person here inside. Did you dockerize your DVC pipeline? Then you know these lines. You need to do this. You need a git in it. You need a DVC in it. So this is boilerplate code, and I'm happy that I don't have to write boilerplate code. That's it. Okay. So far, so good. now I think linear pi really saves you times and nerves to get from a notebook to a pipeline because it does a lot of things in refactoring saving your lives when you edited or deleted cells which you needed for your solution and I I have to say, I will never review a notebook again. Reviewing a notebook is a, well, it's a story for itself. So there are some solutions, but I'm not happy with them. And I've just discussed this with my team. So I think before I review a notebook again, I would transform it with LinearPy into a pipeline. Because then I see the relevant code, which reached, which made the model and I can really just read the code which is important and not around and I think this will save time for me, definitely. And the question is whether you should now use notebooks in your CICD because when you You get a tool which automates the step from a notebook to a pipeline. Why not just check the notebook in and let the pipeline produce in the CACD? So this sounds nice and can be the future, but not yet. We need more contributions for this. So at the moment, LinearPy saves you time to get from there to there. you have to check it by hand afterwards because the pickle thing it will always break something I think yeah but it really helps you to cross the line from dev to ops and it's not the only feature so they're working on more features they're even working on a platform to use linear buy and just follow them on linkedin join their own select so there are a lot of things which will happen in next few months which I think it's worth to follow and yeah you can also think about contributing because they're really community focused they're really nice to work to work with so it was a pleasure for me to work with Andrew on the code and I really learned a lot and well that's where I think we end now also I have also to say thank you to the vulgar which the company I'm working with they enable me to do contributions during my working time doing open source thing and this is really also something very important I think a lot of companies use open source but they don't allow the workers to contribute in the working time which I think it's a pity because it's not only a giving because you also get something back you get experience back I I just saw different way to work also because everybody has a different style of working using other packages which you are not aware of so it was really cool to do this and yeah last but not least I want to thank you to the organization team because there are a lot of volunteers in doing a great job and also the stuff here on the backside the camera and the audio yeah so thank you all I hope I helped you I I hope this helps you in your work, that you're saving a lot of time and nerves using LinearPy. Thank you so much for your amazing talk. You just shared to us a lot of knowledge and new libraries. Everyone likes Open Source, so I believe a lot of people now are up to try LinearPy. maybe making some pull requests to help develop the product so that's cool and I just want to say that we still have time for questions we have five minutes which is great you're perfect in time and guys online you can also write some questions to slider we are monitoring and you'll be asking it here to Thomas so we have So one of the things about notebooks is that sometimes you execute a line and then you execute other lines and then re-execute that line before and that might affect the variable in different ways. How does the tracking of that work with LinearPy? Yeah, so LinearPy really watches you. It's like a big brother. So when you did a cell execution, it doesn't say, oh, you executed a cell, but you executed the code inside the cell. It remembers it. Even when you deleted it, edited it, or manipulated it, it tracks what you're doing and not what the cell is like. So this is the thing where LinearPy really helps you. Yes, thanks. I have two questions, actually. The first one is how does it track these lines? How does it save them and what kind of objects? And the second is do you have, I assume it only reads the code in the notebook and it doesn't go a layer deeper, right? So is it possible if I import my own functions from a Py file, for example, to see those lines also represented? Okay, so I'm not quite sure, but I will start with the second question, because the first one I just had problems to hear was, so when it's outside the notebook, a library lying outside the notebook, which is not imported by a package management system, I think at the moment it's working not, but I'm not quite sure. So if you join the Slack community, ask a question. I cannot answer this. The first one, I just had problems because so many persons were leaving the room. Can you repeat it? Yeah, these code lines that it tracks and shows you if you request them again in your artifact, how does it save them simply? Is it in external files, or how are they tracked and saved? I don't know. Sorry. No, I've not looked inside really the core of LinearPy. But it's also a good news, because if you want to contribute and implement your own framework, you don't need to know something about how the core of LinearPy is working on. But also here, join the Slack community and ask the question. I even think there is a podcast where Doris and Andrew explained how this works and intrinsics work. So it's really join them, ask them, and rule post, I think. You get an answer there. I was just wondering how you are using LinearPy. So what is your single source of truth? You told us that you are usually only looking at the generated code because it's easier to understand than the notebook itself. I will do it in the future. Okay, you will do it in the future. But the question is, if you have this generated code and it's easier to understand, will you ever go back to the notebook? Where do you want to develop further? So, I got nerfed by notebooks two years ago, go and I tried not to use them but I came back because when you're not using notebooks as fuzzy as they are you're missing something you're missing solutions and so the notebook it fulfills so many criterias of the first field of development iterations getting feedback etc that's really worth using it but I'm lazy I don't want to do that boilerplate things and refactoring my my notebook this I want to automate and this is my use case because I got a lot of small AI project where I only have two weeks time to show results and this saves me a day or two days of work that's it in two days of work is fine you can go to the mountains Thank you, just one adding to this question actually so what happens when you have you had run linearpy to create the output of files and then you do a minor change in the notebook and rerun it. Is it so intelligent so to say to kind of bring it in an intelligent way that it doesn't mess up the whole a different story, so to say. Thank you. I think that's really the point which I mentioned. So will LinearPy allow you to check in the notebook and generate the CACD pipeline automatically? So the thing is when do I switch from a notebook to a pipeline, then I'm pretty sure that the solution I found won't change much and afterwards normally I'm working on the hyper parameters or doing slightly changes in the code and the pipeline code so this is the way I would use it but in case you want to work in the notebook back again so we are in future what we've talked about which will be perhaps problematic I would do a git merge and hope that git just does the things I did afterwards the last time again on this newly generated notebook so this would be my hope at the moment to if you would like to stay at the notebook also for well the next two three months of your project