Driving down the Memray lane - Profiling your data science work

Speaker 1 [00:02]

Thank you for coming for my talk. So first of all, I always said this is the most important slide because it got the link to the whole slide deck. And also, there's all my social media presence. You can kind of, if you have questions, for example, tomorrow when you wake up, you have a question, you can still DM me on Twitter. Mastodon is the same handle, but I don't check as frequently as Twitter, sadly. But we check both, basically. So, we are going to profile our data science work, and how it's going to work, I'm going to show you. But first of all, so this is me, I'm Cherk, I work for Alaconda, I used to be a data scientist but now I am developer advocate at Alaconda. I love open source projects, also like this morning you have heard that we have to support non-focused projects, so please talk to them in the booth. So I help organize a lot of conference because, again, I love the community. So who have been there? Who have seen this error, memory error? Yes, we have all been there. Every time we saw that, it's almost as bad as seeing the blue screen of death, right? It's like, oh my god, I have to run it again. So yeah, that's why we have to do memory profiling. And what is memory profiling? So before I really show you how it can be done, you may, for those of you who've got really good eyesight, you may also see that I have a tab open, there's a Jupyter Notebook, but I'm going to show you after. Before that, I want to talk about what is profiling, for some of you who haven't heard about the term before. In software engineering, we do profiling, which is actually investigating our program's behavior. So usually it's kind of something that we kind of like inspect your program when it's being executed. So that's why I sometimes also call it dynamic analysis, because it's actually what is really happening. It's not because of your code, you kind of analyze it because of how you write your code and stuff, but it's what is actually being executed. And so we have tools to do that. We have tools to inspect or monitor your code when it's got executed. It's called Profiler. So there are actually many, many different types of profiling. you may have heard of C profiler or this different thing that people use, but today we're gonna focus on memory profiling. So, like I said before, profiling, we're monitoring our code, and for memory profiling especially, we are looking at how the memory is being used when we execute our program. So it's about memory allocation. One thing that's always been talked about is the garbage collection, because when some memories that has been used and is not needed anymore, is supposed to be free and like get rid of the stuff you store there before is, well, that's garbage and you have to collect them so you free up the space. But you know, it's not always going to happen perfectly. That's why it's always got talk about because things may go wrong and we really want to get it done right. So when the program gets executed over time, we want to make sure that all the memory is being used efficiently. There's, you know, you're not going to get that memory error again. So that's why we want to investigate, we want to see what's going on. So why we need a memory profiler? Especially, I think there's something that has been overlooked when doing data science work. You know, we are all like kind of, you know, data scientists, we all care about the statistic, about like getting all this analysis, telling the story from the data. But sometimes we forgot that we are also using the computer and we should actually think about how the computer is doing. Is it going to surface perfectly or is it kind of having some hiccup somewhere in your code? So, for example, data science work, you may have done it before, you load a super big data set, we have to handle lots of data, sometimes we run out of memory very easily, especially on older machines. Nowadays, you know, like machine or cluster, whatever you use on the cloud are very powerful, you can always spin up a bigger one, but you may be paying more if you're spinning up a bigger cluster, so you may as well see if your code is, you know, doing the proper thing and optimizing all your memories. So Python apps, you know, we all love Python. Python is very easy to use, but Python, because it's so user friendly, sometimes like we kind of obstruct and didn't look at, like at a low level, is our memory being managed properly? We can't really have the transparency to see it. So a lot of times, you know, when we're not careful, we get that memory error. Or even worse, because we are doing scientific work, a lot of the tools, the libraries that we're using actually have a lot of external plugins that's probably not written in Python or something that is written in some other languages. So sometimes we may get something that, oh, it takes forever to run, but I don't know why. At the end of the day, you don't have to patient, you have to queue it. It may be because actually you're running off memory, but because of the communication between the plugin and Python, and then somehow it doesn't get raised, the error doesn't get raised. So you're just waiting there forever for nothing. So it's not good. So we need to know what's going on. So before you kind of deploy your work, scaling it up, we have to do our profiling. So a lot of terms getting involved when we're kind of starting to do profiling. You may see them online all the time on Stack Overflow, whatever, when you ask a question. People talk about this all the time. heap and stack memory, what are they? So I am trying to like, this is what I found out, trying to do a comparison. So heap memory usually means that it's just like a big kind of bubbles or sea of variables. They're global variables, they can be stored anywhere, there's no structure to it. So that's why you need a reference, otherwise it's kind of lost in the abyss and then you kind of don't know where they are and then you don't know that they are not needed anymore and you forgot to collect them, which is the garbage collection. So these need to be free. If you, you know, if a tool or a program that's, like, do it properly, then all this garbage will be collected and free up the memory so you can do more stuff. Or if you kind of, for example, again, something happened and you have no patient to wait for the program to finish and you killed it, then, well, it will be free because your process got terminated. It's basically done by the operation system, so it's, like, something that is not about your program. It's about, you know, you killed everything so they all got free up. On the other hand, there's, like, stack memory. Stack memory usually is for local variables. So that's why local variables, when it's outside of the scope, you can't access those variables because they are stored in stack memory. And, you know, they will be free when your function returns. So outside of your function, it's gone. You don't have access to it anymore. But those, like, you know, there's no problem of, like, you know, forget to collect the garbage because, you know, they are in order. So once your function is done, all of them will be erased. There's less problem of this kind of leakage of memory. Another thing that got talked about all the time is the residence set and virtual memory. Wow, it's so fancy, these terms, I love it. So residence, what is it? Residence means how much is actually used on your computer RAM. Sometimes it's like something that, you know, virtual you may compare like all physical and virtual, so it's something that's actually used because nowadays computers are very smart. If you are not doing something yet, for example, you haven't loaded a CSV yet, you don't have to hold up that kind of space, or sometimes even more low level, your computer will swap all this memory around when it's not being used, it's going to be storing something else before it got to be used. So basically it's just making it more efficient. So sometimes you can run a program that actually requires more memory than actually the RAM you have on your computer because of this, like, resident and virtual memory thing because things got swapped around. So residents about, like, what is actually being used at the moment. Virtual memory is the whole process, how much memory is needed for the whole process to be finished. So that, like, sometimes, like I said, maybe bigger than what you actually have because things got swapped around and stuff. So that's why sometimes looking at it, you have to bear in mind that, you know, what you are really using, what is really needed. So for example, residence memory, how much it's using, you can actually like got a really reliable estimate of whether you're running out or not. But for virtual, you know, you may compare, you know, program which one use more memory at the end of the day, but, you know, it may be not reflecting how much memory you're actually using at any certain point of time. So to make it more complicated, different operating systems may be different as well. So I'm just giving a general idea here. So what actually is happening really depends on your computer. So it's very difficult to explain everything. So I'm just giving an overlook at this point. So how does Python do all these complicated things? You know, Python can be installed in my machine, in your machine, on different operating systems. So Python actually did a pretty good job of making things work on different operation system with the memories. So I hope it's not too small. I don't know why it becomes so small. But anyway, so Python actually store objects with a very, very interesting mechanism. So first of all, it will have a private heap. So I will explain more in detail. So there's like things called arena and then all these Python objects are actually blocks. you can imagine as a Lego blocks that they kind of like fill up this arena. So in Python, we have a, so you may heard this name before, PyMetalog, which is actually the Python like memory manager that is built in. It is kind of like the higher level kind of things and then below it, there's actually other like C APIs that's like, you know, so complicated that I don't even understand, so I won't explain that to you. So it's like PyMem, Raw, Metalog, that kind of thing. So I won't look at the C code because I am not very good at C. And then after that, it will actually talk to your, you know, your machine's management system, so virtual memory management manager. So those are actually, like, OS-dependent. So different type of Python object, actually, they would have a different size. For example, so this is how PyML log work. So actually PyML log is designed for small objects. So the maximum size of the objects that can work with pymylog is 512 bytes. So that's a defined amount of size. It can be smaller, can't be bigger. So that's the maximum size of the block for your Python object. So all these blocks, like I said, they are filling up this arena. So everything is kind of managed, like block by block, so it's a very organized way of doing it. The advantage is, you know, very easy to implement, very quick, you know, to access and stuff, but may not be the most optimised things to do. And also, if it's bigger than 512 bytes, then it will fall back into using other more complicated things like the pymem, raw, malloc, all the other things. So also the size, it kind of needs to be a multiple of A, so it's from H to 512, because they are blocks and they can only be stacking in units. So that's all the technical details, but we want to know how we can actually use it, right? How are we going to do it when we are using our Jupyter Notebook doing some data science work? What are we going to do? So I found a tool that actually can do the memory profiling, and it will work in your Jupyter Notebook. So this is memory. So I'm not involved in this project, and so I'm just telling you because it works with Jupyter Notebook. let's see how it works. So I have this code here. I can also put the link for this slice and all this code in this code later, so don't worry about it. If you want to try it at home, you can, you know, download the code or something. So I'm gonna show you the first one, which is explaining some kind of basics. So when you're using it, how I'm gonna look at it, how, you know, using is very easy, but how to interpret it is a little bit, you know, require a little bit of knowledge. So, like I said, this tool works with Jupyter Notebook, so there's actually a plug-in that you could use. So, you know, we have all used this percentage magic before, just like, for example, Netpalib, you have used the percentage magic, so it's kind of like that. You load in an extension. Of course, you have to have memory installed. It's taking quite a while. I don't know why. Maybe I'm running off memory. So, yeah. Why? Please. Okay, I'll just run the cell and hopefully something will happen. Yeah, why is it taking so long? I think maybe I'm... Oh, yeah, it's starting now, yeah. It's very slow, usually it's not that slow. I think I opened too many tabs, like you'll see at the bottom, you see? I'm trying to demonstrate what happens when you run out of memory, okay? So, no, I'm just joking. So this is an example that they show in their official documentation. have showing you how it looks like when you use this percentage magic and to get a frame graph. So what is a frame graph? A frame graph for people who like developers who have done this kind of like profiling work before, of course, they're not, you know, they're familiar with it. That's not new for them. But for data scientists, it may be something for you. And this is Yeah, I'm zooming in too much. But don't worry, I have a solution for that. But anyway, look at this beautiful frame graph. We have everything is blocked. So you can see that actually for each function, so I'm looking at it like kind of top-down, you can revert it, like you can flip it over if you want, but what I'm doing it here is to showing you that, for example, if this function is calling something, that's kind of like it's showing below it, so for the most bottom one, for example, this one is actually the deepest call, if you look at this, right, there's a very, you know, kind of, you know, iterated, iterated but like layered on kind of structure, you can see the last call are C and D, so it's like A times N. It's very silly but it's a demonstration. They say these are C and D, right, so they are the deepest call because, you know, everybody is calling them layer by layer. There's how it would look like in a frame graph, so if you want to look at what is the deepest call, look at the bottom of it, right, or you can flip it around, I forgot how to, oh, or you can flip it around like this. This is more like, you know, a tower, if you can imagine, so the other is like a well, this is is like a tower. So to look at something more complicated, let's run this example. Okay. So here you see that this is a little bit irregular because now we are doing things that are more complicated. They are different things, not just A times N all the time, but sometimes they're different sizes. So that's obviously that they will be using different amount of memory, that's why it's kind of irregular in this example. So we call this edge that has nothing on, well, let's invert it because the term makes more sense if we inverted it. So this we call it a roof because there's nothing on top of it, right? There's the deepest layer if it's got inverted. So we want to look at those roofs, reason being that roofs kind of show you what is occupying the memory. For example, if you see that, for example, this one, so this is actually the call, if you believe me, right? It's kind of too big, but it's actually F. So F is calling G. So basically F is doing nothing except calling G. So G is doing something, right? So you can see here that this is G, this is F. So G is sitting perfectly same size on top of F because how much memory F is using is how much memory G is using, right? So, you know, it's just doing nothing but calling G functions. So that's why it's this. So when you look at this this way, then there's nothing for you to optimize in F because, well, if you can optimize G, you can basically optimize F, right? But for a counter example is something like this one. So this one, so if you believe me, because the box is too big, is actually D. So D is doing a bunch of things. So you can see D here. It will have other functions as a roof. For example, this one, I believe, yeah, this one is E and this one is F. So E and F is another box to sit on top of D. So they are not the roof of D because there's something on top, right? But D actually got its own roof here. This small edge is D's roof. So it means that you can actually optimize D. If you can optimize D in some way that get rid of this roof, then it will be more efficient. Then it's just up to E and F. So that's why it's very important to find out this roof and to see where you can improve or optimize your code. For example, you write a bunch of things and then you see that, oh, actually you got a big roof somewhere, then maybe I can improve that a little bit, change things a little bit and see if that could improve things. That's very abstract. isn't it? Let's look at something more practical. I have another thing. I love ice cream, by the way. I always use the ice cream as an example. I'm going to run a bunch of cells because it takes a while because of my fault. I'm so sorry. Let's look at the code first while it's running. I'm doing something. I have some CSV which is about ice cream review and products. I'm going to load them in using pandas. I'm just It's gonna run this bunch of things. It's just pandas, nothing very interesting. And look at this here. Oh, it's done, okay. So you can see it's just a pandas data frame. We are all familiar with it. So this is something that may be something similar to what you do day-to-day, you know, cleaning up the data. For example, this one, for example, I want to get yearly stat, right? Because, you know, all this review, they are very detailed, but they are also like a different date. I want to congregate them into different years so I can see how each ice cream is doing, whether they get more popular this year, less popular this year. I want to have a look at that. So what I'm gonna do is the most straightforward thing is like because this is a string, I haven't shown the D type here, but if you show the D type, this is a string. So I can actually cop out the first four letters, they will be actually the year, right? So I'm gonna, ah, okay, goodbye using that. So that's what I'm doing here, right? But there's another way to do it, is how about I do it properly, in quotation, convert that string into a date, and then I just get the year of it, right? So this is the second way of doing it. So I have two versions of doing the same thing. And I'm going to, you know, so this is doing the other join and stuff, you know, I hope you are familiar with it, because you're all doing data science work. So yeah, so it's just, you know, doing a bunch of like group by and then join together. So this is the first version that is just copying the first four letters of the string. So let's see how it performs. So you can see the frame graph here. Oops, I think it's, yeah, it's this way it's very complicated, right? So because like Panda is doing a lot of complicated things that we are too scared to look at. But you can see that we have this thing, right? This thing we know because it's calling get year stat. This is using the first method, which is getting the first four letters of the string. And I want to compare to the version 2, which is converting it into a date. So how is it different? I'm doing the same thing. I can have a look and comparison. So this is here, this one. So you may see that, oh, my God, this is like, how can you look at this? Well, if you really don't want to look at it, one thing good about it is that all the the results actually are saved as an HTML file. So for example, this one, if I look at it, it's saved to, so tmpa something, something. So can I find it? Yeah, this one is actually the same frame graph. It's in an HTML file. So this is easier to see, I hope. Yep, so it's easier to see. So yeah, this one is the one that I get it, you know, convert into a date and then it's a year. So you can see, compare these two. So this actually, you know, is causing, How much is, it's still too big. Sorry, let me zoom out a little bit to see if it's better so I can see. No, it's not. Let's revert it and yeah. The one thing I don't like about this, because it's too long, I really can't see much. And then I can't really zoom in and out to have a good, better look at it. But you can see, I want to compare this one. Okay, what I can do is to run it again because sometimes the order will be swapped if I run it again, yeah, now it's like kind of in a hacking, hacky category, right? Okay, now you can see that it's using 3.2 megabyte. So, is it mega, no, it's MIB, I don't think it's megabyte, it's not using that much, but anyway, 3.2, right, remember this, and then we compare it to the first one, which is this one here, it's using 1.2, so oh, actually version one is better, I've just caught the first four letters and it's like, actually if I convert it into a Python data object, it's actually using more memory. So now you know which one you should do. If you don't care about doing it properly using the date object, if you just want to quickly do a group by, you know what to do. It's very, you know, quick to just, so my instinct was right. So this is what I would normally do, just to cop the first four letter and do a group by like very quickly. And this is actually the more efficient in memory perspective, more efficient way of doing it. So now you know, and you have proven that, oh, after years of data science experience, I'm actually doing it more efficiently, yeah. Yeah, so this is just basically how you do it. Of course, you know, it's a bit difficult at the beginning, it's a bit scary, you know, so many calls and stuff, but I hope you can try doing it. It's, you know, it may help you to solve some mystery. Sometimes, you know, why is using up so much memory? Because, you know, sometimes you're doing some kind of, you know, pivot table or, like, you know, joining different data frames and then why it's got frozen up. This may be why. And you may investigate and somehow you may be able to squeeze, you know, your CSV into a small enough memory to be working in memory instead of, you know, spinning up a bigger cluster and cost more money so you can tell your company, I saved money, right? So this is quite good. So I think that's all I have to say about memory profiling. The last thing I want to tell you is about EuroPython, which is, you know, a conference that I've been help organizing in the past few years, and this year it's gonna be in July, and it's gonna be amazing in Prague, and tomorrow I hope someone's going to tell you more in the lightning talk, I hope. So, or if you have any questions, talk to me. I have an Alaconda pen. If you talk to me about Alaconda, I'll give you some swag. So that's it. Thank you so much. Is there any questions on slide?

Speaker 2 [24:04]

Thank you so much for a very memorable talk. I'm so sorry I had to.

Speaker 1 [24:09]

Back to the top for you.

Speaker 2 [24:11]

There are actually three questions, four now, five. They're currently coming in, but maybe we can already get started with one. Can MemRay show runtimes as well on the X-axis?

Speaker 1 [24:24]

Do you know that time on the X so the x-axis you mean this one, right? Yeah So this actually now is not in order you can see that if you run it the second time, you know The order will be flipped. So they are not actually the order that has been executed It's just like if if a function is calling another function, so that's the order of execution is on the y-axis not on the x-axis So yeah, the x-axis doesn't mean too much. It's just for comparing how much each process is causing And if the box is longer, it means that it's using more memory. So that's the x-axis. The y-axis is actually the calling order.

Speaker 2 [25:04]

Then the next question would be how does this work with parallel implementations? Is there a fancy?

Speaker 1 [25:11]

it actually you can I think there's like a thing you can flip to look at the parallel as well if I remember correctly let me have a look at one of the things see oh I remember no actually there's some settings that you could allowed so if you look at the documentation right memory memory documentation they would show you that actually there's some like parameters that you can add to this percentage magic to turn on my polarization all these other things that that you could, you know. Because I know that a lot of data science tools that actually is using a lot of optimization to do maybe parallel or like, you know, pre-compile stuff. So those parameters may help you to investigate more.

Speaker 2 [25:56]

Is memory the most commonly used memory profiler and what are other good solutions?

Speaker 1 [26:03]

do come across others, but I haven't looked at them in detail. I look memory in detail because they work with Jupyter Notebook. Because when I get interested, I've heard one of the core developer announced that memory now have a Jupyter Notebook plugin. So of course there are other tools that can do the same thing, but it may not be as easy to use on your Jupyter lookbook as memory. There are also other profilers as well which like not just looking at memory but also like maybe the speed other performances but you know so but I kind of like memory because it worked with Jupyter notebook.

Speaker 2 [26:46]

Is there a way to process memory output data non-visually? Somebody wants to export it to a data frame.

Speaker 1 [26:53]

I think you can export it. Again, look at the documentation. I think you can export the report as well. Now you can see that all these frame graph is generated as an HTML, but there's other format that you can kind of output all the results.

Speaker 2 [27:07]

related to getting things a little bit less cluttered. Is there a way to only say profile my code and ignore the underlying functionalities?

Speaker 1 [27:19]

But there's actually like, you can see there's like, oh, hide these system frame, but it doesn't, it helps a little bit, but it doesn't help much because again, a lot of them are not considered as system frames, it's just pandas doing panda things. So yeah, it helps a little bit, but not like perfectly. I know what you mean, like, oh, what if you just show the code I've explicitly written here, right? Unfortunately, it's not gonna be, but if you check this, that it will hide some of the stuff that's even more complicated than the pandas in China.

Speaker 2 [27:55]

I guess we have time for one last short question. Do you know if there's a spider plug-in for memory?

Speaker 1 [28:03]

I have to confess, I haven't used Spider for a long time, so I don't know. Yeah, sorry about that.

Speaker 2 [28:10]

Okay, thank you so much