Get to grips with pandas and scikit-learn

Speaker 1 [00:02]

So, we'll do the pandas part this morning. There may be a cancellation this afternoon. If so, I stepped in and said that I could do the second part of my tutorial. But if you've already done pandas, this is not for you. And, yeah, it's really pandas for beginners. And then, yeah, maybe scikit-learn this afternoon, maybe not. I send a message to the organizers to know so like I don't want you to lose your time so if yeah once again and so you won't need much this morning anyway so here like even this morning like for this morning you won't need psychic learn yellow brick I don't think either I'm not sure I don't remember but yeah these are the main things that you will need like pandas and jupiter I could dismiss that. Thank you. Did everyone already use Jupyter Notebooks, or did some of you never used it? Which is not a problem, but just... good for everyone? Cool. Thank you. Okay, so for those who just arrived, I'll say it again. I will just have the time to do the pandas part this morning. And it's pandas for beginners. So if you're already using pandas, it's probably useless for you. And, yeah, maybe. And please let me know if you need any help to start a Jupyter notebook because you've never done it Yeah It's not a big deal, we're just gonna use it like for one or two cells so So I can switch my environment one time now, I'm not sure. I'm using the install, but now I'm trying to install it on a conda. So I'm trying to switch my environment from on a conda one time now. Did you install conda in your virtual environment or not? On my parking system. Yeah, but then each time you do a virtual environment, you need to reinstall everything. You just have Python in your virtual environment. So you don't have... Well, I don't think you... I don't know. Now it's in this environment. I'd like to use this PyCon D environment. So I can just pip install, I guess. Yeah. I think you can pip install anyway. Load the con environment. So load the con environment, and then you can pip install in the con environment. So I can load this environment, yeah. So I can switch from this environment to this environment. It's just conda load environment. Let me find... Space activate. Is that what it is? Then I will try to install it with pp install, but probably because there is a package. Yeah, my virtual environments are not conda, so I'm not sure about that. So source space activate space the environment name. Yeah, it's the same as the source. Space, activate the environment name. So... It's like, usually, okay. So now I can install it all. Yes, now you can install everything. Can I install it? Can they install it? I mean, PIP should work as well, but... Yeah, that's... Yeah. It's recommended. Cool. Okay, thanks. You're welcome. Thank you. if you don't have yellow brick don't worry I'm not even sure I used it in the first notebook and even though it would be in one or two cells it's not a big deal oops where did I use it No, Yellow Brick is not... We're not going to use Yellow Brick this morning, so just... No, it's another... library, I believe? Yeah, no, it's another library. It's not part of Scikit-Learn. So just before we give it a start, for the last one who arrived, we're just going to do the Pandas part this morning, because an hour and a half is not long enough to do Pandas and Scikit-Learn. And if there is a cancellation this afternoon, I stepped in to do the second part but I don't know yet so maybe maybe not we'll see so everyone get a get the opportunity to download or like you don't really need to clone download is enough and open a Jupyter notebook or do you tell me if you have issues and then we'll kick off okay so i'll just super quickly present myself so my name is sandrine pato i'm french live in london work as a data scientist at qb insurance and i'm a former teacher so hopefully my skills are useful today so if let please let me know if not everyone is there and have a notebook open so we're gonna we're gonna start with the first notebook and that's mainly what we'll have the time to do this morning, which is data wrangling with Python, with Pandas, sorry, which is built on Python. Everyone knows, once again, how Jupyter Notebook works globally or not at all? Yeah. So it's Sandrine P. Let me show that again. And then if you go to my, where is my mouse? If you look for Sandrine P, then you will find, like I managed to have it, it's the first one on my homepage. So like if you just google Sandrine P GitHub, you should find it easily. So like once again on the installation side you will need Jupiter, Pandas and Matplotlib, maybe Seaborn, not Scikit-Learn, not Yellowbrick, so like don't get worried if you can't install For your brick. Okay, we'll see. All good? Okay. So, let me go back to the first notebook. So, if you don't know Kegel, that's a website, like that's a data science competition website but there are also lots of open source data set that you can find one advice is even if it's on Kaggle if I were you I would check that it is open source and what's the condition to use the data and also remember that we are in Europe and it's not because data is open source that you have the right to do whatever you want because GDPR so that's an advice so this one is definitely open source so we're going to use the titanic dataset which is a really classic one when you start to work with pandas and with scikit-learn so you can find I put the link so that you can you can have a look look at it but I've already put it in the folder so no worries so first things we're going to do is import pandas yeah you'll see like most if not all the cells are empty and my way of working is for us to fill it together because i believe that when you type you learn more than just running cells so we're gonna import pandas as pd some people do that's a super common way to import pandas some people just use pandas completely like the full pandas name it depends you do whatever you want as you know another thing that i have to warn you about it's it's my french laptop when i usually work with an english one so there will be probably lots of misspelling with q's instead of a's so that's usually the fun side of changing laptops Okay, you could import numpy as np, but it's already within pandas, so you can also do np equals pd dot np, and you will have loaded numpy. The first thing we're going to do now, The next thing we're going to do now is to import a data frame. So we're going to call it df. And to do that, it's a CSV file, so we're going to do pandas.read underscore csv. And we'll put the path. That's the other problem. There are the queues and the brackets and the quotes. So, it's in the data folder, then there is a titanic folder, and the first one is called train.csv. Okay, there is... Oh, 2Ds. yes yes is it big enough or not do you want me to make it bigger or yeah yeah yeah okay you're welcome I don't know if I can move that okay so there is another data frame which is the holdout so we're gonna say import it so pandas.readcsv and it's in the same folder and it's called like this the only difference is that it's called test.csv. So when you're doing data science, what you want is to have a holdout, so part of your data that you're not going to look at until the end of your process to check that your predictions are good. Okay, but you don't want to look at it. You don't want to be biased. You don't want to see what it looks like. You don't want to see if it looks like the rest of your data. If we had a time series, we would, for example, take the first two years in our training set and in our holdout the last six months or something like that and we would train on the first two years and we will keep the holdout the last six months and like untouched and we don't want to look at it and then see if our prediction or predictive model works fine on this last six months sorry okay so same for your data preparation you don't want to be biased, you don't want to say, oh, here there is something in my holdout that I should pay attention to. No, if there is a change in the data, you shouldn't know about it because in real life, you won't know about a change that is coming. Okay. So, there is in the Jupyter Notebook, you can look for help. So, if we want to look At the help of the first function that we use, so panda.readcsv, we can just put a question mark and then write our function, readcsv, and it will open down there the help and give you all the parameters that you can use in your function. you can also use shift tab when you're on the function ups normally here and you will have it here and you can just add the same thing here okay I just put So that's something that I found on the Keigo website. So you have the data dictionary. So that's the different columns that we will have in our data frame. So we'll have different information about the passengers. Well, first, if they survive or not, that's what we would try to predict with the model. So that would be our target. So first thing we're going to do is to have a look at the first five rows using the head method. So our data frame is called df, and we're going to do df.heads. And by default, it will show us the first five rows. We can also decide to look just at one if we want to just look at the first row, at 20 or whatever. We can pass it a different number. So, similarly there is a tell method, so same if we want to look at it because we want to To only look at the last 3 rows, we can again use our ?pandas.dataFrame, because it's applied to a data frame. So if you don't put that, you won't get the help you're looking for. And so same, it will tell us like the default value is 5. So if we do df.tell , we will only see the last three rows. Then we can look at the shape of our data frame. So we can use the shape attribute, and it will return the number of rows and the number of columns of her dataframe. Another useful one is df.info, the method, not the attribute, with brackets. And it will return the name of the different columns of our data frame. And it will also give us the type and the number of non-null values in the column. So, like, we can see that most of them have lots of non-nulls, and we have, like, cabin that is not well filled, and age that is not completely filled. After that, if we want to have a look at the numerical columns, we can also use the describe method to see the distribution of these columns. So df.describe, and it will return the count, the mean, the standard deviation, the minimum, the first and third quartile, the median, and the maximum of the different numerical columns. Here I put a note, so like there is another library which is called pandas-profiling, which is quite useful as well, but it could take super long to run, depending on the size of your data frame, but it will give you more information on each column. For example, knowing that you have a column that is completely filled, so no non-values, that's good. But if they're all the same, it's not really useful. If they're all equal to zero, well, you'd like to know that. And Pandas profiling will tell you that. It will give you more information. But I didn't want to install that and to run that. But once again, big data frame, it takes lots of time. so if you want what I usually do is I use it on certain columns and not on all the columns of the data frame or you make it run and you just go do something else that's another option you have to install it that's another library it's called pandas profiling but it's not in pandas so you'll have to pip install it or to conda you can probably conda install it as well after that we can get a list of the column names so So, df.colons, and if we want to put it in a list, we use toList, because otherwise, if you have a too long list, you wouldn't see everything. Oops, toList. What is that? Ah. Yeah, without underscore, I was correct the first time, and it's that I have a double M. That's better like that. Okay, so that's our 12 or so columns. So after that I will use the option to display a smaller number of rows than the default one. So if we want to lose at the default number of rows you do pandas.options.display.max underscore rows and it will tell you that the default one is 60 and what I want to do now is to set it at 15 so you just give it the value 15 so now we will maximum see 15 rows when well Well, 15 rows will be displayed. Subsetting now. So, there are different ways to subset a data frame. The first one that we will see is just to get a colon. So if we want to look at the p-class column, so we will just pass p-class into square brackets, and we will see the first 15 values, well, not the first 15 values, sorry, but we will see 15 values, the first and the last ones, of the p-class column. And it also, at the end, tells us once again the length of the column and the type of the data. So after that, if we want to look at the 12th observation, there are two ways to do that either we use I lock or dot I lock or dot lock so here it's going to be the same because the index and the position are the same I stand for integer not index so it's going to be the position of your rows or your colon when we work with it here it's the same. So we're going to just do... pass the number 11 to of the twelfth value, the twelfth row. And so if we do a df.lock with 11, that's going to give us the same thing exactly. So, after that if we want to look at the name of the last three observations. This time we're going to work on rows and on columns at the same time. So if we use iloc, first we're going to give it the rows position so we can start because again it's position we can start at minus 3 until the end and for what was it the name so we'll have to look the name is 0 1 2 3 so the third colon the fourth colon sorry so with integer 3 and we will have the last three observations and we'll get their names. With .lock we'll have to give it an index so the first index we want is 888 until the end and we can pass it just the name of the colon and we will get the same thing. It shouldn't work because the thing is here the 888 is the index it doesn't have index minus 3 so it's probably starting at 0 and gives you everything No, I guess. Yeah. No, for me, like I didn't try, but normally with loc you use index, so index of the rows index, so name of the column, and the other one it's all position. So you can try, but I don't think it will return you anything if you give 3 because it doesn't know colon call 3. So after that, it all depends what you want to do with your colon. If you want to do a loop and need an integer in your loop, and then it's possible. Well, it really depends. It's possibly easier if you want to use the position of the row to use that, but then you have to make sure that the position of your column is the correct one. Yeah, it depends on the situation. So both are useful. Okay. Finally, have a look at the age and fare of the 45th, the 1st and the 8th observation. So if you're there already, you can give it a try. So one will be with iLock, one will be with Lock, so you'll have to check the position of the age and fare column. iLock, sorry, which is integer, is position. Lock is labels, so labels for the columns and index for the rows. so what did I say 45th and then the first and then the eighth so we can just pass it a list yeah a list of positions and then same for the columns we said what age and And fair, 5, 5 and 9, am I correct? Okay. Enter for luck. The same list, 4407, and then age, and fare. so it depends what you've done if you've done if you df.ilock if you pass it just like that 44, 0, 7 well, it doesn't understand because like for him the 44 corresponds to the rows the 0 to the colon and then the 7 it doesn't know like if you pass it this way it works you have to pass it a list so that it knows that it has to look at this list of things ok ok so the label or index integer only refers to columns and not rows no, it's their index, so like, it's label for column, index for rows and the other one is I for integer so position for both yes I have no idea but that's a good question anyone knows? I don't know if there is a difference in speed or for me it really did like I use it I use one or the other depending on my needs, but yeah. Okay, another thing you have to pay attention to is iLock, .ilock and .lock doesn't work the same exactly. So if you use ilock, the end value in the range is included or not. Sorry, I forgot how I started my sentence, so I don't know how to finish it. So if we want to look at, I don't know, like the rows from 5 to 10, If you do it with lock, you can see that the row with index 10 is included. If you do the same thing with ilock, it will stop at 9. All good, or you have a question ? No ? Okay. So we're going to clean different columns. The first one, so we're going to label this one, the sex column. So the first one we're going to look at is the count of the different value in this column. So, we want to look at this x-colon, so we do df, square bracket, and then we pass, like, in general, the name of the colon we want to work with. So, like, once again, like, it's the same as, I don't remember who has the question, but, like, if we wanted to look at two colons, we would have to pass a list of colons, so, like, we would have two pair of square brackets. And we're going to do a value count, so dot value underscore counts, and we will see 577 male 340, sorry, female. So one way to labelize this column, to change that into numbers, have male and female, but have zeros and one, is to use a map function. So we're gonna create, sorry, a dictionary. So you call it the way you want. For me, dico makes sense because that's a short way to say dictionary in French, but I don't know if that makes sense to everyone. We're gonna do others, at least another one, so that's why I give it a name corresponding to the colon and so we're gonna give it, where are my curly brackets, male the value 0 and and female the value 1. And we're going to apply this dictionary to our column. And we're going to create a new column doing that. To create a new column, it's super easy. just pass the name of your new colon to your data frame so this one is going to be called sex lab for label yeah I know I made that to make to make one of my friend laugh and I'm I'm videoed saying sex lab so like please think of me I don't care it's fun anyway so we're gonna do so our first colon so the sex colon and we're gonna do a dot map and we're just gonna pass it the dictionary so and we have a new colon created that simple and we're gonna check the values that we have in our new colon so you can redo a value count and apply it to the sex lab colon. And we can see in this column we have zeros instead of male and one instead of female. Actually I know that you can labelize, there are other ways to labelize colons. I like to decide the way it is labeled, so usually I use a dictionary, but there are other ways to do it. For example, for me, for the unbound colon, what I want to do is to get the labels in order where people went on the boat. So, like, what did I do? One will be... so that's why I looked in which, like, what was the first stop. So, Samtham will be labeled one. After that, we have what? Cherbourg will be labeled two, and and Queenstown will be labelled 3. So, first we're going to look at not the number, not the count, but the proportion of the different values in the embarked column. So, embarked... And so the only thing that you have to do, we're going to do a value count, but we're going to pass it NormalizeEqualTrue and instead of returning the number for each value, like a count for each value, it returns a proportion. So, same, if you're already there, you can also start to labelize the colon, so we're going to create a dictionary, so, deco end, oops, underscore would be better. So I said for S, S will be 1, C 2, and Q 3. Too far, so after that I said C 2, Q 3. Ah, crap. And same, we're going to create a new colon, embarked lab, and to do that we're going to a map or dictionary to the embarked colon. I did, and that's a mistake, thanks. It should be a capital S. Thanks. Is there a way to make it under K-speed independent? Oh, I don't think so. No, no, no, I don't think you can do that. And same, you can check the proportion, but this time I'd like to see the proportion including the non-values, because there are some non-values in this column. So on top of giving it normalize equal true, we're going to pass it drop and equal false. Because by default it drops the non-values. Yeah, I forgot that. And we see that the values are a little bit different than the previous one because this This time, we have included the NAND values. Because there are NAND values, NAND is a float. It's a count, so these are the values that you have in your columns, it's not... Yeah, but my point was that, because there is a NAND value, NAND is a float, so that changes all our columns, even if we give it integers, because of the NANDs, it turns everything into floats. And so talking about the NAND values, we want to replace them with zeros. So we can do that, so we do... So if we just do that, I'll show you, df, embarked lab, embarked lab, whoops, no, crap. So, if we just do fillna, which is the function to replace all the non-values with something, so I want to fillna with zeros, if I just run that, it will return me the colon with the na field, but it won't replace the na in the original data frame. So, there is, I believe, in place equal troop. I've heard from a core developer that it will be depreciated or deprecated, I don't know how it's said, at one point. So, I take the habit not to use it and to just reaffect that to the original column. Ah, except that an underscore would be better. And so this time, I replaced my colon EmbarkLab with the EmbarkLab with the NS replaced with zeros. So now, like Sam, if you want to check your value count instead of the NAND, you will I love you, Rose. so like talking about the holdout originally like imagine that there were a force a force port where people embarked but in our training we just have three we want to be once again biased and know that in the holdout there will be a force one because right now i'm not supposed to know about it because i'm not i'm not supposed to know anything about the holdout so i do my prep on the training with what I know about the training set and then I will have to adapt and I will work only with what I know on the training set ok so the h colon we can plot things directly with pandas so we're gonna first use the magic common matplotlib inline so that's from the jupyter notebook matplotlib inline, and then we're going to do df.age.ist, oops, I know, that should be there. and we have an histogram so ok there is no title no labels nothing but it was simple and easy of course you can do better than that I'm not the best at visualization and I use it rarely I have to say I know that now you can change and use other stuff than Matplotlib so if you want to use Seaborn you can set Seaborn as the default things to plot, library to use to plot that allows you to just plot that in a notebook you don't need to import No, I didn't import it. I don't need to import it. It's a magic command from the Jupyter Notebook. There are other magic commands, but this one allows you to plot directly without importing Matplotlib or anything, and it just displays it. It's the magic commands of a person signed before them. You have, like, different... I'm sorry. You can also use like the command line or some stuff like that. If you look for that, you will, yeah. So time allows you to time the running time of your cell. So the matplotlib one that we just used. Load is quite useful as well. Allows you to load something that is in a different file. System I don't use often. Yeah, that's all there. And there is one that allows you, I think it's... I don't remember. To use the command line. There is one to use the command line as well. Usually I use a... Oh yeah, yeah, yeah. It's not a magic command, you're right. Yeah. Usually I open a terminal. Because you like it same, you can open terminals so I prefer to open a terminal and do everything in the terminal. Yeah, I think there is a difference between when you use it and when you don't, don't ask me what again, like, but there is a difference, I believe, okay, cool, I'll look for it, I know there, yeah, I'll look for it again, find the number of observations with none in the age colon so there is if you just want that you can just use the isNull function So df, we said age. So crap. If we just do is null, it will return a list of booleans telling us if it's null or not, if the value is null or not. If we want the number, we just have to sum that, and it will return the number of true values. Okay. So what we're going to do next is to replace the missing value, these 177 missing values, using the median, but not the median of all the columns, but the median per p-class. Because it was true then, it's still true now, like depending on your p-class usually you don't have the same age. class are usually not the same age than first class, third class, sorry. So, I decided, and like once again, yes? Is there a difference between an NAN and a NAN? Like, the NAN in Pandas, Pandas is built on NANPy, so it's a NANPy, it's a NP.NAN. So it's, as we said, it's a float. NULL, like same, that's more Python thing than a pandas thing I think null is like just an object I will have to check but that would be the type would be different but like in a pandas in pandas you wouldn't have null you would have nonce yeah yeah yeah it's cool like that but because you could have also when you have dead time colon I didn't put anything with daytime I think but then you have it's not nine none it's an 80 so not a time and then if you use is no that works as well okay so we're gonna do a pivot table so we're gonna give a name to this pivot table so So age underscore median equals df.pivot underscore table. And so we want the median of the h-colon, our index is the p-class colon, that's not correct index equal p class and the function we want to aggregate with so So ACFUNC is median. So if we want to have a look, where am I? At our pivot table, it looks like that. So the median h for the first class is 37, second class 29, and third class 24. And once again, because there are nones in the h-colon, that's why we have floats. So, after that, what I wanted to do is to replace the NAND values in the H column with these medians. What I've done is not really beautiful but I didn't find another way to do it and I wanted to use my pivot table and not just do a pivot table like that. There might be a nicer way to do it. So what I've done is that I applied a function to two columns. So I want to work with the agent p class colon. So once again, like what we said earlier, the two pair of square brackets are necessary, otherwise it wouldn't understand what we're talking about. And we're going to apply a lambda function. So if you're new to Python and don't know lambda function, don't bother, like just type it and that will be fine. So what we want to do, yes, sorry, thanks for reminding me, but I don't know, I've never done that? How do you do that? Control space B? Control shift B? Okay, so that's one line. Still better. That's something. F11? No. F11? That looks good. Oh, cool. Oh, thank you. That's probably better in the back. Learning something new every day, that's cool. Okay, so what I want, and that's here that was not really convenient to do. I want to take the median age corresponding to my p class and if I do that I also have to look for the value because otherwise it will just return it won't return the what's inside the cell like the not it's not exactly a cell but like you won't return exactly what's inside the cell so if we want to do that we have to do a dot values with a it's better and a zero like I have to pass it zero because it would journalists, even if there is only one thing. So I want to do that if the age is null, otherwise Because I want to keep the edge. Axis equals 1, otherwise it will work on the rows. And then you can do like a value count or like something to check that you have no nones or you can do, oh no, I've put, sorry, plot an histogram of the H-colon, and then we'll see that it's a little bit different from the previous one we've done. Just two times. And if you look at the previous one, it's a little bit different. have more value around, like, 20, 30, because it corresponds to the 177, like, they were, what, 24, they've been replaced by either 24, 27, or, was it 37, or something like that? Okay, we'll create other columns. So we're going to create a column with the number of other members of the family. If you look at the information about the dataset that I put there. So we have Sibsp is the siblings and the spouse, and we have Porch which is parent and children. So we're going to use these ones. These two ones to know how many other people from the family there were on the boat. And to do that, that's super easy. You can just add two columns. So same, if you just add the colons, you won't create anything, so we're going to create a new colon called family, and we're just going to add the two colons we were talking about which are sibsp, so sibling and spouse, plus the parent and children, and we can have look doing a value count. Okay, the next one, it was more to show you another thing that we could do. So this one I called socio, so like, because in the family column you have other members of the family but you don't have the person themselves. So I wanted to just add the person, so this one I called socio. And what we can do is... So I'm going to put brackets because we're going to do two things. So we can add the person themselves to the family colon. And to do that, oops, with a y would be better, you can just do a plus one and it will add one to all your colon, to each value of your colon. And then if you multiply by the class, the p first class, it will take them one to one. The first one multiplied by the first family plus one multiplied by the first p class and so on. So it will do it line by line. time it will give you like different values like honestly this one is more like an excuse to show you that you to show that what we can do but it was more like when in the family members you have everyone so it would there would be my family members but not me counted in the thing so I added one because like I'm still part of the family so yeah and multiplied by a class because whatever it's just an example to show things that we can do with columns so here we can and see that we have like definitely different values. So another thing we can do is to, instead of looking at the head, for example, of your data frame, is look at the sample. uh... it's it's good i can go down yep sorry I think it does that row by row, I don't know. It gives you the value, I don't know what it does. Because it returns, I think it's more or less returning the column. Yeah. Yeah. I think it's just the column. If we do that. No, it's not the colon, I don't, yeah it is the colon, 0, 0612, yeah, it seems that, yeah, yeah yeah exactly, like if I just look at the colon you have the same thing, it tells you that it's a method. I don't know. It returns information about the method, I think, not about the yeah yeah okay so oh yeah I was on the samples so sometime it's more interesting to have a random sample of your data either to make sure that the first five rows are not have not something specific in it that you can see different different rows at the different place in the in the data and sometimes it's interesting to play with the random state to have to see different cases so if you want to do that DF so we're going to look at the cabin column and we do dot sample by default it will just return you one row I don't know what the random state is by default or if it change its time it does it say it? Random state none so it's probably changing each time. So you can either put the bracket at the beginning or you do shift tab and then Then you use the plus here to extend it, this one to lower it. Oh no, this one puts it in the back. So first thing we're going to pass it is the number of samples that we want, so 15, and I like to give it a random state, because sometimes you will see something interesting, and if you want to reproduce it, it's good to have your random state, or if you want like one of your colleagues to look at the same rows, same. Good to have a random state here. And so you can see we have like random indexes and the corresponding values of the cabin colon. Okay, so what we're going to do is create a dec colon with only the first letter here. Okay, so, DEC, so new colon called DEC. we're going to apply a lambda function to the cabin colon so same we do but like this time it's easier because we have just one colon so the function will be simpler so if If x is a float, meaning that it's going to be a none, so if I could have done, maybe is null, not sure. Is instance x float, then I want to keep it, I want to keep the none values. Else, I want only the first letter of the cabin. I want to have a value count to see how many value of each deck I have. So now, if I'm not interested in having low values, I could put all of these, so A, F, G, and T, in an other, like label them as other. So what I could do is use NPware. So, dfDec equals np.where, where the dec is in a, f, g or t. So, there is a .easing function to check that. And we pass it just the list of values that we want to check, so A, F, G, and T, here. So if my deck is equal to one of these values, sorry I should have put that up there, then I want to put, I want to replace it with other, else I want just to keep the deck. And what I want to do with that is do a get dummies. Yes. Oh, yeah, yeah, yeah. 15. Yeah, I think we're fine. Thanks. So get dummies will transform. Well, will not transform. Will give you for each of the value of your colon. it will create a new column and we'll put 0 and 1s to tell you if the value of this row was like, so for example what do we have C or not, we'll see it will be easier if you see it probably, it's not that clear, so that's creating a new data frame so deck underscore dummies equals pandas.get underscore dummies and we're going to pass it the column that we want so df deck and we're going to give it a prefix that it will put at the beginning of each new column that it creates and our prefix will be dek. And then we can look at the head of this new data frame so dek underscore dummies dot head that's what it looks like and it's easier to explain when you have it here Oops, sorry. Okay. Okay. Okay, and so this one, again, created one new column for the values that we had left in our deck column, meaning B, C, D, E, and other. And for each row, it will tell us, okay, so this one was a none because there was everywhere. This one was C, this one was a none, this one was C, and so on, and so forth, and so on. and so what we want now because it's a new data frame what we want is concatenated with our original data frame so we're going to do df equal panda.pd.concat and we're going to pass it a list of data frames that we want to concatenate So we want to concatenate df and the decDummies, and we will give it axis equal 1, because otherwise it will concatenate regarding rows, which won't be exactly what we want to do, meaning that we will have like a bigger data frame with lots and lots of NANDs, which is not really what we want. And if we check, we do df.shape, and we can see that now we have 22 columns, so we added some here and there. So like the labellized one, then the the family one, the social one and the five from the get dummies. So if we want to drop columns now, some columns are not going to be used because I get a message and I will do the second part this afternoon, don't know like it's going to be updated in the schedule. So for the machine learning side, we're not going to use certain columns, so we can drop columns. So we're going to do a toDrop, we're going to give it a list of columns that we want to drop, so name, tickets, cabin, sex, embarked, and deck. Normally, I have done, there's probably misspelling somewhere, it would be too fun otherwise, too easy. And, oops, what we're going to do is df equals df dot drop, and we're going to pass it the list that we just created, so to drop, and same the axis. Because I think by default it's 0, and put the bracket at the good place will help. And so now, like, same if you want to do a shape. we know of only 16 columns. Yes? Is that undoable? Like, it's in memory, right? So if you've got the columns, can you run a command to store them? Is that okay? I'm not sure you can do that. I'm not sure you can do that, because I over, yeah, because like here I, the way I've done it, df equals blah blah blah, I have overwritten my data frame, so, that's why usually the thing is, here I don't care, and really I know I'm not going to use them anymore, but Otherwise, the advice would be give a new name. Like, when you want to keep your data frame and want to apply something to it, just give another name, create a new object. Yes? Can you what, sorry? Oh, yeah, I don't see why you wouldn't. Yeah, yeah. I didn't do that, because in the subsetting, we could have here a list comprehension, you can have anything. You could use a start with, or like, yeah. Yeah, yeah, you could do that. Okay, one way to reorder your colon if you want to do that, or that could be another way to drop columns like if you have 100 columns and you just want to keep 20 you can just decide for a list of 20 and pass this list to your data frame and then you will just keep these 20 columns so like for me it was just to give an example how to do that so just to avoid having to type everything I'll just do a df.columns that will show all my columns, and the, so I'll just copy that, call list that, and the only thing I'm going to do is to move survived at the end, because I like it, it's my target, I like to have it at the last column, just personal preference. And we can just do df equals df, and pass it, ha ha ha. list. And if we do a head, we'll see that just the first two ones, oops, I was surprised not to have put a Q somewhere instead of an A. Here, and we have the survive colon, which is at the end. So another one we can do is to check at the correlation between the numerical columns. So here, because I find it nicer, I'm going to do import seaborn as sns and create a correlation matrix for my data frame and I'm going to just display it as a with a heatmap okay try again so sns dot heatmap So, my correlation matrix, and I'm just putting some other parameters, so red, yellow, green. What is it on that piece? Ooh! Ah, it's G-N, not G-R at the end. Here it is. So obviously the diagonal is super green, dark green, because, well, all the values are 1s. And the rest show us how much correlated are the different numerical, well they're all numerical now, columns. So we can see that here, which makes sense, socio and family are highly correlated. If you remember the way we created them, that makes complete sense. And if we want to look at a subset, we can also just pass it a list. So double square brackets. If we want to look at family, socio, and all the like sibling and spouse, and the parent and children, up, ah yep, bad, bad, bad, that's not going to work. a square bracket, and .core, what is it, family, I don't know why I want to put, I am not a wired family, we can see that there are, some of them are already correlated, because I'm conscious of time so like the old out what we would have to do here is to do all the transformation that we've done to our training set we would have to do it to our old out we're not gonna have time to do that right now but that's not a big deal anyway everything like i've saved another data frame so like if you come this afternoon there won't be a problem we're just gonna see how to save a data frame I'll just do that so that can be collapsed there so if we want to save DF we do DF dot do to sorry underscore CSV and we pass it the the path where we want to save it so I wanted to save it and done and sorry in in the same folder as previously, so data slash titanic slash mytitanic or whatever you want to call it, mytitanic dot csv. we've saved our data frame with all the changes. Okay, well, it's noon. So, as I said, so apparently I'm taking over someone this afternoon, so I'm gonna do the machine learning part. I don't know which room, but it should be... it will be updated on the schedule well no five is not a tutorial room because seven okay cool then well if you want to go on please come this afternoon otherwise see you somewhere around in the conference and have a good lunch