(Serious) Time for Time Series

Speaker 1 [00:04]

All right, welcome everyone. So our tutorial is about time series. It's called Serious Time for Time Series. It's time to take time series seriously. Had to practice that a little bit. So my name is Marisha, I'm from Amsterdam. So I came in from Amsterdam. I am a data science educator at Go Data Driven. And besides that, I'm a data scientist and a Pi data organizer as well. So if you're ever around Amsterdam and want to visit a meetup, please do so. And this is my colleague James. We're doing the tutorial together, James.

Speaker 2 [00:30]

Hi, so my name is James. I'm also a data science educator at GoDataDriven. I have a background in education, so I used to be a high school teacher. Hopefully, I won't be dealing with any moody teenage behavior from the audience today. And yeah, so like Marisha said, I'm a data science educator at GoDataDriven. So GoDataDriven is a data science consultancy based in Amsterdam. So we do a range of consultancy projects helping to get companies started with their data science projects, upskilling, and we also have an academy which does trainings on data science and data engineering topics.

Speaker 1 [01:16]

Yes, so we are the data science educators that means we regularly do trainings as part of go data-driven Academy We do a variety of topics So it's trans data science data science with Python deep learning all those kinds of things and one of the other ones we do is Time series, which is also why we're here, of course So that is a little bit about us and why we're here. We're also we have a quite a big group over here so I want to invite you to join our poll over here. So you see the QR code or you can visit the link to get a bit of an audience impression. Okay. Okay. Oh. So the link was deleted or suspended? It's a great start, of course. How do I switch to the... Well, you can also visit the link that's up there, pollev.com. Hey, there we go. So we have a couple of questions here. That's also for us to get an idea of how the audience is because we're quite a big group. The first question is, of course, what did you enjoy most about the conference so far. So we have people, meeting people, lightning talks, really like the lightning talks myself as well. And the food, well the food is pretty good. So did you do a no question? Can you go to the no question? Uh-huh. Oops. Yeah, so the second question here is, what is your job title? We have, of course, quite a few, with both the PyCon part and the PyData part, quite a diverse audience, I think. So I think I see a data scientist up here, software engineer, analyst, senior, more engineers, or a researcher. All right. Consultant. All right. And the next question is, where are you from? So we came over from Amsterdam. It's about a six-hour train ride to get here. I met a lot of people from the conference who are from Germany, but also quite a bit More spread out from Europe, so quite a diverse audience. I wonder who traveled the furthest. Next question, what is your current energy level? So medium. See one medium there. Someone's very energetic. Most people very energetic. One person in the audience who really doesn't want to be here. But most people are still quite energetic, which is, of course, a good sign, because we are doing a tutorial, so we have to do some stuff ourselves as well. Another question. So how much Python experience do you have? So you're just starting out, zero to one years of experience. Are you quite comfortable in Python? You remember working with something that wasn't 2.7? There's quite some very experienced people here then. Nice. Also good for us to know, because we are doing our time series, our tutorial about time series, we will be covering a little bit of machine learning. I do expect like a lot of people here at PyData maybe have some experience with it, which I think I also see from the poll. Build many machine learning models, most comfortable with the topics, don't necessarily build it themselves, and some of you still starting out. And the last question, of course, is for us to know what do you hope to gain from this tutorial so do you want to know a little bit more about data analysis of time series because you've never worked with them before are you comfortable analyzing time series but would like to learn more about modeling or are you already very familiar with the topic so the connection interrupted everything goes really well to start out the morning I guess that everyone's very familiar with the topic and we just like to pick up a few tips and tricks so So that's a good point for us to get started. All right, then we'll go to what?

Speaker 2 [06:31]

Ah, here we go. Oh, there we go.

Speaker 1 [06:32]

There we are Quite quite a few people are very familiar with time series already Just just for my impression. So there was a tutorial about time series on the first day as well Can I see a raise of hands of who was with the psychic type see someone in the back? Oh quite a few in the middle as well. All right, so you must be a little bit familiar with time series then I think that's a good transition to what we'll cover in this tutorial then

Speaker 2 [07:02]

So, yeah, obviously we're starting off this tutorial with a bit of an introduction. Afterwards, Marisha is going to take you through how to perform time series analysis with pandas. Probably most of you have used pandas before, but you may have not taken advantage of all of the great things that you can do with pandas when you're working with time series data. Afterwards, I'm going to do a demonstration about time series modeling and being able to model seasonality in data. And then we've also got some challenges which we can hand out to you guys so you can have a go at. Probably we won't have time to cover them in this session, but be nice to practice in your own time before we finish with some closing remarks. We'll also take a 10-minute break there. We'll take a five-minute break about halfway through. And just before we get started, just want to make it clear what we will be covering, what we won't be covering in this tutorial. So as I said, introduction to using time series data with pandas and some seasonality modeling. Obviously, this is just an hour and a half workshop, so we're not going to be covering, we're not really going into deep learning. Especially some of you are new to machine learning, so we're not going to try to do a crash course in deep learning and then cover time series. We're also not going to be focusing on things like profit or maybe some of the other time series libraries. We really just want to talk about the fundamentals and the actual concepts in this workshop. Okay. So it is a workshop. We do want you to code along with us and have a go yourselves. So there's two options for doing this. You can either access this material from GitHub, or if you follow the second link, it's going to take you to a sandbox environment where you can access the Jupyter notebooks there as well. So we'll just give people a minute or so to get set up before we begin.

Speaker 1 [09:16]

Yeah, so we will be we will be going through this tutorial with Jupyter notebooks And we will explain so I will cover the pandas part James will cover the seasonality modeling part And we'll talk about a couple of concepts there And there will also be a lot of exercises in there for you to try out things yourself and see if you can Find some answers to the questions or the exercises there So we do really recommend that you visit at least one of these two links to to make sure that you have access to the Jupyter notebooks and can actually code along with us All right, so we are a technical company, but we are having some issues with screen mirroring and the presentation kind of stuff, so don't mind me if I keep on looking over there because that's my screen as well, so I don't know what's going on. Apologies for that, I'll just scooch over here so I can see the screen in the meantime as well. You should be able to see this notebook. This is the very first notebook that we have. It's the notebook that's called Time Series Analysis. The goal of this notebook, I'm going to take you through this one, and the goal of this notebook is to get you familiar with the Pandas functionality that there is to analyze time series. What we'll cover is some time utilities in Pandas, how you can read in your time series data and parse those dates properly, and especially some time-based manipulations and smoothing. And these are basically kind of prerequisites to the seasonality modeling that we'll be doing later on. So of course, as always, the very first step is... What's your shortcut for running a YouTube or a notebook, so?

Speaker 2 [10:58]

Shift and enter.

Speaker 1 [10:59]

Shift and enter.

Speaker 2 [10:59]

Shift and enter.

Speaker 1 [11:00]

Okay, it's control enter for me.

Speaker 2 [11:02]

I need control of it, so.

Speaker 1 [11:04]

All right, so first thing of course reading in pandas and We are going to be using pandas with timestamps, so All of your you probably Python users, so you're probably familiar with date time Timestamp is kind of the pandas equivalent of using date time so Using using timestamp we can parse dates As you can see over here by just creating a pandas of timestamp object we can see what the date is and what the time is. Does this work? A little bit more? Alright. So the advantage of using Pandas Timestamp is that they support a wide variety of operations as well, so sure, we have the date, it knows how to parse the date, but it can also tell us a lot of other things. For instance, it can tell us the amount of days that were in the month, so right here we have, it's difficult for me to read, but it's February, so we can see that there are 28 days in February in total. We can also find out the week of the year that it is and what quarter this date belongs to. Again, all kind of neat functionality from Pandas would be very difficult to extract yourself. So that's where timestamp comes in and another couple of operations that you can do is, you know, check what the difference between two dates is or what the name of the month is and these kind of things come in really handy when you want to create your own features based on the date that you have for your data so in this case we can see that this date happened to be a Sunday so the very first very quick exercise we're going to spend too much time on this but investigate the timestamp features and methods to see if you can figure out what day of the year it is today so we can see here already ready to the current date, simply pd.timestamp today, and that should give us the date of today. We can check whether we're in a leap year, and we can check something like how long is it until a public holiday, for instance, Christmas. So let's take about 30 seconds to one minute for this to get you a little bit familiar with using these timestamps, because these are, well, this is what we're going to be using throughout the rest of our data as well. All right, like I said, we're not going to spend too much time on this exercise specifically. Hopefully, it gets you this impression of things that you can get from timestamp information. But of course, the reason why we're covering this is because this is a tutorial about time series, meaning we want to deal with data that have timestamps. So an important next step is reading in the time series data. And the data that we're going to be reading in is the data that we're going to use throughout this notebook and the other one, which is about air quality in Californian countries between 2007 and 2017. So in this data set, each data point indicates the average air quality at that point. And of course, if you've used Pandas before, you know that when you have a CSV file, it is located in the data folder. You can read in the file like this. But as we can see when we read in this data, we have two columns. We have the index, of course. We have two columns. One of them is the information that we're interested in, the air quality index, AQI. And the other one is the date local. And yes, these actually do look like dates. But if we check the information about this data frame, we can see that this column by default is not parsed as timestamps. So this is just strings, basically. And we want to make sure that when we read in the data, these are actually parsed as timestamps, so we can do some nice indexing on it and some other operations. So when we read in a time series data set, we adjust it a little bit. So we did make two things we do slightly differently. First of all, we set parseDates to true. So in this case, that's what you can see over here. parseDates is true. That will automatically detect whether something is a date or not. There's also, of course, options to do it with the specific formatting that you're interested in. And the second thing that we do is that we set the index not to the 0, 1, 2, et cetera, but we actually set it to the date column. So if we read in this data again now, you can see that it looks slightly different. We only have one column, which is the air quality index. And the index is no longer just 0, 1, 2, 3, 4, et cetera. It is this date that we have over here. And it might seem like a very minor change, but I'll show you why this is important and why this makes your life a lot easier when you're dealing with time data. And you can see here that we now only have one single call on the error quality index. So why is this nice to do? Why do we set the date as the index? Well, if you know the log method, this allows you to filter on your data frame. And in this case, because we set the date as the index, we can use the log method to get a range of dates. In this case, we're looking for all the dates between the 1st of February 2008 and the 10th of February in 2008. We use it as a kind of an indexing method. And as you can see, all the dates that we have now are actually do fall between that range that we had up there. And another really nice thing about it is that when you create a plot in Pandas, by default, it will take the index as the x-axis and all the columns that you have as a line for the line to draw. Well, in our case, our index is the dates, and our column is the information that we want to plot. So that means that when we do create a plot, we very neatly get all the dates here on the x-axis. And we can see a little bit of, so there's already some trends in here. So we see what's happening with the air quality over time. But of course, this is just about reading in the data. The nice thing is that Pandas provides a lot of functionality as well to get more insights from your data as well. So for instance, right now, the kind of information that we have is that we have it per day, right? So maybe you're not interested in the air quality index per day necessarily, but you're interested in, let's say, something like the average per year. Well, that's where resample comes in. That's a very easy aggregation that we can do. Essentially, what we do here is we take our data frame, we resample it by, well, in this case, by year, but it could be anything. And then once we resample it, we need to perform some kind of aggregation, so in this case, the mean. What does that mean? We basically take every individual year and then take the mean over that year. And if we execute the cell, we can see that we have some kind of information per year about the error quality index. You see that the date that we have here is the very last day of that year, so the 31st of December, but it is actually about 2007, this is the information about 2008, and so on and so forth. And the nice thing about resample is that what we did right now is just resample by year, but you can take a lot of different kinds of periods, so for instance, we can also do it per four weeks, for example. Gives us some more information. You could do it by month, by business month, by quarter, lots of different options, and if you want to check out all those different options, you can check out this link over here for the complete list of options there. So what we did right now is we took the mean over that period of time, but maybe we're interested not only in the mean, but some other forms of aggregations as well. So that's where the .ag method comes in, where we state that we want to use our data frame, our air quality index data frame, we want to resample, in this case, by month, so look at every individual month, and we're not just interested in the mean, we're actually interested in a couple of different things, in this case, that is the mean, but it's also the variance, and these are things that were built in, so we create a column called mean, and the content of that column should be the mean, it's just a string that we pass there, same for variance, but we can also create some custom aggregations. For instance, what we do here is we create a column that we call spread, that's the first part here, and the second part is how we actually calculate that spread, meaning in this case we take the maximum value and the minimum value, subtract those and see what the spread is over that month. And doing this gives us a data frame back that on the index has the date itself. Then we're interested in the AQI column, that's why it's all the way over here. There is only one column in our data frame, so we only have the AQI column, but if you would have multiple columns, you would have the mean, the variance, and the spread for all of those different columns. So that's what we have here. And then we have that mean column, that variance, and that spread. And And the names of these columns are determined by, well, the first element of this tuple and the actual content of the column is based on the second element of the tuple. So in this case that were built-in methods and here we defined one ourselves using a lambda function. And the advantage here is that we have our time series data set and using all this Pandas functionality we can create, we can amend it with some other information. So let's say in this case we only have the index, which are the dates, and we have the air quality index, which is the information of course, but let's say that we wanted some time-based features there. We want to know whether this timestamp was what quarter it was and what day it was, etc. And it's pretty easy with Pandas to create that information. So in this case we take that air quality data frame and we use the assign method, so this may not be the method that you usually use for creating new columns, it's the way that we prefer to do it. But essentially what we're doing here is we use the assign method to create a new column. We create two new columns. One is called quarter and one is called weekday. And how do we create them? Well, we take the data frame and then we use the index that is what holds our dates. And we use some of that information from those timestamps. So every timestamp has information about what quarter that timestamp was and what the name of the day was there. And using that information, we can create kind of an augmented data frame which doesn't only hold the information about the air quality index, but it also holds information about the quarter. So we see here that this date was the third quarter, and it was a Wednesday. And here we had the third quarter, and this happens to be a Sunday. So these are some Some nice functionalities I think Pandas has to offer, we have a couple of exercises here as well, so just to get a bit of a practical experience with doing this, I mean some of you may have done this before, for some of you it might be new, but what I want you to do is look at these exercises, so there are two here, one of them is compute the average AQI, Air Quality Index, per week, and find the week that has the highest, or therefore the worst because the higher it is, the worse it is. And the other way, the other exercise here is to find the weekday with the worst air quality. So which day of the week had the worst air quality? And as a little bonus, it's nice to make a plot about that. So let's spend a little bit of time on that. If you really don't know what to do or you're not sure what the answer is here, we also provide the option to load the answer. I would say don't do it until you've tried it, but of course you can remove the comment here, load the answer, and then it will show you what we did. There are of course multiple ways to do this. And if you are really a little bit quicker than most of the people in the room, there is a bonus section here on shifting that we will not be covering, but I think it's pretty interesting to maybe have a look at if you have some time left. All right, All right, so I see some people are still working on it, some people have already finished with the exercise, so I'll talk you through the first one already, because I think there's a nice little thing in here that you may have not seen before. So the question or the exercise here was to compute the average air quality index per week and find the week that had the worst or therefore the highest AQI. And the interesting thing here is what you've probably tried to do is use resample, and you saw that we could do that by year, by month, et cetera, and we can also do it by week. The thing, however, is that if you just use the W, it will assume that there must be one day that's the first day of the week, right? So by default, that would be Sunday, and I don't know about you, but I think the week starts at Monday, not necessarily Sunday. So by adding this little dash mon, what will happen is it will resample not by assuming that Sunday is the first day of the week but assuming Monday is the first day of the week. You could also of course change this to Wednesday if you would like, but that would be madness. So what we do here in order to answer these questions or what I would do is I take the data frame that I have, I resample it by week, so in this case Monday specifically, but that might also be Sunday for you and what this will do is it will separate the data frame into the parts of the weeks that belong together so you have so you have a separate data frame for for each combination of seven days that starts at Sunday and that means that you have seven values for each of that each of that part and we need to aggregate these some way and what we do here is you can take the sum but in this case we're taking the mean over all of the different weeks in our data set but we don't we're not just interested in the mean we're actually looking specifically for the worst air quality and that means that the easiest way to find that for me would be to sort set sort the value so we take the mean we sort the values and well let's see what the what weeks were the gave us the highest values well that was this week in 2007 that ended on the 24th which is and this was the mean air quality index over that week so these were the worst weeks and these were actually the I would say best weeks in terms of air quality it's nice to see that it's a little bit more recent is better air quality longer longer time ago is worse air quality and the other exercise here was to find the weekday so which specific day of the week has the worst air quality. And a nice way to do this, it's not the only way to do this, but a nice way to do this is also to create a little plot there. So let's see how we approach that. So a little bit more elaborate, so we take the data frame, we have the data frame for the air quality, and in order to find the weekday with the worst air quality, we are going to create, maybe you did it in a different way and you got to the same answer, but the way that we did it was we created two different new columns, and these are very similar. They basically contain the same information, but one of them has it by name and the other one has it by number. So we have the weekday in terms of the name of the day, Sunday, Monday, Tuesday, et cetera, and we have the associated number, so Sunday being zero, Monday one, Tuesday two, et cetera. And why do we do that? Well, we want to use that weekday information, So what day of the week it is, what string is associated with it, to do a group by statement. That means that we can gather all the Sundays together, all the Mondays together, all the Tuesdays together, and get the mean over all of the Sundays in our data set, all of the Mondays in our data set, and all of the Tuesdays in our data set. Why do we also create that weekday num? Well, it wasn't necessarily really necessary, but you can see that if you just, let me comment this out really quickly and this one as well if we just do it like this we do have all the information that we're looking for it's just that the days are ordered in a way that's not really natural I would say like it's it's alphabetical order I think and I prefer my Tuesday to be after the Monday and then the Wednesday afterwards so that's why we created this other column here the weekday number because the Monday is 0 and then it's the Tuesday is 1 and the Wednesday is a 2, so we can sort by that weekday number, sort the values, and now we have them neatly arranged. And we have the average air quality index for every day of the week. And of course we're looking for the worst one, so we're gonna plot this to make it a little bit more insightful. It's only seven values, so I could probably find the worst value as well, but it's always nice to make a little plot. And we can see what the worst day of the week is here. That would be this one over here, Friday. So for those of you who got around to it, there's a little bonus section on shifting. If you didn't get around to it, you can always talk to us after the tutorial as well or have a look at it yourself. The material will stay online for quite some time. You can have a look at it. It's quite interesting. But what I really want to tell you more about is smoothing. And it's also because this smoothing is kind of a prerequisite for the seasonality modeling that we'll be doing later on. And the idea here is that we have our data, and we want to get some insights in what's actually happening. So there's a lot of noise in the data set there, of course. In this case, we're going to focus specifically on one single year, otherwise the plots will become a little bit too big, but you can do this for the other plots as well. So we're going to focus on 2016. And we can see here that, well, it's a lot of spikes, a lot of noise, but there do seem to be some trends here, right? So it seems like over the summer holiday period and also in the summer, in general, it seems like the air quality is a little bit better. It's lower, lower is better. And in the winter, the end of the year and at the start of the year, the air quality is a little bit worse, right? So this is something that you can see from this plot. But what we'll try to do is make sure that that's a little bit more easily visible. And the way that we do that is with smoothing, because right now we see a lot of different spikes and I kind of had to eyeball what the trends here are. And we want to get a smooth line through this to make sure that we can really visualize that very neatly. And the way that we do that is with rolling average smoothing. So this helps us discover patterns in the data that are maybe not as easily noticeable when it's just all the noise that we see there. So we create a smoother pattern over time, a smoother line based on the data points. And with rolling average smoothing, we basically take a window and average in that window. So in Pandas, we can do this with the rolling method. So we see that over here, the rolling mean method. And we can set the size of the window. That means the amount of, in this case, days that we want to average over. So we can see that here. We take that data frame, specifically the 2016 part, and we create a new column here to visualize, and we create that column by doing the lambda function, getting our data frame, and doing a rolling average smoothing over a 20-day period, if I see correctly. And we create a plot with that. So you can see that this is a line that goes through the data that kind of shows what's happening here. And how smooth that line is is really dependent on the window that you choose here. So if I change this to 7, for instance, you can see that it's a lot more spiking, a lot more closely resembles the original data. Well, if I make it a lot bigger, let's change it to 50, for instance, it's a very smooth line. There's no right number to choose. It really depends on your data set what gives you the nicest insight. Let's change it back to 20 for a second. And this goes through our data and it shows you what your data is approximately doing while removing the noise around it. But what you can see, and especially when we use the 20-day period, meaning we take 20 days in total, we average over that, that's going to be our data point, so we remove all of those spikes. But we can see here, for instance, yes, our line does follow the data, but it lags behind a little bit. So we see, so here there's a bit of a spike in the orange line, and that spike is after the actual spikes in the original data. So why is that happening? Well, we take a 20-day period, but it lags behind, so only 20 days later we will actually see the spike because of that higher time point over here. And that's maybe not what we're looking for because it doesn't really it's not really nice not really nice fit to the data so one of the ways that we can offset that is by using centering and by centering essentially what we're doing is we manually shift the data points backwards so they do fall in line with the data and there's two ways to do that well one of them is using the shifting method that we saw before for those of you who did spend some time on that but actually rolling mean when we calculate the rolling mean it also has the option to center so what we do here is we take our data frame we create a column that we just saw so that orange line over there which does lag behind it's it's a 20-day period but we also create a second column and a second line there which sets center to true and a third one which is going to be exactly the same just to illustrate that it's it's done in the same way where we manually shift back the values and as we can see when we plot this they are still they're just they're taking the same number of days so they are the same lines it's just where they occur in the graph is a little bit different so the blue lines are the original data points the orange line is rolling mean that we feel like that we gathered without centering and then both the green and the red line well you don't see the green line because it's exactly the same as the red line, we shift back the data a little bit to make sure that when we have a little peak in our data here, our smoothed line also has the peak in the same place and not 20 data points later, because that's not really what we're looking for. So rolling average smoothing is basically a very simple way to isolate the signal from the noise in the time series data. There are some drawbacks to this, so first of all you have to choose a window size, we set it to 20 days right now, you could have set it to 50 days or 100 days, or one day I guess, but that would just be the same as having the original data. So it's highly dependent on the window size, and if your window size is too small, you have a lot of noise, still have a lot of noise, if it's too big, you can remove the actual signal that you're looking for and not really get some insights. And it always lags behind the window size unless we center it, which is something that we can do, but you don't always have that information up front. So there's another exercise here. So previously, we did the weekday with the worst air quality. So in the previous exercise, we saw that the week that started on the 24th of December had the worst average air quality. But we did that by week, right? So we said, okay, which time period, starting on a Monday and then ending on a Sunday, had the worst air quality? But that might actually not be what we're looking for. Maybe we're not looking for a specific week, defining a week to start on a Monday and end on a Sunday. Maybe we're just looking for a seven-day period that was the worst. And maybe that seven-day period starts on a Wednesday instead of a Monday or starts on a Friday. So that's the exercise here. find the seven-day period that has the worst average error quality and As you can see also from the name of the the Python file that holds the answer. There is some rolling average moving coming up here Another section that we won't be covering here today because we're going to move on to seasonality is exponential smoothing But it is quite interesting to have a look at if you're really quick with the exercise Yeah, so just to repeat your question, so you're looking back 20 days, so it's easy to calculate a value for the 20th days because you take the previous 20 days, but what do you do on day 19 or 18 or 17 when there's no 20 values to look back at? Well, if I remember correctly, but correct me if I'm wrong, it just takes, so let's say you're on day 15, you want to look back 20 days, you want to calculate the average, it will take those 15 days. So for the very first data point, you do have some data, but that is the actual value that you had on day one. Day two would be the average of day one and two. Day three would be the average of day one, two, and three, if that answers your question. Oh, so you're talking about, like, you have this peak over here, I think, and then the The orange one is lagging behind, but then the red one is a little bit soon in a way. So I think this red peak over here has nothing to do with the peak over here. This red peak has to do with the peak that we see here, and we don't see a peak afterwards because if you take a 20-day period, sure, the values are pretty high over here, but right before that, the values are quite low, so they average out to a very noticeable peak. So I think the peak that you see here is not about this data that's happening afterwards but it's actually about the data just before it. So this peak over here. Yeah, yeah, exactly. Yeah, so that's what you do when you center. That's what we specifically say when we do centering, when you do a 20-day period. I don't think it's, well, it's not 10 days before and 10 days after, it's 10 days before and nine days after or something like that. But of course, by default, when you're dealing with data, you don't want to look into the future, right? So it makes sense that by default, it just looks back because that's the data that you usually have and you don't have data about what's happening next. It's very difficult to calculate a smoothing average over the next 10 days when it's just today and I don't know what the value is tomorrow and the day after and the day after will be. So that's why that option is there. So we can censor if we want, but we're looking into the future. And in this case, we've collected data from 2007 to 2017, so we do have that data. But if I wanted to do that for today, I can only look back. I cannot look in the future. All right, let's load the answer here and make sure that we have enough time left for the seasonality modeling. So the question here, and again, there may be a different way that you did it, but the way that we do it here is the question is slightly different from what we've seen before. Before, we were looking for the week, where a week is a strictly defined Monday until Sunday period of time, or Sunday until Saturday period of time. In this case, our question is slightly different. We just want the seven-day period where the values are the highest, and therefore the air quality is the worst. So this is where the rolling method comes in, and because we're looking for a seven-day period, we state that to seven days, and we calculate the mean, and if I'll just comment this out very briefly, we can see that we get a data frame where every day has a value associated with it. In the original data frame, every day had a value that was the value of that specific day. In this case, that value is no longer of that specific day, but it is the rolling average over the seven previous days so that what we see here and we're interested in the seven days the seven day period of time that is the worst so how do we get that seven day period of time that is the worst we're looking for the highest values here and we can do that multiple ways you can sort for instance sort this and see what what date it is or we can check the n largest which gets us the largest value based on the air quality index column over here and when we do that we see that the period of time the seven day period of time that was the worst in terms of air quality is the seven day period of time that ends on the 20th of december in 2007 because this value is a value that's not just associated with this particular day but also the six days before that so this was the worst seven day period and you could actually check if you're interested to see if this is not a Monday, but a different day as well, if you're looking at the index. So what we covered so far is that this was the first notebook that I wanted to take you through. This is some functionality that Pandas has to offer to reading your datasets, but not only reading your datasets, also get a bit of insights in your dataset. There's some really nice functionality that it has to offer with the timestamp being the Pandas equivalent of date time, and we saw how to properly read in the time series data set, so I really strongly advise you to always put the date as the index there, because it allows you to really neatly make plots and find a filter on the data that you have, and we saw that you could do time-based manipulation, so you can do aggregations, for instance, with resample, which is a really neat method, and do something like shifting and getting information about the days that that you have. And the last part that we saw was smoothing with rolling averages. And if you also had a look at it, also exponential smoothing or weighted smoothing or expanding windows if you feel like having a look at that as well, which removes the noise from the signal. So that gives you some insights into what your data is doing. And based on the techniques we saw so far, you can already answer quite a lot of questions about your data. So for instance, which year had the worst air quality? which five-day period has the largest decrease in air quality between the first and the last day, or something like, does the air quality improve over the weekend, can be really interesting questions to have a look at and answer. And just with the tools that we discussed so far today, you should be able, or you have all the tools to be able to answer these kinds of questions. But of course, there's much more to do with time series, and that's when I'm going to hand it over to James, who will tell you more about seasonality modeling.

Speaker 2 [42:54]

Thank you. Okay, we'll take a three-minute break just in case anyone's desperate for the bathroom or needs some water, and then we'll continue. Okay. If we have a question, can you just repeat the question first? Because we have all that audience, and they didn't understand the question. Sure. I know it's annoying, but maybe just... Yeah, yeah. Oh, sure. Okay, so we will get started again.

Speaker 1 [43:55]

Very dramatic.

Speaker 2 [44:02]

Okay, so can you just give me a thumbs up at the back if you can read the notebook okay? Yeah? Cool. So, Marisha's been discussing with you how you can perform time series analysis with pandas and some of the great functionality and tools that it has. We're now going to move further on to discuss modeling time series data and specifically how we can model seasonality in time series data, which I'll discuss a bit as we go forward. So just loading in some of the libraries we're going to be using here, and I'm going to reload in the data that we were working with before, so we don't need to introduce that again. It's this air quality index data where we have the higher the value it is, the more polluted the worse it is. Okay, so let's get that plot back. We've got the plot from the full data set now, and you can clearly see that there's some regularities or some patterns going on in this data. Okay, so yeah, obviously we're going up and down quite a lot as we go through the different seasons as we go through the year, so as Marisha was saying previously, it seemed like air quality was worse in the winter than it was in the summer. Any other things that we can see in this data, maybe we can get from the audience? You can just shout out. Yeah. Downward trend. Pardon? There's a downward trend. Yeah, exactly. So we can see that we're kind of oscillating around between 6 and 10 to begin with, but But then as we go towards 2017, it looks like we're more between 2 and 6. So in this data set at least, it suggests that the air quality index is getting better. So nice to have some nice news about climate for a change. But yeah, so we've got these patterns. And generally, when we're talking about time series data, they can be categorized into these four things here. So trends, like we've seen here, a downwards trend. Seasonality, predictable repeating cycles. So here we've got yearly seasonality in that it's going from January to December. We see this repeating pattern. You might have daily seasonality. So if you're talking about like power usage in a day, probably you use more power in the day than you do at night because then you're sleeping. You can also have cyclical components, so patterns with no set period for repeating, and then also some residuals. It's going to be almost impossible to model, so even though we do have some repeating patterns and there's some clear trends in this data, you can also see it's quite noisy, and it's obviously going to be pretty difficult to completely model that noise. Okay, so seasonality is very common in business data, the data that's collected by businesses. However, it can actually often obscure the signal or what you're looking for. So often something that can be really helpful when you're modeling or trying to understand the data is try and decompose it into these different parts. So maybe you want to access a trend. So you want to ignore the seasonality, because that's just repeating year on, year on. And you want to see the actual general trend that's happening. Now, one naive way of doing this, or one maybe simplistic way of doing this, would be to take a rolling average, like Marisha demonstrated before. So you can see here, I'm taking a rolling average, I'm centering, and I'm using a window of 365. and so corresponding to days in the year and you can see when we do this with that large window we're able to essentially smooth out all of the the seasonality so we can access the general trend and ignore the the seasonal repeating patterns what would be the issues with this if we're thinking about about modeling our data or thinking about using it to predict in the future. And again, feel free to just shout out or raise your hands. What if you don't know how long that seasonality window is? You can just guess this year because you saw it there. Yeah, exactly. So I mean, I've assumed that it's a year, because that's generally what I've been able to pick up from the data. It might not always be so obvious. I mean, typically with time series data, you can associate it with a regular window, but that's not always going to be the case. Any other suggestions? Yeah. You can also have a different seasonality for different intervals. So you can have both, for example, yearly seasonality and monthly seasonality, and then they're all the same. So for the benefit of people watching online, you can have maybe multiple seasonalities, right? So you could have a yearly seasonality, monthly seasonality, that would be maybe hard to access, just like this. Yeah, summertime, wintertime changes, yeah, indeed. And I guess, yeah, if we're talking about modeling this data as well, then this is going to be insufficient, right? We've got no equation that we can then, we're able to plug in values and extend this beyond the window of data that we have at the moment. so we don't end up with any actual equation that we can use we're also using centering there was a question from the audience previously about why don't we automatically center well I am centering in this occasion here but obviously I can't use this then for predicting anyway because I'm trying to access data in the future which we wouldn't have if we're forecasting and looking ahead in time. OK. So what we're going to do then is instead try and model this. We're going to use a very simple model just to begin with, just to really demonstrate how powerful it can be. So we're just going to base our modeling on a linear regression model. So, what we're going to do is make some kind of equation where we're going to have some kind of equation where we're going to be able to have maybe a number that we learn. We're going to be able to put in some information from the data. We'll probably add another term. This is the general form of a linear regression model. And then from this, we're going to be able to predict that air quality index, OK? So what am I going to input into this model? Well, I'm not going to be able to put a date as an input to a linear regression model, because it's just the date, at least, as it is at the moment, because this isn't a number, right? Obviously, we could maybe try and separate this out. For the time being, what I'm going to do is just consider each date a different time point. So this is the first date in our data. So I'm going to give this a corresponding number 0. This is the next one, so I'm giving it the corresponding number 1. So I'm going to have a very simple feature matrix target vector. So my input X is going to be this 0, 1, 2, 3, 4. And I'm going to be trying to predict... Well, I'm going to be trying to learn a model where the target vector is like so, the air quality index. Fitting a simple linear regression model with scikit-learn here so I'm passing in I'm choosing a linear regression model passing in the data fitting it and then I can use this model to get predictions from the data back again and this is going to correspond to the linear trend so you can see here this is the linear trend that we're predicting if we plot both of this information you can then see in blue the original data which has the seasonality and in orange the linear trends which yeah is it's not got any information about seasonality so it's just giving us a simple linear trend notice what we were passing into the feature matrix was zero one two three up up to however many pieces of data there are in the data set. So I've not given the model, at the moment at least, any information that it could have seasonality. So of course, we're expecting just a simple straight line here. One thing that you might want to do when you're modeling time series data is put in some break points. Often, you'll have a particular event that will happen, and that affects the trend. and the trend changes from then on. Specifically in this context, we're assuming that we're maybe aware of this particular point in time, so that might have been a change in the business, it might have been an external change, but something we're able to know and input into our data. So, for example, summer 2014, there was some kind of hurricane, etc., and we see we see a bit of a change in the data you can maybe see there's a it's the the the gradient or the the general trends begins to change around there so what I can do or what I'm going to do here is add in an indicator variable for summer 2014 so scroll down a bit so the first variable I've added is this after summer 2014 indicator variable so whenever the the dates are below August 2014 we have zeros whenever it's beyond that we have ones and I've also got an interaction term as well what that's going to do to my linear regression model, well, it's no longer just going to be y equals mx plus c. It's going to be mx. So we're still going to be inputting our days or whatever. So there's 0, 1, 2, 3, 4, et cetera. We're going to have another term, which I'll call this x1. I'll call this x2. Hopefully, it's not getting too mathematical. So we're going to have this other term here, which is going to be to do with that breakpoint after 2014, before 2014. If it's before 2014, that's going to be timesing by 0, and it's not going to be included. If it is after 2014, it's going to be timesed by 1, and we're just going to add on an extra term. So let me I won't include the interaction for the moment Just so we can talk about that So now we've just got the after summer 2014 If I include this when I do the linear modelling You can see at 2014 in the summer When we get to that point we can then suddenly going to add on an extra term you'll notice obviously we keep the same gradient at the moment because all we're doing is adding on an extra value when we're after 2014 if we want to be able to change a gradient then we also need to make something dependent on the what timestamp it is so that's what we're doing with this interaction term so if I add this back on and we include this information, then we're going to be able to add a term that's proportional to the date of the year it is or the time point it is. And so you can see the gradient starts to flatten out now after summer 2014. Okay so we've seen how we can model the trends, we've seen how we can take seasonal data and effectively ignore the seasonality part of it, just capture the trend and also deal with breakpoints and shifts in that trend, what we're now going to have a look at is dealing with seasonality. So at the moment, we've just been able to model our data for the trend, not include anything seasonal, but that's because when we were passing this data into our model, we didn't give it any information about the season. There's nothing it can gleam from this data that indicates that it's in January or it's in December or if it's in May. This is exactly what we're going to do with this next part here. So I'm going to use some feature engineering which Marisa demonstrated previously. And I'm going to add on to the data some information about what month it's in. So we're then going to include this information when we go to train our model. Now, obviously, I can't pass this data in at the moment, because we've got this data as strings. It's categorical information. So we need to choose some way of encoding it. I could choose to ordinarily encode this data, right? So I could say January is 1, February is 2, December is 12, but that's going to encode an incorrect bias into our data because that's going to suggest that January and February are close together. January is quite close to March. We're talking about January and February 1 and 2, January and March being 1 and 3, but January and December are very far away from one another because then we're going from 1 to 12. But actually, that's not what we want to suggest at all when we're doing seasonality because after December, we come to January again. So we want, if anything, December and January to be close to one another. so just converting this to numbers is going to be insufficient instead what we're going to do is one hot encode the data so if you're not familiar with that essentially we're going to give each of the months its own coefficient so we'll have a coefficient for January a coefficient for February, et cetera. And so they're all going to be treating these months independently. And that will mean that each month is as close to the other month as well. Yes? Why not do something like a step hold? Something like you do the fallout for the next? Sorry, could you repeat that louder? Could you do like fallout for the next, but each month you do the same step for the next and the third month? I'm not exactly sure what you mean by polar. So the question was why not to use polar coordinates, why each of the months are as close to one another. I think essentially with the one-hot encoding, that's what we're going to do. Because what we'll end up with, if I... So, with our data, instead of having just a month column, right, where we maybe have January, February, December, hopefully you can read my terrible writing, we're going to convert this so that each month is going to get its own column. So we'll have January, February, December, and then obviously all of the months will have their own thing, right? So then for January, we've got 1, 0, 0, 0, February, 0, 1, 0, and then this has the effect that each of these points will be as close to the other. I think maybe that's, maybe we're kind of referring to the same thing with the polar coordinates. So, okay, so different, but still, I guess we're going to have the same result there. Yeah? I think the problem with polar coordinates is that still, once you transition from, say, 60 degrees to 0 degrees, or, like, 40 degrees to 70 degrees. Okay, right. There's still the problem, but there's a whole, like, field of statistics, but actually, my question would be, is there any machinery that you know of? I think, so that was a question, again, to do with polar coordinates. I'm not going to answer that now, obviously, because I think we would get a bit sidetracked, but maybe we can discuss that at the end of the session. Okay, so we've discussed what our model is, or the data we're going to be sharing to our model. and what I'm going to do here is some nice scikit-learn pipelining using the column transformer to just tie this all together I'm actually using drop first one hot encoding so I'm only going to be making 11 months rather than 12 and I'm dropping the variable of January theory, not to go too deep into the theory behind that, but it can be harmful to the model if we don't do that. So encoding these variables, passing it through, fitting my linear regression model. And let's just remove that so we plot the whole data. Okay. So you You can see now that we've, because we were passing in seasonal information, we're able to capture the seasonality of the data as well. Obviously our predictions that we're giving, it has these shifts as we're going from one month to another, and that's to do with the way we've encoded the data, right? You have either in one month and then you're suddenly in the next month, so we do have these shifts as we switch between the months. So it's not smooth, but we are able to capture the seasonality of the data pretty well here. I'm not getting into any metrics for actually assessing how out while we're capturing that at the moment just for the time restrictions. But obviously, maybe you'd want to calculate some metrics on this data as well here. But with this, we are able to separate the trend and seasonality components of the model. What I'm getting here are the coefficients of that linear regression model. So when it's February, we're adding minus 0.3 to our predictions. When it's March, we're adding 2.9 to our predictions. When it's April, we're adding 1.8 to the predictions. And when it's January, we're just adding 0. We're adding nothing. But what we can do is take the average of this. So on average, over all of the different months, we're adding 0.86 to our predictions. So averaging out these seasonal effects, so not actually worrying about what month we're in and just always adding 0.86. we can get back the trend from this model. So even though we encoded the data with seasonal information and we got out a model that gave seasonal predictions, if we average out those seasonal effects that we learnt, we can get the linear trend back. So now we have predictions which capture that seasonal information and the trend, we've been able to isolate the trend. if we then take if we then subtract our thing what from the trend we can then isolate the seasonality so and let's just make the plot so you can show you what I can show you I mean so So I'm plotting here just the seasonality. And I've made the seasonality column. Let's just show it without the plot to begin with. So this is the seasonality column I'm making. And I was able to make that column by taking the linear monthly and subtracting the trend. So taking our predictions, which included seasonality, subtracting that trend that we were able to isolate, and then we can capture just the seasonality by itself. So you can see we've got zero here. Just for reference, so there's no trend going on at the moment or very minimal trend going on at the moment. You'll notice that the seasonality effect seems to be greater during the winter months and less so during the summer months. So the impact of being in the winter months is felt larger than in the summer month. That's what we were able to learn from the data as well, which is an interesting insight. OK, so just see where we are with time. What I'm going to do is just briefly go over the next section to just give you the highlights of it and give you the intuition about it as we are coming towards the end of the session. And then we'll have some time to just wrap up and take a few questions as well. So at the moment, our predictions are nice. We're able to capture our trend and that seasonal patterns in the data. One issue at the moment, though, is that we're switching from one month to the next. So we feel a certain effect just for January, and then as soon as we get to February, that effect completely changes. Whereas, actually, probably that's not what happens in real life. It would be much smoother. So you maybe feel some effect in January. As you're getting to the end of the month of January, you're feeling that effect less. you're maybe starting to feel the effect from February. And then as you continue, as you get to the center of February, maybe that's when you feel the effect the most. But maybe you still feel some effect from January then. It's not just switching from one month to the next. And as soon as you go to the next month, you feel something completely different. So how you can encode that into your data is with a technique called gradual seasonal filtering. So you can do this with a few types of filters. One would be a linear filter. Just have some code to make this filter here, just as a demonstration. So I've chosen, so I've got 365 days here. I've chosen 15 just as an example. So if you really felt an effect on the 15th day of the year, maybe you, yeah, so on that 15th day, you really have a peak. And then as you go on with the year, that effect gets less and less and less. And likewise, when you were getting towards that day, that effect was getting larger and larger and larger. Because of the particular settings I have for this filter, you can see it's actually starting to have an effect right at the end of the year, getting larger and larger and larger until you get to its peak. What we would like to do then is have some of these filters for each of the different months. So have a filter for January, February, March, April, May. So for each of these different months, you fill an effect. and it's not the case that you just feel an effect singularly from one month. It can be the case that you're feeling the effect from a few different months. Okay, so then this would be the data that I'm choosing to add on, or I would use to model with. So I'd have a time point variable, and then rather than having just a binary indicator showing what month we have, we have something where we're going to feel an effect depending on what month we are so you can see here we've got the first of August so you can see at that point we're kind of halfway through the well we're about equal distance from the middle of July and the middle of August so the effect we feel is rather similar as we go further on in August we feel less of an effect from July and more of an effect from August. Alternatively, if you want something a little bit smoother, you might want to use a Gaussian filter. So it's going to be the same idea, only we're not going to have something linear. We're going to have a Gaussian. And there's a small exercise here where you can do some modeling including these Gaussian filters. I won't go through the code. I'll leave that exercise to you, But just to demonstrate what you can produce with that is something a lot smoother when you're talking about capturing that data. OK, so just to summarize those gradual seasonal filters, they are a simple feature engineering trick. It still allows your variables to be interpretable and your seasonal effects to be quantified. Obviously, with all of these things here, you are encoding some bias. You're assuming some seasonal effects. I would say for time series data, you do as a modeler have some information though already, because you know that effects will happen and not happen, depending on what time of the year it is. And actually, I personally really like having that opportunity to be creative and try and model those effects which I know are happening. We've seen how we can do that with just dummy indicator variables. That gave us the shifting effect depending on what month we were in. We can do it a bit smoother with the gradual seasonal filters. But you could also try maybe adding a sine curve. That could also help capture some seasonal effects. And then once you have this, as well as using it for forecasting, you can use it for outlier detection. And obviously, something we haven't discussed just through time restrictions is model evaluation. But that might also be something you want to look at and so you can properly compare the models you're creating. OK, so just a few closing remarks, and then I think we'll have time still for a few questions. Yeah. OK. So just to briefly summarize what we covered, Marisha took you through timestamps, time-based aggregations, creating time-based features, And smoothing time series data, we then discussed time series decomposition, modeling time series data, and engineering seasonal features. So obviously, this was just an hour and a half workshop. So we tried to fit in as much as possible. But obviously, there's a lot more to cover. You'll find in here, we also have a third notebook, which has some extra exercises that you may wish to cover. But yeah, if you're really interested in time series and you'd love to learn more with us, we do have a two-day course which allows us to take a bit of a deeper dive into this. So there's a short link there on the time series training.

Speaker 1 [76:59]

Yeah, we're based in Amsterdam, of course, so we do have in-person trainings, but we also do a lot of trainings online So this is something that you could join remotely as well if you're interested

Speaker 2 [77:09]

And, yeah, we do have a booth here, so some of you might have seen us already. We had the Drop Blocks Challenge. The R2-D2 has been handed out now, but it would still be great if you come up and have a chat with us later on. Yeah, I think we have some time for one question, so make it a good one. Pressure's on. Yeah, alternatively, we'll be around if you want to come up afterwards. Up to you.

Speaker 1 [77:52]

No questions whatsoever?

Speaker 2 [78:03]

Thank you