Common issues with Time Series data and how to solve them

Speaker 1 [00:02]

So, good afternoon everyone, and I'm delighted to see such a crowd, so it kind of proves that the topic is relevant, so let's see if I can meet the expectations. First, before I actually start talking about what the problems are and how we solve them, a bit of background, because of course, who am I and why am I somebody who can tell you about time series and how to address this, right? there's a lot of a lot of things out there so I as was mentioned I'm from the Nellis I work for a company called Xebia, Xebia data specifically and we're essentially our consulting company helping in the domain of data so that means in my day-to-day life I join different companies usually sort of maturing data teams that are doing great stuff but kind of lagging behind actually putting stuff to production effectively so that is essentially identifying issues they have, helping them address that, and well, just very often it happens to be time-serious type of problems, so over all the time I gather quite a bit of sort of common problems that are simple after you actually know them, but not that obvious at all when you actually just skip it over. So that's a little bit what we'll see today. but before that maybe someone got in the wrong room or just never heard about time series in general so why should you in general care well as you as you probably know and i think it was mentioned in the previous keynote talk right essentially time series are a major part of pretty much any company's agenda because we care not about just today or yesterday we care about things in dynamics, right? And time series problem is essentially anything that you care about measured not only today, once, but measured continuously over time. And that essentially gives us a beautiful ability to learn about what might happen in the future based on what happened in the past, right? But for that we need to take into account all this complicated time-related stuff which makes the problem sequential, so not just a bunch of random data points like you usually see in sort of more traditional machine learning problems. And it also happens that there are a lot of sort of interesting artifacts and problems that originate specifically from the nature of time series and timestamps themselves. And as it often happens, so we data specialists, we're quite versatile, right? So not everyone works exclusively on time series. So it's quite easy to overlook some of these things. Further, I should mention that not only it's a very prevalent problem, it's also a very important one, right? Typically, it coincides with things like predicting your revenue, the demand, and a lot of other things that you kind of want to get right, so that means that all these problems are not just interesting, but they're also pretty important. And that's pretty much what we will try to address today. I'll just mention that, again, as I often see, most of these issues are kind of downplayed are considered unimportant, so hopefully we'll prove it otherwise. And specifically I'll walk you through just one particular large project that I worked on last year that was at an energy company called Vattenfall where we worked on energy demand forecasting, so quite a common but very interesting problem. And actually it coincided that their team kind of stumbled upon a lot of little problems along the way that I think kind of nicely captures typical things that I see elsewhere, so hopefully there will be additional narrative to what we will see in a bit. And specifically we'll cover four different problems, each in a different stage of working on developing a solution to a time series problem. And for each of them we'll discuss, well, where do they come from, how can we best address it, or how should we not address it, and for a few of them I'll also provide some simple code examples, even though that's not the major point of this talk. And yeah, you will see a lot of time series plots, which is great. All right. But first sort of a bit about the dynamics of how a typical time series project typically goes. And I think a lot of you would immediately think, oh, time series, let's do profit, LSTM, just train some fancy stuff. But that's a couple of steps further, right? There are some important steps that we kind of forget about because they're not the most exciting ones, right? You need to get your data together from different sources. The quality is probably questionable, there are a lot of artifacts you need to detect, and then for all of these detected problems, you need to choose a way to address them, and there's kind of no rule of thumb solution in this world, and all of that kind of piles up into a lot of things that, as you can imagine, all affect your end result, right? Because as you know, garbage in, garbage out. So actually for three out of four issues, there will be all before training models and not after or during. So let's start from a rather simple one, which I think most of you can relate to, but yet it will show an important lesson that we'll see sort of reoccurring throughout this conversation. For instance, seeing a problem and spending a bit of time to understand what is really going on instead of taking an obvious shortcut which often will mislead us. And that is essentially about a common problem where you need to connect two different, well let's say data sets or pieces of information sources together. Usually it's based on the time, right, so essentially when did each piece of information come from. And here I have a little token example of essentially observing energy costs over time simultaneously collecting data on temperatures. It doesn't matter here they are together it could have been also two disjoint data sets that's not the point. The point is we need to make one important step and that is figure out okay how do we exactly match today's energy costs with the exact temperature there was today and as you can see here it doesn't exactly match but from a first glance you can see like it looks like it's just lagging behind so probably the temperature sensors data is uploaded the day after while the costs are known immediately, which could have happened. And again, our goal is to address this and really make sure that things match as they should. What are possible issues that can trigger this type of data quality issue? Well, very often, as you probably encountered in a lot of time series, data doesn't just come from your company or some kind of reliable source, very often it's governmental data. I've actually worked quite a lot with data from the Dutch statistical agency and you actually see a lot of it is even sort of hand inputted because I saw like 20 different ways of writing February, for instance, which is kind of remarkable and it could be a lot of other things, right? So different types of sensors, different types of data providers. So I think it's a safe assumption to just think probably the input data is a mess, especially if we have different sources. So back to this problem. What is kind of the obvious assumption that probably many junior data specialists would make here? Well, it's probably just lagging, right? So you kind of looked at the head and you saw that maybe you just saw all of them actually looking like they're from yesterday. So the first idea, well, it's lagging. Let's just take a delayed version of that and the problem is solved. Look at it for a second because there's clearly something not going right, don't have to shout out the answer, it's just reflected, and as you could have noticed then, we actually should have had an observation in the second row, but instead we're missing it, right? So you actually see that the dates are not matching if you carefully look at it. And again, like here it's obvious, but if you're just looking at a bunch of random rows from a large data set, that's really not something you would immediately spot, right? So the real mistake was to make an assumption that you could have verified, but you didn't because it seemed like a trivial problem. So what could be a better solution? Make it explicit. Right? So my biggest tip, and it applies not only to this problem, don't take shortcuts, don't think that obvious things are always the way to go, always take a step back, look at the full problem, and think about what is the most reliable, explicit, and clear solution to this problem. So even to the simple problem, the most explicit solution is to make sure that every date matches by construction. So when you join based on the date, right, whatever is a problem will explicitly come out of the result of this joining, right? So you see here that since we requested it to be joined on the dates explicitly, whenever there's a mismatch, so if a date is missing or something is duplicate, it will be reflected respectfully. And that means you can directly then measure what are sort of the data issues that originate from this operation, and most importantly, you don't just leak some kind of flawed human logic into your data, and as a result, into your model as well. An additional little tip, it's also just here, of course, it's available, but it's great to have timestamp for every piece of collected information, right? is also what I've seen is you get temperature and it kind of looks like almost a full month, but maybe a couple days are missing and you actually don't know when it was collected. So if you in any way control where the data is coming from, getting timestamp for pretty much every step that you can measure is usually a good idea. Also short remark, if you have questions, because there'll be four blocks, maybe just make a mental note and we'll come back to it in the end. I think this will be better for the flow of the conversation. All right, and to sort of wrap up this first block, let's have a look at a potential code solution, which is perhaps not super exciting, right? Because I think the majority of you have done joins and merges before. Anyone who has never done or seen it? Great. Maybe something less obvious, right? So also make it explicit. So which, what kind of columns you're merging on, or maybe even some additional checks. And I don't have it here, but also additional tip testing, right? So whatever you're doing, make sure there's a test that kind of measures the result of the operation. But if you did it this way, instead of a lag, you wouldn't have any issues with this problem. All right. Then if this one was clear, let's move to the second step, right? So we've just seen what could go wrong potentially during the data collection stage. Again, don't think that this is the only thing that can go wrong. Of course, there are multiple others, but hopefully each of these blocks kind of gives you an idea of what you should typically expect so that you can take this time yourself for your problem. So the second step, and again, also don't take it as ground truth always. For you, this might look a little bit different, but again, it kind of gives you an average truth of what I would typically see in a time series project. So the second step here is inspection, analytics, and finding issues, right? so essentially you've got everything you need, but again, as I said, just assume that there are problems, because almost always there are, and when you think there aren't, that's even worse, because it means they're very hidden. So in this case, we'll see something that might look like there aren't any issues at all, but in fact, if you would look at it a bit more carefully, you would make your life in the future much easier. So let's have a look at this time series, right? So you see some, this is generated data, by the way, and no company data was exposed. But this is a pattern that you see very, very often, right? So essentially some cyclical fluctuations, right? A lot of business data in time series has this sort of repetitive seasonal pattern, right? Think about energy consumption throughout the year, more in the winter, less in the summer for the heating. Unless you're in California and you use crazy amount of air costs. Either way, right, you have these fluctuations, but you have something else here, which isn't very obvious, that is the increasing magnitude of the fluctuations. Sometimes it's just part of the problem. It's just sort of what happens behind the hood. You have no way of identifying what is actually happening. So I've seen a lot of data specialists kind of trying to explicitly tackle this problem. So if you train models, you could do something like multiply your features by the time feature to sort of add this dynamic component to the problem. So you could, in principle, solve this with some kind of, like, increasing magnitude sign curve translated in the future. But if you, again, take this fast brute force approach, you're actually trying to solve more problems than you potentially can solve, or at the very least you make your life much, much more difficult. And if you think about this, right, so this is energy usage over time. Again, think to yourself, don't have to shout it out, what is probably going on behind the hood, right? Let's say we're like a young, emergent energy company that's becoming increasingly popular over time. Well, that's what every company wants, right? We're growing, so more people are joining in. And if this happens to be aggregate data, it's quite natural that this is essentially a function of the number of customers that we have. And very often it would indeed turn out that if you look at something like number of clients that you have over time, it's something like this, right? Which essentially translates into that increasing fluctuations that we saw before. So if you took a step back and thought about that and identified this cause of the issue that we saw before, you can actually now disentangle the problem and say, okay, I'm not going to brute force it and solve sort of aggregate energy consumption over time that incorporates the growth of the business in it. I will separate. So we'll essentially see for, essentially if we divide the consumption by the number of customers, we have one problem that is interesting for us that is just, let's say per household, what are typical seasonal fluctuations of the energy consumption, which is one problem, and another problem is predicting business growth over time, right, which are both interesting, relevant, important problems, but this way maybe two teams can work in it, both have their own approaches, issues, and then you can always combine the results together. And of course, debugging and evaluating this will be also way, way, way easier. Let's see. So if indeed you divide it, in this case, you get this amazingly equal generated data. I don't have a code example here because if you all did merges, I think everyone has divided one column by the values in the other. So allow me to skip this one. All right. got the third one and it's going to get a bit more advanced and interesting perhaps if so far you've been warmed up enough. So that's the third step, so that's right before you get your data ready for actually doing something with it, that is when you have already got stuff together, identified all of or at least most of the potential problems in it, and then you're trying to address it. Again, there could be all kinds of issues, I'll show you one interesting one because I think there's quite a bit to learn there, and it also generalizes well to related types of issues in time series. And also what is nice in time series is that the solutions that you know from elsewhere don't really work so well. Specifically, that will be about imputing missing values in time series. As you probably know, for regular tabular data, it's not so complicated, just replace with the mean, median, most common value. Now here, well, you could technically do that, but of course, remember, everything is dynamic, right? So it's important not just what the value is, but what are the neighboring values? So where are we in the whole dynamic over time? And in this case, you see that it also unfortunately happens that all these missing values are kind of clustered together, which also very often happens in reality. Think about things like weather making impact on things. So a real example, we're also looking at generated wind power or like maintenance of the windmills in the ocean. As you know, there's like a million of them between England and Netherlands. And if there's like a huge storm or something, pretty bad is going on, then this is also when you will get all kinds of data anomalies, right? So what's important, they all cluster together, so if you try to use some kind of simple brute force approach, not only it will not be a very good solution, you will actually introduce something that is additionally problematic. So actually, you'll do something that I would call anomaly propagation, so you'll basically, will introduce a problem and exaggerate it over time, which kind of only makes things worse. So for example, right, if you take a look at this three missing values over here, if you just take a common approach, which is just kind of backfill, so you take the previous value and copy it for the following subsequent missing values, you can immediately see we kind of generate this relatively high value flat curve, which is probably not something you would observe in reality, right? Well, again, here I happen to actually have the true values, and this is actually from a real data set that's, I believe it's from 2008 French household energy consumption data set that you can look up on Giggle if you're interested. But essentially, what we want to do is to replace it in a meaningful way that kind of addresses the whole dynamic pattern and not just what happened yesterday or the day before that. So again, we're gonna take a step back, look at it from this perspective, and realize that we need a more flexible dynamic approach that will address the actual issue instead of some kind of duct tape and glue as we've seen elsewhere. And there are multiple clever approaches that you can take here, so I'll just show you one that I personally like quite a bit. And that is using something called exponentially smooth values as a replacement. Now that's something not everyone of you might have heard, so bear with me, I will explain it one by one. Don't look at the bottom graph, so let's focus on the top one that will make understanding a little bit easier. So what you see here, the dark blue values are what we actually know, right? And there is the, I believe, the bluish border and white inside, that's what we're actually missing. And the green ones, again, this is something special, so we're doing something called exponential smoothing. So most of you must have heard about at least something like moving average, right? This is kind of a more sophisticated solution in that family of smoothing algorithms. Again, the math is a little bit less trivial, so I'm not going to write out whole equations, but there is a simple idea behind it, is that every exponentially smooth value is essentially a weighted average between yesterday's smooth value and today's actual value. There's like a parameter you can play around with, I'll show you in a second in the code. But again, the idea is that it's something that we kind of start growing from the beginning, And it takes sort of a geometrically declining history into account at every single point. But at the end of the day, it's always a balance between today's value and some sort of smooth value yesterday. So if you got that, then you're already with me. There is an additional problem, though, right? So you start growing the smooth curve, and then it encounters a missing value. So we need to add an additional solution to that. And this is actually where it's okay to use the previous value, just not the value of the actual observation, but actually the previous smooth value would be propagated for the subsequent missing value. Right? Remember, we're waiting today and the smooth value yesterday. If we don't have today, well, let's just copy the smooth value from yesterday. That's why you see, for instance, here, right, it's the green point that is being propagated three times, not the dark blue point. All right? So if you understood this, now actually just a little part is left. That is essentially taking the green points every time we have a missing value and replacing the missing value with those points. Remaining green points, well, here we don't really care about them. But as a side note, well, exponential smoothing is kind of a nice tool to keep in your pocket anyways. So you might use it elsewhere. So now to the bottom plot, because here we're actually combining three different things. So we have, again, the dark blue points, the actual observed values. Then we have the red points. Those are the actual unobserved values. If we knew the truth, that's ideally where we want to be, right? So the closer to that, the better. And then we have sort of two candidate solutions. So yellow is forward fill. I think I called it backfill accidentally, so no, forward fill, right, in this case. So from yesterday to tomorrow. And that's exactly what you see here. And in green is our new favorite, essentially replacing with exponentially smooth values. And as you can see, actually most of the time it's doing quite a lot better. Of course, this is like kind of a cherry-picked example, but trust my word, if you do it for a bunch of problems and you compare the solution, which is something you can actually measure, so feel free to play around with this as a sort of exercise, maybe to try to prove me wrong or find whether that applies to you or not. But very often you'll see that this works quite a lot better and I think it also has intuitive sense to it, right? We're not just looking at whatever happened yesterday, which might be also affected by a lot of anomalies and other factors. We're looking at the dynamic history and that's what you care about in time series. All right, then let's have a look at this code example. It looks complicated, But actually not so much is going on here, because in Pandas it's quite easy to implement exponential smoothing. Essentially all you need to do is use this method here with a parameter that is the weight between the two, and then this will ensure that we forward fill the missing values during exponential smoothing. Once you do that, you get your green curve, then everything that is left is essentially filling, and this is actually very easy with this method that is combined first. So if there is a value, we don't do anything. There's a missing value, we get it from the second column. So quite neat. So quite a complicated approach, all fitted in just two lines of code, essentially. All right, well, then the last but not the least, probably what we all thought about in the beginning, it's the actual modeling. That's kind of a chapter of itself, right? So I'll bring one just to kind of make it the closed-up story, but I think this is where many more issues would be found. I think the first three are kind of the more authentic one in this talk. But here's an interesting one that will also make you think of the same idea, right? It's essentially taking a step back, looking at a problem, and only then starting to solve it. So here we see, again, that single household energy consumption over four years. And we're trying to predict it essentially on a daily basis. this, and this is exactly what the model output would look like. I would kind of omit what the model is here, that's actually not critical. What is important, we're just using the raw data as the input here, essentially. And that's also very often how people go about it, like, okay, we have the raw data, let's try to feed model over it, they don't really think about, okay, how does it actually translate into what we're trying to address. And as you can see here, there's kind of a lot of weird things with the model trying to pick up all of the individual fluctuations, that might not be something that you really care about overall. So this is where I would, again, recommend to take a step back and realize that, well, maybe those crazy fluctuations is not something you're actually trying to predict. Maybe you just care about how does an energy consumption pattern look like over time on average, right? And that actually should have implications for what you're feeding into the model, right? Because if you're telling the model, try to predict all of that, it will try to succeed at all these crazy peaks, and that's actually not something you wanted. So if you took a step back and you kind of reformulated the problem, you might now actually use smooth values, and here you can again use my favorite exponential smoothing, because maybe that's actually something you care about predicting, right? So essentially we figured that we can transform the problem and not only make it easier for the model to predict over, but we actually get closer to the truth that we're trying to address, which is arguably something we should aim for in a time series project. All right, we're kind of short on time, I have here a little example of how you would do that in principle, but actually it all just breaks down to replacing your train set with the smooth values rather than the original row values, so allow me to skip that one because I still would like to address some questions, and maybe just a little side note. If you find it interesting or fascinating, just reach me out, connect with me, and I'm happy to address more issues on that matter or just time series in general because that's for sure a fascinating topic to talk about. So the last bit is, again, take a step back, look at your problem, make the right choice, and this will get you two steps forward. Thank you.

Speaker 2 [26:11]

Thank you very much. There are six questions. I don't know if you have time, but let's start with the first. It says the second topic was disentangling of two time series. Do you have any tips on how to disentangle if the underlying relations are not so good?

Speaker 1 [26:26]

Yeah, that's kind of a million-dollar question, right? I don't think I have a rule of thumb here. I would say the best tip is talk to... So the question was, how do you disentangle two-time series like we did with aggregate energy and customers when it's not obvious? Again, there is no rule of thumb. I would say my best tip, talk to people that actually get this data, be it, I don't know, workers building the windmills or analysts collecting the data. That's the best way to get to the truth, but probably the best second advice here.

Speaker 2 [26:59]

So another question is about you talked about exponential smoothing for imputing

Speaker 1 [27:04]

Uh, you know,

Speaker 2 [27:05]

Can you take uncertainty into account if you do other methods?

Speaker 1 [27:11]

Can I take what into account?

Speaker 2 [27:13]

uncertainty into account if you use other methods than exponential smoothing

Speaker 1 [27:17]

Yes, so there are definitely more sophisticated ones, right, so this is kind of to find the middle ground for a general audience, right, there are more sophisticated methods where you can kind of construct your own type of smoothing which indeed would take sort of the local uncertainty into account as well and if you really want to be precise that's indeed a good idea, but I'm not going to throw around technicalities here, I think that's too much for an average person.

Speaker 2 [27:43]

Another question is about how would you in a real world modeling problem where you do not know the missing data point values decide which smoothing algorithm works best for filling?

Speaker 1 [27:54]

also great question the best solution is I would say hyper parameter tuning so that's something you can actually parameterize just like there was this alpha remember the weight I nobody asked right but that's also like why 0.5 and not 0.1 parameterize right so use it explicitly as a parameter of your machine learning pipeline you can do like a custom transformer and so I could learn for that and you can perfectly determine the answer empirically right doing your favorite cross-validation and grid search or whatnot.

Speaker 2 [28:25]

So maybe a very last one, how would you train a neural network when time points with data is heavily irregularly spaced, impute missing values also over long time period?

Speaker 1 [28:37]

Well, it's a great one, but I'll say that could be a great topic for another talk, so I'll address it at the next opportunity like that. Thank you.