Predictive Maintenance and Anomaly Detection for Wind Energy

Speaker 1 [00:03]

Right, okay, so hello everyone. Thank you for coming here so numerously, I guess. So yeah, as he said, I'm Tobias Hoenke. I have been working for EMBW for like 1.5 years roughly and work there on predictive modeling and anomaly detection and basically something that's called condition monitoring. I'm going to try to explain to you in this talk what kind of problems we face and how we're planning on solving and how we're solving them currently and all the challenges that are ahead and so forth. So this is a table of contents first a little bit. At first, I will talk about condition monitoring at EMBW. So I will talk about an application called EMBW Asset Radar, which is like a proprietary software that EMBW develops in-house. I will talk about data sources, So what kind of data do we have available about our wind turbines to monitor them? I'm going to talk about anomaly detection first, like in a more general sense. I will define what that actually is, and maybe you'll see that it's actually kind of difficult or more difficult than you would assume. I'm going to talk about predictive modeling, then what's the idea behind that, how do you do that, what kind of challenges are there. And then I'm going to talk about that topic specifically and how we use that at EMBW, how we do it, and then about future plans, a concept we coined as anomaly space and how we use that. And then a hopefully very helpful example to understand what we're doing and why we're doing exactly that. All right, so let's get started. So first of all, I want to talk about AssetRadar. So AssetRadar is, as I said, like a proprietary application. And the way it works currently is it monitors roughly 450 wind turbines that are located in Germany or some other parts of Europe, mostly Sweden, I think. And the idea there is that it monitors all relevant components. That means not every single one of them, but only those that are judged to be important for maintenance. We use multiple detection methods, so it's not really just like one thing we do, but we actually have like a set of either established or customised methods we use to keep track of our units, and all that is, as I said already, bundled in proprietary software, and it basically collects data, like in this little graphic on the bottom there, you see the parks collect data, send that to a park server, and the server then sends that to the cloud, and everything that's in that little bluish, I suppose, cloud, that is pretty much what Acid Radar does. And Acid Radar then sends alerts to diagnosticians, and those are actually humans, right? So these are real engineers that know wind turbine energy and actually look at the data and confirm whether what is spotted by Acid Radar is in fact a a real concern or not at all and then they will decide then how to proceed with that like usually they would indicate stuff like on-site follow-ups or stuff like that all right the objectives overall of acid rater are to minimize on-site maintenance that's really important so we want to go there as little as possible and as early as possible because the rule of thumb is the earlier you repair something the cheaper it is so and you really want to have those cheap ones because basically if you wait too long the worst case is you have to basically replace the whole thing and you don't want to do that except for the tower that's usually not something that breaks very easily yeah and we want to monitor all units operated by EMBW that includes actually solar power plants and water power plants and so forth we want to use all data that is available to us us, so we don't want to skip on anything or miss any defects, and the main goal is continuous development and improvement. So we don't want to be stuck on like one state and just ship that forever, but we want to keep going and use the knowledge we gather from the diagnosticians and put that into the application. Right. So that is about that. Now I'm going to talk about data sources, and one of the most important data sources I'm going to talk about here, that's called SCADA. It's short for Supervisory Control and Data Acquisition, something I only learn when I talk about it in talks. But yeah, we usually call it SCADA. And what SCADA is, it continuously collects data from multiple sensors like temperatures, pressures, currents, all kinds of physical quantities, and then aggregates that in 10-minute intervals. So we don't really look at the raw data, which could have any sampling rate. I don't even know actually what the raw sampling rate is because we only look at SCADA data. And these 10 minute intervals, they are summarized in the mean, the min, the max, and the standard deviation, and that's pretty much all we work with. And that's an example, I guess, of what it looks like. So those are just two really random signals. The top is like a bearing temperature, and the bottom is like a wind speed that is measured by something we call an anemometer, which is like an instrument to measure wind speed. So the shaded area is like everything between min and max, and the straight line is then the mean. Usually we have a standard deviation too, I didn't put this in here so you can see anything at all. So that's the data we're working with on the SCADA side. Then we have oscillation data that's a tad more complicated. So that's basically acceleration data collected in strategic locations in the machine. Really the way you can think about this is imagine like tiny little microphones that measure what kind of noise the machine is making. And those are not continuously measuring because you need very high sampling rates for this, like up to 50 kilohertz or something like that. You can't really afford that on these tiny computers there on the machine, so you do that irregularly. You just kind of like trigger a measurement when it's useful and then you have a few seconds of like an audio clip of what the unit sounds like. It's kind of like when you sit in a car and something's wrong with the car and you can kind of hear where it comes from, that's basically what this is. And that allows for a very specific analysis of the faults, because basically the way it works is you have the drivetrain, so the drivetrain, I have to explain maybe, is everything that rotates in the wind turbine, that's part of the drivetrain. So the rotor blades in the front, and then the generator in the back, and then the gear in the middle. And you have a lot of translations of these rotations into different frequencies, so whenever you see, maybe I could just go to the next one, and whenever you see like a frequency that peaks, you know which part of the turbine is creating that peak. So when you see a growing peak over time, you usually know like, okay, something in the turbine that is rotating with that frequency is having some trouble there, right? So here, maybe for some more explanation, top is the raw data, so really like an audio clip. You could actually listen to that. You can actually listen to that in ACID Rater, I think. And the bottom is just a spectrum like an FFT if that says anything to you. So really straightforward processing there to analyze the frequencies. All right. So now I'm going to talk about anomaly detection and why it's such a hard problem. So I try to, like, explain this with as little formulae as possible. Let's see how I fare. So basically, when you would ask yourself, like, what would I consider an anomaly, you would probably say something like, oh, yeah, well, an anomaly is something that's not normal. And you will probably catch yourself using that word in the vaguest of terms, because it's actually very hard to define what is normal. And usually, all kinds of things go into a definition of normal. Like, it's not just, you know, something very objective but it's actually something like the business case is interwoven into your definition of normal and in the case of a wind turbine what do you want the wind turbine to do right you want it to produce electricity and you better hope that is as much electricity as can be produced right so that is your concept of normal but it's still it's not that simple really because it's a very technical term and a lot of complexity is involved in there so that's why i said it's very much non-trivial. Statistically, maybe you would define something like a density or something like that. Here on the right, you see I called this some feature. It's just some toy example of data. You can plot a histogram of that or something like that. Then you can maybe fit a PDF to that model. That's the orange line. You would say an anomaly is an observation you made in a region where that orange line is very small, basically. So everything above 1.5 or below minus 1.5, you would maybe consider an anomaly in this case, or an outlier, depending on how you phrase that. So in this univariate case, this seems to be very easy, even though calculating a density from data alone is notoriously difficult, as some of you might know. And it gets more complicated when you look at multivariate data. So here I extended the data from before by another dimension, and you can see here this orange star in the middle, you would maybe consider completely normal in the univariate marginal distributions or these projections, really, but in the joint view of both features at at the same time, this is actually kind of obviously a really odd one there in the middle of this strange curve there. So and of course this problem gets harder and harder the higher the dimensionality of your data. And another problem that was actually kind of obvious with the Univari case too is the decision has to be made like when is p so small that you will actually frame this as an anomaly, right? Like that is actually kind of a very subjective decision, and it's kind of tied to the question like how often do we expect anomalies to occur? And also a follow-up problem of this is if you can never really objectively decide whether something is an anomaly or not, how do you find like a, you know, a data set that is actually completely normal that you can train any algorithms on, right? And the time series, we have time series in this case, and time series are actually especially difficult because really every data point kind of depends on all previous data points and you can actually easily find samples that look completely fine when you look at them in an isolated way, but the way it evolved over time is completely off. Right. So one way out of this or like one treatment of this, I suppose, is predictive modeling and I'm going to present to you like the idea behind it in the context of wind turbines. So really the expectation is that if you have a healthy system, then pretty much there should be very stable and robust relationship between certain signals or certain data that is provided by that system. In the case here of the wind turbine that you see here on the right, what does the wind turbine see? It mostly sees wind speed and direction, it sees air temperature and air pressure and so forth, and it really just reacts to these environmental features. It kind of works autonomously for the most part, and then you should actually be able to predict all other signals from this, right? This would be like the wishful thinking or whatever. And so there's really no reason why you couldn't replace that wind turbine by a statistical model. So why don't we just do that? Try to predict every other signal, and if we get significantly worse than usual or than in the training set, then we know something's off, right? And those deviations from the usual predictive performance we could basically just label a defect, right? That would be basically the idea. So the way we use this at EMBW is fairly simple. So this is an example of, like, three input signals and one target signal. In gray, the shaded area is the training period, and the training period is mostly determined by those diagnosticians I mentioned earlier. They would look at the data. They will see, like, okay, this first year of aberration, for example, looks fairly innocuous. Let's use it and maybe we'll cut out like a few weird looking bits of it or whatever and then we'll have our training period It's very important that a training period by the way is one full year of data because it turns out that seasons are a thing And you kind of want to account for that So this training period then we split up in a train and test set like this maybe we usually use a one to six ratio, which makes no difference, but that's what we do and and on the train set then with the train set we train a regression model we for the most part use just GBRTs and train regression model on that and then evaluate that on the test set and from the test set we then get like a set of model deviations that the model has never seen right on samples that the model has never seen and that test set of model deviations we can really use to gauge our our system and to answer the question like how when is a model deviation unusual enough to give an alert or whatever right so this is basically then the result we have model deviations on the test set those can you know theoretically have any shape that can be skewed or long tails or whatever and then we want to decide like what is unusual the way we do this right now is we just do a make a statistical modeling of this of some kind that's the orange curve here and then we basically set like a lower threshold for this and an upper threshold for this that can't be crossed uh based on really pretty much like a quantile that is indicated by the statistical model all right so that's what we do with that okay then uh one well a problem really to to to solve is this trade-off here between specific and generic. So what this really means is we could theoretically choose a model that is super generic. We just dump everything into the model and try to estimate from that dump of inputs everything that comes out. You could, for example, think of an autoencoder of some type, right? That would give you the advantage of very few models, in this case maybe even just one. And you have very complete monitoring because you can just dump everything in and you're done, basically. But the problem here is it's extremely hard to interpret these models. For the most part, because of, you know, you may think something like, well, every signal that I didn't reconstruct probably is the signal I want to look at, but usually you have issues with cause and effect in these models and it's not that easy actually to do so. Also another downside is we actually have engineers on board at EMBW, they know their machines very well and we did not use that knowledge at all, right? So we basically look at everything, not at the things that the engineers are interested in. So and then the trade-off on the other end is specific models with few inputs. So that of course gives you then a lot of models. In our case It is a lot of models, like 20,000 or something like that. So that's, of course, sort of a maintenance problem as well. And of course, not every signal that the machine records is necessarily monitored, right? So only the things are monitored that the engineers decide to look at. And the hope is, of course, that the engineers know their machines well enough to know that we won't miss anything. On the upside, though, each model indicates a very small subset of diagnoses. So we don't just know something's off vaguely, like in the case of the autoencoder, but we actually kind of know which relationship is violated in the machine, and we can actually make very specific follow-ups and look at these things in this very specific way. Which is, of course, not perfect. still have like you know a number of input signals and you never know like okay who's the culprit here you always have to check and you always have to check maybe with other models too and so forth so there's still like some manual work to do but at least not as much as with the autoencoder and of course yeah well the engineering knowledge is now there I think that's kind of obvious all right so then on this kind of empty slide you see our future plan of automatic diagnoses so So really the problem we're facing right now is we have a lot of units right now. We have very few diagnosticians. And in the future, we assume that there will probably be more units to come, right? Like way more, we all hope, I suppose. And so we kind of need to have like some sense of scalability. We want to know like how much can we scale this without basically linearly scaling the number of diagnosticians at work. So what we really need there is we need the models to give us, or we need a model, really, to give us more precise pointers on what to look at to facilitate the process of getting to a diagnosis. And the idea here is basically what we want to develop is some type of meta-model that learns from previous defects, puts things together, and kind of like, you know, mix and match certain previous defects with new ones. So but there's of course like a huge number of challenges with this involved like so this cluster of words here Is supposed to give you like a sense of just how diverse the data really is we're working with so it's not like we all Have like one type of wind turbine that we put everywhere, and then that's the end of it, and they don't They they're not all equipped with the same kind of sensors, and they don't have all the same type of specifications in fact like even stuff like components here right like there are wind turbines that don't even have gears so a gear failure on a gearless wind turbine is probably not a very likely diagnosis I suppose so and like that we have all kinds of other things like let me mention like weather and seasons that's also a really complicated one if we for example see a defect and it was used to be like kind of stormy or there were like a few wind gusts there kind of unusual maybe for the season or something like that, then whatever meta model you train on that, that probably will point out that every gusty season is a defect. And you don't want that. So that's kind of difficult to account for too. Like, you know, in the best case, we actually would just place defect wind turbines with all kinds of defects all over Germany and just let it run for a year and then we had enough data. But that's not possible or at least it's not very economical. cycle, so we kind of have to find a way to kind of ignore everything that's irrelevant, so that's a very tough problem to solve. The next challenge is labels, so we have diagnosticians and they do their job, of course, but we still have kind of incomplete labels, because not the whole lifetime of every unit we have is fully monitored in the same way, so we're missing some labels here and there and so forth. And also defects are really rare, right? Like as a data scientist, I can say it's unfortunate as everyone else, like normal people, they will probably say that's actually really good, right? So we're missing labels. I pointed out a few like rough units here. 450 units we have, seven years of average lifetime and roughly 80 monitored signals per unit on average. Actually, it's more, but 80 is just the number of signals we monitor. So that would combine into like 252,000 years of data. With a 10-minute resolution, that's like a few billion samples, really. And that's, of course, a lot. And you would think like, oh, yeah, that's a lot. So just train a huge neural network on it or something. But that's actually not really possible because of the heterogeneity problem, right? So of that data, it's only 0.05% which is actually in any way labeled to have a defect or some kind of, I don't know, unusual behavior. So that's very, very little because you can barely see that red little bar there, right? So that's another huge problem. We just don't have the labels, really, to train anything more sophisticated, and that's also kind of a problem. So our solutions to this problem or our solution to the problem right now is the anomaly space and something we coined I don't know where it exactly comes from but the idea is basically we don't actually look at our units in the sense of their absolute measurements like oh the bearing temperature was 50 degrees or something like that like nothing absolutely specific like this but rather Rather, we just look at every component of our unit in the context of the unit. The question isn't what is the bearing temperature, but the question is how unusual is that bearing temperature, given the current conditions? And we basically collect all kinds of anomaly detectors like that, that always give us a normalised view of how unusual something is, and we work with that. And the reason why we need to do this, again, I talked about heterogeneity, but there's There's also the problem of like slight differences in how the sensors are installed, even in units of the same type, so it's kind of difficult to work with that. And from that, we hope to extract a signature of every defect, and from that we hope that we can actually recognise defects from earlier. And then the question is, of course, why do we need so many detectors? you can see here like this on the right here is like a collection of heat maps for every single detector we have available for like i think one unit and that's just like a little crop of the actual full list so why do we need so many and i have an example here that's hopefully kind of helpful to understand like also what the diagnosticians are really doing so here is an example of a power output model so what that does is it takes like the wind speed and the temperature and like the position of the wind turbine relative to the wind and so forth and tries to estimate from that what is the expected power output of that unit right it's very fairly simple i think and we see here in this region it has actually very unusual deviations from the norm right like before that before this shaded area it looks way different from what it looks like inside so you think like oh something's wrong with the power output of the unit sounds really complicated and then you look maybe you check first okay how does this power output actually compare with the neighboring units right because they all see that like roughly the same wind so you'd expect like on average they would kind of cancel out to like have no difference at all and as you can see here that's actually the case here so there's actually nothing wrong with a power output of that unit because it has the same power output as the rest of it So that's kind of disheartening. And then you would probably go on and look at this here, which is a model we created. So think about the wind turbine as a giant wind speedometer, or an anemometer, we call that usually. So it just measures the wind speed in a very robust way. And then additionally to that, we usually have two mechanical anemometers. It's such a difficult word to say, by the way. We have two anemometers installed on the unit that redundantly measure the wind speed. So we have three things that measure the same thing, so we can kind of, like, tell whether one is odd. And in this case, and it's kind of, like, shown in the bottom of the two plots, sensor two of the two sensors is actually defective. And so, you know, what at first maybe looked like, you know, a problem with, like, a complex problem with the operation of the unit actually turns out just to be a defective anemometer because, you know, some bird takes revenge or something like that. You laugh, but it's actually something that happens. All right. And that, I think, concludes my talk already. So here's some summary. You can read it, I guess. And let's move on to the Q&A, I suppose. So thank you very much for listening.

Speaker 2 [25:28]

So your last chance to vote, if you haven't done it already. The first question, maybe I'll come over here. The first question I have from the audience is, how do you handle time-windowed features if you cut out certain periods in your training data?

Speaker 1 [25:47]

Well, it really depends on what we're doing, really. Sometimes it's sufficient, depending on what we do again, sometimes it's sufficient to set them to zero, fill them with some zeros, and sometimes we can interpolate these or whatever, but usually it's so dependent on what we do exactly, so it's really difficult to answer that in a generic way. Yeah, that's my answer.

Speaker 2 [26:14]

Okay, so there's room for further discussion after the talk.

Speaker 1 [26:17]

I guess so, yeah.

Speaker 2 [26:21]

Okay, this one seems to be fairly high upvoted. How do you evaluate your model detection performance? Do you have a set of anomalous data?

Speaker 1 [26:28]

Yeah, that is a very difficult question in so far that that is a problem we're still

Speaker 2 [26:28]

Yeah, that's...

Speaker 1 [26:34]

working on, of course. We have some labeled data, right? And we have some attempts at evaluating our models on that. But it's a very interesting question to ask because you would, for example, think, okay, we have predictive modeling and we can now estimate the performance of the regressor, for example. And maybe that is a good indicator of how good the model is. But that usually turns out not to be the case. Those are really independent things. The capacity to detect defects and the capacity to just predict whatever is happening on another input signal. So yeah, we have some ways to benchmark our models just by checking them against defects in case we have some, which is not always the case, of course, but that's pretty much the end of it. also not the case that we do some extensive hyperparameter tuning or whatever. That's usually not something we attempt because it seems like a waste of time, really.

Speaker 2 [27:34]

Okay, thank you very much. Don't you need to train on two years of data to learn about seasonality, somebody asks.

Speaker 1 [27:43]

Well, I would then ask like why though but Yeah, I mean like very often we actually do extend our training period longer than one year That's actually kind of common in case we feel like there's not enough data to back up certain weather Conditions and so forth. But yeah, I'd be curious to know like why exactly it should be exactly two years and not one year like Maybe we can talk about that after if you can find me somewhere

Speaker 2 [28:11]

Can you elaborate on the infrastructure, where and how the models are run, how does the data get there, do you use streaming, API calls, batch predictions? Very long question here.

Speaker 1 [28:24]

Yeah, well, I'm not an expert in that, I have to say. I'm mostly just a data scientist and use the stuff that the data engineers shows over to us. But we work in the Azure Cloud, of course. Of course, I don't know why I said that. Really, I don't know why I said that. Anyways, yeah, we use everything. We use, I think, Azure functions in the Azure Cloud, and that's pretty much the extent of my knowledge. Yeah, and we use an influx DB to store all the signals and query them.

Speaker 2 [28:57]

Let's see if we have time for one more question. How do you deal with anomaly score spikes versus gradually increasing anomaly scores? Do you use the same threshold or separate ones?

Speaker 1 [29:10]

or separate ones? That's another really interesting question, because when we ask diagnosticians, what are you looking for in these model deviations? They usually will tell us we look for trends. We don't usually look for outliers or whatever, like little spikes, but they usually look for trends. So very slow crawling up from around zero or whatever. And we do have basically like we have some monitors that actually work by specifically looking for trends instead of spikes in case that you know these spikes are not really very useful to look at we also try to ignore you know outliers actually so which is actually kind of counterintuitive because most people would think outliers and anomalies are kind of the same but we really look for robust trends we look for like actually like paradigm shifts in the way the unit operates not really for statistical fluctuations in any sense.

Speaker 2 [30:08]

Then I would like to thank you for a very very interesting talk. Maybe we can have an yeah