Time series anomaly detection with a human-in-the-loop

Speaker 1 [00:06]

Thank you for the introduction and welcome to time series anomaly detection with a human in the loop. I'm presenting this on behalf of Zeiss and we'll start with a short introduction of myself and also Zeiss and then come to time series analysis in the industry. Why do we do this? What do we try to do? What's maybe the challenges? Why doesn't it work in some cases and so on? And then the main part will then be scalable outlier and anomaly processing at SAIS. That's some initiative we do to improve the usage of time-serious data within SAIS. Let's start with the introduction of myself. So, that's me. I have a background in particle physics and I used to analyse lots of data of particle collisions at the LHC. Afterwards, I worked as a data scientist for three and a half years, roundabout at at Umlaut and later on Accenture, mostly in the automotive industry, before I switched to Zeiss in Munich in 2023. Over that time I've been working with Python regularly for now roughly over 10 years. So much to myself, now let's talk a bit about Zeiss. While you probably already have heard the brand name Zeiss or maybe visited the booth over there, You might not be fully aware what they actually produce. Zeiss is a portfolio company with different segments. We have the four major segments here. So over on the left here, we have the semiconductor manufacturing technology, SMT. They produce parts which are used in machines to produce high-end microchips. So probably your smartphone will have a microchip which has been produced using Zeiss devices. Then the second segment is industrial quality and research. One example here is high-end research microscopes, which you can use to do, for example, biological research. Then the third segment, medical technologies. So if you ever have to have a catheter surgery, some eye surgery, it's hopefully done using with the help of test devices. And then the last part, consumer markets. This is maybe the most tangible. So this ranges from eyeglasses to binoculars to camera lenses and so on. If you want to find out more, come visit us at the booth. It's right over there on the other side of the building. But over all of these segments, we try to use machine learning and AI applications more or less everywhere. As you have seen from the introduction, size revolves somehow around optics. So everything has some optical application. And what we are trying to do with ML is then there's a large focus on, of course, computer vision methods. But on top of that, there's also stuff like production optimization, image analysis, as I said before, but also recommender systems for different parts. And there's many, many different applications. A lot of them have to do with computer vision. But today we want to talk more about time series analysis. So why do we talk about this? Looking at time series data is not a new thing. People have been doing that successfully for decades, but it's still a very relevant topic today because, in the end, time series data is everywhere. This starts at your manufacturing sites where you have manufacturing lines, where you have sensors at all machines, or can have sensors, and goes up to all the devices you maybe produce and sell to people. So the question is now, Now, suppose you have gone through the really huge effort of actually collecting all this data in somewhat central places in a somewhat usable format. You are still left with the question, what do you actually do with that? Because in industrial contracts, you don't do this for fun. You want to make something more efficient, cheaper, faster, and so on. So the question is, how do you gain any value of your data? And if we look at a very popular example, which would be anomaly detection, So basically, you want to have a look at your data and see if it looks weird. So there might be a problem inside, indicating that something is not going into plan. And you would want to get notified about this. If you want to do that, you could go to your experts. So your domain experts, they know the machines they're working with. They know the data. They can have a quick look at it and tell you, that looks good. That looks alarming. That looks, well, I'm not sure. We have to dig deeper. So these are the people who actually have the domain knowledge and know most about the data, but the issue here is that while this is technically possible, it will get more and more less likely that you can actually do this because the amount of data grows so much that it's in the end impossible. Also, this is a very time-consuming endeavour, so it will take some time. With more data, it will take more time, of course, and then you will probably not have thousands of experts running around your factories, so their time is also valuable, so this is also expensive if you have to block them with just looking at data. One thing which you can then do instead is, or what many people hope for is, okay, let's just train a model which can predict or find these alarming behaviours and then we're fine. We have a scalable system, it's automated and so on, and that would be fine if it works, but there's a big caveat, you need labels. So you need somehow a label data set to do like the traditional approaches and these labels are not there from the get-go. So in the end the question is where do you get these labels? If you have a use case with data which is more or less common, let's say you have images with cats and dogs, you can maybe outsource this and give that to some other people to label the data. But in an industrial context, usually your data is highly specialised and only these people really know what's going on. So you have to go back to them and ask them to label the data. You can do this, but again, in the context we are talking, we have a lot of different data, we have a lot of different people coming to us and ask us, we have task X, problem Z, can we solve that with machine learning? And the answer you have to give is, yeah, well, maybe. We have to look at it. And if you have to label data for six months to even try, if it's worth doing it, then it's less attractive, of course. So, this is somehow a big issue, which we have to face here. So, what you could do instead, instead of relying on labeling data beforehand, is you could go to unsupervised machine learning models. So, scrape the labels, use algorithms which can just look at your time-serious data and somehow flag out these parts which look different than the rest. Which works out fine, and you will get predictions, but the question is, you still need to validate somehow if your predictions are useful for somebody or if it's just like random noise which is not hurting anybody. And then the question is again, okay, how do you get this validation? Of course, you have to go back to these people. What's the takeaway message from that? So basically, it's not so trivial to get any value out of time-serious data and the involvement of the experts is mandatory. So you have to get these people involved and tailor your workflow around them because this will make or break your project. So and with that let's jump to the main part which is the scalable outlier and anomaly processing at SAIS. This is some initiative we are driving towards robust efficient and scalable use of time-series data within SAIS and and what we want to achieve or what we want to do with this is listed in these three main boxes. We'll go over them in detail, but in short, okay, we want to somehow get the humans involved, so we want to get the experts to give us their feedback quickly. We want to combine this with some robust infrastructure so that we can offer this service to as many business units or as many departments as possible, and we want, in the end, to enable future use cases. by paving the way to label data sets so let's have a look at that thing first what we want to do is we start with unsupervised machine learning we predict anomalies we want to get feedback of the experts and this feedback will in the end be our labels so what we want to do is we want to start right away providing something to them and not having them label data for the first month and then see if it works but we want to get in contact with them as early as possible possible, and just get feedback from them if the models are useful to them or not. And this will, in the end, provide the gateway to more fancy approaches. For example, predictive maintenance, we would look at the timesheets and try to predict if the machine will fail in some time, let's say in a month or so. Or root cause analysis, so if your machine failed, why did it fail? And for that, the labels are very useful. So this is what we want to get to. You can see what we are currently doing in the diagram here. So here we have a time series. So you have the time on the x-axis, and then some sensors on the y-axis. It's multiple sensors, a multivariate time series, and what we will do is we will start from historical data from today, basically, and look back, and then use some algorithm to make predictions. we'll flag some periods as anomalous, and then we'll go back to the experts and ask them, hey, how does that look? They will then give us feedback and say, OK, this one is something I want to be alarmed about. This one is not relevant. This is not relevant. This is maybe something which is very important to us. So we'll get some feedback. And then after a few iterations in the beginning, we will then be able to just say, OK, we'll go with that model for now, put it live. And then what we'll then do is basically ingest new data, all data which has been recorded up till then, do the same thing, predict anomalies and get feedback, and do this over and over again. And over the course of time, we'll get to some data set which is labeled, but during that period, we already provided something to them which is hopefully of use to them. What we are currently not doing or not focusing on is basically making predictions to the vast future. So, depending on the algorithm, you make predictions for the next step of a time series, but for now we are just focusing more on the anomaly detection part and not on really forecasting. This is a bit what we want to achieve and what we mean with enabling future use cases. Let's talk a bit about the machine learning side of that and the machine learning infrastructure mostly. What we want to do is we want to provide some reusable tooling. We want to do this not for one customer, so customer means like size internal, but we'll have many people who could use this and we want to provide more or less the same service for them without having to reinvent the wheel or having a startup phase which is extensive in the beginning. So we want to have reusable and scalable tooling, minimize startup time, cost, and effort, and of course, support some flexible choice of methods and models because the data will be very different depending on the customer. With that, we also want to provide automation as much as possible because this will make it easier for us to maintain, and it will also, of course, make things easier for everybody involved. How do we do this right now? We have a simplified architecture diagram here, which we will go through in detail part-wise. So everything which is in the diagram should be explained on the legend on the right here. As you can see, we are working in Azure, but the same thing would of course also work in AWS or GCP, and basically all our code and all our orchestration is done currently in Azure DevOps, which is the GitLab or the GitHub of Azure. This would of course work also in different systems. What we assume which is given to us is the data source. This is of course not a trivial task to actually gather all the data and make them somehow accessible in a somewhat usable format, but in our case there is a neighbouring team whose job is to do this. So this we assume is given for us and for our use cases, and what we do then to have some automation is we use pipelines to basically call Terraform and provide all the infrastructure we need for a given project, and then if we have that, we can come to the actual machine learning core, so this is all which is visible here and as you can see there's kind of a lot of stuff going on down here. So what we do to reuse tooling as much as possible is we will have all the code related to time series manipulation, so pre-processing code, imputation and stuff like this will be in a separate repository which will be reused for every customer and we basically just build a Python package out of that. That's our internal package, which allows some reusability. And then for each customer, there will be some additional repository where the actual machine learning workload is defined, so how we actually do that. And as you can see from that repository, there's a lot of these rockets, which are the Azure DevOps pipelines, going up to this blue box. And this is actually the orchestration of our machine learning workloads. I'll talk about this in a minute. But let's first have a look at the actual pipelines which we run. So there's three. It's the colored lines here. So we have a data ingestion pipeline that's the green one. It basically goes to the data source, fetches new data, and stores them in some other data source for us in a format which is usable for us. So we do this because usually in this data source there's a lot of additional data. This is not simply put up for us but for many different use cases. So this is why we basically have another data ingestion pipeline also just to get updates. The next step is then the preprocessing pipeline. There's the yellow one. So what we do here is we go to the data have we fetched, apply the pre-processing we need for our algorithm we chose, and then store that again. And then in the end, there's the red pipeline, which is the anomaly detection pipeline, which does then the actual prediction of anomalies. As you can see, more or less everything revolves around this bluish box. And this bluish box is Azure Machine Learning Studio. There was also a nice talk about that yesterday from getting from IDEA to basically MVP in one day, which was also heavily using that. What Azure Machine Learning Studio is, it's basically an end-to-end machine learning framework from Azure. You could do the same with different frameworks for sure. What it provides is it allows you to register and create computes. So depending on your algorithm or model, you might need some CPUs or you might might need a GPO cluster, so what you can do here is basically just create them in there programmatically or also by clicking in some user interface. It also allows you to register data stores so that you can easily access your data and it handles all the authentication for you. It has environments so where you have all your dependencies listed inside. But the main part for us is that it has some ML pipeline building framework, so where you can take different snippets of Python code or different Python files or modules and stitch them together in some form of a pipeline, which you can then run and basically reuse, and it's well-defined what you're actually doing. It's also built around MLflow, so it has all the capabilities which MLflow has, like tracking and versioning of models, pipelines, and so on. And then, which is also nice, if you're setting it up, it has some graphic user interface where you can go and look for running jobs, compute storages, so it also has some interface, which makes life a bit easier there. What we then do is, I said before, we do basically all the orchestration parts, so when is which pipeline triggered in Azure DevOps, We have, for now, a very straightforward schedule or concept. There is a schedule running in DevOps, which will periodically trigger the data ingestion pipeline. The period length will depend on the project, so it might be weekly, daily, hourly, but it will always be some time apart, so it will not be live streaming data. It will always be batches of data. and this will then, if that completes, that will trigger then the pre-processing and anomaly detection. These are two separate pipelines code-wise but which is useful for debugging or for developing or testing new stuff if you just want to play around with different pre-processings but here they are just triggered directly one after another. So we used this in Azure DevOps because we found that for our use case it worked better than the scheduling capabilities which are inside AML Studio, which also has similar stuff like schedules. But for us, Azure DevOps worked better here. And that leaves us with the left part of this infrastructure diagram. And as you can see here now, finally the user or the human is doing something and he will mostly interact after some authentication with that blob, which is a web application. So let's recall again what we want to do here. We want to get these humans in the loop. And we need the domain knowledge, basically. That's the key point. If we can't use their domain knowledge, it will not work. So we need to get their feedback on models and predictions as quickly as possible so that we can iterate quickly and know if we are doing well. And we, of course, need to use their time as efficiently as possible. And for that, we need to provide easy-to-use tooling. So if they first have to use a complicated tool, they will most likely not like it. So what we try to do here is to have a simple UI where they can just click. And what we're using for that is Label Studio. We have some customizations, but very little. It's an open-source data labeling tool which supports basically all major data formats, image, audio, video, text, and for us important, also time series data, and this provides some easy-to-use UI out of the box where you can explore data or label data, and the other important part for us is that it provides several ways to interact with it programmatically, so you can trigger webhooks out of Label Studio and then do anything. It has a Python SDK and an API. This is why we chose to build on top of Label Studio. What we want to provide here is basically mostly three main functionalities for now. We want a feedback mode. We want to get feedback on the ML-based candidate anomalies from the experts. This is the core part. Apart from that, there should be also some annotation mode, so if an expert is noticing something in one device during his day, he should be able to tell us, oh, this is something I want to be alarmed about, so he has to have some way of really annotating data or anomalies. And then, of course, there should be some summary mode where you can review past candidates and feedback which was given. system, and the way it works right now is AML Studio will basically use the Python SDK to write all its anomaly candidates into some Label Studio project, so projects are way in Label Studio how to, let's say, basically group different kinds of data, so here we are just using one project where all the new anomaly candidates will be sent in, and then we want the experts to have a look at that. We'll see on the next slide how that looks. And have a look at the candidates and give us feedback. And right now, every time we get feedback from an expert, this will trigger a webhook, and the webhook, with the webhook, that will trigger an Azure function, which is the AWS Lambda, basically. And there you you can do anything. You can run Python code, for example. What we do in there is many things, but one of the things we do is basically if we get feedback, we will transfer that feedback from the feedback project to the summary project so that it's directly updated, more or less in real time, in Label Studio. That's the workflow which we have here. What an expert would see if he looks at Label Studio is something like this. So here you again have some time series so you have the time here and different temperatures here on the y-axis this time and the as you can see they are more or less stable and at some point start to wiggle and the algorithm here thought ah this looks weird I will report an anomaly so the bluish things are then the candidate anomalies predicted by the algorithm. And here the expert can then basically click on that and classify this. Okay, here we chose three different categories. You can say, okay, this is a relevant anomaly, we want to know about this. He could say, no, this is not relevant. Or he could say, yeah, okay, this looks weird, but it's something we know about and we don't want to be alarmed about. So he could basically Go there and classify that but he can also do more so he doesn't only have to accept this He can also basically change the boundaries so he could say okay This is a relevant anomaly, but maybe it should be a bit longer it should go up to here or maybe it should stop here, so this is some additional feedback, which we can get out of that and This is basically then the thing we want to achieve with the scalable outlier and anomaly processing initiative in size and And if you're wondering what this has to do with Python, basically a lot, we're using Python more or less everywhere. So as I already said, basically all the tooling for data processing and data handling is basically built into Python code, built into Python package we used in the machine learning repos. All the pipelines are written in Python, so there's also Python code. And then the Label Studio has a Django backend, so more Python code, and then all the function apps we run again to do all kinds of manipulation within Label Studio based on the webhooks are also basically Python code, so more or less this is I would say 90% of the code which is relevant here is written in Python. And with that, we come to the summary slide, so I'll just leave it here. We talked about time series analysis in the industry, what we are trying to do and what the challenges are. We talked about our scalable outlier anomaly processing initiative and talked about a bit how we actually set this up from an infrastructure point of view. So thank you.

Speaker 2 [25:27]

Thank you. It's amazing. Thank you. For me, it's wondering if you use also Label Studio as an open source project, but it's also used by MTAs.

Speaker 1 [25:40]

Mm, yeah, yeah.

Speaker 2 [25:42]

Okay, we have, wow, a bunch of talks for you. Okay. How much of your Python code have in-house created and how much you have from abroad?

Speaker 1 [26:01]

So basically the label studio thing is basically an open source tool, which we just just use All of the pipelining which we do which we write is custom code from written by us Then there's a lot of custom code about reading and writing data again But for the actual algorithms depending what use we don't implement them ourselves, but also use open source tooling

Speaker 2 [26:28]

Amazing. Cool. The next question, also, I want to ask this, but thank you. Which advantages you use or you see in Azure Machine Learning Studio?

Speaker 1 [26:43]

So...

Speaker 2 [26:45]

In comparison with MLflow?

Speaker 1 [26:50]

Yeah, is it about Azure ML Studio or Azure itself?

Speaker 2 [26:53]

itself? Azure ML Studio.

Speaker 1 [26:55]

Yeah, so basically we tried this AML Studio out and it does its job, so it's usable.

Speaker 2 [26:56]

Okay.

Speaker 1 [27:07]

You can quickly get up running. So for our use case, it's a suitable choice. It has some disadvantages that it's not entirely clear how, So if there will be new features or which parts will be maintained and which won't. So this is a bit ambiguous. So, yeah, I see that there's also a lot of different tools which might be a better choice here. But the nice thing about AML Studio is it, of course, integrates very well with all the other Azure components. And you can get started right away pretty quickly because you can do everything programmatically. But you could also just click it if you just want to get something running up fast. It's it's a nice thing

Speaker 2 [27:54]

Amazing. Thank you. We have a huge bunch of questions. It means your talk is very interesting, not only for me. But it's interesting, two questions together. Do you have real-time use cases of streaming data, and how are you processing your time series, how you extract extraction techniques?

Speaker 1 [28:22]

Yes, so in the department I work, we don't have any real-time time-series data applications, but I'm not sure if that will definitely not hold true for all of Zeiss because it's a huge company. So, yeah, try to get through the booth and ask people there. They might have a use case there. And, yeah, for the other questions, so currently we were using a thing for the demo things here. I think it's using darts as a background, as a package and then some of the pre-processing is done in, I think we're using like internally sklearn pipelines because it just allows to stitch transformers together pretty easily.

Speaker 2 [29:13]

Last question about Snowflake database for large time series. Are you planning to use it or something?

Speaker 1 [29:21]

Currently we don't have any immediate plans to try and test this out because we're more focusing on the part here on the left, but if there would be a use case where we think okay we would need to use this or this would give us a real advantage right now, then of course we could also test this out, but at the moment there's no concrete plans.

Speaker 2 [29:49]

Okay, thank you for your answers.