Towards Intelligent Monitoring: Detecting Degraded Flame Torch Nozzles

Speaker 1 [00:07]

Hello and welcome everybody. Welcome to my talk about degraded nozzle detection of flame cutting machines. We'll talk about this proof-of-concept initial tryout we did for our customer, Frau Mark, who was nice enough and let us actually present the data and what we did. Maybe before we start, we are the RISC software, GmbH, we We do a lot of data intelligence in industry, in medical informatics, and other parts of the field. To move towards the topic, our customer has the issue that sometimes a nozzle actually breaks, which is bad. It's quite dangerous, a broken nozzle. And at this point, the customer needs to send it to an operator who actually replaces it, tests the nozzle. And after that, he gets a new one or a repaired one. That's like the general use case the customer wanted to approach. Maybe before we dive into the topic, who here knows what flame cutting is? One, two, three, three people. OK. Imagine a big machine, so like three meters tall, with a large flame torch, so a large lens of flame. You used to cut material, quite thick material. Normally, you would use a saw. However, a saw can only cut material up to like 30 centimeters. Everything above that is really specialized and hard to do. And if the material is moving, like here, it's super hard to do, actually. And instead, you use like a big flamethrower, basically, with a concentrated flame to cut through the material instead, which works quite nicely. What the operator does when he gets back such a nozzle is he wants to differentiate between intact nozzles and degraded nozzles. Degraded nozzles are usually visually not different, but the flame can look different, and they sound differently. So the way the gas comes out from the machine and the flame sounds while burning can be different. So a thing here is normally you like to use cameras to tell visual differences. Our cameras are really hard to use here. This is a dangerous environment, the flame is really hot, light conditions vary a lot, especially because we're using flames to cut material, and the machine has moving parts. The flame is not standing still, it's actually moving around. So capturing that with a camera is hard. So we came up with the idea to instead use a different sensor wake to to actually do what the operator does and we was we were hoping that we can use the audible spectrum to actually differentiate that which was not the given so we started out by coming up with sensors and testing those sensors and if we can differentiate it so initially we had a lot of questions first of all how do you look how do do degradation and defects actually look. Normally you have clumps which are forming or you have material which is slowly degrading from the heat near the nozzle. That's like the usual issue here. And you have additional conditions you have to be aware of, like sensor safety and people safety with such machines. It's a hot flame, it gets like a few hundred decrease near the flame and that's quite dangerous for you and also for the sensors and if you are not careful you're likely gonna burn or destroy a sensor which costs like a few thousand euros and additionally you have some environment condition and different states of the machine on that later more so the danger zones we defined if you can see the front of the machine on the top left you have like the flame part and the nozzle part which is actually like quite dangerous and you don't want anybody here after that you have like a larger zone where you have material which is spraying around you can be spraying around and of course the flame and heat and quite a big zone this last zone around the machine is actually still quite dangerous for spraying material we defined that to look where we can place a sensor which will not be broken within a year. And initially we wanted to have sensors in the zone 3 or maybe zone 2, but in our experiments we saw that zone 3 is pretty much not usable. The sensors don't get enough feedback from the flame, you can actually differentiate it. So it's zone 1 or 2 at least. So the sensor parameters were set on zone 1 and 2 mostly. The machine itself has also different states it can run through and the states are like cumulatives, so you turn on the water cooling as a first step, you turn on the pilot gas as a second step and we wanted to see how the different states actually influence the sensor. Mostly what we were worried about is that for example the water cooling introduced too much noise into the system and we actually couldn't use like the data so we did that and we had a few processes we defined on the machine which are which happen in production or are part of testing routines and wanted to take a look what can be used to automatically differentiate like what's a good nozzle how does it look when it's good or bad so we came up with a few experiments designed those and afterwards we took a manual look at get the data and looked what's sensible. We didn't get a lot of data in this initial phase. It was more like a day worth of data generating, so a few gigabytes of the sensor data. But the goal was to set the sensor position to get the right sensor and to at least know if it's possible. And when we are looking at the initial results, you can see that we had quite a bit of clipping. So this is the spectrum here on the top right, and the clipping happened because the sensor was too close to the nozzle, which actually introduces this clipping effect in the data. You don't want that. The other thing we had is if the sensor is too far away, you actually have noise like water cooling and similar stuff, which polluted the sensor output so much you cannot visually tell any difference at all. Additionally, we tried also some environmental effects, so we had loud, large machines running near our lab setup, and those introduced noise as well, but that was handleable. We ended up with Zone 1, quite at the edge at Zone 1, where we put the sensor. The reason for that was that we avoided clipping. kind of safe at least from the flame and it got us good data we got good feedback and for most part we saw the process actually in the data not labeled at all but we had an idea that it might work here so with that we had the sensor position we had the sensor selected we wanted to get get our data so get at this point we wanted to have like a process one of those we defined earlier we could repeat on good and bad nozzles to get our data and actually trying to set up a test set to do that we define the test basically the test has like two phases and we particularly chose this one because it's actually a test that happens in production that was done so if this works we can actually like use production data and roll this out to production if it works it's quite easy to test and it's also one of the things the operator told us we tried it and do that to test nozzles and repair them at this point it was like a year and a half ago i believe the company framark started to get old nozzle generate data gather the data and we started to actually look at the data and maybe try to come up with some sensible regime to label it which was quite hard because when you look at the data and you can see here on the left side this is one measurement so one operation the operator does to test a nozzle and on the right side you see one part we are interested in so on the left side you see you have multiple of those processes within one measurement and you need to split that. Now this problem is not quite trivial if you think about it. Because this is done by humans you have different phase lengths or the unmaterial length where we have a different behavior in the sensor. You have different gap length so the length between those tests is not the same. You have some noise and artifacts this is especially true at the end of such measurements and at the beginning those are different in length as well and all in all it's kind of hard to say how many tests we have per measurements as this is not a fixed number so at this point we said it might not make sense to do labeling like automatically as this is not an easy task as well and we wanted to focus on the proof of concept So, and we looked at some broken nozzles and it's even worse to actually differentiate here, especially automatically. You can see there's quite a bit more noise in the system. So at this point we wanted to say, yeah, we need those regions, we need annotations, and before we do any other stuff, then we do feature calculation in some way or form and set up our classification to answer the question, can we differentiate between degraded or intact nozzles. So how to get this annotation? For that we used a piece of software I learned from last year's PyCon. It's Label Studio. Who knows here Label Studio? Okay, quite a few people who used it actually. Two, three people. So what we did, we have multi-dimensional time series input. We load it up into Label Studio and what we can actually do is to actually label regions we are interested in so we took our measurement and basically split it up into test regions we used as an input and such a test region in later in the talk will be called a patch now that was really nice as you could actually do that with multiple sensors at the time and you set the test region over one joint time axis so this was actually really usable and nice for us and the customers with that we had annotated about 879 regions we had to skip a few files because nobody could tell if we are the test actually started and how it looked like and this was reviewed by the domain expert of course then looking into the data we have like multiple measurements from the same physical nozzle and on average we had like nine or ten tests in such a measurement. Now the thing is, what makes it a bit dubious or hard to tell if this is absolutely true is that we have a bit of a problem with physical nozzles. For broken nozzles we had many different ones. For intact nozzles we had exactly one physical nozzle we could use. At least we got a few more tests here and the distribution between glass, glass distribution was actually pretty okay. Also the length of the tests differed by quite a bit so you can see here the duration and most are really short like 1.5 to 2 seconds but you have some longer ones with 3 seconds and 3.5 seconds. This is especially the case for for broken nozzles where the operator actually is longer in the phase two, the unmaterial phase, and it can be that it's just dangerous to go off the material really fast. At this point, we extracted those patches for faster parallelization. This is what it looks like. On the top row, you got intact nozzles. You can see the jump from phase one to phase two quite clearly and in the bottom you got broken nozzles it's a lot harder to tell what's happening here mind you in this data you also have background noise effects from water cooling from the sensor itself and you got quite a bit of data points we measured it with a sampling rate of 2 megahertz which returns in a lot of samples even for such a small patch and one of the things we wanted to do is actually downsample it and see if we can still tell the difference. Next up, we calculated our features. For that, we used the FFT, quite classical, quite simple for a proof of concept. Who here knows FFT? Who here has implemented FFT? Or at least used it I guess yes a few so we use that on the patch we used it on the crop that's the selected red area in the middle and we used some combinations where we split the patch in equal parts and compute that FFT on each of those parts to get a spectra at this point we use the spectrum and calculated either stats as a very simple baseline or we use the spectrum directly. One of those two for models later. Secondly we also did a spectrum feature set so we really calculated the whole spectrum with the sliding window as you know it from standard FFT which gives you this spectrum on the right side and used each column as a feature. This approach turned the idea was to get a lot more input features so you have similar input features but a lot of them and it might make it easier for the model the difference to the other approaches is that the selected area where the spectrum is generated is a lot smaller so it might be harder to see for the model we are actually a difference is okay we did that another thing we have to talk about is the test train split so remember we have those measurements and each of those measurements is of one physical nozzle so you have multiple tests per physical nozzle it's like in medicine when you have one patient and you have multiple tests blood samples whatever from one patient you have to be careful to not mix the same sample so same sample from the same patient into your training and in your into your test set because that makes a difference otherwise you are mixing distributions So what we did is we split the train test data so that if a measurement is selected for the testing set, it's squarely put only into the testing set, never into the training set. You have to be a bit careful here. Additionally, we of course used cross-validation when applicable, so everywhere, and we used the test data only for evaluation. The ratio, the glass balance ratio, is around 0.55 to 0.44, which isn't too bad, but we still only used adjusted metrics for evaluation. So the ratio here shouldn't be a big problem, but for metrics, we look at balanced metrics in some way or fashion, or the F1 score, not the accuracy. So next up, we set up the computational study. all of this, computing the FFT, computing the feature set, doing that with quite a big parameter grid, because for FFT you also got different parameters like window sizes and other stuff, and sampling rates as talked before, and then the classification, which each model also has parameters. We had to set up a computational study. The computational study was actually quite simple. First of all, we converted it to a standard format and into our patches. Second of all, we did the FFT with the parameter grid and third, we did the classification pipelines. We measured the time to get an idea how long this takes. You can see the FFT transform with the parameter grid takes about 20 hours on 29 cores, so paralyzed. In total, we burned around 36 CPU days worth of compute. The result is actually quite interesting. If you look at the different feature sets you have, you can see that on the left side if you use the statistical stuff, it doesn't work very well, and if you take a bigger window of the patch, it gets really good, and it works really well if you take the whole spectrum as an input. It's better than stats. We used some simple models, a simple neural net and a support vector classifier, which Simple baseline models, but that's good enough to get a really good balanced accuracy and f1 score for this Yeah, and with that there's like the Outlook and it works. It works quite well a bit too Well, we want to have different physical intact nozzles in the future just to verify the results this is important and We kind of realize that which area you select from the patch and how to select it is not trivial so that's in the future part of the problem how you do that and can you set the sensible and big window size and where to put that because that makes a difference for the classification for the result and is quite crucial in practice also it might be hard to generalize because different nozzle sizes might introduce more or less noise into the system Yeah, I think that's it for the most part. Thank you.

Speaker 2 [19:11]

Thank you, Dominic. That was quite fast. We do have a couple of questions and sliders, so I'll just go through them, and you can add more. We do have time. I would say the first one is, I would say two in one. Could you describe the sensor in more detail and also what kind of sensors and microphones did you use and what parameters did they measure?

Speaker 1 [19:35]

These sensors are structure-borne sound sensors, which is like a standard sensor in industry. Imagine a microphone. Normally you have sound waves you measure with a microphone, like the one near my mouth right now. If you don't measure sound waves in the air, you do that in a body, it's suddenly structure-borne sensor. So you measure waves which are permutating through somebody. They can be like any machine or a table or anything else. Those are interesting because if you have a process where we produce a flame, you automatically have some vibrations stemming from this process. And these vibrations are what we measure here on the left side. And this sensor is a very high-frequency sensor. so per second we have around 20k samples we get per second and that's like the main idea here

Speaker 2 [20:36]

Next question. Did you consider active learning for labeling?

Speaker 1 [20:41]

We considered it but the thing was for the proof of concept it's unsure or it's not easy to say can we tell the difference especially on degraded nozzles and we wanted to make sure that if we do a proof of concept we know that it either works or doesn't. So at that point we manually labeled it which took four hours. So I guess that's fine.

Speaker 2 [21:05]

Okay, give me a sec. So, what prevented your customer from supplying multiple intact nozzles to enable a better sample of good?

Speaker 1 [21:15]

Not much actually, he just simply forgot and initially he wanted, you have like multiple lab nozzles of multiple sizes you can test of course and we had one dedicated for the size we were going here. So initially he just measured those and he wasn't aware that this could introduce a problem. It's not a big problem afterwards to repeat the experiments here, but you have to be careful I guess, yes.

Speaker 2 [21:46]

Okay, it's not a couple of questions regarding your Model, so did you try to train a model to learn the baseline including water cooling to remove noise before calculating the spectra?

Speaker 1 [21:58]

We didn't because the initial experiment of differentiating degraded and intact nozzles worked really well. That was the backup plan to have data about water cooling and how the noise looks like to maybe subtract from it. But we didn't need to do it. I mean the results are really, really good. If you look on the right side, the patch spectrum is around 0.97 to 0.99 balanced accuracy and F1 score. and that's really good. Otherwise that would have been the plan.

Speaker 2 [22:31]

So in your train test split shouldn't you do the split on device level?

Speaker 1 [22:37]

Yes, generally, but we are working here with the LED machine, which is exactly one device.

Speaker 2 [22:46]

There are a couple of more if you're still there.

Speaker 1 [22:47]

Yeah, sure, shoot.

Speaker 2 [22:50]

Let me check. Okay, so what kind of changes in production may affect the quality of the features or model? Also, does the type of material to cut make a difference?

Speaker 1 [23:05]

Yes and yes. The changes in production, that's still an open issue. We have to imagine those are industry production halls with many machines working on the same shop floor. All of those machines introduce vibration into the system, which is on a very large scale level, the whole shop floor basically.

Speaker 2 [23:06]

Yes.

Speaker 1 [23:27]

So there's still a bit of uncertainty here if those noise, those extra noises from other machines might pollute the signal a bit. We are hoping that that's not too big of an issue as the machine is for the most part in its own area. So it's not too bad, but you have a bit of uncertainties here. The second part of the question was again.

Speaker 2 [23:50]

Question? I forgot.

Speaker 1 [23:52]

Yes, we tried that in the initial experiment. It made a difference, but it wasn't too large, especially with the materials we tried, which was metal, some kind of metal. So those are rather similar. I, however, expect if you change the parameters of the materials, so how thick the material is you are cutting, it might impact the sensor by quite a bit. Because for larger material sizes, you actually need a bigger flame, which likely introduces more noise and a different signal. That's why I said in the last parts of the slide, generalization really depends on a lot of factors and it will not be easy. I doubt that.

Speaker 2 [24:49]

You're still good? Yeah, sure. Good. Since a degraded nozzle shows such a different measurement, can't you just use that fact to detect them?

Speaker 1 [25:01]

I mean, that's what the model tries to learn, I guess. It's a lot more apparent in the FFT signal, so transforming it, than in the general signal, especially because we fear that the general signal also contains quite a bit of noise, which you can at least control with some kind of preprocessing, as we did.

Speaker 2 [25:23]

So, what were the challenges in feature extraction?

Speaker 1 [25:27]

So the the challenges were the noise mainly and getting Getting the feature extraction to actually compute but that I mean, it's just a lot of compute you need to smartly parameterize and do

Speaker 2 [25:43]

Couple more questions regarding tooling and methods. So the first one is Did you also look at time series classifiers from Python packages like esky time?

Speaker 1 [25:53]

No we didn't. I'm not sure that this really is applicable for this kind of time series because we are working with high frequent data. If I think about those classifiers I'm mostly thinking about low frequent data you use those with. So no we didn't.

Speaker 2 [26:15]

What neural network architecture did you use?

Speaker 1 [26:18]

Standard neural network feed forward network in this case, yes.

Speaker 2 [26:23]

Maybe one last one and then we'll leave it here. Is the customer keen on turning this into a production system? How much faster is validating with the model versus a human?

Speaker 1 [26:41]

He is very keen on doing that, you need to understand that the process of actually taking apart your machine, so stopping the machine, taking the nozzle, sending it back to the customer, replacing the nozzle takes time, and time is money in industrial cases, so this is actually helpful. Additionally, a degraded nozzle is actually quite dangerous. If you have a degraded nozzle, it's a lot more likely that material is going to be sprayed around in the shop floor and that's really dangerous for humans but also for other machines and other sensors and if you damage a sensor and machine in production on the shop floor you're not losing hundreds of dollars you're losing thousands or more

Speaker 2 [27:28]

I think one more. The model seems to be really good to detect bad nozzles, but what about okay nozzles? Did you set a threshold?

Speaker 1 [27:35]

That's another issue with the data package right now. We set it up so we can try to differentiate between good and intact, but it will be interesting when a production machine is first used with this system to see how the degradation happens over time and how well our system reacts to it. But I listened yesterday to a talk about actually kind of changing your models a bit to maybe try to detect OK nozzles well, or at least avoid those if possible. But this is an issue in production we have to tackle in the future.

Speaker 2 [28:14]

Thank you, Dominic. Thank you, audience, for listening, for your questions. I would say let's give one more round of applause. Thank you.