Data science complexity and solutions in real industrial projects

Speaker 1 [00:03]

Hello from my side. I'm glad to be here. It's very exciting. Who am I? I'm a data scientist. I'm working for Rosen since four and a half years now. I'm working there on a wide range of problems like machine learning, optimization, computer vision, and stuff like that. And you can find me here. The slides we will share on our GitHub repository. And you can talk to us at our booth. Rosen develops and manufactures equipment, software, and methods for the inspection, diagnosis, and protection of industrial structures in a wide range of industries. Because damage can cause serious impacts. and that's what we don't want to see we want to prevent we want to prevent these accidents so yeah as you see rosen is working on a wide range of fields but our main business is still inline inspection what is inline inspection it is a service for oil and gas companies who want to know more about their pipelines so they want to know what the state is where the defects are and stuff like that and for that we are building tools so-called pigs which we put into the pipeline and they have a lot of sensors all around the tool and these tools are pumped with the mediums through the pipeline the data we are collecting is similar to image data, so we often have 2D data in one axis we have the circumference and in the other axis we have the distance, so it looks like an image which is really, really long and in this data or with this data we try to infer information from the pipeline wall, we try to find for example cracks dents and corrosion So we have classification problems, regression problems, and these tools collect a huge amount of data. It can go up to multiple terabytes per run, per line, but it depends on the diameter and the sensor counts and on the length of the pipeline. We are working on a high risk in a high risk environment. That is because if we do mistakes, it can cause these accidents which we saw and this is really bad for the environment, for the client and even humans can be killed. So we address the challenges with machine learning and with Python. There's often the misconception in data science that most of the time is spent tweaking the machine learning models and training these models, but a lot of time is most often spent on just on data wrangling. Why is it the case? Real data is much more complex than toy data sets, and what is missing in those toy data sets often, you have to collect the data. In our case, it's really, really expensive to get data and we data scientists don't decide what kind of data we get. We just have to take what we get. Then data cleaning is often an issue. You have to standardize it, you have to normalize it. You have missing and imbalanced data and scaling is also an issue. For example, if you have a lot of data, You have to store it somewhere, and also you want to make it possible that other data scientists also can use your data. So we have a classic approach and now an improved approach to work with data. The classic approach looks like this. We are constructing and manufacturing tools, which we then create first measurements with laboratory measurements and also pull tests and pump tests. So pull tests, we have pipes at our company site, which you can see at the image at the right. we have a lot of pipes with different diameters and these pipes have artificial defects which represent for example corrosion or cracks but these defects are relatively simple like ellipses and when we do a pull test such a tool is pulled with a wrench through the pipeline with this we create data which we use for tweaking and training our machine learning models. As I said, we have these defects and for example, we now try to estimate the depth of these ellipses which is similar to the depth of corrosion. Then we go to the client, We prepare, transport and launch these tools. These tools take a lot of measurements inside the pipeline. And with our trained models and with data analysts, we are processing these data and analyzing these data. It's important to say that these data scientists are not the same people which are working on tweaking and training these models. They are only working on client projects. And based on what they find out, they write a report which goes to the client, and the client excavates and repairs the pipeline where it is needed. So only the most severe defects are excavated. Now the client is interested in the quality of our work and also just to know what is in fact the state of the pipeline, they are verifying the defects. So an old approach is to use a pit gauge, a mechanical measurement tool. But nowadays there are modern approaches like using laser scans and X-ray computed tomography. here at the top middle you see a laser scan so the basic idea is you just have a laser beam and a camera pointing to the pipeline and by triangulating you can calculate the position of of your laser beam. At the right, we see our tool measurements exactly at this position. Laser scans have a high resolution, but only of the outer pipeline wall because it is non-destructive. We don't cut samples out of the pipeline. Just from outside, we are doing the laser scans, so we cannot see corrosion, for example, from the inside. But we get a good image of corrosion. We can see the depth. We can see the shape. And it's definitely better than these artificial defects. Then there's X-ray computer tomography. It has even higher resolution. You have 3D data of your sample, but you have to cut out a sample of the pipeline and you have to put it into an X-ray computer tomograph. This is only done to see cracks, because you cannot see cracks in laser scans. You cannot look into the pipeline wall, but with X-ray computer tomography you can. It is very, very expensive. Why? It's because you have to shut down the pipeline, since it's destructive, And that's the most expensive what you can do. Now we have seen that how we deal with data is very straight. So we didn't get verification data from our clients. We only used our own data which we produced. And we have seen strong potential to improve the quality of our models by using these verification data. Because, as I said, real defects are a much better representation of what we have to do than these artificial defects. When we started implementing this feedback loop, we had a lot of verification data in-house, but it was very distributed. So if you were in one project, you couldn't that simple get the data from a different project. So, we started implementing it and we have seen a lot of open questions and issues. For example, how to get the verifications from the client to the data scientist. Then we data scientists don't define what is verified and the data is not clean and it is not aligned. Well, we have found solutions for all these problems, but we have seen that it was much harder than we initially have hoped and thought. First, how to get the verifications from the client to the data scientist? We need some kind of delivery chain. So the client creates verification data, then it has to go to the project manager who has the contact to the client, then the data has to be standardized by some kind of data engineer, and it has to be labeled and aligned by, so that's manual work by an annotator, and then it goes to a data scientist. structures had to be defined so there were no data engineer and no annotator when we started at this in this chain yeah these processes like what is expected from the data engineer had also be defined these peoples had to be trained and for example the project manager had to be informed that verification data is yeah that we really really need a verification data and that these persons have to ask for more verification data from the client what we saw what really helped in our case or first we had the problem that these people had a lot of other stuff to do and we are the ones who wanted to get the data so yeah we wanted them yeah we wanted to get the data they had different stuff to do it was not so easy to get the data and what helped was helping solve their problems and then they had us for example now we have yeah we have the right tools with Pythons we have a lot of libraries we can for example automate a lot of stuff like Excel tables with pandas and so on we can visualize stuff, especially the project manager often really liked interactive bouquet plots which they can show to the client and they really like how our machine learning models improve with these, how they improve to show this to the client if we put these certification data also into our models. Yeah, it's simple, automate the boring stuff. There's a book from us where we got a lot of ideas from. For example, we have written a small script or a small tool in PyQt where you can define hotkeys and these hotkeys are bind to a Python script And with this, it was really simple, for example, to select an area in Excel and visualize it with one hotkey or two. Yeah, that multiple buttons are pressed. It doesn't matter which program just with one hotkey. We don't define what is verified. that's for us it's a real problem as I said verification is really really expensive and the client at the antique decides what is verified as you see at the image that's right we have depending on two variables we have a lot of samples but these samples are not distributed very well we have a lot of of examples in the middle and the classes you can see different colors respond to different classes so there are certain classes where we nearly have no examples so what to do we had to somehow deal with it and how to deal with it we wanted to fill the gaps so we can still use these classic approaches like pull tests and pump tests and laboratory measurements, that's still valid. But nowadays we are using more and more synthetic data. My colleague Hendrik Niemeyer had last year a talk completely about this topic. So what you need if you want to create synthetic data, you need your ground truth data and you need your measurement data. this is done in our case by first creating the ground truth data that means we are creating a blank pipeline wall with no defects then we put defects in and we can do it for example by using laser scans and distorting these laser scans by doing them deeper or shallower, longer or wider and this way we can we can fill these gaps we can use for example basic geometric shapes like ellipsoids and what's even better we can simulate corrosion growth in a physical way or physical plausible way and we are doing it with a 3d cellular automator so now we have our ground truth data our pipeline but we also need our measurement data and we can calculate it with FEM simulations. The problem here is one FEM simulation takes about 15 minutes with the quality we require on one core. So we somehow had to distribute it and we have done it by putting this FEM simulation into a Docker container and distribute it on our, yeah, we have a DCUS and Mesos and Marathon cluster where these Docker containers are running. Okay, so next point is that the data is not clean. All the clients are using different, they have different data formats and data containers. That's an issue because often they are using software where they have paid for and we don't want to buy all the software. So we have written flexible converter tools in Python to convert from their data to our in-house data, which is often simply CSV if the data is small and simple. and otherwise we are often using HDF5 as data container because it's really nice if you have a lot of scientific data or measurement data you can store it very performant the metadata is stored in MongoDB and like client information, the date the path to the HDF5 files, path to our inline inspection data so now we also also had to write proper interfaces for data access like filtering reading writing the data and we had to think about data storage the data gets quite large we are using an isilon cluster which creates one file system which is really nice or one huge file system which is really nice because we are using this huge HDF5 files and you don't have to forget that you need IT support for that. Now that the data is standardized it's still not comparable with our inline inspection data for example. For example the wall thickness, the depth of defects is always specified relative to the pipeline was so that means for example a hundred percent defect is basically a whole therefore we have to calculate the wall thickness out of our data another thing is the pipeline curvature here in the XCT scan data as I've told you these are 3d data and this is only one slide of such a scan but we see the pipeline wall here our data is most of the time 2d data so we had to map the 3d data somehow to our 2d data and the pipeline curvature was a problem here so we had to somehow normalize it we have done it with psychic image and open CV first we had to filter the noise away then we have used edge detection to to calculate the outer or to find the outer border. And we have used half transformation to find the circle which represents the pipeline wall best. And with this information, we were able to interpolate the data back so that we had, again, so that we had a straight outer pipeline wall. Now there are only simple transformations to get these important formations which we want. As we see at the right, we have these cracks. We want to know where these cracks are, the position and how deep they are. In our case, we were quite lucky that these labels had been annotated already before by the company who, which created these XCT scans. Okay, problem that, it's also normalization problem, but it's a problem which we definitely have, maybe others don't have it. We have no GPS data in the pipeline. There's no signal, simply. And we have to somehow align the, tool measurements and the verification data. How have we done it? We have written in PyQtGraph a scan alignment tool. We've called it scan alignment tool, where at the top the laser scans are shown and at the bottom we have shown our tool data. And you can now manually select points and say that these points have the same position. and this information is then stored into our mongo database and that's basically the alignment the problem here is of course it's tedious and it's time consuming but it is how it is today we are experimenting also with an automated alignment so the idea here is to use template matching from circuit image as I said we have basically a long 2d image so we have a template and we have an image and we simply compare pixel by pixel at each at each position the yeah how similar the data is of the template and the image if you would do it in this naive way it would take forever but you can speed it up with fast for your transformations and stuff like that the highest response then would be your your best alignment the problem here is that we cannot simply compare laser scan data and MFL or tool data because these are completely different kinds of data so again we are using here FEM simulations to simulate our laser scans so that we get simulated tool data which we now can compare with our tool data summary I've talked about data science challenges in inline inspection we have machine learning problems classification regression we are working in a high risk environment i've talked about an often done miss or often happening misconception in data science data science that most of the work is spent on treating tweaking and training your models then our i've talked about our classic and our improved approach with the feedback loop and feeding back verification data is hard. That's what we found out, but we have found various methods to tackle these challenges. To the conclusion, it is hard to feedback validation data into our models, but it is definitely worth it. This is not a machine learning talk. It was just about data science. So here are no numbers about our machine learning models, But nevertheless, we have seen a strong increase in classification and in our regression accuracies. And now we are even possible to write completely new algorithms which weren't possible before because we have now this new kind of data. Thank you very much. I think there's time for questions. Thank you very much for this interesting talk. I see there are some questions.

Speaker 2 [25:20]

Thanks for the presentation. And then from what I understood, that you generate the data for training machine learning models, right? Yes. Synthetically. Yeah, which means that it will be different from the real data. And then if we, okay, just generate it, then it will, I mean, it will be overfit to this, maybe formulas that you use to generate it. And then so how is it generalizable to real client data? And then do the clients also allow you to use data to retrain it?

Speaker 1 [25:54]

Okay, and to the first question, it is only to fill the gaps. So we try, the best thing is what you can do is to use real certification data and it's only to fill these gaps. And we have put a lot of effort to do it as physical plausible as possible. Yeah, and we are testing only on real data. so we can we know it will be better yeah or we can we can yeah as I said we test it on real data so we know it works we know it works yeah to the second question I think that's a little bit harder to answer yeah I don't know I think maybe we can talk later about this this question. I'm not 100% sure to answer this. So we have one last question.

Speaker 3 [27:01]

thank you for the great talk it was really from the real life I think your clients are very conservative the oil industry and so on yes and we worked for years in the finite element thank you for this American also but a finite difference or the computational fluid dynamics and if you're dealing with turbulent problems validation is a real hard task what about validation of your models and acceptance on the customer side. Even the final element is complicated, but it is more complicated, I think, and they have to believe in this. They have to pay for this. And what about this? This is a new technology. This is rocket science. This is the rocket science of the 22s, not 90s or 60s. Can you comment on this?

Speaker 1 [27:47]

At the end, there are always data analysts looking into the data. So we don't have fully automated sizing procedures and classification procedures. They are always looking at people which are really experienced, which have experience, right, thank you, and domain experts, right. so it's not fully automated and I think we are even enforced by law to have people which are looking into the data yes ok, thank you very much again