Why E.ON Loves Python

Speaker 1 [00:07]

Thanks a lot for having me. I think this is the first session in this room, so let's see if the technique is with us, and we take it as it comes. My name is Christer Friberg, and I work as a senior machine learning engineer at E.ON Energy Distribution in Sweden. I would like to take you on a journey of innovation and collaboration where Python has played a vital role for us being successful in this. And yeah, sure, this is yet another action figure, I couldn't resist doing one myself. You can have your own box, it's next to the unicorns down the hall. My journey with Python started out, or at least with open source, about 30 years ago when I started doing theoretical particle physics research. There it was natural to work with open source, and what you did was, of course, natural to share with others. What we did was high energy event generators producing code for simulating particle physics collisions, and the event generator we used at that time, which still exists actually, is Pythia, and to my happiness I saw that when I checked out just before the conference Pythia is now also having a Python interface to it and well I moved on to after finishing my PhD working at the startup for where we were doing online handwriting recognition and it was for embedded software handheld devices mobile phones, etc. We were doing machine learning, computer vision of all kinds using various programming languages. I'm mentioning one here in text at least. I will catch up on that one a bit later. But I want to highlight a few things we did with Python at that time was to build the CI-CD system from scratch since we didn't have that available and did really work with the version control system we had at that time. So I sort of did that with my left hand and using Python for it was just great and very simple. And a couple of acquisitions were followed after that. continued on a track on online handwriting recognition, doing sequence to sequence models, more and more deep learning, also of course cloud computing and all of that. Here I also sort of as a side project migrated from a Stone Age version control system to Git and also wrote this sort of little snippet in Python for doing that migration so traversing the whole commit history of this huge like 10 years old repository ending up in a 10 gigabyte of git repository with unchecked source code so it was quite a huge project doing that for all the different open branches that we had. Since about three years I have been working at Aeon and there I have been doing some image analysis projects and I will give you some examples on those later on. So here's one of the problems, as I see it, with just an arbitrary programming language. You ask some developers to solve a problem. Here it's just a simple thing to get the last character in a string, and you ask a dozen of developers and you get sort of a dozen of different answers on how to resolve that. And if you ask the different coders, they don't really understand each other's code, but they have their favorite things of doing their coding, and they have their little special things that they often want to use, but it's not easily understood by others. And of course, using Python for solving issues like this, there is often a Pythonic way of doing it, so you sort of end up with one way. This is of course an exaggeration, but still I think in this community it's fair to say that this is, yeah, at least in the community people strive for having a Pythonic way of solving their things, so it ends up with a fewer ways of doing it, and it's also built into the language itself, I say, simpler to understand by others. Okay, so great, we have sort of one way of doing it, but what happens then, you know, you get this reality check, you actually need to document this code and you need to have some comments, type hints, shall you use that or not, what about variables and function names, how to cope with that, what line length you want to have in your code. All of these details also makes, I mean, code will differ between one developer and another. The way of resolving this, of course, is making sure that you have the code quality that you agree on by setting up coding guidelines. And not only coding guidelines, but also how to manage your code in projects, Make sure you have guidelines for that, that will facilitate, if you have many projects in your organization, how you work with it and you agree on how to develop your code. And also make sure you have some naming conventions, not only for the code as such, but also if you sort of leave out some configuration to environment variables, keep naming conventions on this as well. Having this all set up, it's easy to set up a continuous integration for actually enforcing this kind of best practices that you agreed on. You can do your linting and code formatters automatically. What's the point of doing all of this is, of course, that when you hand over the code to someone else, to your colleague, then you want them to focus on what the code actually achieves, not the tiny details of should there be a new line here or whatever. Just agree on the best practices and force the tooling to sort of set that up for you. Then you can focus on the real code. And at Eon, Python is really the go-to language for all our data science work recently in my team we migrated one data science program programming language to to python and of course the benefits are are plenty one is of course if you have new joiners it's easier to have just one base programming language for them to to get to know and it's easier to get consistency in your code if you can keep to one coding language so the collaboration will also sort of foster by having more streamlined development environment and having fewer coding or languages that you need to keep up with and if you do integrations with other toolings You do this once. So, I mean, there is obviously reduced overhead of doing this. And further benefits of using Python is, of course, the straightforward modularization into packages and modules. This really simplifies maintenance and lineage. In particular, if you're working in a cloud environment, it's important to have a lineage and tracking of what actual code is is running in different parts of your cloud services so this i would say python really helps keeping this in place and python also has a very robust tool chain for for doing this code quality checks uh setting up testing unit testing integration testing also for packaging the code there is a great tools for doing this. And all in all, this all then serves the envelopes principles that we want to have implemented. Here's an example of the training loop. We have a script for doing training of a model. We can have for each commit of this code we had can have a CI CD pipeline running CI CD pipeline running doing the unit test linting and code formatting for us we can build this as a package if all the previous steps were successful when we have this specific version of the package we can build this into a virtual environment like in this case we use the docker image for setting this up so we can automate this build process so we have this available and we can push this into a container registry which we then can have making use of this in a machine learning environment which in turn can be used by a machine learning component where we also need to configure some computes what script to use what kind of data we take as input what results we have as output and of course then we can use this component as part of a machine learning pipeline where we put together several components that can work together. So keeping this modularization facilitates development and this then corresponds to our model development environment that we use. going into some details here the model development can in practice and look like you have some data you do some pre-processing you may actually land this pre-processing do you see the cursor yeah in data assets that your version so you can sort of have that done once and for all and you can start off from this data asset when doing some training optimizations and so on but you can also do this from the start of course and when you have trained your model you can evaluate this compared with previous versions and if it's better you can register this model you can have it versioned for being able to to roll back to an earlier version if you want to do that. You can continue this in a new iteration by improving the model or just having better data quality or more data available, so you want to have retraining and hopefully a better model in the end. You can then put this together in a scoring script, so you take the code and the model that you have developed, so you can run inference on this, and you can put on some business logic. In order to have it available for the business to use this, you can set up a real-time endpoint. And here, the endpoint will route anything you send to this. It's usually REST APIs. If you send something to that endpoint, it will be routed to deployment. and this deployment then has like the scoring script and the models used for doing that in this deployment and you can have a new version of your deployment that works in in parallel so you can have mirrored deployments you can have this in production and actually test out the new version of your deployment make sure that it works and when it works as you expect you can just switch the green deployment instead and take away the blue one and it's also easy to roll back if you have lineage in your in your system you can roll back to previous version quite easily by having this blue And green deployments. This end point then you Typically send something to the end point through a rest api. Here we have a cable cabinet image. You get some results back. This is quite cluttered, but i Just want to show you the model development inner loop that we Talked about here to the left. And then we have different different staging environments with a model deployment outer loop in this area here. So when we are happy with a model that we want to have running in production, we can zoom in a bit, we can have a gated approval where we have a human in the loop to make sure that, okay, this looks reasonable, we have some tests in this environment, and we can bring it into the next pre-production environment staging environment here where it can do yes so this environment is then production like so you have more data and all and other components that are as identical to the production environment as possible you can monitor how the model performs in this environment and again if you are happy with what you see you you want to stage this to the production environment and again we use gated approval here so we usually use four eyes principles which means that not a single person should be able to take something from development environment all the way to production without having someone else reviewing and approving the code and models ending up in production and of course also in production we keep on monitoring data model and infrastructure let's see I don't my clock is not like sorry three minutes. At E.ON we are working in like 15 different countries and those that are shaded here are where we have energy distribution companies. I think in Germany alone we have nine different DSOs and we have similar problems that we want to solve so we have started out to collaborate more around code and we have something called data and AI hub which is a comprehensive platform for advanced analytics, so without going into too much details here, it supports data as a product and AI as a service, and you can say it's like hugging face for AOM. So that's the main goal. Now I want to switch over to some MLOps in practice, showing some examples. sort of resembles to some of the previous parts where we have a live endpoint for doing image analysis on cable cabinets and here we send an image we have a user interface through a web app we send this image to our machine learning online endpoint which has a deployment hosting several models and we give the results back and feedback to a field technician to see if there is something that they need to correct or if it aligns with what's expected when doing service on a cable cabinet. This is what it may look like, an image of the outside of a cable cabinet. We send it through the model. We use an instant segmentation model here. You have a hatch of the cable cabinet that it detects, the Aeon logo, the warning sign, and the cable cabinet ID, which is masked here so you don't see it. On the inside, it gets a bit messier. You have a lot of different components. There are some devices here in green. And you also have some signage on all of these devices. We have some business logic that we want to check that all the devices have signage as expected. and don't be scared of the details here, it's just to show some of the business logic. So here we lump together these three components, they are considered as one, so they are counted from left to right, and then we have another device, and this is device 3. We take a closer look at this one, compare it with the documentation system that we have, It confirms that this is an SLD triple zero, and this device is dividable, so we need to have two different IDs on it. We see that it has that, and we extract those objects. We run OCR on this, get a digit out, and confirm that this is actually the position three as expected. all green for this check. So this is just one thing that we check here, but there's a long list of things that we do at the moment. Some of the projects is we are using drones for overhead line inspections. Here we also use object detection. This is in collaboration with Archeon. So after the drones have taken images and processed, Archeon processes this for us and sort of provides this interface where we can have a domain expert sitting at the office, sort of taking a closer look at those red boxes. Here's an insulator that is broken and needs to be repaired or switched. So, this is very useful. We have a project SAMS, where we do ecological vegetation management. Here we use satellite images and LiDAR data to optimize and validate clearing in our power line corridors. So, we use a semantic segmentation model here. having satellite images and LiDAR data and from this we take the 3D point cloud of the LiDAR data and map it to a two-dimensional image where we say each pixel is it vegetation or no vegetation and then you have a threshold at something like three meters if it's above it's vegetation And by doing this, we can automatically annotate and have our images, satellite images, labeled. So, satellite images are, in general, much cheaper to get hold of, and the LiDAR scanning is done perhaps once a year or something, or even more seldom, to just get some training data for this. So the idea is then that we have, when we get the satellite image, we can then run this through the model and get the prediction for where we have vegetation in our power line corridors and then we can do planning or validate if there has been clearing and if there are still some trees that needs to be considered. Another project is smart meter image analysis, where we have changed, we have, like, one million customers in Sweden, and the existing meters were exchanged to smart meters the last few years, and here the field technicians were taking photos before and after switching the meters. We were doing object detection to quality assure the switching of meters, making sure that it's the correct meter that's at the correct customer, that there is ceiling, for instance, and reading off the display for various things. So these checks were flagged and very successful. We have some open source initiatives with AI Sweden, just to mention. So we have some collaboration with DSOs in Sweden, and we want to do federated machine learning to get the electricity demand forecasts in Sweden for various DSOs. The idea here is to keep data at the DSOs, and then you have a shared model. So the model parameters leaves the company, but the data stays at each DSO. Keeping the data and sharing the model. And yes, we are doing all this with Python, and I think this is actually my last slide. So I think, yeah, to sum up, Python is not just a tool, but a catalyst for innovation and collaboration at Aon, and I think the rapid prototyping is the thing I enjoy the most with Python, so I hope that we can continue to explore, innovate, and collaborate using Python to create a brighter future for us all. Thank you.

Speaker 2 [24:59]

I don't have a microphone, can you?

Speaker 1 [25:01]

Yeah.

Speaker 2 [25:04]

tell the question and then the answer.

Speaker 1 [25:06]

yes let's see how do field technicians provide feedback that improves the model and which

Speaker 2 [25:06]

Yes, let's see.

Speaker 1 [25:13]

mechanism are in place to incorporate this feedback into the model yeah so right now they get the work order and then we have some expectations on what the cable cabinets are supposed to look like and if there are deviations they can report this usually it's i mean a way of doing it when they build cable cabinets from scratch but there could be things that were on for yeah you couldn't foresee this so they needed to do some adjustments and then they can motivate this in a free text field just saying okay i needed to do this because cable were short or too stiff to get this device in the right position so then you manually take care of this and update the documentation accordingly so that's sort of the feedback

Speaker 2 [26:16]

They counted my numbers.

Speaker 1 [26:17]

We have a mic for live questions, so if you want to ask a live question, that's also an option. Okay. Take this one first. What were the main challenges in migrating apps from R to Python? Were there more of technical or organizational nature? I think both. There were, of course, people who were accustomed to only working in R, so there was a journey journey for them, actually upscaling using Python instead, but as I see it, if you, I mean, Python is really simple to start using as a beginner, and I think the people who were affected by this were actually taking or embracing Python and were looking forward to doing this upscaling. So I think instead of being isolated of doing this in R, there were fewer people handling and coping doing that. We could have them in to the whole team. So I think that was just positive in that sense. And there was, yeah, any other challenges? We, I don't think so. I mean, there is more to do with the code. The code is not that nice shape at the moment, so we need to do some refactoring and it's sort of legacy code still, although it's migrated to Python. But it's working and producing the same results as before. I have one question here in the audience.

Speaker 2 [28:07]

This is the last question.

Speaker 3 [28:08]

Okay. Well, thanks for the talk. One question. What were the main interesting difficulties when onboarding people that have Python knowledge into your projects? If you employ new people or have students coming in, what's difficult?

Speaker 1 [28:23]

I think if you have programming experience, then taking on Python is really easy. I think that's not only my view, but the people I've worked with that have done that. And I think if you take people that have just graduated, they may have used some Python already in school, and more and more people are actually using that and also universities are switching to using Python more and more because of the friendly usage of it.

Speaker 2 [29:04]

Yeah, thanks, Chris. I think you can talk to him in the coffee break maybe and discuss this even more. And yeah, please, thanks. Give an applause to him.