Optimizing Energy Tariffing System with Formal Concept Analysis and Dash

Speaker 1 [00:07]

So let me introduce myself first. My background is in physics. I was studying physics at Moscow State University. Then I specialized in astronomy and did my PhD at Max Planck Institute in Heidelberg. And then I left academic path and now I'm working as a data scientist in industry and specifically I work in a company called DSC we are an SAP partner specializing in industry solutions for utilities so we are a consultancy company consulting energy providers companies like Stadtwerke so the utilities companies who are using SAP and now a bit more about the stock today so the goal today is not mostly not to show my work but just rather to let you meet this math that I saw an application in my company and I thought that I did not see that before and I was curious so my talk will consist of mostly three parts or I will present you three parts the basics of this math then the implementation that I want to make and also the use case because I think it It is important to understand where to use or where it's not applied, actually. So, formal concept analysis. Please raise your hands who heard about formal concept analysis before. Okay, I see a few hands, and this is quite cool, because formal concept analysis was actually invented or developed in Darmstadt in 1980s. and how I want to present the basics of formal concept analysis to you is I want to start with the type of data it's working with. So the data in formal concept analysis is called context or formal context and it consists of three things. The objects, which we have here, those living beings. then the attributes here the attributes of the living beings are they need water to live they live in water or on land etc and the relation between them or the incidents here we are having the binary incident so true or false and now the definition of formal concept analysis that i like is that formal concept analysis formalizes a concept of concept. So what is a concept in this mathematical framework? It is a pair of objects and attributes with a star. And this pair is called extent and intent. And we'll talk about the star a bit later. But let us first look at the table, and we see that there is some structure, or there must be some structure. But it's quite hard to comprehend just by looking at it, even on this little table that is an example. So what patterns do we see here? We see that all living beings need water to live. And, I mean, maybe we also see that the creatures which can move, they also, the creatures which have limbs, they also can move. So some obvious things that we can just grasp by eye, but not much more. So let's ask a question. For example, what does a frog and a dog have in common? So we can highlight a frog and a dog and mark what attributes they have in common. And we can call this pair of the marked objects and attributes an A. But then, is this a concept? To check this, we need to understand the star. So, the star is that this concept, it needs to be such a pair that the objects, the attributes needs to be those attributes that the objects are sharing and vice versa. So, we can check now those attributes and see how many objects match this description, this amount of attributes. And we see that it's a frog and a dog, and there is one more creature, a cat, that is actually similar to the dog. So now we can say that A is not a concept because of the cat. Now we can mark frog, dog, and a cat, and now we know that this pair, which I call B, is a concept. But what we also can notice is that dog and a cat, they are mammals, so they have one more attribute in common. And we can then mark dog and a cat and say that this pair is pair C, and we can check that it is also a concept. And in a formal concept analysis, we also can say that C is less than B because C is more specific. So C has less objects and more attributes. Or we can also say that B is more than C, or B is a more general concept. Now we can also mark the green stuff and ask ourselves how the pair D, which is also a concept, compares to B and C. And obviously D is independent from B and C because it doesn't have the objects in common. So we figured what is our context for the data. We can also calculate or mine concepts out of our data. And the next step would be to visualize it, because now we know that we have the concepts and they have some relation between them. So it would be nice to plot a graph. plot a graph, I would love to use Python. And for this demonstration, I used the library that is called fcapy. There are a bunch of Python libraries for formal concept analysis. I'm not going to compare that, but I started with fcapy because it has nice documentation and tutorial, I think that it's easy to understand the math from this library. So I can read data for exactly this table and see the Pandas data frame out of it. And now this was the class formal context. So this is my context. And now I can compute concept lattice. lattice is the this structural group of concepts out of my context and I can see how many concepts there are for this data set I can also print the concepts and see that concepts are indeed this pair of objects and attributes which are called extent and intent. And now the last step is I'm going to plot a graph, which they have a visualizer based on network X. And the graph looks like this. So here we see the structure behind this data. I think that I did not explain a few more things, namely, yes, namely the top concept, so the number 0. The top concept consists of all objects and maybe attributes that all objects have. Then there is also a bottom concept that consists of all attributes and maybe any objects, if there are some objects that have all the attributes. So the next part, now I have a table and a graph. But what I am missing is interactivity. Would be nice to interactively work with this type of data. So for implementing this, I need an interactive table and an interactive graph. So I chose dash. Now please raise your hands who used Dash. Yes, imagine how cool it would be if Dash was also developed in Darmstadt. So Dash has a really nice also user guide documentation. There are open source components and enterprise components. I only used community components. And it's based on Plotly, so I think even more of you know Plotly. Now, what I wanted to show when I'm talking about Dash is a minimal example of a working web application. So I have the demo, which has 30 lines of code. A large chunk of it is just how to pick photos from the folder. So I basically just say that app is a dash, and I make my layout. Let's try to run it. And then we'll see how fast it is and how cool it is. So what I did here is just a navigation bar with a carousel of photos. And you saw it, it took Python code of 30 lines. OK, and now I am going to show you the actual app for formal concept analysis that I use. It starts with the same example. And I did here two tables, the attribute table and the object table, and they are interconnected. So when we move something here or change something here, deselect attributes, this also changes. And now we can look at the graph. And this is not just a graph. This is a hierarchical structure. So to make it more understandable and user-friendly, I made such restrictions for movements because this is a graph, so in order theory, it's called Hasse diagram. So the most general concepts are at the top and the most specific are at the bottom. So the order matters. and we are not allowed to put those concepts below. Now let me try to upload here some data that is familiar to you. For example, our PyCon schedule. And now let's try to select all the objects. And you see the preview here. so the preview shows that this diagram will not be readable at all so I think the schedule is very nicely represented as a schedule form but I wanted to show you here that this is a very flexible tool and for example if I want to just select the talks that are on visualization and Jupiter and see how many there were I can see that there were only two talks and I can see when they were in which room and everything there so now let us switch to the let us switch to that tariff in system or the application that is actually useful here And I just wanted to tell you a bit about the naming. So imagine you have a meter in your cellar. And this meter measures your electricity consumption. And it has a number in SAP or whatever system. And it is connected to the installation. That is more of an abstract thing. but the installation connects the actual physical device to the billing system. So the tariff type and tariffs, they are connected to the installation, and the type is more of like how we are going to bill, and tariffs are the rules which we are going to apply for the billing. So now I'm going to switch back to the tool and upload the tariffing system example. So here as attributes, I have the tariff with their names and the descriptions. And here as an object, I have not only one thing, but three things. the tariff type, an example of installation number, and the count, how many other installations have the same combination of the tariffs. So in this combination, it is very useful for the application. Let me check the specific tariff, which is for electricity, annual billing, and then 01, and choose all of those. So I did this because it is impossible to view all the tariffing system at one structured graph. So therefore, per client, they have like 1,000 tariffs. We do like 10, 20 kind of pages of those structures. So in here, we need to rearrange the descriptions a little bit so everything is visible. And then we can, why is it useful? Because with these pages that are for the specific tariff type or the specific selection, we can discuss if this structure is correct. So that if there is something that client does not agree with, then it might be restructured. For example, here we see this concept, and we have its intent and extent over here, and we can see that only this tariff type and installation example, and there are only two of those, which have this specific combination. maybe it is intentional maybe not and maybe I didn't say it clearly but the tariffs are the specific rules with the specific prices or discounts so maybe it is something to revise and also the implementation is when there are some migrations and if you know SAP you know that it often comes together with the keyword migrations and then if you need to rewrite the tariff and system completely then you want also to test if all the tariffs they work properly with the combination then the question is do we test all the installations that it's too much what is the minimal amount of tests that are sufficient to check if the entire tariff system works properly. And now in the formal concept analysis, we can take the most specific concepts, so the most specific that are here on the lower layer, so that go to the bottom concept, one two three four five six six six installations for this structure so this is the application for for this and I also maybe wanted to share you wanted to share you how it was implemented and I wanted to say a few words about what it under this interactivity. So interactivity in Dash is made with callbacks. The callbacks are here as green rectangles. It has some inputs or triggers and some outputs that then go further. And here this is the entire tree of callbacks for this application as you saw and I grouped some of it for example I have upload and download functions the tabs I actually have English and German support language support I have the two tables that you saw and the graph preview which is quite important because you see that informal concept analysis If there are too many concepts, then it's hard to visually comprehend the structure. So the graph preview interactively shows you maybe you need to select fewer objects. It has a graph and graph control menu, also concept content, so intent and extent for each concept. And I'm mostly proud of the user-friendly graph movements. This is what I showed you, that the concept should obey the hierarchical structure. So there are also some applications of formal concept analysis. So there are articles popping up in medicine, in biology, and we can see here social media habits, loneliness, sleep. So in different areas. So I think formal concept analysis is alive. I also had an idea to show you where I took this graph of callbacks from. and this is when you run the application in the debug mode there is a tree of callbacks but when they are generated like this they are a bit messy or random so I grouped that for you on a slide but that's actually a way to explore it and see what callbacks are called and how many times and how much time they took so I want I have a few moments and I wanted to show you a cool example that helped me understand the structure of those graphs or diagrams so I made an example which has three attributes X Y and Z and the objects are the vertices of this cube and then you can see that your concept lattice is just a projection of the cube because I was looking at the different a different concept lattices and there are certain rules for the lattices which are that every concept should have more general concept and that there should be interconnected but I thought that they all look something like crystals and then the idea of that a concept lattice is a projection of that cube in an attribute dimensional space it actually helped me to understand some of the rules for the for the concept lattices okay so as a takeaway message I just wanted you to know that formal concept analysis can reveal structure from your data from a new and interesting perspective and that this tool is very flexible maybe it's not a suitable for every application as I showed you with a schedule but for some cases maybe especially when you have some sparse tables and as you saw in the tariff in system application there were an example and account so that then it can show you you the structure of the data where you can see that this is your full structure and be certain that you didn't miss any patterns. Thank you for your attention.

Speaker 2 [24:27]

Thank you for your presentation. We have a few questions this time. So the audience was working While you were talking Like you were mentioning some other use cases they are curious about like clustering similarity optimization of product assortment But can you tell us more?

Speaker 1 [24:48]

So, this is hard, because I wanted to dive into the use cases a bit more, and I struggled a bit. I found a lot of theoretical books which showed examples, but those examples looked to me like a display from a book, and I wanted to find something real. And I found some exciting papers, exciting by the title, but they were closed, especially like in medicine or biology. Yeah, meaning that you need some academic license to get them or request them for a price. So this is why actually I'm giving the talk here, because I wanted people to look at this and try to think if this applies to their problems as well and maybe play around and check it out because um i think that uh not a lot of people know about this water just

Speaker 2 [26:00]

You are spreading the love for FCA.

Speaker 1 [26:03]

Yes, exactly. Or spreading the question. Does it apply? This is actually my question.

Speaker 2 [26:03]

Yes. So, don't love it immediately, just decide if you want to love it. Are there any evaluation metrics of general or specific concepts, or?

Speaker 1 [26:22]

Well, so you have the structure and there is a hierarchy. I'm not sure about, like, if a concept has only one object, it is quite specific. But I mean, this is all relative to your data set. I think this is very relative to your data set.

Speaker 2 [26:48]

The next question.

Speaker 1 [26:49]

question

Speaker 2 [26:51]

Can you represent Don't Know in the context and how it would work?

Speaker 1 [26:57]

Can I represent? Don't know. Don't know. So if there is not only black and white, true and false, but something in between. This is called FASI, formal concept analysis, and yeah, this is the entire other story. I didn't try it myself, but it exists.

Speaker 2 [27:24]

Can you tell a little bit more about the use cases of FCA in your company?

Speaker 1 [27:30]

This is the use cases in my company. For example, if the client wants to revise the tariff and system and The tariff and system is then this

Speaker 2 [27:40]

huge table. It's the only area he is

Speaker 1 [27:44]

For now, yes, and checking, so if you are trying to revise it, migrate and restructure it, so actually somehow view it, and then also to find the test cases to test the entire tariffing system, not to miss anything, but not to do millions of tests.

Speaker 2 [28:09]

Related to that, could you also use hierarchical clustering for the tariffing in your company?

Speaker 1 [28:18]

I did not try, maybe this is something to look into.

Speaker 2 [28:27]

maybe last question does the FCA help in finding bad data quality fastly by showing illogical concept orders

Speaker 1 [28:37]

logical concept orders or so if there is data and there is some some error in your data I think when you view it in a table you might miss it but when you view it as a structure it might be visible and if you are familiar with those concept lattices and this might be also the case

Speaker 2 [29:05]

Okay, one last question. Is this related to the Netflix recommendations that have very different categories why things are recommended?

Speaker 1 [29:15]

Does Netflix use formal concept analysis? I cannot tell you 100%, but I mean, I strongly doubt that.

Speaker 2 [29:25]

that was a wrap then thanks a lot again for your talk