Driving Trust and Addressing Ethical Challenges in Transportation through Explainable AI

Speaker 1 [00:07]

Thank you very much. Yeah, thank you for coming to my talk. I know there's a lot of interesting talks happening right now, so I feel very honored that you're here. Thank you very much. Yeah, we are going to talk about driving trust and addressing ethical challenges in transportation through explainable AI. So a few words about me. My name is Natalie Bayer. I originally studied psychology and then transformed into a data scientist through many Python conferences and tutorials and things like that. And I co-founded Lavrio Solutions in 2016 and a small company based in Karlsruhe where I work as a data scientist and all kinds of business related stuff. And in order to foster more European AI development, I recently founded another company based in Malaga in Spain. So we are also doing AI there. If you want to know something about library solutions or antenna data, you can come to me. we are doing all kinds of interesting AI related stuff like trainings but also project guidance for enterprises that want to start with AI and implement all kinds of artificial intelligent solutions. So today I am going to talk about a very interesting use case that we developed in a big research project and it's about AI support for dispatchers. So at first we need to figure out what actually are dispatchers and some other terms and then I'm going to tell you a little bit about the reservations that we have seen and that we were facing and how XAI was helping us to solve those problems and some risks of implementing explainable AI. So the research project is huge. It's a big competence center. It's sponsored by the Federal Ministry of Education and Research and as you can see from all the different logos there are a lot of interesting people involved. We have many interesting research institutes but also on the other hand small and bigger companies who are all involved in this competence center and it's called CAL and it stands for artificial intelligence in work and learning in the region of Karlsruhe. There we are tackling many use cases so those are just four of the examples that we are developing there and one of the use cases is what I'm going to talk about is the one with the tram where we are working together with INIT. It's a big company in the Karlsruhe area and the goal of our part of the research project is how to make the public transport more smooth by using AI support for dispatchers. So we are working together with INIT, they are providing a software for dispatchers. So who are those dispatchers? There are people that are monitoring all the trams in the city, so INIT is doing the whole public transport regarding to the trams in Karlsruhe and in many other cities so they are monitoring and the trams in a control center in like a tower where they can see every single trend that is going around in the city what is happening there and they can intervene in case of disruptions like accidents or blockages from from the rail or some kind of delays so for example in this picture you can see that a little bit further behind there is a accident with a car but it's very close to the tram so the tram cannot move further without damaging all the surface so the tram driver is giving a call to his dispatcher and is asking what shall I do I have a problem please solve it for me so the dispatcher can then initialize this positive actions like reroutes and short turns so another term that we are going to figure out what it is. So this is a map of Karlsruhe. Maybe you remember the PyCon in 2017 and 18. It was taking place in Karlsruhe. Very exciting time for me of course. And this is a map of a small part of Karlsruhe. And you can see the tram route of one particular tram. So the black line would be the route that has been already driven and then the dotted line is where the tram is supposed to go next. So it's the planned route to just go from the Mühlburger Tor to the Europaplatz and then an accident happens. Something like in this area there are cars driving around, there are many pedestrians, it's a little bit of chaotic zone in Karlsruhe. so an accident happens and the tram driver is getting an information from the dispatcher you cannot pass without making a huge traffic jam because all the other trams are also going to the other trams are also going to come and then it's going to be just a wall full of trams so the dispatcher is saying to the person that is driving the tram, just take a short turn, go for example in this circle and just go the way back where you came from in order to release the system, to not force a traffic jam. So those are short turns and one of the dispositive measures that the dispatchers can do. But also things like that, that would be also a planned route. And then the tram driver can go into the depot. What needs to be happening is that the tram is no longer functioning as a real tram. So all the people have to go outside in the last possible stop. And then the next tram is going to pick them up. So maybe you can imagine what it's like to work in one of those towers where you are facing six to eight monitors, everything is blinking, it's an intense job, and those people require an extensive experience. So in order to be a dispatcher, you need to have a certain amount of years of driving a tram, because it's like, yeah, it's intense. It's very complicated. You will face so many different situations. There's not a handbook that one can read and afterwards you can be a dispatcher. It's very complex. So they have to parallelize and prioritize a lot of tasks. And maybe you can imagine it's very stressful. So our goal in this research project was to suggest this positive actions. Because the goal was to allow the dispatchers to make calm and well-considered decisions, reduce the complexity and condense the information. And hopefully in the end it will lead to improved working conditions. Because also one of the problems that Inet and many other companies as well are facing is that people are not eager to work in a super stressful environment. So they have a lot of fluctuation. So this could be one of the solutions to actually use AI in order to help the people. So we have quite a lot of data. We have the operational data, so we have the positions of the trams where they are in this particular moment and where they should be. So where the plan is and where they are right now. And whether a dispositive measure was triggered in this particular moment. So we made a machine learning model with all the data that we had. And we thought that was really great. So we were able to show the dispatchers a mockup that looks quite similar to the user interface that they already know. And we could show them, well, this is a very big delay. Here would be a nice point for a short turn. And there's also the possibility of a short turn and things like that. So we were super excited to actually show the dispatchers this cool AI tool with all of these suggestions. And they were like, yeah, that's nice, but where is actually the close button? That would be the most important information for me. We're like, okay, that was not how we planned it. So we came back to the drawing desk. And we're like, okay, you have all kinds of restorations. What is happening? So this is what I want you to take away a little bit of this talk. How to think about the user that is actually using our tool at the end. And maybe not ask them very late in the process, but as soon as possible. So we tried to listen to them and try to hear what were actually the problems. And one of the dispatchers told us, well, I don't like the feeling of being told what to do. I feel like it's giving me now is a place, now is the time for short turn. Do it. I don't like it. I'm looking for the close button because this doesn't feel like a suggestion to me. It feels forced. So we're like, okay, we get it. We did not anticipate it. We thought that's pretty nice. We're very hyped about our solution. And, yeah, we tried to have an open dialogue and be transparent. So we told them those are all the data sources that we have. We try to be as transparent as possible and we were also using explainable AI because I mean nowadays maybe if you are into the field of data science you know that there are many cool ways of doing explainable AI. So we did that already. We have seen things like feature importance, what is the most important part of the data set for our machine learning model. We can actually also go down to every single prediction that we are doing and see which of the data set parts were responsible for which direction of our prediction. We also know that explainability is important in order to be now to fulfill the EU AI Act because there's this mandatory clause that we need to be transparent. One of the ways could be to use explainable AI and for us was like the new thing to not just use it as a data scientist to all those explainable AI tools and visualizations but to actually show parts of this to the domain experts so then trying to yeah be as transparent as possible in order to give them the feeling of involvement. The next stage of our mock-up is now looking a little bit different. They can now see all kinds of data sources that we are having access to. For example, how many people are right now in the tram? How big of a delay is there in this particular tram? Is there a connection guarantee? So sometimes there are trams that have relationships with other trams that are waiting for them in order to, you know what I mean. But also is there an opportunity to actually do one of those short turns? How far along are we into this route? How often does this particular line come. So it's very different whether this tram is coming every 10 minutes or every half an hour. If it's one half an hour maybe the people that were forced to go out need to wait for 25 minutes maybe. So yeah that would be a blue factor in our SHAP visualization. But also when is the next tram that is exactly the same line going to be there. So if the people are coming outside in two minutes there will be the next tram. And how many stops are there until this tram ends. And last but not least of course the most important part, the close button. So the mock-up looks now a bit like this. The dispatcher can now take a look. It's highlighted that there is a very big delay. You should do something about it. Already by hovering over it, he can see some or he or she can see some of the factors that are very highlighted or then he can take a look and at this point they're really just men and you can see this particular visualization of all the factors that are important and And he can say, yes, I want to do now a short turn. And we actually did a tiny evaluation whether they liked it or not. So big thanks to Betül. She did a bachelor thesis about that. And she asked 11 of those dispatchers, how did you like it? Because one of the main reservation that we thought we are going to face is that the people will not like it because it will take a lot of time from their already stressful life. We are adding more user interface, more information. And from those, it's like not a big group of people that we ask, but those are the relevant people that we wanted to ask whether they like it or not and in a real scenario would you have enough time to look at the explanations and all of those people that were answering this question said yeah I agree in some degree so um yeah for us it was very interesting to see that it actually does add value to use explainable AI and it's not just on top an information that nobody's using. But also does the explanation, the explainable AI in this user interface, does it help you to understand why this positive action was suggested? And some of them were like I'm neutral to that, I don't really care but the most part of them also did agree that this is actually helpful. So for us as a European company we think it's super important to have some kind of accountability that we are still trying to just give support to the people and to just give suggestions and not automate or try to listen to them what is actually the problem when they are not using it and so the authority to make the final decision still stays with the dispatcher. We are using our artificial intelligence to condensing all the available information to make a helpful suggestion and I thought back to 2016 when I started we called it just data science, and nobody was really comfortable saying artificial intelligence. And I was thinking, like, wasn't that initially the goal of data science, to actually discover patterns in data and condense information? Wasn't, like, that the whole, yeah, goal? Maybe we can, yeah, in some of the things that are happening right now in the world, maybe, yeah, we can take a step back and not automate everything that we can, but try to figure out when is automation useful, when is it also just a condensation of information. For me, as somebody who's coming from a scientific point of view, I was like, that's nice, but it doesn't feel like scientific proof. So I wanted to find a paper that actually is facing a little bit more the collaboration part that explain about AI is really helpful. So I found a great paper that I want to share with you and it's from Senona et al. And they did a great experiment. They took 48 factory workers from Siemens Smart Infrastructure in their real work scenario and they showed them real images of electronic products. And in like a real world example, because it's quite comparable to their daily work. So they were showing them 200 images in 35 minutes, which I found very intense. And they had to figure out whether this particular image was faultless or whether there was a defect. And they showed them the picture and a quality score that was made by an AI with zero, this is most likely defective, and 100, it's most certainly faultless. And they did two groups. So the one group just did see this quality score, and it was just a number, so they knew the artificial intelligence is more in favor of it being defective or faultless. And the other group was being shown a heat map next to the picture where you could see on which pixels did the artificial intelligence actually put the most attention on in order to get to the decision whether it's a defective or faultless electronic part. And it's super interesting that they found that the group with the explainable AI was significantly better than the group that was just using the artificial intelligence. So I thought, now I can rest my case and explain why AI is great. So now we maybe no longer are looking forward to this human AI collaboration. Maybe it's a little bit more of a human XAI collaboration. And so our goal was to make the life of this dispatcher a little bit more relaxing. We don't know about that. Is automation of any task ever led to a more relaxed lifestyle? Did we ever experience with the last few technological advancements that we are having more time at the beach? I'm not sure about that. There are also risks of implementing explainable AI. So we still have some dispatchers that will not accept our cool XAI tools. They are not always as hyped as we are about our solutions. And the question is also, are we losing competency and knowledge? So how are the next generation of dispatchers going to behave? How are they also going to have, are they ever coming to the same experience level as the dispatchers right now? Yeah, things change. But also, on the other hand, we have a shortage of skilled workers. So we need also to find a way to solve this. I don't know about that. It's like an ethical talk. I have a lot of questions I want to talk with you. I want to discuss with you. So I don't have solutions or anything, but just, yeah, some things to think about and to talk about. So, yeah, but also one problem could be that some people would just accept all the predictions without checking them. Because, yeah, it seems valid. Why should I take an effort and think about what's happening there? And there's also a risk of oversimplification. So, it's also not the solution to everything explainable AI. And, like, for example, believing that one factor or this combination of factors is the true real reason for the short term, it could be also something else. So, there could be always some information, some data sources that we do not access. So, for example, the dispatchers know so many things that we don't have access to. like specific tram driver schedules. They know all kinds of things. Also, if an accident is happening, they will get a radio call. So this is a thing that is not digitized. So we don't have this super important information. But it's the case with every machine learning model. We don't have all the information and that could be the real reason for any kind of output or outcome. And also, Scott Lundberg, the author of SHAPE, did also say it himself that with SHAPE, we can just make the correlations transparent and make them visible, but it doesn't mean that we have found the real true reason. So, I wanted to just show you a little bit of some ethical challenges that we have been facing. Maybe it will lead to some thinking. And we have an explainable AI for higher involvement and better understanding in order to amplify the acceptance and trust. Also, if you are interested in more research, I have also some other cool papers that I'm happy to share in some other point. And I think this is the European way of implementing AI, that we are keeping the human agency that we try to suggest and not automate the people out of the process. And I think it's one way to useful human AI or XAI collaboration. So I hope I inspired you to consider some ethical issues in your daily work. And I'm happy to share all my slides here on my GitHub. But also you can contact me with any kind of questions. And I'm looking forward to a discussion and to all your questions. And thank you very much for your attention.

Speaker 2 [25:40]

Yeah, thank you, Natalie. Great talk. And, yeah, there are a bunch of questions. First question, where is your data coming from? Adding to that, how do you make sure it is updated and includes new training data? Yeah.

Speaker 1 [25:55]

Yeah, and so yeah, we are working together with in it, and they are providing us with all the data so it's coming from the software and Pretty much nothing else so at this point. We don't use any other Data sources like for example you could think about how to get information about demonstration maybe also inside and this is now work in progress. It's an ongoing research project but yeah of course we have also some ideas but if you have some more I'm happy to hear them and to talk with the person who

Speaker 2 [26:36]

Okay, the next is about the bachelor thesis, this is open access. I'd like to take a look at it for a university research project.

Speaker 1 [26:46]

project. Yes, I will be happy to give the contact of Betül. I would forward the contact. I'm pretty sure she would be happy if somebody is reading and actually enjoying the bachelor thesis. I mean, I would have been if this would have been my bachelor thesis.

Speaker 2 [27:05]

Okay, next one. How do you define a good prior? Let's say you want to define a prior that encompasses performance relative to group of questions type. How do you define this?

Speaker 1 [27:25]

not really sure what that means actually

Speaker 2 [27:35]

Maybe you can discuss this in the Discord later.

Speaker 1 [27:39]

or like I'm also here in the car

Speaker 2 [27:41]

On the coffee break, there are much more options. So let's go to the next one. How do you start the bi-json modeling process? Do you write down the equation first?

Speaker 1 [27:53]

No, we did not use a Bayesian method.

Speaker 2 [27:53]

No.

Speaker 1 [27:57]

We used a light GBM, and it worked pretty fine for us. And we were not trying to automate, so we are not looking for the super 100% accuracy or precision or anything. So we were fine with our light GBM model. It's super small. It works very good. And so we were like, yeah, that's enough for us.

Speaker 2 [28:25]

The next one is a practical one, which features of your solution were most helpful for the dispatchers and what were they missing for their daily work?

Speaker 1 [28:38]

Yeah, I mean, like they are seeing all of those different tiles and they are seeing all the highlighted parts. So they like the part that they got the highlight of. So they could see in every prediction the combination that led to this prediction. So, yeah, I think it was the highlighting part of that, if that answers the question. Yeah. but also like yes I mean they they know what they are looking for so for example for them it's super important to see how many people are inside of the train so inside of the tram so you could see maybe one time we have 120 130 people inside so for the AI that's a reasonable amount of people but maybe for them if If they know it's a lot of traffic at the Europaplatz, there's a lot of things happening, they will not do the short-term because 120 people on top of the existing people that are waiting in line. And those are all the things that they know and that they are looking for into the data set. So yeah, things like that.

Speaker 2 [29:56]

Last question. Given that these real world phenomena have nonlinear interactions, how reliable are the explanations from XAI that compute contribution of individual factors?

Speaker 1 [30:12]

Yeah, I mean, how can we check that? I don't know. How can we figure out whether it was really the right, we can just figure out whether it was the right prediction, but whether the explanations were right. If you have an idea how to correctly check that, I'm happy to talk with you.

Speaker 2 [30:38]

Okay, this was the last one, and yeah, please again, applause for Natalie, please.