Algo.Rules - How do we get the ethics into the code? Keynote

Speaker 1 [00:03]

Thank you very much, and hi, everybody. So why am I standing here today? And I didn't know whether there would be a little introduction. Usually when people go through my CV, they forget to mention my very first job, which is actually the one I'm most proud of, which was formatting floppy disks at the computer store of my dad when I was six years old. so I've been interested in technology and digitization my whole life I'd say but I started working on the ethics of algorithms which is the project I'm leading at the Bettelsmann Stiftung not because of my interest in technology or not solely because of that but actually because I'm interested in human decision making and the flaws of human decision making and how human biases are introduced into societal structures and how those in turn reinforce human biases. And there's, well, us as humans, we like to feel like these super rational beings and we know from cognitive sciences that we are everything but just rational. We are led by the weather, our personal experiences, our mood, and fortunately stereotypes, whether we like it or not. And I started working on this issue because I actually am convinced that automated decision-making can help us make more efficient, more consistent, and also fairer decisions, and that it can help us create a more equal society. However, as you all know, obviously, algorithms aren't just falling from the sky. They're made by humans, and they can contain mistakes. They can reproduce biases by the people who use them, who decide over their use, who set their goals, who interpret the outputs by algorithmic decision-making, and they can actually scale human biases and make things even worse than before. and usually my audiences I have to convince them more of the opportunities that lie in using the technology particularly when you're in Germany people are rather techno-skeptic and I believe that today I have a bit of a different job I would like to show you some examples that show that the line between good and bad is often a very thin one and that also show that we need to look at code through the way it is actually embedded in a socioeconomic context. Some of you might have heard that Amazon started, I think it was in 2018, to use an algorithmic system to automate part of their hiring process. And first of all, I thought, wow, this sounds really like a great idea because we still know that the labor market is one of the most unfair parts of society. For example, in Germany, when you apply for a job with a non-German sounding name, your chance of being declined rises by 50%. However, shortly after they started this trial, they realized that the system was consistently rejecting women who applied for jobs at Amazon. And it's quite clear that here the problem wasn't actually the code, it was the data the system was trained with, which was actually data on current Amazon employees from the past 10 years. And as you might know, Amazon, just like most big tech companies, employees are to a large extent still male. I think it's about 20% actually of people in technical roles that are female. So the system was trained on this data, it learned this bias, it started rejecting women, and even when the gender variables were removed from the CVs, it learned to pick up, through proxy variables, the information on the gender. For example, because people attended specific schools that were only for women. You might also know, actually another example that I like to use a lot is from the field of facial recognition, where an MIT student about a year ago, Joy Boulamwini, I really recommend watching her TED talk. She was working on the quality of facial recognition and she realized that her own face couldn't be detected by any of the systems she was working with. And why was that? Because she's a woman and because she's black. And she found out that IBM, Face++, and Microsoft's facial recognition systems work much better on men than they do on women and on white people than they do on people of color. My next example brings us to France, where back in 2009, the French Ministry of Education started to automate the matching process of students to universities. And this, of course, is a process that's really prone for automation. However, the problem was that the way the matching happened and the criteria that were being used for this matching process were kept secret. In the US, the use of these kind of systems had been really successful in increasing the happiness of universities, students, parents. However, in France, it started evoking criticism among students because they wanted to know what was actually behind the system, what kind of values were leading the matching. And based on the French freedom of information law, they sued the French ministry to open up the code. And what started was a legal fight that went on until 2017. And in 2017, they finally won. And a funny note on the side, what the French ministry did was they printed out the code and sent it via post to the student union. Thanks again to a lot of volunteers that they found through GitHub, they were able to re-digitize it. And what they found out was that one of the key factors for matching students to universities was actually the proximity of the address to the university's address. And if you know a little bit about the French education system, you might know that all the elite universities are actually located in fancy areas around the center of Paris. So this system was reproducing social discrimination. And here, this is a super simple rule-based matching system. It wasn't the complexity of the technology. There's usually this narrative of the black box algorithm that we cannot understand because of its technical complexity. This was not the issue. The issue was actually the narrative that was created around this black box problem and a conscious decision to keep the system secret due to trade secrets, due to maybe worries that it might evoke protests. My last example brings us to Australia. where the Ministry of Social Welfare started using a system to detect discrepancies between income reports and information they held on social welfare money that was received by people. When a discrepancy was identified by the system, it sent out automated warning mails to people. And the idea behind using the system was to increase the efficiency of finding the people that were wrongly receiving welfare money. So this actually could be in the interest of whole society. You save tax money. And they were pretty successful in ramping up the efficiency. They sent out 200,000 of these warning letters within a week, something they would have needed a whole year before. And in the testing phase, 200,000 of these warning letters were sent out to people shortly before Christmas, stating, well, we'd like to inform you that you owe the state 16,000 euro. And what they found out was 20,000 of these 200,000 emails that were sent out were actually wrong. So people were asked to pay a lot of money, start paying tomorrow, and they were not informed that there was an automated system behind it. They were not informed how they came to the decision. And they were not really offered any forms of appeal and the ministry that had used the system because of its efficiency at the same time was not prepared for this backlash and they weren't aware that efficiency also means that mistakes scale and actually if you go on twitter today it's been three years since they started using the systems and you look for the hashtag not my debt or robo debt you can still find stories of people who haven't found justice, who are left alone, and where the burden of proof is moved from the ministry to those people affected by these mistakes. What these three cases show is, like I said, that the thin line between a system that's used with a good purpose that can actually create more harm than good is often a fine one, and that it's not enough to just look at the code. We also need to look at the data that's being used to program it. We need to look at the underlying goals behind the system, what values went into it, and we need to look at how it is embedded. And, well, something I should maybe mention in all our work is that, of course, when I talk of algorithmic decision-making or algorithms, I don't mean the system that's used to sort screws in industry, or the one that's recommending you what dress to buy. When I'm talking about algorithmic decision-making systems, I mean those that are used in areas that have a profound impact on people's lives. In the debate of how to create more ethical algorithmic decision-making, there has been one idea going around for quite a while. It's the one of creating professional ethics codes for programmers. such as the Hippocratic Oath in medicine or the press code for journalists. And we thought about this, and we felt like this was actually a short for it of the complexity of the development. And the examples I just showed you, well, they basically underlined this thought because they show that it's not enough to focus on the code alone. So what we did instead is we came up with a process-oriented set of rules, the Argo Rules, which we developed together with the Berlin-based think tank iRights Lab. And those nine principles, which I will present to you shortly, address everybody who's involved in the development process and the use of algorithmic decision making. So people who are the decision makers and actually decide whether a system will be used at all and what kind of values will go into the system, what it should be optimized for, people who create the code, but also those who collect and categorize the data, also people who design how an output of an algorithmic system will actually be presented, because a lot of these systems in the areas such as education, health, and security are actually assistant systems. So at the end of the decision-making process, there's still a human that gets a prediction, let's say, and uses that to make the final decision. and the way the algorithmic system presents its output has a profound effect on how independent that person remains in their decision-making process. Because we address such a very diverse group of people who have also very different levels of understanding of the technological bits or the socio-economic bits, this was really a challenge because we wanted to create rules that were as concrete as possible while being, well, understandable for this whole group of people. And we tried to do so by involving as many and as diverse people as possible. We formed a diverse expert group. We conducted surveys. We had an open survey where around 300 people participated. And I actually think there are some people in this room here who are part of this process. And we're really thankful because the creation of the AlgoWheels was really a joint effort. and it is until today. And now I need like a dramatic drum roll because here they are. Those are the nine algo rules. They're not perfect. They're not final. They never will be. They're actually open for everybody to use them, to adjust them to their purposes, to try, apply them and make them fit. And I think what they really are is a start for discussion among different people in an organization, and I want to walk through them real quickly. So the first one, strengths and competency. What we see oftentimes, and particularly in the public sector where a lot of these application fields happen, is that the people who decide over the use of the system don't really know how they work. And on the other hand, the people who design these systems might not be aware of the social impact they might have. So what we need is more of a conversation between the different stakeholders involved and competency building on both sides. This is also necessary for the second rule. Define responsibility before a system is being used. It needs to be clear who is responsible for which parts. We also demand, rule number three, that impact assessments happen before a system is being used. In the three cases I gave you, it is pretty clear that people's lives are affected by these systems. But there might be other areas of application where, when you look at it at first, this might not be so clear, and only when you think, for example, of possible side effects, you might become aware that this is actually having ethical implications. Guarantee security, I think that one is pretty self-explanatory. Provide labeling. You might realize that one of the big buzzwords of the debate, transparency, does not appear anywhere. And we consciously decided actually against using it because transparency can mean so many different things. And it depends a lot on who transparency is provided for, what kind of systems it is used for, and in what part of the development process. So we tried to narrow it down by saying we need labeling. When an algorithmic system decides over whether I get access to university, maybe what kind of treatment I get in a hospital. I need to know so. I need to know when a person actually used predictive algorithm to make their decision. Secondly, this is another dimension of transparency. We need to ensure intelligibility. We could also call this explainability. So we need to try to make the way this decision was made understandable to those affected. And this one doesn't only have a technological dimension. It's actually a lot about cognitive science. And we know from data protection that when we just open information to people, this might not really lead to an increase of their understanding because there's this phenomena called information overload. A lot of people actually don't read when you provide them with information of how their data is being used. So it's really key to consider how much time people have, how much knowledge they have on how the system works, and make things as understandable as possible. Safeguard manageability. I think in the development process, this was probably the criteria that changed the most. At the beginning, we felt that there should be the opportunity to kind of make a wrong decision unhappen. But of course, that's not always possible. When a car goes against a tree, you can't make that unhappen. So this criteria became manageability, which basically means that you should always have the opportunity to switch off a system if something's going wrong to detect what made it go wrong and to change that. Monitoring impact, something else these cases showed, that oftentimes the impact only becomes visible once they're being implemented and once they're being used by the people who work with the systems. So it's not enough to do an impact assessment before using the systems, but you have to do it on a regular basis, particularly if your system is a dynamic one and it changes. And the last one, which I think became pretty clear in the Australian case, you need to establish complaint mechanisms. If you inform people about a system, making decisions about them, if you make it intelligible, this doesn't really help them if they don't have a way to actually complain when something went wrong. At the moment, we are actually working on making sure that these rules are also being applied, because it's great to put them out there and to say to people, you can use them, but it won't have much of an impact if we don't work towards the implementation. So we're in an experimental phase here at the moment. We're seeing whether they can be integrated into a quality criteria catalog for health apps. We're working on integrating them into the norming processes, standard-setting processes of algorithmic decision-making systems. And we're also asking for public sector institutions and programmers such as you, designers, to help us personalize them. Because, of course, things such as intelligibility or security mean very different things depending on whether you're the person writing the code or the one deciding over the organizational structures in which the system is being used. So I hope you enjoyed the presentation and I hope that this was maybe helpful for you, for your work. that you can use it to start discussions with the people you work with, to ask questions, because the skills you have are more and more in demand. And I think you should never underestimate the power you have to also make demands in the organizations you work for to talk about these kind of issues. I know that there will be a Q&A session. I'm happy to go more into detail on the different rules, on the ways of application, and I'm happy to learn from you as well. Thank you very much. We do have time for a few questions.

Speaker 2 [19:13]

for a few questions.

Speaker 1 [19:16]

yes hello so my first question would be that you have these rules but you have any kind of best practice best practices for them is it available and if it is where we can find them um good question uh really important we're actually collecting best practices at the moment We've come up with a couple of them, still collecting more, so bring them on. And you can find them, not now, but probably in a month or two, on the Algorules website. So go to www.algorules.org and you get all the information there. This is also where you can find our contact address and where you can submit ideas or best cases. Thank you very much for this great presentation. I have seen so many useless certifications for companies over the years, but this would be one that I would really like to see implemented in a lot of companies. Are you planning on doing a certification thing or something like a progress update for companies or something that they can? Yeah, so we're talking a lot with IEEE, with ACM at the moment, and a couple of software companies who are willing to test these kind of things with us. We've also had the idea of creating like a matrix system of classifying different softwares based on these rules. Because, of course, depending on the impact the system has on society, you don't need the same level, let's say, of explainability. So we kind of need two approaches. We need a classification of the systems based on the rules, but also based on their societal impact. And if you have ideas, if you have contacts, let's have a chat afterwards. it's on ah it is um you were talking earlier about how um the algo rules were created as a joint effort and i was just wondering how did you ensure that you had diversity within your own team because i can imagine that was that's very key to what you're describing here thanks um well by making it a topic like by consciously deciding that diversity should matter by um looking at who's part of the discussion i mean we had an expert group of um around 10 people where we try to make sure that different academics perspectives and the room that people without an academic background are in the room, that people of both genders are in the room, and of course, we didn't succeed on any levels. I think diversity is really a process, and you can always become better, but I think we did an okay job there, and of course, when we opened up the algorithms in our survey, this is where it becomes tricky, because there's always a self-selection bias it's people who participate who are already kind of aware of the issue and who have the time to deal with this so this is obviously where it becomes harder but now that we're testing these systems we're again trying to involve as many different people as possible

Speaker 2 [22:42]

Hi, thanks for the great talk. I was wondering, are you aware of, and I mean, there are loads of examples.

Speaker 1 [22:50]

Can you stand up?

Speaker 2 [22:50]

Can you stand up?

Speaker 1 [22:52]

Great, thank you. Hi.

Speaker 2 [22:54]

Are you aware of monitoring?

Speaker 1 [22:54]

Are you?

Speaker 2 [22:56]

Because we always come across these stories and how they go wrong and how they affect people. But I haven't come across something that kind of collects and also looks at what happens next. Because there might be a company that makes the mistakes and then starts adopting these algorules and actually produces best practices based on their mistakes. It's not always that things go wrong and then nothing happens. Is there kind of like a monitoring service like that?

Speaker 1 [23:32]

I mean, to some extent, the idea is that we put this out there and that we continue working on certain fields of implication, but that in other fields, we depend on people to take this on and make it their own project. So we're definitely not going to monitor all ways in which the agri-rules are being implemented. But something we've been doing, because I actually agree that there's not enough exchange of knowledge on what went wrong, what went good, how people dealt with issues, we're creating this report called the Automation Society Report together with an organization called Algorithm Watch. You might have heard about it. It's kind of like Food Watch but for algorithmic systems. And they're also one of the organizations behind the Open Shufa project together with the Open Knowledge Foundation, which was a project that used data donations to make the Shufa algorithm more transparent. and we've published the first version which collects good and bad examples from across Europe and makes them available to the public and we're currently working on a second version of this report which will be published in March next year you can go to our blog Ethics of Algorithms and you'll find information there thanks a lot, thank you everybody we're going to have the rest of the QA session

Speaker 2 [24:56]

the QA session.

Speaker 1 [24:57]

upstairs Thank you.

Speaker 2 [24:57]

upstairs in the lounge.

Speaker 1 [24:59]

Please be mindful that the tutorials are gonna start at 11

Speaker 2 [25:02]

start at 11.30 and the talks start at 11.50.

Speaker 1 [25:05]

and please do take your cups.

Speaker 2 [25:07]

Cups and glasses.

Speaker 1 [25:08]

and glasses with you outside the hall. Thank you a lot. Thank you, everybody.