Machine Learning Models in a Dynamic Environment Keynote

Speaker 1 [00:08]

First of all, thank you for the invitation and putting me on this huge stage in front of this amazing audience. I'll try to calm down. So welcome to my presentation on machine learning models in a dynamic environment. I will only be telling stories. I'm not going to show you any equations or mathematics. Just also no silver bullet solutions, but just stories. During this presentation, we will have an interactive part where you will need your mobile phones, but we'll get to that later on. So thanks for the introduction. I'm, as he mentioned, co-founding director of the Innosource Commons Foundation. If you like working in an open source way, Innosource tries to bring that into the corporations in order to teach your colleagues how that works so that they can be effective upstream committers. Why would I do that? Because I'm a member of the Apache Software Foundation, and I know that in open source we need helping hands. And I'm open source strategist at Overpace. Hands up if you've trained a machine learning model ever in your life. Yay. Awesome. How many of you have deployed it to production? How many of you, after deploying it to production, have experienced cat and mouse games with your users, users trying to game your model, trying to get around guardrails. Not so many anymore. Maybe you just didn't see it. So we will spend a little bit of time on history. My mom made me put away with a gray hair last weekend, so you can't see how old I am. And then we will share some recent stories, including your experience. The stories that I share are stories of naivety, of tinkering with technology, but also tinkering with communication, a story of discovering as we move along and discovering how naive we've been. A long, long time ago, we created a system where we could publish texts, webpages, link them to each other, create hyperlinks, move along those hyperlinks and discover new knowledge. make knowledge freely available over time we published so many of these web pages that we didn't remember the URLs anymore so we created manually curated collections of linked lists at some point in time manually curating these directories didn't work anymore so we created search engines automated search engines. And at some point, people wanted to move up in those search rankings. How many of you remember keyword stuffing as a methodology of search engine optimization? Hands up for all of the old people, please. So for those who don't remember it, there was a time when search engines would only look at the match between your query and your text. So putting more of your target queries into your text helped. Putting them into headers, into titles helped. What happened with web pages was that they were stuffed with keywords. To counter that easy hack towards search engines, some of these search engines started looking at hyperlinks, thinking that if a lot of pages point towards your web page, it's certainly more important than others because there was a certain cost to creating those web pages called PageRank. Hands up if you still remember Google Bombs. So what happened was that for each hyperlink, you have a link text. Now if the target query doesn't appear in your web page, but your web page talks about a certain concept, the idea was you take the link text and you add it to the text of the web page. Now imagine a few friends having fun creating link texts which have no relation to the target page. You can easily put that into the search index and rank the web page very high up. So the funny thing is as soon as people compete for attention, as soon as people compete for money, there will be people trying to compete against your model to work around it. There is a story about a search team putting out a search engine, optimizing for returning users. Actually optimizing that function made the search experience worse, because if you make your search worse, users return more often. So you need to be very careful with what you optimize for. The goal is to make users return, but preferably with new questions. At some point, people realized that natural content is something that's good for users because they get their answers. It's good for search engines and for content providers. So even today, that's a fragile balance. I'm going to tell another story. At some point in time, we started sharing messages with each other, unencrypted, with no ability to verify where the message is coming from. It was just messages sent. It was cheap. It was cheaper than sending advertisements printed on a sheet of paper to your post box. So even if just a tiny fraction of your people clicked on a link, it still provided enough revenue. So how many here still remember learning, training, machine learning models to counter email spam? A few more. How many of you remember having to retrain because spammers adopted and adapted to your same cat and mouse game, right? In order to end that, we needed to move beyond machine learning. We needed a concerted effort between male providers to collaborate. It needed domain experience and machine learning experience. I still remember one friend of mine building a machine learning model to counter email spam. He spent weeks and weeks and weeks to get across the target metric. He didn't make it. At some point, he created an Excel sheet where all the features were listed, ranked in order of how helpful they were in order to counter email spam. He took that list and went to the actual networking engineers. So they came up with additional features. He did that round trip a couple times, and it took less than a week to get the metrics up. So you need that combination of domain and machine learning expertise. Essentially in both of these examples it was a competition for money which motivated people to get around models. How many here have a credit card and have it been taken locked for spurious behavior only to realize it's been me traveling too far, too often, whatever. Same use case. Sometimes it's not only money. Sometimes people are after information. You will see adversarial attacks on malware detection systems. Think your virus scanner. You will see adversarial attacks on network intrusion detection systems. It's always this kind of cat and mouse game. Now let's think a little bit bigger. Let's imagine we are building a site where people can put messages publicly, where they can share them with their friends, where they can follow friends. In order to grow your reach, you put a little bit of news in there. At some point, users get so many messages that they can't read them, so you want to rank them by how interesting they are supposed to be or you want to recommend interesting content. What could possibly go wrong? Many many years ago I wondered why one of the social media websites I was using was getting more and more unusable. And I got a quote from a friend who studied communication science. My background is computer science, so I didn't have any clue about that. The quote he gave was, it should be clear that improvements in communication tend to divide mankind. Coming from Harold Innes, Changing Concepts of Time. I found it surprising. How can communication improvements divide people? So I googled after that. And I tried to find the publications that he referenced. And the publication gave some scary examples. One of the examples was about the invention of the printable type. Printable type suddenly meant that leaflets were much faster to be created. Who here remembers what happened afterwards? No history people? There was this new church. So I got that quote in 2016, I first referenced it in a presentation in 2017, so it's not a new kind of concept. We've known that even in computer science. We've also seen time and again how people used automated ranking systems and recommender systems and automated targeted advertisements in order to sway public opinion. We've seen examples in 2016. We've seen talks in 2016 as well. So I would like to take you on a journey. Imagine you're building one of these sites. I would like to ask you, I would like to play a game with you, like which ranking factors what you use, and then I want to have you play Mr. Evil Cat. And Mr. Evil Cat wants to abuse your site, and that is where you need to scan that barcode. We should end up on the Mentimeter slide. This is where I'm getting nervous again, because I've never used Mentimeter before, let alone at such an audience. Ah, it's working. So on a social media site, which factors would you use for ranking messages that you want to show to you is this. It's the obvious stuff. It's number of likes, number of views, how many followers the person has, number of comments. Makes a ton of sense. If you go to the next slide, now you're playing Mr. Evil Cat. I've given you just five examples. Which factors do you think Mr. Evil Cat would be easiest to influence? You can just rank them. What is easiest, what is second, what is third? And while you do that, you can think back about your most favorite social media posts that you've read this morning or yesterday evening. Now think about your social media sites. Which path could Mr. Evil Cat take to share their narrative with a large audience? I love the buying the medium. A couple of years ago, I wouldn't have thought that would be possible. Lots of bots there. Some things that became obvious to me only two years ago maybe, there were some large accounts and if you follow them and if you make the mistake of reading the comments, that's a nice attack vector as well. If someone has a huge following and you want your message out to many people, just put it as a comment underneath these people. Now, imagine that you are using that site. What could Mr. Evil Cat do that makes that sock puppet account more trustable? Which factors do you look at when you follow an account in order to figure out whether it's something worthwhile following, or whether it's something that you should ignore. You're being way more conservative than me. For me, it would have been avatar picture and description. But being at a PI Data conference, you likely know how easy those are to generate today. I've got three more for you. To find Mr. Evil Cat's sock puppet accounts as a user of a platform, what would you be looking at? Thank you. Changing perspective, you're not the user of the platform, but you are the platform. What would you be looking at to find Mr. Evil Cat? Anomalies and postings is very nice. Human-like API usage, human-like but not human. step. Imagine those platforms would be cooperating. What would you be looking at now? Now it's hard to imagine those platforms working together. The funny thing is we do that already and security in some states in Europe, where if one bank is being attacked, they tell other banks that they have been attacked for very simple reasons. The attacker doesn't care if it's your bank or another one, they just care about the money And the same applies here. So you should have access to this afterwards. So which service do we target? If we are Mr. Evil Cat, we will target the service that has popularity with our target audience. So your tiny little thingy over there on the side, it won't see these patterns. It will target a platform that is easy to influence and one that has more user features. Why more user features? Because each feature that a user can influence is one targeted tech vector. Which tools help me if I'm Mr. Evil Cat? Anything that looks natural, LLMs, anyone for image creation, for text creation? What helps me is if business metrics look, for instance, at engagement. What helps me is in transparency when the platform doesn't tell its users that something's going on. What helps me is lack of collaboration between those platforms because Mr. Evil Cat doesn't care who he attacks next. He only cares where his target audience is. So what does that mean? Anyone here remembers that we have to do input validation to avoid SQL injection? Yeah, pretty much everyone. Same for machine learning models. Whatever we put in there as training data, we need to understand what training data we put in. We need to check that user behavior actually passes the sanity check. We need to check that if we roll out a model, does user behavior change, because this may point towards users that are trying to circumvent what we do. We also want to look for groups of users showing surprising behavior, and that's something that we can't do manually anymore at that scale. We may also want to search for users that interact with our content in surprising ways because this may point to new types of content. Now at FOSS Backstage last year we had an interesting talk about what I referenced before, the information security space. Here, I believe it was in the Netherlands, certain sectors are required by law to collaborate if they are under the attack. Because attackers don't care if it's bank A or B, they only care about the money. So if an attack is happening here and this is being deflected, it's likely to move on to the next bank. Something like that should be possible here as well. And the other recommendation is there have been these kinds of we circumvent machine learning models for a long time. That's why I started with search engine optimization and email spam before. There are publications that talk about adversarial machine learning where you talk about having a machine learning chip to production and users trying to play with it and play with it in interesting ways that you may not want. There are publications which talk about covariate shift, where if you train on a certain distribution of training data, and you put it in production, and the distribution changes, how do you detect that, and why? So essentially, we've seen a lot of hands up when talking about training models. The pipeline is clear. You have a model that you train in the middle, you have input data, you have training parameters, and then you have users probing the model. What gets interesting, for instance, is if you train your LLM on the Internet, on crawled pages. The first interesting part is what happens if users take the output of that model, publish it on the Internet again, and thus create a feedback loop. The other interesting thing is you could ask yourself, what are attack vectors? Well, it's content created on the Internet. Any bias that you have in your training data, you will replicate. So it's easy for outsiders to push a bias into your model if they wanted to. What are unintended side-effects? A very trivial one. When automation has side-effects at large scale, there was a project called Wordfrag, which crawled the internet in order to create statistics on how often people use certain words. These distributions after publishing LLMs and GHGPT were no longer trustworthy, so they had to shut down because what they were replicating was the distributions that LLMs were outputting instead of what's really out there for human language. Now you can go back to our little interactive site and I'll give you a minute to tell me of the side effects of LLMs that you can think of or that you've seen. Thank you. So essentially, as we've seen, the input is interesting, and if we train on the Internet, we get all of the Internet's biases and influences. Everyone who can influence what we take from the Internet has influence on our model and what we output. If you train on social media, same thing applies. Also symmetric itself can move behavior. Whatever moves an email across the barriers that we set up moves how people shape emails. I still remember people training on the text of the email only to wake up to filled hard disks because spammers started putting the advertisement text into images attached to the email. So when you build a pipeline for your machine learning system, what you want to think about is where could someone try to game the system? I remember talking to people working within Twitter thinking about such and such a feature would be nice to have, wouldn't it be nice to see such and such in your timeline about your friends? Yes, it would be nice, but it also opens up the door for someone trying to influence the ranking. So typically, when we roll out such systems, we operate within a loop of observe, orient, decide, and act. We observe the world, we orient ourselves, we decide which way to go, we build the system, and then we check whether we've achieved our target metric. If we deploy machine learning models to a dynamic environment, what we also want to check for is outside of the target metric. What is surprising? What changed? Did the data distribution change? Did user behavior change in ways that we didn't expect? Do we see contents that we didn't expect and don't want? or is there content that we want and that we want to reinforce so we want to look a bit further now a good friend of mine Ellen Friedman in 2007 when it was in its infancy visited Facebook what she said was the ability to look implies the ability to influence and I found that surprising that was 27 I come from a technology field I don't come from a communication science field I didn't see it coming back in 2007 within technology we tend to be in this happy little bubble when I created Berlin Buzzwords it was a conference on big data on search, on scalability on data analytics and I didn't see how it should be co-located with something that talks about impact on communication. It was organized by the same organizers of Republika. Republika is this digital new media conference. I didn't see how these two fit together. It took about 10 years for me to realize what this technology stuff has to do with media and with communication. But then a couple of months ago, I talked to a friend, and they didn't get it in their head that what they had learned about adversarial machine learning so many years ago had an impact on what they were working on on a daily basis. So I started asking in my network of friends, is that somehow unusual that people don't see that. A, we've got a shift. We've seen that earlier when I asked people to raise their hands. Not everyone remembers this cat and mouse game of spammers and email spam classifiers. Not everyone remembers the cat and mouse game between search engine providers and search engine optimizers. So we do have a generation shift here. The interesting thing is though that we are engineers. We are closest to the data. We see what happens if you look close enough. So what I would like to leave you with is that if you build machine learning models to not only look at your daily goals but also to look outside of the box, what can go wrong? What is the impact of what you are working on? The other thing I would like to leave you with is for a long time I have heard people identify as tech, not explain too much what we do. I've met someone who I had to explain like basic concepts of how search engines work to despite the fact that they were married to someone building search engines. That shouldn't happen. We are the ones who have that knowledge and there is this one saying that you only understand a topic very, very well if you are able to explain it to somebody who has no background. So I would like to leave you today with a wish, and that is to explain to the people around you what you are doing on a daily basis and what the impact of that is. We've had the keynote yesterday in the morning about how we need a discussion at the societal level of what we should allow AI to do. To have that discussion, we need an informed society, and we should be part of the people informing society. With that, I'm happy to take questions.

Speaker 2 [32:18]

Thanks, Isabel. I forgot to mention that we would be taking questions via Slido, so we didn't get any, but so I can open for the room and I can bring you the microphone. If you have any questions, raise your hand and I'll come to you. But we're also taking the questions via Slido. You guys got used to it already, right?

Speaker 1 [32:36]

I've moved people to Mentimeter, sorry.

Speaker 2 [32:43]

Help me, anyone?

Speaker 3 [32:48]

It's maybe a bit of a meta question, so I'm also involved in doing some 3D printing, and these people, they have also the situation managed that they have to find a new way to think about doing things. And my feeling is somehow that this also is the same kind of change, the large language model, now on a different community. Can you say something about that, your feeling on this?

Speaker 1 [33:19]

It is pretty much the same thing. You have similar discussions there in terms of, for instance, what are the things that people should be allowed to print or what models should be allowed to be shared. So it's a similar thing there and it's a similar education thing there.

Speaker 2 [33:43]

We have one question here in Slido. Do you think Mastodon is more secure against attacks from Mr. Evil Cat?

Speaker 1 [33:51]

It depends on who runs your Mastodon instance and what they employ. So if you take the ranking approach, it's only time-based, so it's at least one feature less. But I would assume that as it grows in terms of who uses it, it will be targeted as well.

Speaker 2 [34:15]

Another question coming. How do you inform a society which doesn't want to be informed?

Speaker 1 [34:26]

standing in front of all of you and asking you to do that. So the funny thing is we can talk to each other so that we are informed. But then every one of us is part of communities. We are part of the schools of our children. We are part of clubs where we play sports. Just talking to people helps and even just educating your own family also helps a lot.

Speaker 2 [34:56]

Take one from the audience

Speaker 1 [34:59]

Hey, thank you. Do you think social media will inherently lead to a further divide of society? Do you think social media can be used as a good platform? I suppose. I've seen it used in a very, very positive way as well. That's why I've been there.

Speaker 2 [35:18]

There was a comment here. Thank you for not using AI generated images

Speaker 1 [35:23]

What role should Governments have in protecting

Speaker 3 [35:36]

protecting society.

Speaker 1 [35:37]

And regulations. Whatever word we as society give them.

Speaker 3 [35:43]

Do you think we give them enough or

Speaker 1 [35:45]

all

Speaker 3 [35:46]

And I didn't they have enough control

Speaker 1 [35:47]

have enough control over the platforms? I think some people in government could be better educated, but at the end of the time that again boils down to us doing the education. I mean, within the circles where I am, people tend to be frustrated with talking to politicians, but there are people who do listen. They have a very full day, they have lobbyists talking to them, but they also tend to listen if you talk to the right people. And yes, it is very tedious.

Speaker 2 [36:25]

Lots of questions coming now. Is the bubble effect possible with LLMs like with social media?

Speaker 1 [36:34]

What do you mean by bubble effect?

Speaker 2 [36:37]

What do you mean by bubble effect? It's an anonymous question.

Speaker 1 [36:44]

So I would compare it in a way that it's susceptible to attacks as well, because what you put into these LLMs, as long as this input is not controlled by you, there are ways to change your input. And right now a lot of these models are being trained on the internet. The other thing that I can imagine, like, if you think about what data code generators are being trained on, it's open source data. Who stops Mr. Evil Cat from creating an open repository, slapping an open source license there and putting susceptible code in there? What's already happening is that code generators create dependencies that don't exist. They make mistakes. But those mistakes, if they happen often enough, what already happens is that MrEvilCat creates software artifacts, puts them into NPM or into whatever repository is out there so that if you take the generated dependency into your code base, you also get susceptible source code in there. So they are being targeted.

Speaker 2 [38:01]

And how do you notice increased attacks on your model when it's a good time to decide an act?

Speaker 1 [38:16]

It depends on your business model, it depends on what you want to achieve, how you define the attack. So you need to know which way you want to go.

Speaker 2 [38:29]

And can you name any positive effect of generative AI powered bots in social media?

Speaker 1 [38:35]

Powered bots, yeah, sure, was automating tasks. I don't need a human to generate a weather report for me.

Speaker 2 [38:49]

Maybe one question from my side. What do you, in your opinion, how do you see how now that internet basically AI generated content, how this is going to influence the way we consume information?

Speaker 1 [39:06]

So the very first effect I see is that it's just an increase in information, and in a way, we need a better understanding of which information we trust. There was a time when Wikipedia wasn't a trustworthy source that you could cite. Right now, people start relying just on automatically generated answers. I still come from a generation where I try to check the sources and where I try to figure out where that answer came from. because I don't want to put toxic stuff on my pizza just because it was put into Reddit and then fed into an LLM.

Speaker 2 [39:46]

And do you think we screwed the www?

Speaker 1 [39:50]

I don't think so, we just need to find new rules. We didn't screw publishing just because we invented movable printing type letters.

Speaker 2 [40:01]

And one more question. If you could invent the social web again today, what would you change?

Speaker 1 [40:07]

decentralize

Speaker 2 [40:12]

Maybe one last question from the audience before we close the session.

Speaker 1 [40:17]

I've got one question for you who will go out here and try to educate people around you Come on don't be shy Thank you

Speaker 2 [40:30]

Thank you, Isabelle.