Unsupervised shallow learning for fraud detection on marketplaces

Speaker 1 [00:04]

Hello, everyone. Yes, I'm going to guide you through this wonderful journey we had at Adgen for finding fraud in marketplaces. A quick intro about myself. I joined Adgen around three years ago. I'm heading data science there. Before, I had a career in aerospace for around ten years. I started in Barcelona engineering, and I'm from Mallorca, which I think is relevant, given that we are in Germany. If you don't know Adgen, just let me, just two minutes, sorry, one minute to say something about it. Adgen is a European Again, the company is a fintech, we do the core processing of payments, we also offer financial services, all is API-based, everything is digital. We are all around the world, and we are quite big. Last year we processed more than 500 billion euros through our service, and we serve Ubers, Spotify, Netflix, Delivery Hero, Flixbus, for example, these companies. And I'm here to talk about machine learning for fraud detection. And when people think about payment processing, they immediately go into transaction risk, which is a setting where you have a well-known seller, say, Flixbus, and you have people shopping onto them, right? In this setting, it could be you always trust the seller. That's the important bit. And it could be that the shopper is malicious. Say, I don't know, they bought a batch of stolen cards on the deep web. We have algorithms at the gate to make sure they don't go through. We also have an offering for marketplaces. On this setting, think about eBay. For example, we process all the transactions on eBay. In this setting, you have another data point, which is sellers. But the seller, you don't know that well. You do a KYC process, which is everything automated, you gather information like the bank account, the email, the passport, And then they start processing, and you see this interaction between shoppers and sellers. But in the setting, it could be you don't need to take for granted that the seller is non-malicious, so it could be that actually there's some bad people doing selling, right? So we built something for that to help our merchants, we call our customers merchants because we are B2B, and we built a thing called Score that has this flow, just this very high level overview. The idea is that the main agent platform is built on Java and Postgres, and that's what exposes the endpoints to our customers, and then it's served by a big data platform where we massively use Python to help on the decision-making, right? So in this setting, we could use KYC databases, accounting databases, and we do some sort of online community detection, think about it as some quick and easy graph algorithm. And then we would also, from the big data platform, use algorithms to find weird patterns that could point out to malicious behaviour. We send those signals, and then we compute a risk score that our customers can see to see if the sellers are actually malicious or not. On this talk, I'm going to cover this bit, which is the part we do in Python. We use big data a lot, so we're very heavy on Spark over Hadoop. And, yeah, I'll guide you through the journey that we did when we tried to solve for that. First, I want to tempt you with a little quiz. So, here's a toy data set, and I want you to think for a moment what you think here is odd. What's an anomaly? Now, most of you actually have seen five and six being very lonely, miserable there, and it's true. They are what we call local global outliers, right? The groups at 0 and 4, they are pretty well-behaved, same shape, same colour for everyone there. Group number 3 is weird because it's the same colour but different shape, so we cannot say anything, but group number 1 and number 2, you really see there's a data point there that doesn't fit, right? That's what we call a local anomaly, and that's what we try to find out with those algorithms. We're not trying to find out the global anomalies, we cannot say anything about them, they're just lonely. We're trying to find who is different within their context. These are the anomalies. To do that, I'm going to use an example, which I think is the best way to guide you through that. A significant shopper. What is a significant shopper? You expect shoppers to have a certain revenue on a seller. If someone actually buys 95 per cent of the sales of a certain seller, that might not be adequate, right? So, yeah, so I think here in the joke, if you buy €10,000, that a lot? Well, if you're buying a diamond ring, maybe not, but if you go to Febo, maybe yes. And if you're not Dutch, probably you don't know what Febo is, but they're proud of their cuisine, and basically it's this. They serve cold stuff fried from a long time ago. They love it. So, yeah, don't spend €10,000 on that a month. So, anyway, what you really see there immediately is that we can use the average transaction value, which I will shorten for ATV. If it's an Uber ride, it's $15. If it's electronics, $500. And the merchant volume, how big is that store? And you can use these two features to say, okay, what's going to be the shopper ratio that I expect for a normal shopper, right? And we're trying to do that. And that's what we did. And then, like a coyote, we just went there and just hit plants and failed miserably until we got it done. Right? So the first thing you want to try out is what basically pumps 95 per cent of the artificial intelligence systems of the world, which are if statements. So, you build a rule, and you say, well, a bucket, if it's that ATV, and this ATV, and this volume, and this volume, but that, as you can see already there, is going to not end well if you want to have some performance, because you need to put a lot of nodes. For the record, just it's going to be handy later, this basically means that this toy data set, you're going to slice up in different areas through lines, right? What's with rules? Well, if you want some detection performance, you need to put a lot of nodes, and if you put that in production, that's not going to end well. So probably rules are not the best choice. Detection performance, even with a lot of nodes, is never going to be the best that you can get out with other techniques. The good thing about it is that we humans don't trust machines, and we feel good about understanding what they do, so a rule is always understandable, so, yeah, it's nice. Regulators love this, for example. Anyway, that's not the plan. Let's move forward, try something different. Clustering. The idea with clustering, if you're familiar with those algorithms, K-means or HDBScan, you put together all these points that are similar, and the algorithm handles it, right? What's good and bad about that? Well, what is good is that you can put more features. You're not any more bound to use ATV and volume, because you can just put more in the matrix, and that will work. Why it's bad is around the boundaries. Let me make a metaphor there with the billionaires in space last summer. I don't know if you followed, but it was a bit ridiculous they They were having this fight about who's going to space, because was one going to 80 kilometres up and the other was going 100 kilometres up, it doesn't matter. The atmosphere fades out, right? So it's just a convention. So when you have this fading out, it's difficult to establish a boundary. Clustering kind of gets a bit lost there. But moreover, we thought, hang on a second, they didn't ask us to detect clusters. They wanted us to detect anomalies. So this is just like a means to an end. Maybe we're just complicating our lives. So we said, OK. Move forward. Next plan. Then we try to say, well, what's better than a rule and is not clustering? A very simple linear system. What is nice about it is that even if you're allergic to equations, you can understand that one. And basically, you have two coefficients, three coefficients, an offset, and one coefficient for the volume, and one for the ATV, and you try to establish a bit where you should be and have a prediction on the ratio, right? So what is good and what is bad about it? The good thing is that, well, it's really explainable because you can see A and B tells you how much you're putting in the decisioning for each feature. Good and bad, it depends. If you have borders, linear systems are not going to behave well. If you don't have borders, and you're bound to have non- polynomial features, then you might get it. But the detection performance wasn't good. So what we tried next was random forests. And random forests are nice, and if you recall the title of the presentation, was called shallow learning. So the idea is you might be thinking random forest is not an unsupervised learning algorithm, it's a supervised learning algorithm, I need labels. What you can do is try to predict the ratio based on the past and do an average, sort of average or shallow learning on the labels. So you want to be very precise on your context, that would be the features like volume and ATV, and you want to be imprecise on your target, say, the shopper ratio. So you do that, and we like that a lot. There's a certain explainability embedded, because you can do feature significance. You can put more features. We use PySpark ML for random forests, which works nicely, especially for shallow cases. You can embed categories. If there's boundaries or no boundaries, you can still do something. Detection performance is actually quite good, and it's fully unsupervised in a way that you can just throw the full dataset onto it, and the labels are actually what you observed in the past. Let me give you an example for actually a production case. This is a real-life significant shopper. We have 800 million rows, 14 input features, one target variable, and we distributed over Spark, 16 executors, and 36 minutes training time. Of course, this is a visualisation that is downsampled because you're not seeing 800 million rows here. Ended up looking like the UK. Totally fortunate. And I'm asking here, where are the significant shoppers, or big bubbles, or maybe people sticking at corners. No, the algorithm finds the stuff all over, right? And this is the magic of it. So, did we solve the problem? Well, not fully. We still need to validate the model, right? We just took a blind guess and threw the whole data set into an unsupervised setting and tried to get a prediction that made sense, and manually inspecting everything looks good, but we don't have proof. And what What happens if the behaviour is actually more than one dimension? Because here we have colour and shape, but a random forest, you have one target variable, only one. So what can we do about it? And we went into the next plan, which was using, we are data scientists, we like complicating things, so we went into neural networks, and we tried to do the same trick with shallow learning over neural networks. And I know what you're thinking, you're saying, hang on a second, neural networks, tabular data, doesn't work, right? But it does. Because if you're using heavy regularisation, it's kind of these cases where you can actually use it. You can look up this paper which has been recently updated. It's a great introduction on when you can and not use neural networks for tabular data. Very few cases, by the way. We use an architecture, you might be familiar, called an outencoder. The idea is that you have a bottleneck that's imposing regularisation into the dataset. Sorry about that. Shall I off and on again? You guys see it? Thank you. Thank you for hanging on. Hopefully we fix it soon. I asked, they didn't allow me. All right. It was worth a try. up. It was worth a try. Shall I...? Okay. So... So we wait? Okay. So we hang on. All right, okay, so sorry we just try to wait a couple of minutes if they get it fixed I'm not going to move, I'm not going to do anything. So I was saying, sorry about that, out encoders, right? So out encoders is an architecture of a neural network that basically imposes a middle layer that it's reduced, it's a bottleneck, and that basically creates embedding that compress in a way the input signal, then you decode it back, right? Because we wanted to have this trick around learning very well our context and learning our behaviour on the average, we actually kind of gave a twist on it, and we customised the loss function and we created a scoring equation that could help us find those local anomalies. We did that with the Keras interface over TensorFlow, and we used over Spark and Python. Well, we solved the problem. We haven't yet validated, right? It's still the same problem as before. We just have a bit of a leap of faith that the model is doing a good job. Now we enter into scalability problems, because you cannot really use PySpark ML to do autoencoders, and if you use tabular data, it might be that you have certain volatility with the gradients on your neural net. We are engineers, we went and tried to solve for this. Volatility, we used ensembles, so we run 16 next at the same time we average out. And the scalability, we get in there, we experimented with barrier mode on Spark that could help us by shuffling the weights. But we haven't yet validated the model, so that's something that we did last year. How do you validate unsupervised learning? How do you hyper tune unsupervised learning algorithm? That's a hard question, right? So, we created our own test chamber. We created a data set that had statistical properties crafted to the things that we wanted to see, and then we tested the algorithms to have a benchmark. We were very happy we did that because we actually found a bug, and that bug was around the treatment of our categoricals, so we had to actually change the equations, and, yeah, I put the meme there because people ask me whether I just put this from Wikipedia just to wow people. These are true equations, so these are the stuff that is running in production right now. And, yeah, maybe you want to see how it works in reality, this thing. There's some cases that we have seen through Score and how we can help marketplaces fight fraud. This is one where we see that the account is very young, that we have seen unusual chargeback ratios. A single shopper did 90 per cent of the volume. That's already quite a hint. Low ATV and repetitive amounts. What happened there is people inflating their rating as a seller through just little volume themselves, and then they have a great five-star score, and then scam people. That can happen. That's platform abuse. Another case is account takeover. Basically, they stole your credentials, they changed possibly your bank account, and here we see a change of KYC signal and a sudden change on your behaviour, in this case a spike in sales. Could be, in most cases, fortune is not the case, it could be money laundering, outputisation rate, prepaid cards, it could be a fraud ring where we see people actually that are very connected through sharing, say, emails or bank accounts or whatever, and they actually, well, they have little, they don't have shopper variety, they have refusals and stuff like that. So what is next for us? Did Did we completely, fully solve the problem? No, we didn't. Because if I started, right, that was the example case where I started with a sellers and shoppers network, a very simple network, what we did with these algorithms was basically to isolate every seller and look at their surroundings, and then found an embedding or find some sort of prediction around it, which you can totally see as something like that, as a disconnected network, right? Same thing. But we're losing some power there, right? If you go back here, you might say say, hang on a second, there was one shopper that was a fraudster that was connected to a seller that was also a fraudster. What if they are the same person? And even worse, what about the people that transacted with them, right? So that's why we're going full in with graphs. So investing hard with that. Ideas that we can also mix up KYC attributes and behaviour and have way more prediction power and have our algorithms do a better job. And that was my last slide. Thank you very much. Yeah, that's my LinkedIn. If you want to connect and just shoot a message, say that you've been here, so you don't try to sell me anything. And yeah, I'm happy to answer any questions. Thank you.

Speaker 2 [18:56]

Thank you for the amazing talk. We have some questions online, so I'm going to read them. I wasn't aware of unsupervised random forest. Which library did you use?

Speaker 1 [19:07]

Yeah, PySpark. So PySpark ML. That's nice because it distributes on all the machines that we have.

Speaker 2 [19:17]

Can you explain what you used as a target in the random forest? As a? As a target in the random forest.

Speaker 1 [19:23]

Yeah, so what you use is you have seen in the past the the ratio of the shoppers right in this case So that data point in the past is that you would use you would try to predict that right? Just the average not very you don't want over fit. That's the key

Speaker 2 [19:38]

Isn't random forest a supervised model? Maybe you meant isolation forest.

Speaker 1 [19:43]

Random forest is a supervised model, but again, if you use the labels as I want to predict in the average this, you can treat it as an unsupervised thing. You don't have a label to say that happened, right?

Speaker 2 [19:58]

ML machine learning library are you using?

Speaker 1 [20:00]

Good question. For GraphML, we're going big with PyTorch Geometric. We like it a lot.

Speaker 2 [20:08]

Did you try to make the problem a supervised learning by using mechanical torque to create labels or refund and customer support ticket? Yeah

Speaker 1 [20:16]

So we do have in the end labels coming back from our customers when they say all that actually was a fraud case this isn't That the question so the problem was is twofold, right? The first is a cold start problem when you start putting this thing out You don't have any any feedback So you need to have something there and the second is well They're gonna be labeling things that they have seen and also they have some biases when you have humans in the loop They why you know label things here or there What we use it we use those labels to track performance, right? So if we see detection method that for example has a very low we call it interest rate So we have a thumbs up down flow. So if we see that the detection has a low interest rate, then we try to iterate on it and see what's going wrong.

Speaker 2 [20:55]

happens when a fraud is detected? Are you contacting authorities or just sending information to your customer?

Speaker 1 [21:01]

No, this score is for customers. So for example if we see that happening on our GoFundMe for example Where we pilot it with GoFundMe to build this stuff They could take care of it. Is that is a problem?

Speaker 2 [21:12]

All right. Thank you. So I don't have enough time for the rest two of the questions, but you can ask from here. Does anybody in the room have any questions?

Speaker 1 [21:22]

We also have a booth upstairs, and I'll be sitting there, so if you want to just come and chat, that's also fine. It's really hard to understand you. Yeah, of course. The thing is that at Agile, we have a different philosophy where we don't jump straight into vendors. We try to see for ourselves. What we did, for example, was to use what we had, in this case, Spark and GraphFrames on top of it, try to do certain analysis and see what would work. But we're figuring out. Neo4j could be an option we go for, it might not be, we'll see. Any more questions? So the idea is that it's low, right? But... Yeah, but it's not my fraud, right? It's my my customers' frauds, so I'm not happy to disclose. Yeah, but we don't train on it, right, if you're going there with imbalance. It's more to say, right, from all these cases, how many of them were false positive? That's our biggest right and then if we have a big high false positive ratio is when we try to see okay we need to do something better here yeah can you use like some of your cross clients information to help with the validation of frauds excellent question. At Agin, we're super open and transparent in the way how we deal with data, and we consider our data to be in the hands of our customers. We just use it for them. So it's an opt-in scenario where they say, yes, we want to check, and then you can merge and see what's going on with all the exposures that we have. Think eBay, Wix, GoFundMe, and all that. But we We don't push it or we don't do shady stuff with it.

Speaker 2 [23:55]

All right, one more question.

Speaker 1 [23:58]

I have a question. How did you handle PII information like the KYC details for the customers, especially when storing it in the graph database? Indeed. That's a very, very good question. So, fortunately, most of the cases you can just tokenise and stream tokenised data, and then you still do anything around it. Sometimes you miss some prediction power if you don't decompose your feature. A classical example is your email, if you want to subtract the domain, for example, of the email. So we take care of that, but we also, I mean, in Agile we iterate a lot, so we build all the stuff all the time. We're also spinning out new flows to train on PI data straight. It just has different requirements for, well, regulation and all that.

Speaker 2 [24:42]

Thank you for questions and thank you for the talk. Thank you.