Transformers for Game Log Data

Speaker 1 [00:07]

Thank you very much and good afternoon everybody. I'm really looking forward to talking to you guys here today. A brief agenda of my talk for today. I'll start with a short introduction about the topic itself. Then I'm going to quickly review Transformers and GPT. Then I'm going to describe to you how you actually would adapt data from games to work with a GPT model. Then I'll explain or describe a little bit how I train different models. And then I'm going to show you how I use the inference to actually apply to game data science use cases. And then I'll give a short outlook and a conclusion. Just very briefly about me. I studied computer science, then did a PhD in machine learning. If you happened to attend Christian's keynote yesterday, he was actually my supervisor. And as it was already mentioned, today I'm head of business intelligence at Lotum. Lotum is a mobile games publisher here from Hesschen, they are actually from Bad Nauheim. Besides that, I do co-organize a machine learning and AI meetup in the area of Cologne, so if you're around, feel free to join, there are also often Python topics, and I'm also a lecturer at the Technical University of Applied Science in Cologne, and I'm teaching a course about data science there. I think the common theme is that I have always been working with data, AI, and gaming, and this is really my passion. Just briefly about Lotum, who we are. Like I already mentioned, Lotum is here from Bad Nauheim, which is a little bit north of Frankfurt. We are around about 50 people, and we create mobile games that millions of friends and families play together every day. And we celebrated quite a milestone in recent weeks where we had the one billionth install of our games. Our games are typically hybrid casual games, mostly puzzle games, trivia games, those kind of games that are, or we have four franchise titles that cover the majority of our installs or downloads, which is four picks one word, which is probably the most well known one, which has more than 400 million events. But now let's dive into the topic of today. So what was actually my motivation for this talk? And I thought about, okay, transformers are everywhere. And if you attended some of the talks the last two days, you'll also have seen transformer examples in various different talks. There was, of course, Christian's keynote where he had a graphic showing a transformer model. You had then the guys from Get Your Guide who really gave an interesting talk on how to use transformer models for ranking. And they're essentially everywhere. And I was just wondering, okay, if there's such a transformative technology technology and used in so many different areas, is it also possible to apply it to games log data? And I also have the deep conviction that if you really want to understand how things work, you've got to deep dive. And if you want to really have some innovation, you really need to understand the technology, then it's not sufficient just to do some prompting, for example. And there's also then this vision that I have how you could use those models in the future. So by now, I think we all know what actually GPT does. You give it a input sequence, essentially a prompt, and then it generates more words. So it just does it by the next word prediction. And I was curious to understand what would now happen if I take game log data, so events that are sent from the devices, from the games that are expressing some action in the game, you train a model on it, and when you then put some data into your model, what will it predict? Is it actually possible then to have a machine or a model that predicts the future user journey of players. And that's actually still what I'm really interested in and I want to work further on. And it's definitely a model that would have various application in industry. If you look at the area of game analytics or game data science, there are various topics, such as player segmentations. So we want to know who our players are, how they play our games. You want to do something like churn prediction. So you want to know in advance if players are potentially leaving the game. And there are also applications such as purchase prediction for monetization or dynamic difficulty adjustment, which is also tied to term prediction. And just to also frame what this topic or this talk is about, I also want to just briefly say what this talk is actually not about. So I will not have the time to give a detailed introduction to machine learning. And I will also not give a tutorial on neural networks. So I'm assuming some basic understanding of both topics. And it's also not about a production-level system. So this is more of a pet project of mine. This is nothing that we would already apply in or at LOTOM really in our games. And I will also shed some light why this is maybe right now not yet the case. But it's still a promising way to go. So let's start with transformers. In general, I think probably also this graphic has been seen by many of you. In general, what is a transformer? in general it's just a deep neural network and what's special about it that it's a sequence to sequence model so you give it an input sequence and then you get an output sequence and there's also one property of the transformer which is also important for this whole language generation stuff that it's autoregressive which means it consumes the previously generated tokens when it predicts the next token and initially that was actually introduced for machine translation and in a very simple example, you would have a transformer architecture, you had a sequence or input sequence of four tokens, something like, why are GPUs important, and then you would get an output sequence, which is the translation, which says, why are GPUs important? And in general, I think the main takeaway is just to keep in mind transformers are mapping one sequence to another one. However, we are now, in this setting, not really interested in something like machine translation or mapping one sequence to another. we're rather interested in the case of generating information, more similar to GPT. So if you take a look at GPT, which is doing next-word prediction, you give it again an input sequence, and then you get an output sequence, which is predicting the next word. As you can see here, essentially, okay, you can see my pointer. So the output that you are training your model with or that you're expecting to get is just the input shifted by one token. And this is also the reason why I'm here focusing on GPT, which is essentially an instantiation of a transformer architecture, because I'm interested in next event prediction, so to speak. And GPT is nothing but a generative pre-trained transformer. And if you take a look at the original Oatmeyer iPaper, you'll not find all that much information about the architecture itself. It just simply says, we use a multilayer transformer decoder for the language model, which is a variant of the transformer. So we should better understand what the transformer does to really understand what GPT does. The transformer architecture itself, I said on the last slide, it's a multilayer transformer decoder. And if you look at the transformer architecture in total, you will see one left part, which is the encoder part, and the right part, which is the decoder part. And essentially, we can here entirely ignore the encoder part because we're only interested in the decoder one. In the following, I will give you some Python code snippets because I really want to highlight the fact that it's not all that difficult to build a very simple GPT model yourself and adjust it to your needs. And my code is actually heavily inspired by the nanoGPT by Andrej Karpathy. It's a really nice YouTube video where he explains in around about two hours how the original GPT model worked. And as you can already see, two hours, that's not the time we have, so really rush through it that you just get the basic understanding. And then if you're further interested, I would strongly advise you to take a look at the repository. That being said, I'm even further removing parts here from the code just for brevity, so I'm not talking about dropout, and I'm also removing all the GPU-specific code. So in general, I showed you a similar graphic graphic to the one here on the right-hand side when I talked about the transformer architecture. If you just wanted to implement this kind of GPT model in Python, you would define some neural network class, and I'm using here the code that is also based on PyTorch, and the definition is probably not a lot more than those five lines of code. However, initially maybe the embedding you obviously heavily rely on what PyTorch is already offering you. You have the embeddings for the input, and then you have a list of layers. I will give a bit more details on the layers in a second, but essentially this is this template structure that you have here. This blue box is a layer, and you have these end times. So most of the computation lies within the layer, and I'll give you some details in a minute. And then at the end, you have this language modeling head, which is the green box on top. And this essentially takes care of transforming the output of the layers into text. And then, as it is a neural network, you have a forward pass. And there's also not much magic that's happening in the forward pass. You initially get the input indices, which is just a tokenization of your text. Or in the case of game log data, it would be indices of events. And then you just calculate the embeddings. And then you add them. And then you move the sum of it into the layers and push them through every layer one after another. Maybe what's briefly worth highlighting is here the embedding, because initially when you put your data into a transformer architecture, you may have a list of tokens. And essentially, the embedding class takes all care of mapping this to a embedding which not only reduces the dimensionality and makes it more compact, but it also adds some semantic meaning to it. However, if you use an embedding, you lose the position information. And it might be quite important to have the order of words correct. And this is why you have this positional embedding, which then adds, again, position information to the data. Now, I already mentioned that probably most of the magic happens somehow in the layers. And if you then look at the code of the layers themselves, yet again, not much magic. There is a multi-head self-attention, which is depicted here, which is probably the most important part, which I'll talk about in a minute. Then you have those layer norms and the feed forward network. Most of this is everything you get out of PyTorch, more or less, for free. And then the forward pass, again, just transforms the data. And then you also see here those skip connections. You essentially take the input that you send to the mass multi-self attention and add it again to the output, which is done for regularization. Coming to the most interesting part, probably, of the transformer architecture, which is the masked multi-self-attention. So the original paper you might have heard is attention all you need. And attention is really what makes the transformer architecture work so well. If you look at what the multi-hat attention actually is, it's nothing more than a concatenation or several self-attention hats in parallel that you just concatenate at the end again. And if you look at it, how it looks in code, It's also fairly simple, because you just have a list of those heads again. And as you can see, essentially moving the complexity from layer to layer to layer. So we started at the head, then we moved to the multi-head self-attention. So the next thing that I will explain to you is the actual head itself, because this is where the computations happen. If you, again, I refer here to the graphics from the attention is all you need paper, you will find this graphical representation of the attention calculation. If you look at it in detail, then you will see it's a series of matrix multiplication, and there's a masking component. So really, speaking on a high level, this attention mechanism allows the model that the tokens can look around and see what's in their neighborhood, essentially, then to make a prediction what the next token is. And the query and the key are essentially computating a relevant score for different tokens how likely they are to come next and then you multiply them with the value which is the tokens or which are the tokens themselves at the current position one thing that's fairly important in particular for text generation is the masking because if i now want to predict a sequence of words and want to predict the next word i cannot look into the future because i don't know words that i haven't produced yet essentially opposed to translation, for example. And this is why you have this masking where you limit the relevance scoring only to previous words. And if you look at this in code, then you have three linear projections that you define for query and value, and then you have those matrix multiplications here. I'm not going to go into the details here. It's essentially you have a matrix multiplication, you scale it, But then it's the interesting part again, the masking, where you then limit on how the tokens can actually look in what direction. And this is essentially, you can think about it like that. If there is a matrix and essentially this is always the token you want to predict, you can always only look back, but not in the future. And yeah, that's it actually already all about how to implement a transformer architecture and let's say 50 lines of code or something like that. Now, the interesting part is, I hope you have a general idea now on how this works for natural language processing, but how are we going to go about it if we have game log data? In general, the idea of using sequential information and sequential models in game data science is not entirely new. Probably around 10 years already, there was a paper that won a churn prediction competition back then, which used an LSTM component to model sequential data in a user journey. So this idea of having various events following each other and using this information from it was already somehow used there, but it was combined with a pretty complex neural network that encoded other features, such as counts of players, metadata, so on and so forth. year, however, there was this work which was called Player2Weck, which is a collaboration between King, the other well-known mobile studio, and a university in Sweden. And what they essentially did, they trained a long-form model, which is an encoder model. So GPT, decoder model, BERT is an encoder model, and they applied this to event log data from some mobile game as well. If you then think about, okay, what are the different components and how they map to each other, you will see that in natural language processing you have certain ideas like the the use cases text predict or text generation. In game analytics you have something like behavior prediction. Then in natural language processing you have words. So every word is, not every word is one token, but one word may correspond to several tokens. In game data, you have events that are fired from your devices, from your games, something like app started, level started, level finished, so on and so forth. Then in natural language processing, you have a structure in text, which is sentences. So I was wondering, okay, what's the sentences in a user journey? And then you realize they're sessions. And from that perspective, when you have a dot in a natural language processing sentences, I also added a token, for example, to distinguish different sessions of a user. And then when you think about documents in natural language processing, which is maybe one level higher, then you also have user journeys in games. And when you think about this slide that I had several slides ago, where I said, okay, what we want to do with GPT, where we want to generate words, we're now in a setting that we want to generate those game log events. If we have a user journey or part of a user journey, which is game start, start level, wrong answer, use joker, you wonder what's the most likely event to follow after this series of events. And in that case, it was a finished level. Now, I'm using in this work that I'm presenting here mostly data from an open source or actually available for academic usage data set from a game that's called Blatant Soul. This is an interesting data set because Bait and Soul is actually a quite well-known game in Asia, and it has also been released in Western markets roughly 10 years ago. And the game itself is a MMOPRG, so it's a massive online game with a role-playing component, and you have two different modes to play. You can either play it, so to speak, single player, you interact with the environment as a player, or you can do a multiplayer experience where you play against other opponents. And it has a martial arts combat mode, and players can level up their skills, buy items. So it has a vast amount of actions that players can do, and it has an entire economy. The data set itself is also actually quite granular in that sense. I mean, the entire data set is split into three. And there's one training data set, and there are two test data sets. We will ignore the test data set here today entirely because it was mainly published to test a concept drift. So the publisher of the game, NCSoft, decided after a while to change the business model. And they wanted to know how previous models would work after they changed their business model. So for us, it's more interesting today the train data set and test data set number one. And you will see in the data set that it has around about 77 events that a user can make. And this is, in total, you have in the training data set around 48 gigabytes data, which amounts to 175 million events. So I think when you look at the data, 3,000 or 4,000 users is not all that much. But if you then look at about 175 million events of those few users, then it's quite granular how the user journeys are described. right? And if you then, if you would dive a bit deeper and take a look at the actual data itself, then you will see a bunch of CSV files. I mean, we all love CSV files, right? And in particular, if there are a couple of million rows. But still, to take a quick look at it, you can open it in something like Excel, but we learned today probably that DuckDB is the more interesting choice here. And then you will see essentially one row is one event. In this example, let's say here we have the log ID, which is 4001. And this could be something like a repair item event. So you know, OK, at that timestamp, a player repaired something. And then if you wonder, OK, what did that player repair? Then you can look at other columns. And then you will see, for example, at the metadata, you know the entity code here was, I don't know, 40. Then maybe that player repaired a weapon. And if you then think about it, OK, this is way bigger vocabulary than only 77 events because you can now start to construct a vocabulary. And that's actually already a quite interesting point because if you follow a lot of the discussions on machine learning and all those kind of stuff, there's always this one camp that says, oh, we don't need feature engineering. And if we use big neural networks, it will come all for free. But in that case, it won't because now you have to think about how you build your vocabulary. I mean, natural language processing, in the case of GPT initially, it had a vocabulary of 40,000. And now we're starting maybe with a very basic vocabulary. We could build a model that solely has 77 events. So very small, definitely not a large language model. Then you have in total maybe 79 tokens, because like I mentioned earlier, you have one token to distinguish sessions. And then you have maybe another token to distinguish or to explicitly say this is the end of a user journey. However, this may be not enough information to really distinguish different use cases. So, for example, if you later want to know if a player died by another human player or an AI player, then you need to distinguish those kind of events. So, in the next iteration, what I then did, I took a look at some of those columns, expanded the vocabulary. Essentially, I started with the 7-7 base events, and then I took a look at some of the columns, and there's something like an item. And then I started to append it. So essentially, initially, my tokens would only be E2113. Then I continued with E2113 underscore 2. And you can continue this game with every iteration. And probably I could have spent enormously more time on creating a more powerful vocabulary, but I only did it a couple of iterations just to see what the effect is. So by now, probably you hopefully have an idea what the transformers are how we convert data from games and feed them into a transformer architecture let's train a model now the fun stuff begins i initially thought but then realized okay now the pain actually starts at the beginning i was quite naive everybody tells you yeah you need gpus for deep learning and otherwise it's not working i said okay i'll start with a small model So I really took the smallest possible model probably, vocabulary of 77 tokens, an embedding size of 32. So this is, for example, if you compare it to GPT, GPT-1 had something like an order of 768 or something like that. I had one head, so no multi-head, and I had two layers. And I took a context window of 1024, which is already quite large if you compare it to the original GPT. But the idea for me was you see those user journeys are very granular if I start with a context window of let's say 32 I almost don't have any information because it's probably just the last quest the player solved or the last combat he or she had I Mean this is around 60,000 parameters definitely not a large language model. No multi head And I then also said okay Let's start with maybe 20% of the data from the training data set and let's train it for 50,000 iterations Yeah, that took around 12 hours on this MacBook and it was quite loud I was quite happy that I didn't have to sleep in the same room and then also I'm not the patient person So I then quickly realized That the results that were generated somehow made sense but it was it was sort of also gibberish it makes sense that the order of events is correct, but then the entire user journey didn't really make sense. Now I realized, okay, now let's start with hyperparameter optimization. And this is when all the pain started. Because then you start like, okay, how large do you want to build your vocabulary? I mean, I gave you some examples earlier. And then the next question, how should I scale the context window, the number of hats, layers? Even the hidden states in the feedforward network are maybe something you want to optimize for, and this is just a model component. Then you also have the training component, where you can play around with dropout parameters, you can have different batch sizes, the number of iterations you train for. It's way too many parameters that you can do an exhaustive search. And this is also these days when there's so much discussion about costs of training. And this is actually what the costs are. all those trial and errors where you try smaller models until you have a certain level of confidence to actually train a large model. And then I decided to play around with some, I would say, mid-sized models. So I played around with the number of heads, with the number of layers, increased the vocabulary. I got to a point where I had, like, models of 300,000 to 400,000 parameters, and it's It's easy to say that I would have needed to wait days to have results here. So that wouldn't scale if I want to train dozens of them to get a better idea. So I moved to Colab. So Google Colab is actually quite awesome because you get a Tesla for free. So I uploaded my code to Colab, started it, works like a charm. It's orders of magnitude faster than a basic version here on the MacBook. There's just one issue. They tell you, yeah, hey, you can run this instance maybe for three or three and a half hours. That's not too much, as I realized. I mean, I was able to train models with, let's say, one order of magnitude more parameters because they take around about two to two and a half hours for 100,000 iterations on Google Colab. But then you quickly get those kind of messages that you're getting disconnected. And that's also one of the pains if you work with Colab. It's not really meant for a long-running background job. I mean, this is something I thought, okay, then I'll need to game up. And then I also, frequently, if you choose a runtime, you also get those gray-shaded other hardware models. And then you get curious. I mean, they tell you L4, they tell you an A100, and then you think, okay, A100, these are the ones that you always hear in the media, and you want to run your code on this, but they want to have money for it. So I then thought, OK, let's move everything to GCP and spend some money and do the YOLO run. So essentially, this is also you hear a lot where people say, OK, try various things. And then there's the YOLO run, where you put all your hopes into a single run. And you have dozens of options. And you play it around. And I tested dozens of models on Colab and other GCP instances. Eventually had a vocabulary of 300-something tokens, stayed with the same context size, embedding size of only 192. But keep in mind, if your vocabulary is only around about 300 totems, it's not worth it to have a huge embedding size, because then at the end, what's the benefit? I realized that three heads and six layers seem to be somewhat of a sweet spot, at least for the analysis that I've been doing so far. So still not super deep, but you then get to a model which has around about another order of magnitude more parameters, 3 to 4 million parameters. I mean, of course, if you compare it to something like LAMA or whatever, it's nothing. But it's a specified or it's a model for a really specific use case. And then I started the A100, did the run. It takes probably around 11 hours on an A100. And all the other models would have run probably in a fraction of a time. The models then was done after a while. And then I did the inference, and I'll show you the results of the inference in a minute. Only one thing I also wanted to mention, one super important hyperparameter you need to take care of is the learning rate. And in the end, I think most of all of those model questions about layer sets and something like that is not as important if you don't have a good learning rate. Because these are two loss curves of the same model with a slightly different schedule for your learning rate decay. And that can make quite a difference if you end up at a loss after 100,000 iterations, which is maybe around about 0.5 or 0.6. But now we have the model. We had the YOLO run. And then I was super eager to test the model on two different use cases. So the first use case was player segmentation, as I showed or I said earlier. And the second one was predict future events. Because in particular, the second one, today, if I wanted to have a, let's say, purchase prediction model, I would train a separate model. And I would do it essentially for every event. And my hope was, I now have this super powerful transformer. It can do it for all of the events. But initially, I started with the clustering or player segmentation, because this is also the primary use case that is shown in the player-to-veg paper. It is fairly simple to implement, Because what the model also gives you, if you think back at the GPT diagram that I showed you, there was this text generation hat. If you just ignore that hat and you take the output of the last layer of your model, then you have something like an encoding of your current player or the user data that you put into the model. And then you can take this matrix, which has number of rows as the context size and number of columns as the embedding size. And you can apply pooling, so that you reduce it to the length of a vector of your embedding size. And then you have a vector that represents your current input, so to speak. You can use that data. You then first apply PCA to reduce some noise. And then you apply a t-distributed stochastic neighbor embedding that you can plot it on a two-dimensional grid. And for clustering, I use the Gaussian mixture model. And still, my approach somewhat differs from the player to VAC, because I'm not using the BERT model, but the GPT one. They are also using a different way of pooling. And then if you take a look at the results, it's actually quite nice clustering. I would definitely say it's not perfect. And if you take a look at the paper, their clusters are even clearer, but they have probably around four number of digits many events. But still, if you look at the clusters themselves, some of them somehow overlap, but they are also clear clusters that you can distinguish. For example, if you take a look at the gray and green cluster, you can see them here well, you will see that actually those are users that have similarly a lot of engagement, let's say like mid-core players, but the one cluster has a lot more of interactions with the environment, so they're playing a lot the solo mode, they are solving a lot of quests and those kind of things, and I think it was, I don't remember the exact names are probably twice or three times as much of those interactions, and the other players seem to be rather interested in the multiplayer combats. And the same is, for example, this blue cluster here really shows you the high-engaged users while the upper yellow cluster is somehow the low-engaged users. This as a first application, and I went on and took a look at the event prediction. Like I said previously, this kind of event prediction is something that I was interested in because it is quite akin to a purchase prediction or difficulty adjustment if you want to learn if a user will solve the next puzzle or quest or not. And essentially then I took a look for two events with different propensities. So one event is the die event. So I wanted to predict if a user is going to die in the next 1,024 tokens or within that span. And the other event was a kill player character, which has propensity around about 30%. To be fair, if you wanted to do a purchase prediction, you would need to take a look at events that probably occur less than 5% or something like that. And to be honest, in those cases, the models don't work well yet. And I think events with 30 or 70 percentage of likelihood is sort of an easier case, but it's something you can start with. Now the question is, how are you going to do the predictions? And I identified three different ways you can do that. The first one is the obvious. You now have your transformer model. Let's generate the tokens. Let's generate 1,024 tokens, see if the event occurred. Then the other one is, you'll hear a lot about fine tuning. And if you also look into the original GPT paper, there are various fine tuning cases. When the GPT paper came out, it was a lot about fine tuning. And only in later iterations, they realized that those large language models don't need that fine tuning. So I went a bit back and also applied fine tuning to my model. And then the last one is that goes back to the idea of this representation learning again. You can use the same encoding that you get for your clustering and also apply a classification algorithm on top of it instead of a clustering. So when I did the first approach, This is essentially pretty much the same GPT graphic that I showed you before. Generated 1,024 tokens, and then I measured the F1 score. So for the die event, I got a score of 0.79. For the kill PC one, I got a score of 0.38. Those scores are probably not very meaningful to you now. I'd rather use it as a general trend to show you how it compares to the other approaches. And I will give some insight, or I will just give some background on this later. But what's more important now, this initial idea that I would just have this transformer that would produce those tokens, and I'm done, just didn't really work out well, to be honest. So these are not very good scores. And what you realize is, and you then look at the events in detail, that it was often maybe components that make sense, but not in the long run. So that might be due to the fact that my context window was not small enough, and it's very computationally demanding, because you need to generate 1024 tokens. That is comparably expensive if you look at other benchmark algorithms that you would use in data science. I mean, think about a decision tree or something like that. Last but not least, it's nice because you can just take the model out of the box. I still believe that this is the path I want to further investigate, but at this point I said, okay, I don't want to show those numbers at PyData, I need better stuff so then I did the fine tuning so fine tuning replaces the text generation head by a classification head then you train or you have your base model which is your original model, then you change the head and then you do some fine tuning, several iterations and you try to train a binary classification algorithm with it And then you just do inference, but you only need to do one inference, right? Because you only need to go with the data once through your model and then get a prediction. Here the results already looked a lot better with the die event about 0.86, kill PC event still not super good with 0.53. Definitely better results, definitely faster, but the disadvantage is you need another training iteration. and you lose this charm that you have one model for all of your use cases. Then last but not least, because I wanted to try it, I thought, okay, what happens? And also, okay, one thing I need to mention which is quite important, fine-tuning I found super challenging because I had quite some issues with overfitting. So it seemed to me that I probably didn't have enough examples for fine-tuning and that was really not trivial to really get a good fine-tuned model. So from that perspective, I said, okay, let's just ignore the hat, just take the representation of the users again, just train a simple scikit-learn algorithm. This is essentially the best working version so far, where you just take the input that you would previously also use for the classification. I trained a random forest, and it gave me a score of 0.86, which is pretty much the same as previously, but much faster. and the KillPC went up to 0.56, which is the best result I've obtained so far in the past weeks. But it's not really nice because you have a complex transformer model on the one hand and then you train another algorithm on the other side, which also needs sort of fine-tuning, right? But the nice thing is you can train, I don't know, hundreds of random forests in essentially no time. Yeah, and then also I mentioned that earlier, I'll just do that very quickly, I mentioned to you that this masking is essentially one of the key components when you train a transformer model and I also mentioned that the paper player to rec uses a longformer model. And essentially longformers are a way to get a more efficient way of the attention calculation And you achieve this by not giving the tokens access to the entire previous sequence, but you limit it to a fixed number of previous tokens. By doing so, you change your computational effort from squared to linear. I also played around with that and got actually some interesting results. So, for example, if you were to train a model that's somewhat comparable to the YOLO GPT that I trained, you will probably need 30% less training time, which is already, sorry, 30% less inference time if you got the model, which is actually quite interesting. I then also thought, okay, now I can train way larger context windows, so I doubled the context window from 1024 to 2048. That was somehow surprising. I mean, it slightly outperformed some of the representation-based learning results, but overall, results were somewhat comparable. So let me give a brief outlook and summary. So is this now ready for production? Certainly not. I mean, compared to other models that we would use at work, for example, this is way too complicated and takes way too much computational time. Then there's also the question of clustering. It's nice to show you that graph, but the actual proof will be now talking to a product manager or a game lead if that is helpful for them. So this is still outstanding, I would say. Then also the other question is F1 score. Has this been a really good F1 score? This is something you need to really test in the wild where you run an experiment and see if you can really improve the engagement of your game, for example. I mean depending on how you use that information false positives and false negatives can have different costs. Also, when I look into the future what I want to do, I certainly want to improve the word prediction because right now I did it greedy and typically you would do something like a beam search to get better sequences I also want to further expand vocabulary and I have started to train models on our own data at Lotom but with a very small vocabulary and as you can see here in the clustering that's not working well with a super small vocabulary yet that needs some more expansion and yeah then definitely the goal is in the long run to have an A-B test that is actually fueled by a model like that. So what are the takeaways? I think developing your own transformer in Python, it's really just a few lines of code. It's also quite straightforward to get your initial game log data into your transformer, but then it starts to get more complex if you want to have a good vocabulary. Heavy lifting is still based on GPUs. I've started to look into MLX, the library to run code efficient on MacBooks. That also looks quite promising. It's quite nice to have the same model for different use cases, and that's still charming about the approach in general. And I think the results are promising. I mean, I'm definitely not done with this work here. I mean, if you sometimes call something an MVP for Minimum Viable Product, I would call this maybe an MVM, like a Minimum Viable Model. It proves the point, but we are not done yet that this is something right for production. Last but not least, I want to thank two people, actually three. I should probably thank ChatGPT for collaboration, but I also want to thank David Leonard and Rafet Zifa. These are two friends of mine who have published similar work in the past, and we still have regular chats, and their discussions have been super valuable and helpful for me. And that's it pretty much from my side. Happy to take questions. Thank you. Thank you, Fabian, for this wonderful talk. We have some questions on the Slido, and I will address you one by one. How can you model concurrent events? There were events with the same time stamp in the log examples. This seems to be quite different from the text sequences. I mean, in general, for me it's difficult to think about events that really occur in parallel, or at least if I think about a user journey, there are not too many things of that. I don't know who asked the questions. example, let me know. But in general, I would also say that the engagement in the game is in parallel. I mean, maybe there's some state change. Is there an example? Or can somebody let me know? Yes, please. Yeah. I mean, this is also what happens in practice, because there's batching done when you send the data. And what we do at Lotum, for example, we have an event sequence number. And then what we essentially take the time, and then we sort it by the event sequence number. Something like that happens, for example, if you start the app, then you fetch some information about the account, you initialize some experiments that essentially happen in no time, but they still have an order. And then you get the data batched with the same timestamp, but then you add another information for ordering. So in the end, semantically, they're happening in an order. Have you experienced burgeoning issues and the need for deduplication, like report v2.docs, report v3.docs, and what is your experience? So in general, the question was about deduplication. The data set from Blade and Soul, I have not observed deduplication, but in my day-to-day work, this is definitely a topic that occurs a lot. And in fact, we have actually recently started to look into that and also apply things like embeddings to get this information or to get rid of this information but not as part of this work right now and also I think in general the hope is if you train a model and you see data often enough then you realize that something like level solved level solved it's just a duplicate and that you get over it if your if your attention span is long enough at least this is my hope in that case in your experience is a higher granular granularity of the log data better for the transformer architecture? Yeah, I would definitely say so. Just with the very initial examples that I showed for WordBlitz, where I only used the events and no additional parameters, even clustering was difficult. And I do have high hopes, if I spend more time on the vocabulary, that this gets better. And I think also for the blatant soul data set, this is not the end yet in terms of . Would it make sense to have one day or one session as one token? one day and yeah so the day aspect super interesting definitely um so the i use a session token but that doesn't give you yet more time information and having a similar token to distinguish days and things like that is definitely things i find interesting and worth trying okay last question because we are out of time have you tried comparing your approach with zero shot inference with example context data and a classic llm with a long context window so what I have done is as part of my role as lecturer in the last semester I have asked my students where we worked on a term prediction algorithm to actually just use a basic LLM but that was almost impossible to get meaningful results it was pretty much like random okay that's it thank you Fabian once again please give him a round of applause