Reinforcement Learning for Finance

Speaker 1 [00:07]

Hello everybody. I have here prepared a slide deck plus a lot of code that I want to share with you. It is related to my recently published book which has the same title as the tutorial, Reinforcement Learning for Finance. Before we get started, I am of course interested in who is using reinforcement learning at all? Quite a few. Who has a finance background? Who is using reinforcement learning for finance? Oh, just one arm, oh my goodness. Okay, maybe that's a good opportunity to combine the fields for you. Let me get started with my slide deck here. With a quick introduction, a little bit of background, so it was mentioned that I'm author of a couple of books, And here you see six of them. Python for Finance, I think, is the best-selling one. So many people know me for that one. Not only professionals around the world and hedge funds, asset managers, investment banks, but also at universities, which makes me pretty proud that my books are used these days at universities and not just by cracks already working in the industry. Many, many people come to us also for algorithmic trading, for example, and, of course, a little bit of AI and finance. But my personal background with my PhD is actually in quant finance, mathematical finance, whatever the field is called, was called back in the days. So this is where my origin is, not AI, not algo trading. But, of course, we need to cover and I want to cover and I'm really excited to cover all these topics these days. So our main business as the Python Quants is our certificate in Python for Finance, which has grown over the years, I would say tremendously. Just last week, somebody wrote me, if it has the impression of a complete master's program, I would say that's true. So we cover quite a lot what is required these days in the financial industry in terms of Python for Finance. Among others, also how you can apply reinforcement learning for finance in different subdomains, such as in algorithmic trading or, let's say, in option pricing and dynamic hatching of options. We run, for example, also a Discord community for people who are interested in quant research, quant development. Not only necessarily people who exclusively use Python, also other technologists are around to different topics that are discussed. People have found jobs there. So if you're interested in Discord server, below is the short link. I provide all the links to everything that I'm showing you and all the code that I will use a little bit later. We also run meetups since quite a while. So it's 10 years ago that the upper left one was founded in London, Python for Quant Finance, which has close to 4,000 members these days. So recently we haven't been that active, I must admit, but we are now planning with our partners in London from LSEG, London Stock Exchange Group. We are planning to pick up our recent cycle and do more live events in New York as well. And in a few other locations, we have done our live events on site, in person for a number of years, obviously. For you, just as a background, I'm not going through this. If you're interested in what I've been doing in the past, here is the summary. So, overview for the tutorial today. It is based, as I mentioned already, on my recent book, Reinforcement Learning for Finance. It was published in October last year. I have covered, for example, Reinforcement Learning for Finance, also my book, AI in Finance. but here I go quite a bit deeper as the title might suggest and have a much broader view on the different fields to which you can apply reinforcement learning as an American method to approximate optimal solutions. Here's the overview of the book and what I was planning to do today. I'm flexible if there are specific requests or ideas I'm happy to go through maybe other parts here as well so the first part in the book is the basics so learning through interaction i think a pretty interesting introduction in terms of like what does it mean to learn through interaction right many people say that reinforcement learning in that form meaning that you get feedback either positive feedback or negative feedback is how we as human beings usually learn as well or how our pets might be trained right therefore it is very fundamental and as we can imagine when we are back at school we didn't get like a thousand samples right and then try to implement something that is remotely similar to supervised learning no it's just like step by step one by one getting feedback immediately given what we have done so that's the first part then financial Q learning this is all about finance here therefore here I want to introduce the basics and that part I want to show in the next Jupyter Notebook, these will all be Jupyter Notebooks, data augmentation because in finance we have a fundamental problem with regard to machine learning and AI. Generally speaking, we are limited in terms of data. When people now discuss, yeah, for the LLMs to train, the data is already to scars and we need other approaches. again through reinforcement learning some proponents these days argue in finance we have a much more severe situation that basically when you have a look at historical data it is limited and it's not that much even when you change from end-of-day data to tick data this is still not that much in terms of what usually is applied in machine or deep learning context right therefore i want to focus here on the synthetic generation of data which is hopefully as good or close enough to real data so that we can use it for reinforcement learning, even if it's not like the real deal, right? So synthetic data, absolutely important topic these days in not only reinforcement learning, other areas rely on that as well these days, but in finance, generally speaking. And then I have a plan for two applications, dynamic hedging and dynamic asset allocation. Even if you might not know exactly what is going on there, I think the basic ideas should be pretty straightforward to understand. I have other stuff in the book as well, like optimal execution. That is, I would say, a little bit more esoteric, but that's basically the plan. All in the form of Jupyter notebooks and all is shared, will be shared via a link that I'm going to show you, I think, on the next slide. So, you can either follow the link, the bit.ly short link, pycon.de underscore 2025, or you can use your phone and go via the QR code. And that's the gist where I have not only placed all the Jupyter notebooks that I'm planning to go through with you. I also know I need my classes because the resolution here of the screen makes my resolution so tiny. Here, that's the gist. Let me increase that a little bit. So in the gist, what you find, A, is the link here to the slide deck that I'm using. B, there is the link to the repo, to the full repo of the book code. There's the link to the book itself. a couple of social media links, let me see, this wasn't reloaded, there should be a bit more, exactly, now we see here, here for the gist, if you want to go with the original code, you might need to set up a specific TensorFlow environment with older versions of TensorFlow, the code as it stands right now, as it was published in the book, is not executable with the latest versions, or I would say it is executable, but it's terribly slow, so therefore I stick still to it, but what I have added to the repo are PyTorch versions, so all but two have TensorFlow code, so all but two have been changed here, or TensorFlow has been exchanged for PyTorch, right, and these are all versions that if you, I mean, A, you can set up your own environment, probably you have PyTorch already installed, this should run out of the box, there shouldn't be that many issues, or none at all, hopefully, or you can run them, I have tested that on Google Colab, so no need to install anything, right, you can go on Google Colab, wire this link here, and not for all notebooks, but for some of them, there are additional files, like Python scripts modules, that are required for some imports right therefore if you put in the first cell in google colab then the clone operation and then you navigate to pytorch then everything should be runnable there as well the first notebooks that those that don't have any imports they should run out of the box but whenever it complains that there is something not found then you can go that way right here via git clone and cd. This should be it with regard to getting up and running if you want to follow along but I would say there is no I was asked before that if there was a requirement I said no you can just follow along here to tutorial but if you want to go with the code as well these are two options that you have either you set up your TensorFlow environment via the YAML file with Conda or you use the link to the repo here via Google Colab. Then here we have all the files. I have executed them and then uploaded them so that you see all the results. So this is also an alternative when you say well I'm fine with the frozen versions here then this is basically all you will need what is here in the gist all the five notebooks that I'm sharing here are executed so that you can inspect them and they should actually be reproducible in the sense of let's say that chart or that chart or whatever comes later here right should be I hope so exactly the same as in the book so that is usually what I'm striving for, that the code with all the seeds fixed etc. should reproduce exactly what is presented in the book. Of course no need to execute it always that way, but that's at least what I strive for in the first instance. But before we get to the code, let me get back to my slide deck. If you have trouble accessing something maybe you ask your neighbor or you ask me but this is all available via the link QR code via the slide deck etc so dynamic programming let's get to the bone of it to the bones here right on its dynamic programming we say well reinforcement learning for finance is like I recall like many many years back when people were writing me emails and say well what about machine learning in algorithmic trading what about machine learning in asset management i said yeah i don't know i don't know well so we came from the computational finance side where option pricing etc this was driven by completely different considerations right i said well yeah this all sounds interesting so i'm talking like in 20 i don't know 15 16 something like this right and i said well i don't And these days, you hear like AI in finance, and just three weeks ago, I co-organized a conference about AI in finance in Austin, Texas, right? So, AI in finance, something that people simply talk about and say, but it might not be obvious why we should apply reinforcement learning, for example, to finance at all. Yeah, let's get started with a little bit of a theoretical overview. So dynamic programming is something that is used as a method to solve dynamic decision problems. What is meant by that? Yeah, these are repeated decision problems, right? So when you pick up still these days a finance book and you get started in the first chapter, second chapter, third chapter, right, and the basic introductory parts, what you usually are presented with are static models of finance, right? These are two-period models or one-shot models that say, well, consider we are today, and then consider another point in time, maybe the end of the year, right? And now we want to optimize an investment in a portfolio, or we want to price an asset, or we want to do X, Y, that. So the static view is something that has dominated for a long time. You can almost say over decades in certain fields, certain subdomains of finance. But most of financial decisions are simply dynamic. Dynamic programming is a method that is also used since the 60s, I would say, in finance for the solution of dynamic decision problems. This means that we are not having a situation where we say, well, here today, 1st of January, and then we look at the end of the year, and that's it. No, we might have a look here at the first of this month, the first of the next month, the first of the month thereafter, and so forth, and we go forward. And most of the decisions that we face in finance or in life, in real life as well, are not one-off decisions, are repeated decisions, right? Here, I've tried, this is a non-exhaustive list here of typical examples, But the field of quant finance is basically about decision-making over time. And this might be reflected in a consumption investment plan over time, right? When people say, how much should I save versus how much can I afford to consume? Then market timing, going in the market, investing, going out of the market. I mean, we have just had a very interesting period of high volatility in the financial markets, right? where many people try to time the market again, some exited, some were exited due to leverage and margin constraints, etc., then we have the portfolio composition over time. Even if you say, I'm staying invested in the market, I might recompose my portfolio, let's say, with regard to the simple decision between stocks and bonds, for example. Go at 50-50, 30-70, 60-40. These are also decisions made over time. You are not required to stick to something that you have decided yesterday. You can basically always, at least when trading is open, you can rebalance, as I say, your portfolio. Algorithmic trading, day trading, absolute typical thing, right? This is maybe by the minute where you need to make new decisions with regard to entry-exit, with regard to position sizing, with regard to what to buy, what to sell. Then we have dynamic hedging. This is my background. So in my PhD thesis, you will find the expression dynamic hatching, right? This is like the hatching of options in a way that the position of the hatch and the option is riskless over time. This is something that cannot be implemented in a static fashion. There is for typical situations, I'm not saying for all, but for typical situations, there is no static solution to the hatching problem of options because they simply change their values, There are creaks, if you're a little bit in the option theory, right? This needs to be attacked dynamically. Optimal execution of large trades. It's not the case that the big institutions like we small traders, when we say, well, I want to buy like 100 Apple shares, that's no big deal. But once you have a block which is worth maybe a billion US dollar in Apple shares, then you simply cannot go to the market and sell it at once. This would impact the market in an adverse fashion with regard to your goals, probably. Therefore, you would execute it over the course of two weeks, for example. This is where investment banks make a lot of money. Or, again, in the discussion in the US, setting rates over time. When to set rates, how to set rates, to lower interest rates by the central bank, etc. Risk allocation over time, liquidity management of corporations, financing decisions. And we can go on and on and on. And now I want to close the circle. This explains why reinforcement learning is such a valuable method for financial decision making. Because reinforcement learning here in our case, we will focus exclusively on deep Q learning, numerically generates approximate optimal solutions to the dynamic decision problems. And why is this required? Yeah, because for most of the interesting dynamic decision problems, there are no closed-form optimal solutions, right? This is something that is only available or can be achieved in a theoretical manner, but usually not in a closed-form solution. And this also explains why static models are so popular, because for many of these, you can simply write down an optimal closed-form solution, non-numerical methods required. But here, with regard to complex dynamic decision problems, you usually need a numerical method to approximate the optimal solution. And of course, we can imagine that for basically all of these examples and many more, there is usually quite a lot at risk, like billions and billions of dollars, if not trillions. We think of, for example, setting central bank rates over time. So this is the reason why we want to apply this. So what is now dynamic programming? And this all sounds abstract. There's no need to really internalize what is mathematically written down here on the slide. It's just a summary in a rather theoretical fashion of what a typical problem looks like. And when we later on see the real world examples, everything that is here will be much, much clearer. But just a quick run through, right? We have here a tuple, which defines our problem, the finite horizon Markovian dynamic programming problem, right? A tuple with SAT, RF, and a few more subscripts, etc. Basically, what we talk about here is a state space of the problem. This can be, for example, the price of a stock, of a share, right? Then A is the action space of the problem. This can be, in a simple case, buy or sell that share. t, a positive integer, it's a horizon, this might indeed be one year, for example, but we have 252 trading days over a year, and then we have rt, ft, and the feasible correspondence here, where we have the period t reward, given a certain action, the transition functions, how we move from one state to the next, and we have the action state correspondence, which says, given a certain state, we have these options away. This can be as simple as up or down, going long or short, but this can be as complex as, give me a re-evalued number, a floating point number, which represents a position in a stock. Pretty abstract, but this fits pretty much many, many, many, many problems in finance and in other domains as well. And usually the objective is to choose now a plan for taking actions over each point in time in order to maximize the sum of the per period reward. So think again in terms of share trading. If you can buy the stock, you might be able to short the stock. And you do this over one year. And by going long and going short the stock, you want to maximize your return, your profit. As I was saying before, for some problems, we might be able to figure out an optimal solution, but for many problems, that's simply not possible. In other words, if it's not possible, then we might need to resort to numerical methods in order to approximate the optimal solution, because the problems are simply too complex. We can always try to apply proof for it, but if our action space is larger than the number of atoms in the universe, we know that we easily run then into issues, right? So now to break it down to something more tangible, let me put it that way, a very good background book, which is not as theoretical as I was showing it before, but still it has also the theory behind everything is that book. This is still, I would say, the standard work by Sutton Bartow, Reinforcement Learning. There are a few financial applications or mentions of application areas, but basically not really about finance, the whole thing. But most of you might have seen that book as well. It has in the second edition also the early achievements with regard to deep mind, cracking, Atari games, et cetera, but not that much more afterwards, right? So it's around mid of the previous decade when this book was published. So reinforcement learning is about learning from interaction how to behave in order to achieve a goal. The reinforcement learning agent and its environment interact over a sequence of discrete steps. And this is something else than what we see here mathematically speaking, right? Here we would apply some optimization method and would try to really calculate to the 10th digit, if necessary, what is an optimal action given a certain state. And from that, we would derive our optimal policy. This is not meant here. So here we will have an agent which interacts with an environment, getting feedback and learning over time what is good or maybe even best, given a certain state in terms of the action to be taken. A little bit of background. So for those of you, maybe like myself, who wants to read a little bit left and right, I think these two books are pretty good companions with regard to what learning through interaction means. The one by Susan Schneider, The Signs of Consequences, and what consequences, we can think of it like rewards in the context of reinforcement learning, mean to not only human beings, but also to animals, etc. On the one hand. On the other hand, here the book by Shadia Pearl, like the pope, so to say, of causal science, if you like, the new science of cause and effect. here is the subtitle where he clearly distinguishes between three levels right usually what we say and i think everybody's got started with that like ordinarily stress regression you have your independent variables your dependent ones so you have your features and your labels when you do supervised learning right this is like figuring out some correlation maybe there is causation but we cannot know then the next step would be inter action with the environment like do something getting feedback learning from that and this is where i would sort in here reinforcement learning and on the third layer here of the new science of course in effect are like the traditional what if analysis that you have a conscious being who thinks things through without interacting let's say with the environment the hypotheticals that is not something i guess what reinforcement learning at this stage can achieve but beyond the first level it can achieve the second level interacting with an environment so the environment defines usually the problem at hand and this is I guess what my book here provides basically for the different problem areas that are discussed namely environments with which an agent can interact right the focus is not really on the agents the agents are pretty simple and straightforward nothing sophisticated is used in this context but to model the environment That's usually the major thing here to get started with. Of course, it's always a back and forth, but the environment here is the first thing. Then you need to have a state of environment. This can be for financial market as simple as the current value of a share price, maybe the interest rate, whatever it might be. Usually in finance, we have at least a couple of candidates that we can include there. In other contexts, it might not be obvious. Of course, when you apply reinforcement learning or like DeepMind did it to chess and Go, etc. It's clear when you have the board, maybe the whole board, the board position and the history might be what is relevant or in the context of Atari, it might be all the pixels on the screen. Then you have the agent that does something. So we have these two interacting with each other. We model the environment. In our case, it will be the financial part. And then we have the agent, which is able to interact and to learn from the feedback that it gets from the environment. And the action can be, again, as simple as going long or short. So you might have a binary problem or it might be optimal control problem where you need to select from an infinitely large action space a certain number. This is then usually, of course, an approximation. A step, yeah, a step is simply something where the whole thing is moved forward, and this is now something that is usually in a traditional environment, in a game environment, et cetera, is pretty simple, at least when we see it unfolding. But in the real world, of course, when we want to model financial markets, it might be something that we are not even be aware of how this all works, right? If you want to model, for example, the share price evolution of the Apple share, let's say, if we would know this exactly how this works, we could make billions without a reinforcement learning agent, right? So therefore here we have, generally speaking, we would differentiate between two different transition functions that have been on the screen before. The one might be deterministic where everything is clear, right? You do X and then Y happens, or you do Y and then Z happens. But we generally have, for the interesting cases, we have stochastic transition functions where an action of an agent just influences the distribution of the possible outcomes. This is, of course, a much more complex problem mathematically speaking. This is the real hard part to solve. With a deterministic function, there are often solutions that you can derive, but for a stochastic, A, the mathematics is much more involved in the cases for which you can come up with optimal solutions. The space of these problems is pretty small. Therefore, here, a step sounds like a very small thing, but a transition function really is, in the end, what makes the whole thing interesting, makes it solvable or even not, right? So these are the different situations. Reward, of course, there needs to be some feedback. And this is usually called reward or penalty, right? In finance, it's often pretty simple. It's a profit or loss, right? But it can be much more complex, right? I'm not saying that's the only thing, but when we speak of reward and penalty, we can think in a simple manner of profit and loss. For a computer game, it might be points or we lose a life or whatever. And the target, the objective, usually I would say it's also clear, not always, but for example, when I'm an investor or a day trader or whatever, I want to make the maximum profit, maybe at a reasonable risk level, but that's then already it, right? The policy then defines which action an agent takes given a certain state of the environment, right? This is the mapping, what we have seen before. And in a computer game, this might mean I move it to the left, I move my character to the right, right? Or a trading board that observes something in the markets as well, I go long or I go short, or I stay out of the market, I don't know really what to do, right? And an episode would be a full run through whatever it is, full game or the full data set in finance that we want to go through in terms of real or synthetic data or simulated data as another alternative, right? So now deep Q learning to the, I would say, from the beginning, pretty successful algorithm that we can also, I would say, successfully apply to multiple different dynamic problems in finance. And here a little bit more formalism we can get by in the code situation without that much formalism. The reward function R assigns to each state action pair a numerical reward. so we have a state and then the agent takes an action and the agent receives a reward all right so it's like saying good or bad or we all know so many instances where this is absolutely clear thing then the action policy itself is looking similar but it's not exactly the same right here in this context we have again s and a mapped into r but what we have here and this is now for deep Q-learning, the major part that we should remember and internalize. We have an immediate reward and we have a future reward, which is discounted. So we human beings, we live more or less in the same manner. When I eat too much, currently it might taste wonderful, but later on I might get sick or I might put on some weight so we also need to balance our immediate rewards with the future ones or I was talking to a few people during the break right I said well I went out a little bit long yesterday so I'm not that fit maybe at this time of the day so current reward and future delayed reward and why only here the combination of t and t plus one yeah this is then where dynamic programming and the Bellman equations and principle kick in, when we combine these one-step procedures with each other over time, let's say one year trading, 252 days, then we can make sure, given the appropriate modeling and implementation, that the overall policy and solution will be optimal. So that's, quote-unquote, the trick. I more or less need only to have a look at today and tomorrow, but tomorrow takes into account the day after tomorrow and so forth. So all points and time will be connected in this context via this procedure. There are other examples in finance which are not related to reinforcement learning at all. For example, for those who have a finance background, binomial option pricing, where you do the pricing over a tree. basically in the same way step by step and for every step you make sure that you are optimal for this particular step and taking care of every single step in the end ensures that your whole solution is optimal as well so in most interesting decision problems and not only in finance we need a representation an approximation of the optimal policy so the parent example for something that it can do in closed form is tic-tac-toe right pretty simple game not too many options you can write it down in a table what is optimal given the current board position easy to write down but in realistic or more realistic cases think of chess we couldn't write down for every single board position every possible move with certain outcomes rewards right otherwise chess would have been solved quite a while ago and for the time being it will take a little longer because here again the number of board positions is larger than the number of atoms in the universe right so approximation capabilities of neural networks is where here the DQL comes in right the deep Q learning where we will use deep neural networks and in our case here it will be pretty simple what we use so nothing sophisticated not even LSTM or anything like that just simple deep neural networks, or dense neural networks, if you want to call them in that case, are used. Just as a reference, why they are so good, many, many mathematical results are available with regard to the approximation capabilities. So in the implementation, we have the two different modes. The one is exploration, the other is exploitation. For those of you who use reinforcement learning, this should all be pretty clear. Exploration simply means that random actions are chosen and even from random actions you can still learn, the agent can learn and exploitation means that the agent will exploit what it has learned before so it will rely on the then at this point in time current optimal policy or either random action or something that comes out of historical experience and replay is simply the training, the updating of the model. I like to say, well, when you lie down in bed at night, you might think through the day, you might say, well, this went well, this didn't went well, and you update your own policy, right? In that state, I took that action, maybe next time I take another one, right? And a few technical parameters, and then we are good to go. Gamma, just the discount factor, epsilon is the ratio with which the algorithm relies on exploration As compared to exploitation and then we have epsilon decay because usually we would start with an epsilon at 1. We would decay this to a minimum level, let's say of 0.1 or whatever you find appropriate. So now to the applications. But if there are questions right now, I think we agreed on taking them over the system, over the app. If there are some, if there are none. so far and everybody has access to the code and to gist and no I see only a couple of thumbs up yeah it's just waiting for the I could also repeat it Yeah, the question was whether there is any particular reason why I'm just staying with, let me put it that way, very simple network architectures. Yeah, this just like the purpose of the book was to connect reinforcement learning with the different environments, simply speaking. So for me, it was not a goal to have like the best results, the best scores, whatever you want to measure there, or the highest profit in our financial context, just to show, For example, here when you see the first two chapters, here I introduce deep Q learning on the basis of a simple game, the classic card pool game, and I apply the same agent to the financial problem in the next case. The simple network, which is able to learn card pool, is able to master the prediction game in finance as well. But beyond that, what I was saying before, I stick to the simple network architecture, with which I already get quite good results but I'm not implying, not saying that it could improve the architecture tremendously in terms of the results that we get by changing to convolutional neural networks to sequential ones like LSTM to whatever or make it much more complex so that's the only reason, to keep it as simple as possible at this stage yeah no not in this part this is of course oh yeah yeah the question was the question was whether i model other market participants as well so something like a strategic game right who who might be first who might react with what right this of course is also a very important discipline in finance but here more or less is what i assume a small agent as we would say an economic theory right which has no impact generally speaking on the market so it doesn't interact really as let's say a big financial institution against another financial institution the only exception here and this is the final chapter this is a little bit out of scope you might even say is the one with regard to the large proc trade, where it's also not assumed that one agent interacts with another, but that the actions of one agent have impact on the market, price impact. And they're with transaction cost impact. But a strategic game is not a model. So I stick to the classical disciplines of simple prediction trading, of dynamic hedging, of asset allocation, what are the classical disciplines without any strategic element in this context. You have one more? I'm just really trying to understand. So there is this horizon that you define as, from what I understand, like a time step of actions, right? And I don't know if it's an assumption, and I don't know if it's my understanding that it's wrong, but it seems like you have one time step and one action, right? How well does this translate to stocks and trading? Sure, I think it translates pretty well. I'm not saying that everybody in this world is trading only, let's say, end of day or opening of the market. But people sitting in front of their screens right in the morning, they watch what was happening in Asia overnight, What's happening in Europe, when they are US-based, for example, and then they decide what to do during the course of the day. So when you start at the beginning of the year and simplicity, let's assume one year, you usually would have, that's an average number, 252 trading days. And at every trading day, if you are trading, again, to simplify things, you would only be trading Apple shares or Tesla shares or whatever, you name it. And every day you would say, well, is this now a good day for Apple shares or a bad one probably, given recent tariffs that were announced, et cetera. And then you would go long, short, or you would make smaller your position, right? You would sell maybe 50% of your position. You would go short. You would hedge it and so forth. So that, I think, is a pretty reasonable thing. But I'm not saying that there are so many people out there that do intraday trading and others that like the citadels in the world, the market makers, etc. They trade in a millisecond or even microsecond range. This is adjustable. You can replace if you like more like you can say end of day trading is replaced by one minute bar based trading. so you can simply take one day and you have even more than 250 trading actions in our in our expression yeah but this is flexible it can be longer can be shorter you can trade more often less often so in algorithmic trading of course we start usually with the end of day story and then we move intraday and some people want to trade below one minute bars so what you can do as a retail trader usually it's not below one second usually i'm saying so but generally speaking yeah

Speaker 2 [42:00]

Let me interrupt shortly, so we have two options. Either Yves repeats the questions, if it's not annoying for him, or pose your questions on Slido, choose the Ferrum room, so I can read them and we have it for the recording. I might have missed an answer earlier. There is actually one more on Slido, if I may. Are you aware of actual production applications of RL models in, my screen just blacked out, big HFT shops? It seems a lot of them still leverage different parametrizations of linear models.

Speaker 1 [42:30]

Yeah, and this has good reasons, I would say, right? HFT, I always like to say, is a completely different game, right? They are also not the ones that are too eager to deploy the latest and greatest in AI, right? When you are trading on the sub-millisecond level, right, you have completely different requirements as compared to, let's say, a typical SIS trader, prop trader trading maybe intraday. but on a completely different scale. And the algorithms, most of the algorithms that the HFT shops use, like market-making algorithms, they have nothing to do, let's say, with prediction or with more fancy stuff. These are more or less deterministic algorithms. And in that sense, something like reinforcement learning can and probably is applied to some extent. But the execution with regard to linear models usually is the fastest that it can come up with. and which gives you what you need, namely the speed advantage, right? They invest millions and millions and millions in the hardware infrastructure and the co-location to all the exchanges to save the other microsecond and yet another microsecond. Therefore, the models need to be as fast as the task requires in this context.

Speaker 2 [43:49]

There's one more question, so it's regarding a formula you showed if I can get it was a different slide in The formula our alpha me why is it delayed reward? Why is delayed reward limited to step t plus one?

Speaker 1 [44:00]

And this is what I tried to convey, that Weierle-Bellmann principle, when you now step forward, what is t plus 1? t plus 1 becomes t, and then you have, so to say, t plus 2 becomes t plus 1, so you will have a chain of these elements here, Therefore, it is all indexed, right? I'm not like saying this is just this one shot. So, we will, of course, iterate over all the Ts possible. This can be in the example from before something starting at zero today towards the end of the year 252, right? The last trading day. But we will connect every single step with each other. And in the end, we have a chain, right? It's like a single piece of a long chain which will be connected via what is called the Berman Principle, which then ensures optimality in this context. And again, there are other examples where the same principle is used in finance, like Banomi option trees and a few more.

Speaker 2 [45:07]

There are a couple of more questions. We can take them now or later. What do you prefer?

Speaker 1 [45:11]

I can answer as many questions if you have time, but I need to check with regard to the code, but we can focus of course

Speaker 2 [45:20]

We can also do a couple more and then break. You continue and do another round, whatever you prefer. So one question, do you have best practices to choose good rewards?

Speaker 1 [45:30]

This is what people then usually say, and I like to say that as well. This is usually more an art than science, right? So practices go towards the direction of art. But I would say it always depends on how you come up with your financial environment. So what it can derive from that, A. And B, what the goal of the whole operation is, right? You can even work with a single financial environment, but the goal of the agent might be different, right? It could be something like what I have prepared, I want to show dynamic hatching in the Plex Colts model, or it could be a trading strategy. In both cases, you would have completely different requirements with regard to the reward. In one case, you would rather work with P&L, profit and loss. In the other case, it is the penalty only for deviations above or below a certain fixed target. Meaning in the one case you might have maximized the profit, in the other cases minimize the deviation from a certain target. These are completely different requirements, but still, even given these two base cases, you still have quite a bit of room with regard to how you model your reward. For example, you might add other penalties with regard to large deviations, with regard to your positions. your positions just think of it when we say well i have a hundred percent put for you and now i want to invest stick to my apple share example 50 of that in apple shares or 25 or whatever right for the first investment that should be okay but later on it will be costly if there is a large turnover right i said yeah now 50 next day 25 just 75 so all these trades they will incurred transaction costs and this is something that you would usually penalize so you would add another term which penalizes like the deviation from one to the other beyond your original goal right so again this is more an art than science it's not as simple as straightforward therefore pretty good question pretty good question but not answerable from my point of view at least in any generality need to focus on it

Speaker 2 [47:44]

I think it might be a good point to turn on and to revisit the questions later. Just post them. I'll monitor them on Slido.

Speaker 1 [47:50]

to them on Slido. Yeah, post them and then we will pick them up. All right. So getting back with my classes so that I recognize. So the first Jupyter Notebook is about the basics. I think this is pretty simple and straightforward, but nevertheless, I find it so instructive, right? In the book, of course, there's a lot of text written around this. But in the book, I wanted to start really with a very simple example of how to learn through interaction. And I have executed now all my Jupyter notebooks because some of them would run for minutes on end until we get the results, not that one. So maybe I can simply move through that one, which is a pretty fast one. So tossing a biased coin. We get started here with a reference case of an unbiased coin. And we have here the state space and the action space. We had these abstract definitions, but now a state space is a list object with a one and zero. And the action space is also a list object with a one and a zero. And what do we do now? Doing one epoch. So we play a hundred times and the agent chooses randomly from the action space. And the machine, the game engine chooses a state. Whatever the action that is chosen equals the state, then we say here the TR, which is the total reward, is increased by one. Think of a typical game or think, if you prefer that, of the game of roulette, where we negate the zero, the green zero, and we say we are just playing black or red, 50-50 chance. therefore the sequence first you need to bet and then the state is realized right so when we do this here a bunch of times we see here results maybe 56 by randomly betting 47 so everything will be around 50 around 50 means on average law of large numbers we will get here a reward of 50 so this implies that we don't need to pay to gamble here right otherwise we could say well if i need to bet one here i would yeah break even maybe at best in the long run so now my biased coin now in ssp you see like four ones and one zero this means that the one when i randomly choose from uh here the state space is four times as likely as the zero. I would say this is pretty biased, right? But the rest stays the same. So I repeat the whole exercise and it shouldn't come as a big surprise. Or maybe it comes, right? When you see here, have a look at this. What do we get? Yet, that's kind of like, maybe it is surprising after all. Why is it surprising? I could now dive into, I do this later, into conditional probabilities, etc. But let us go through the examples and then in the end I think everything will be clear. We get here with the random betting method. we get like the same the same mean reward total reward than before so why is it yes simply speaking whenever I bet on one I have a 80% likelihood that I win okay so this is what we need to take into account so the expected value stays the same as we see here right so the next one is the checks that the same here this looks like no this is not the same and because there's one single line that I overlooked by glancing at it just briefly there's one line added to the whole thing the rest is exactly the same and the one thing that is added here is this particular line right and this appends the outcome of that game to the right action space so the action space started as before but we now add what we have observed all right so what does it mean that now when I randomly sample from the action space, over time I will realize, oh well, this number one here is much more likely than number zero. But still, still I'm randomly choosing from the action space. Nothing else has changed. So what this is called, this is called probability matching. And with probability matching, so I have like many, many more ones than I have zeros, right? I therefore choose more 1s and with a higher probability for winning with 1s, I increase my average return here. This is the 66.78 that we see. Of course, everything what we do here, when we repeat this, we get slightly different results. But overall, the averages will be relatively close together. The law of large numbers, as the name might already suggest, just lives off very large numbers and here this is all but very large in a mathematical sense but still we see what is happening. So now to something that is much better. I use here just counter for deriving what is the most popular outcome so to say. Same as before four times as likely biased coin I do the following see here is counter asset and I just get the most common one for my action right and the rest now stays the same I still append here the outcome of that round because it is observed that is common knowledge repeating the same as before so what do we get now we get an even higher average so from our random with a biased coin from 50 to what is called probability matching 66 to now choosing the most likely option here in our case it's one close to 80 so 80 would be the theoretically optimal outcome but here of course numerical derivation so what What was happening? Probability matching is better than what we have seen before, but it is not the optimal solution here, right? But you will see it, and there is interesting literature when I was writing up this and doing a little bit of research, there is an interesting literature that really, yeah, researches the behavior not only of human beings, but also of animals. And both in human beings as well as animals, you will see through, there are also arguments with regard to evolution over time in different populations, why probability matching might have paid off in the past, and why, for example, several bird species, I don't recall which ones, typically use probability matching in certain instances, etc. Although, given Bayesian reasoning, which I come to in a second, is the better option. And human beings use it often as well. This has been examined in lab setups, as it seems like hundreds of times right so people don't easily go with the more likely option in that case people still play probability matching and there might also be good reasons with regard to poker and strategies the question worth with regard to interaction there might be other reasons but here in this simple setup for sure what we see that the going with the most common here technically speaking over the most likely option is the best one that we have. The same holds true and I can run through this with a biased die. Here 4 is the most likely outcome here instead of just an unbiased die. Rolling dice here we get 100 with our random policy when we learn through interaction we observe and learn. We get a quite a bit of a higher mean instead of 100, 180, a significant improvement of 80% roughly, meaning here numerically. And last but not least, what we should do here when we go with the most likely option, as we learn over time here when we bet throughout on 4, this is the bias that we have, we get 300. So again, a significant improvement. And this can be calculated and demonstrated in both cases pretty straightforwardly, that this is what is called Bayesian updating, right? This is Bayesian approach that we say, well, after a certain point, we have a prior probability distribution, we have a posterior probability distribution. And of course, once we know what is the most likely one we should go with it. But once again, probability matching is something that is observed in reality as well. So this is learning through interaction. I interact with a game engine or with an environment and then I'm able to update my beliefs or my action policy, the optimal policy. So now apply to reinforcement learning for finance. You can go once again also to the PyTorch version on Google Colab. Here I'm sticking to the book version. The finance environment is pretty simple. We have an action space. As I've pointed out before, that can be as simple as what we see here, 0 or 1. Exactly the same like with our betting on the coin toss. So zero or one, nothing else. This can be long or short. This can be interpreted as up or down. And now we have a finance environment. I retrieve here end-of-day historical data, which has for a number of symbols, has for 10 years end-of-day data. So we were talking about one year, 252 trading days. So, roughly, this has 2,500 observations for every single instrument, like the S&P 500, like the VIX, like Apple, Microsoft, and a few more included in here. So, for the environment, we set the symbol feature, the number of features, action space, minimum accuracy, and we are good to go. This is pretty simple. We just choose, basically, which symbol we want to deal with. Let us stick to Apple Share, right, and the number of features here in a simple case is just the number of historical observations, like 5 days back, 10 days back, 15 days back. So we prepare the data, we need some log returns. Here for the prediction game, we work with the directional movement, whether it goes up or down. And this will be what the agent is supposed to figure out. It plays the game, I don't know, 100 times, 10,000 times, and for every single step, this was the question from before, the agent now has the task to predict whether the stock, given what the agent has observed in the past, given the features that we provide the agent with, whether the stock will go up or down. That's the reason why we call it a prediction game. The agent is supposed to predict what is happening on the next day. And once this is done, we move forward. And once that is done, we move even more forward. So, reset should be self-explaining, I would say. We can always reset the environment to the very start, like on the chessboard. We get to the initial starting position. So, the step here is, I was saying this before, often in finance or usually in finance, this is a pretty, pretty involved part. But here I keep it to the most simple situation that we have a historical time series and we simply move the time window deterministically step by step forward, right? So whatever I argued with regard to the complexity, here it's the most simple case. We have something deterministic and we move forward deterministically, right? And the agent gets a reward if the prediction is correct. So plus one, and if the prediction is wrong, it gets nothing. So this is in line, for example, with the card pool game that I mentioned before. There's just a little twist that is here included, so that the agent is not running always like in absolute bad territories in this regard, right? When there is a minimum accuracy that we have defined, and the agent's accuracy after a crazed period of a number of steps falls below the minimum accuracy, then we stop the game early on. So this is simply to prevent the agent to always go like the wrong path, so to say, or to come up with too many wrong predictions. So you see here, I mentioned already a few of the symbols from before. It's Apple, Microsoft, Intel, Amazon, Goldman Sachs, S&P 500, VIX, SPY ETF, Euro, Yes Dollar, Gold Quote, and GDX, GLD ETFs. So here I go with the Euro, Yes Dollar, which is so important. We have seen a relatively large movement over the past weeks, basically, since the inauguration of Donald Trump, where it all got started. right so we reset the environment right and what we get here are four lags and now we need to recognize speaking of the ures dollar exchange rate that these are normalized values so we cannot directly relate this to current rate slightly above 1.1 so this is all normalized i can sample here either one or zero right when i step it forward with a random action i get back the new state and here the reward so this was wrong and yeah these are the technical outputs with regard to whether we are done or not whether we can move on so starting all over finance environment but But this time with the lagged return. So before I chose the price itself, normalized, now the return. The locked return normalized, here you see, well, easily we are here in negative territory. Of course, we might see negative returns, but not ones of minus 119%. So all normalized, this is what we need to keep in the back of our head. So one typical benchmark in finance in this context is a random agent. So can we at least beat somebody who is tossing a coin or the monkey who is throwing darts on the share board, right? And this does nothing else than taking random actions, right? And you see here with regard to the prediction game, the agent more or less fails pretty early throughout on average the number of rewards is um here 15.83 the maximum reward is shown here it would be 2607 right so for euro yes dollar we have a few more trading days than 250 two per a year therefore here it's a little bit higher than 2500 right so this is pretty bad so random agent doesn't do well right our tql agent let's have a look now with the intelligence hopefully embedded and you see here the brain the approximation our representation is a pretty simple there was a question with regard to the model pretty straightforward I think it can't be more plain vanilla in this context so this is not a trick or a specific architecture or whatever chosen to trick the whole game here now this is plain vanilla we have a couple of parameters that we have discussed before like epsilon epsilon decay the memory here is set to 2000 and so forth gamma is set to a relatively small value i will comment on that in a second the agent either acts randomly or according to the policy this is the exploitation versus exploration elements so exploration here the random part exploitation the one that is based on the brain here on the neural network during replay the neural network is updated given past experiences like what was happening in the past we chose a batch of 32 i think is a number that is fixed here in init and yeah so we go and interact we learn we collect memories past experiences we update the neural network and all of this is basically done here in the learn method where we go over a certain number of episodes so this is like playing the game of jazz is that well let's play today a hundred games or 50 games of blitz for example and here we say well let's play the prediction game a certain number of times when we collect experiences here with a deque object we We will only retain the past, the most recent, 2000, I should rather say, right? And if we are done, meaning we have failed or reached the end, we get a certain output. And last but not least, the test method, which only relies here on exploitation, right? So this is like when you learn for a math exam, right? You might try around and do your best here and there. But when the test comes, when the exam comes, you just can rely on what you have learned and cannot really play around anymore. So that's the test function. And when we run this, for example, 252 times, as you see here, that's for sure not that much. It's for 18 seconds. Again, here for illustration purposes, for sure not as investment advice. This is like in the gist, for example, pointed out. I might emphasize that here as well, given also with regard to the question of what is used in the real world, in the wild out there, right? And, yeah, you see that it gets to an accuracy after the training of 52.5%. This might not seem like that much, but maybe some of you have read the book, The Man Who Solved the Market. There's a story about renaissance technologies, the most successful hedge fund to have ever existed and successful in relative terms, at least if not in absolute terms, meaning in billions, they have had in the past hit ratio, in that sense, an accuracy with regard to their trades of well less than 52.5%, only slightly above 50%. So that's one measure, one statistical measure that is important, but it's by far not the only one if you want to trade on it. Right here for a just prediction game as a starting point, a statistical exercise, if you like, this is, you can even argue, the only measure that is relevant. But if you want to make money in the financial markets, there is so much that comes afterward that is not modeled here that it's hard to judge whether this is now a high number or whether it's a relatively low number. Of course, we would like to see something above 50%, because otherwise, again, we would be close to the tossing of a coin. But still, it's not that straightforward like in many other machine learning applications, deep learning applications, or even then reinforcement learning applications. what is good or bad just to mention one point if you want to trade long or short for example based on the prediction game it is of paramount importance whether you get the large movements in the markets correct or only let's say the small movements this relates once again to fat tails and high peaks in the markets right it's not the case usually how often you are right it's the case if you are right with regard to the large movements in the markets that's the big deal but this is a little bit too far but belongs here to that story so you have seen here one extreme case where I've just worked with a single time series 2000 in that particular one 603 observational points that's for sure not that much and if you would go and would tweak the model I would tweak the whole setup we could make the agent perfect, right? Increase the capacity of the neural network, etc. Then it would play that game with 100%. Therefore, I don't want to present this case because it's from my point of view not interesting. It's just here, then in the end, an in-sample exercise which says nothing with regard to any performance out of sample. But what I want to contrast is that we have here one fixed data set that we work with and this relates to my quote-unquote critique with regard to the application of reinforcement learning and finance because we have only limited data. And the way out of that is, or one way out of that is the generation of synthetic data. And this is, to be honest, not too difficult, right? We have now that much experience with generative AI, so it's impossible, I think, to exhaustively list all the applications. But here I have the one for the financial markets, where we have a very simple setup. For example, here I start with a deterministic function, which is monotonically increasing, so that we can order the synthetically generated data points, where we have the real data. Here is the red line, and we have the normalized data. It's the blue dashed line, just as an introductory example. But I will use the very same models here. Plain vanilla, straightforward, absolutely simple, just two hidden layers, one output layer here, linear function, ReLU activation, nothing specific, for the generator, for the discriminator here, and to take the two of them together here for the GAN or generative network, where you put the generator in, the discriminator in, specify a learning rate so you see this is really most simple setup I think we can come of course we can even delete one more here whatever but I would say this is still on the absolute simple side to simple deep no networks deep because we have two in layers and we have here a combination in the form of of our generative adversarial networks, where we have the competitors, let me put it that way, the generator and the discriminator, and put together, hopefully, that we'll then be able to come up with something that we can work with. So we create the three of them, the three models. Here I train them, so the training is also simple and straightforward. some noisy data generate here based on the current state meaning with the current weights to generator I simply generate output right I sample real data here right for my original data set a sample here the real data of the same size and then I train the discriminator on the real data here with one so that's the real data right binary classification and here on the synthetic data that's the that's the synthetic one with the zeros right ones and zeros original fake counterfeit data right and so we go on and train here last but not least the GAN where during this step you might have noticed here the discriminator is not trained it's just the generator that is trained during that step so so far so good once we put this all together and we get here from our train models function we get the final real data and the synthetic data right this is trained here for two minutes simply sorry simply plotting the data I would say well might be the same might not be the same I don't have a clue there is another way of having a look at the data and this is here wire statistics right so the real data here has a mean of zero given the function the cubic function that we have chosen also the normalized values have a mean of zero and here the first five synthetic ones are not really close to zero standard deviation okay but not exactly the same so we see the statistics are not that good we can now go and sort this and then plot it here and that's the reason why i use the monotonically increasing one so that i can do the sorting and compare it yeah it is not too bad but it's also not really good here right we see here higher peaks and lower lows right so we could now go on and train this longer and more and more and fine-tune and would come up with it so just as a deterministic introduction and the very same approach this is really now straightforward i won't change the code for the for the gun at all the very same approach i can apply now to historical returns data so i don't want to generate price data as often as the case we need we should work with returns data and this is what I want to do here. So I work in this case with a GLD which is a gold ETF and calculate the lock returns and simply transform this here into an umpire and the array object do a normalization here standard Gaussian normalization and you see here no code changes whatsoever to what we have seen before right training the models now on historical returns data normalized right adding this here into the data frame and adding 25 synthetically generated time series when we have a look now here at the statistics the real financial data has of course a pretty small return we are talking about daily returns and of course they are are normalized to zero right so this is all what we need to take into account in other words mean is already close to zero and after normalization it should be overall exactly zero but we do a random sampling therefore it is not exactly zero synthetic data slightly above slightly below zero standard deviation looks already pretty good right the minimum not really here it it looks pretty good. Then we have the maximum, also not exactly, but here that one hits it pretty well. And when we have just this fever curve, I like to call it simply plot the data as a line plot, I would say that's already pretty hard to distinguish. Of course, I have the legend which says what is real and what is synthetic, but we don't really see a completely different behavior with regard to maximum values and minimum values or whatnot, right? So that's, at least for my eyes, not to distinguish. So I have another plot here. For those of you in finance, you're probably used to the returns curves, right, what you see as histograms. And you see the blue one, the blue histogram, which is just one chosen out of the ones that we have generated is a pretty good overlap to the real data. We have a little bit of a higher peak in the 1K, a little bit lower peak in the other ones, left and right tails more or less are also a good fit. I wouldn't say this is already some proof, but it doesn't look too bad. I could show you other charts where the blue one has a completely different shape than the red one here. So this is now also for just one, the first one generated here our sorted plot where you see on the left hand side looks pretty good here so this is not too bad. So the real data here is higher peaks than that one but we have seen for other synthetically generated time series it's not too bad. So that one I think speaks already a nice language. The red one is the original one on which we have trained the model. And then we see all the synthetically generated one. And for those of you, finance, who have worked, for example, with simulated data. Let's say you simulate where I'm on the color simulation, geometric bounding motion. Let's say these charts look pretty similar. Here we have, I would say, more or less by chance, we have a slightly negative mean trend here. But overall, this nicely captures the behavior. In other words, in a simplified manner, we could say, well, this red original line lies in the space of what is spanned by the synthetic data. This is by no means a guarantee that the data is good, but it's at least a first hint at this particular feature that we would like to have. Now coming to a proper test, Kolmogorov-Smirnov test. Who knows what this test is about? Quite a few, four or five. Basically speaking, when you have two samples of data, this test tells you, I mean, we know this already, but you can test whether, from a statistical point of view, the two samples should be from or seem to be let me put it that way seem to be from two different distributions or are more likely to be from the same distribution. Of course we don't know exactly what this is what I pointed out before what the engine is that generates real world data. We don't know it as an object or as an algorithm or what not. But we can nevertheless compare something that we have synthetically generated with the original one right and to this end we have this Kolmogorov-Smirnov test and here for this run here I think as I was pointing out before what I'm not wrong here this should be the exact same stuff that is in the book and in the original notebooks right we can import this from SciPy here from the stats sub package and we do here the KS test Kolmogorov Smirnov test right and we compare every single synthetically generated time series or returns series to be more specific to the real one And in every case we get a p-value, which is related to the question that I tried to point out before, where we said well is it distinguishable significantly or is it indistinguishable from the original data set and we see here in that particular case the p-values are all above 5% or 0.05 so with statistical significance here you see it visualized as a bar chart right with statistics statistical significance we can say that all the 25 synthetically generated return series are statistically indistinguishable according to the ks test from the original time series and this is what we would like to have for this type of synthetically generated data set That we cannot say, cannot differentiate between the one or the other. She's standing up and looks at me.

Speaker 2 [82:26]

It's fine for you

Speaker 1 [82:27]

Oh, yeah, it's already the time. You see I've planned for two

Speaker 2 [82:32]

Might I ask a couple of questions regarding to the Notebook that you just showed? I think there were two specific questions to that. Sure. Maybe. So one specific question said, aren't the markets going up on more days than going down and you could reach 52% accuracy easily by always predicting up?

Speaker 1 [82:53]

Whether the markets go more up or down.

Speaker 2 [82:56]

The question says aren't the markets going up on more days than going down and you could easily reach We could reach 52 percent accuracy easily by always predicting up That's a question

Speaker 1 [83:10]

yeah this is something yeah this can happen yes yes but again i was saying before this is then not what uh actually you would satisfy i have had this to be honest not with reinforcement learning but you can easily get into a situation when you train a standard deep neural network you do your classical uh supervised approach where you take just like let's say here in our case it was four features and the labels you have I don't know 2600 or whatnot samples that you train it on and the deep neural network can easily learn here that in this particular case just predicting ups and downs indeed this might be then a good trade just to go along with regard to the statistics this is what I meant before this still doesn't need to be a good economic strategy because think of you have more up days than down days and when you say, well, the average up movement is 0.1% and the average down movement is 0.5%, right, this would then still be a horrible trading strategy. Therefore, again, I try to emphasize when I'm talking about statistics and profits, so this would be something in the next two notebooks where the profit dimension is added to the statistical dimension. And maybe let me put it again. Once you read the historical literature on efficient markets, for example, they always speak about to make economic profits. They don't speak about beating the markets in a statistical sense. There is much, much more that comes after the statistical game if you are in the game of making money by trading on your statistics. But incorrect.

Speaker 2 [84:59]

Yeah, thanks for the answer. There was another one, I think, explicitly to this trading example. So the question is, what is the benefit of reinforcement learning compared to simple supervised learning when the actions, as in this example, hardly has any impact on the state?

Speaker 1 [85:15]

That's true. That's something that I emphasize in the book as well that The modeling here in particular in that case and even more so in the static case, right? When I assume and there was a question in the beginning with regard to the interaction where I said well the basic setup here is That the Asian is small and doesn't have any impact on the market, right? And this is something where the analogy fails when you say well even with the very simple card pool game of the gymnasium environment where every action of the agent has a direct impact on the next state so in many cases here that we are modeling there is no such impact at all apart from again the final chapter where indeed every action of the agent has an impact on the price of the market on transaction costs and so forth so this is addressed here this particular element in the final chapter but once again let me get back one step when you have a look at the historical finance literature this is a very standard example but still the analogy to other reinforcement learning context fails here admittedly yeah

Speaker 2 [86:29]

Thanks for the answer. There's, I think, one more to this specific notebook, what you showed. Why does GAN synthetic data improve another model's prediction quality? Couldn't it decrease quality since it's supposedly worse than real data?

Speaker 1 [86:44]

I didn't understand the beginning. Why?

Speaker 2 [86:46]

So why does GAN synthetic data, I think you talked about synthetic data, improve another model's prediction quality? Couldn't it decrease quality since it's supposedly worse than real data?

Speaker 1 [86:59]

Yeah, the question is what is worse and what do you want to achieve? I mean, there is and there will only be one thing, namely the real thing, right? Of course, synthetic data cannot replace to 100% the real historical evolution or the upcoming future evolution. But it will give the agent much more experience with similar situations. right and i have one final slide where i want to point you to much more realistic modeling techniques where i say well depending on how i sample the data here for the synthetic data etc there might not be a single large drop like let's say the covet drop in the markets the covet crash right this of course is something that needs to be expected but overall when you say well i I want to like with self-driving cars, right? I just have that many roads where I'm allowed to drive with it. Therefore, I train. This is how they all got started. My engine for my autonomous vehicle in the GTA 4 game, right? Because there nothing can happen. This is the situation that I see similar here. And for other algorithms like dynamic hatching, which we now haven't covered, right? Which I would have covered next. This gives much, much more exposure for these algorithms than just going with historical data. It cannot replace the real thing to 100%, but it can improve the training of the algorithms to a very large extent.

Speaker 2 [88:42]

We have one more minute, maybe a quick answer. Just one more and I think the top voted one is Can you comment about the use of boosted tree models XGBoost instead of DL models?

Speaker 1 [88:57]

What should I say about comment provides statistics? So Usually I would say here as the name says and that contacts deep neural networks are Generally used. I don't know with regard whether XJ boost or whatever alternative we can come on I've done implementations with psychic learn as well moduli a perceptron. Of course you can Replace like the core the brain as I like to call it by other methods but Yeah, I can say, to be honest, much more in this regard.

Speaker 2 [89:30]

Thank you still for your time. We are up in time. Thank the audience for staying, for listening. Let's give another round of applause and say thank you, Yves. You're welcome. And there were some more questions on sliders, a couple of them. We didn't go through all of them. You feel free to find Yves afterwards.

Speaker 1 [89:52]

And one final word here on my last slide. I was happy to be part of the committee of Michael Stutz's master thesis three weeks ago, also in Austin. And he has taken all that stuff that I have presented to not one other level like the next 10 levels with regard to realistically modeling synthetic data as well as the agents. And if you're interested in his thesis, it will be published soon. Slides and a video recording is available from his talk on YouTube. All right, that was my final message here.