Reinforcement Learning: Bridging The Gap Between Research and Applications

Speaker 1 [00:06]

Hello. Right on time, my computer turned off. So, hi, I'm Misha. Thank you for the introduction. I wrote a model card for myself, to introduce myself. And while you're reading that, give me a quick show of hands. Who has some idea of reinforcement learning? Just to read the room. Okay, vast majority. Who has applied reinforcement learning professionally? For the YouTube audience, like four hands I counted roughly, out of, I don't know, 100, something like that. This is what I expected. I actually expected zero, so that's cool. So why am I here? I am one of the five people in this room who have applied it professionally. I worked on an industrial reinforcement learning application for a customer for the last three years, or for the last four years, something like that, and I learned some stuff. And the stuff that I learned led me to contributing to one of these libraries and actually convincing colleagues to contribute to one of these libraries and be optimistic about this future, okay? Now why on earth would somebody want to contribute to another JavaScript framework? Sorry, reinforcement learning library. I know, right? So I will convince you, I will try to convince you that there are good reasons that we can solve some of the pain that I have experienced in the past, but I will do it in a maybe somehow unconventional way. 95% of the talk will sort of tell you that you should never apply RL on like why it will be very hard, why you will suffer, why it might be only the last resort, and still I'm optimistic. Okay, so bear with me till the end. Okay, the obligatory slides. Fortunately, almost everybody in the room knows about LRL, so I will not go into detail at all. There was one, just one thing, right? So the reinforcement learning agent, or you want to learn an agent that interacts with the world, receives some observation and acts upon the observation. When you Google, which I did to get some picture like that on your first slide, what you will actually see that in the majority of the cases, it will not be observation here, but will be state, which already is part of the divide that I will be talking about, because in reinforcement learning research, which is still dominating reinforcement learning, people think about fully observed environments where the agent observes the state. However, in applications, in the absolute majority of the cases, we think about, we don't even know what the hell the state would be. We just have an observation. Almost no software is there to support this massive difference between what you want to do. All right. So why should we care at all about RL? So I went through the effort and I looked at the talks in PyCon about reinforcement learning at all, you know, at all, like from 2018, before 2018 there was none. Like, Rafa, what do you think, how many maybe people raised fingers, like at all, how many talks have there been given about reinforcement learning in PyCon in the history? somebody shows four, good estimate, three. Yeah, three, okay? Three if you're generous. The first talk was this one in 2019, which you say, why you don't see many real-world applications of RL? As you see, not much has changed. It's a great talk. I will not say anything against what the speaker says. It's actually an introduction to my talk, so I really suggest that you go and look at it. I wish you had already done it then. And then the second talk was somebody presenting their master thesis, And the third one was already reinforcement learning for human, for language models, which is really not RL. And then there are lots of blogs and lots of people complaining that it just doesn't work yet. So why am I here? So there are actually some successes, some non-toy successes, like beyond playing chess or controlling some kind of robot simulation. This left one I really like because it's super prototypical. A company has learned to navigate their stratospheric balloons. It's a stratospheric balloon company, so it's important for them to do that. And the reinforcement learning agent has found a better controller than the human-designed fine-tuned controller over decades of experience, and so on and so on. This problem cannot be solved by supervised learning. It cannot be solved in any other way than reinforcement learning, because the strategy that was found is qualitatively different from the previous one, and you simply have no data to know that such a strategy exists in a supervised approach. You cannot learn from experts because the experts are worse than you are. In fact, in the sort of non-toy but still toy examples of reinforcement learning, like Go and Chess, we are in the very interesting situations that humans learn Go and Chess strategies from RL because Arel, in two days of training, has learned more about gold than humans in the 5,000-year existence of gold, which is, you know, impressive. Then the other is plasma-controlled nuclear fusion. It was done in a research facility, so because we maybe, as you know, still really don't have viable fusion reactors. But it was a collaboration of DeepMind with the fusion guys, and that also could not have been done in a different way. Also, my project, which I cannot talk about, but it works. Maybe just the fact that I actually received money for it for three years is already at least some indication that it's not complete garbage. And also, it couldn't have been solved in another way because I'm a very pragmatic person. I would have tried everything else apart from that. I mean, I'm using PowerPoint and not very high-quality screenshots here. So I'm very pragmatic. Fun fact, I actually wanted, as one of the main successes, to show this one. Yesterday I had this slide on this because Google loudly announced in 2020 that they designed their TPUs with reinforcement learning. Very interesting approach. I was like, blown away, really cool, really design a chip. NVIDIA is probably doing something similar. But Nature issued a product warning, which I've never seen in my life before so three years after publication nature says maybe not timing was also somehow suspicious because the paper was published right before a 120 million dollar cloud deal for Google not saying anything but think of it what you want so now the crux of the talk RL is hard which is why there are so few applications It is harder than it has to be. If you take anything away from my talk today, then this is it. RL is harder than it has to be. It will still remain hard after we solved all the problems, but because it is so much harder than it has to be, it's really like a barrier to apply. It is super high. So this is how hard it will always be. The sources are in the slide. So when you look at this thing, which algorithm is better, you will say, ah, obviously it's the blue one. But, in fact, they're the same exact thing, just one has an epsilon 10 to the minus 7 as a division by zero regulator, and the other is 10 minus 5. Now, this is not the first thing you would look at debugging, okay? So that's unfortunate. Also the difference here, like the orange thing, it starts converging at 4 million timestamps, which if you have a slower environment will take very, very long. And if you're only looking at this orange curve because you've never considered adjusting your division by zero regulator, you will think that your problem is unsolvable. This is how it goes in RL. It's not that if you do something wrong and you're unlucky that you get a 5% drop in accuracy, you lose everything. You will think this cannot be done. You will move on to something else. Your boss will cut your funding. Again, fun fact, the three years project, you know, I got super lucky in the beginning. I chose the exact version of Ray and one algorithm that has worked by accident, let's find out, on my environment and produced some results. In any other minor patch version of Ray, nothing has worked with the exact same parameterization, okay? So sometimes I wake up in the night and I sweat and I think, oh, my God, what would have happened if I had chosen not ray 1.2 but ray 1.3, which arrived just one week after I started the project. So then the learning curves, even if things go well, sort of, the learning curves are really, really scary because you see the blue starts off really good and then dies. Then everything else starts later. This is not that much. It's 100,000 time steps. So it may very well be that if you go to 10 million time steps, that the lowest one will actually win. And I'm not kidding when this happens. And this is not fun. Anyway, machine learning may be not very fun, but this is especially not fun. So you will see effects of this. One of the best top-rated papers at NeurIPS 2021 was a paper called Deep Reinforcement Learning at the Edge of the Statistical Precipice, which just said, I will summarize this very quick guys just to please report proper statistics instead of tuned seats and the one hyper parameter configuration that has worked for you this is what the paper said because people did that you could not reproduce results you couldn't trust almost anything in the field and and the paper warned about that say bad things will happen if we continue like that and published a kind of evaluation recommendations and some code for how to do that and you really don't want to read the blog or like when you start reading is you don't want to read a blog that starts with 37 implementation details of whatever because I don't want to know about the 37 details that all need to be there for the things to have a chance at converging okay it's so this is the complexity that is intrinsic to RL and will probably not go away very soon unless there's a change in culture but there is complexity which does not have to be there okay so it gets worse So, first of all, you remember me in my slides. There was no outstanding framework with, like, 70,000 stars where you say, okay, this is the one I'm going to take. Instead, there is 20 of them, and I'm working on one of them. Yeah. So, we are not in a situation where you have, like, PyTorch or TensorFlow market share on reinforcement learning. There's simply no agreed-upon thing, which means everybody's doing their own stuff, which is unfortunate. The environments on which reinforcement learning is typically tested and on which the papers report the results are not representative at all for industry. This is very unfortunate because, you know, so the real research question is the following. I want to find an algorithm that works well on lots of these benchmark environments. And the application question is, I just want to solve my thing. I have this one thing. I have the stratospheric balloon. I have the chip. I want to solve it by any means necessary. This chip is super different from whatever the benchmark thing is. So now I will tie in my algorithm into the specifics of my environment to try to solve it, and I will never publish it. So this is one also big thing that makes it much harder than it has to be. So the researchers are focused on what I was saying is not the most appropriate question now. But also the appropriate questions are not published. So if you have, and this is not because of some evil, if I'm a stratospheric balloon company person and I want to solve this problem, the absolute majority of my time of solving this problem will be data modeling, like modeling the environment. I will write specific code for how to do this in a modular way. And I kid you not, there is zero assistance to how to do that. You will not find tutorials, you will not find software. The code that I have written in the last three and a half years is 80% generic, but of course I cannot publish it, I've just re-solved problems that other people have solved or maybe I solved them worse, maybe I've solved some of them better, I will never know but it's tied to the core IP of the company, they cannot open sources, they cannot open source an environment of this balloon because this is the stuff that they've been working on for the last 30 years, right so the people, even a good faith algorithm developer will not have access to these kind of important environments. They will just have access to toy stuff that people published. But even worse than that, you know, the well-known research successes are not open-sourced. If you think about it, it's unbelievable. Mu0, by DeepMind, was loudly announced as a general-purpose, powerful reinforcement learning algorithm. This thing was used to solve Atari, to solve robot learning things, to solve Go, to solve YouTube compression, to solve matrix multiplication that was actually shipped into the LLVM compiler. So this is a success. It probably, after it was shipped, it optimized thousands of years of human experience just by making some algorithm a bit faster. Nothing of that is out there. The only example I've ever seen of Mu0 in open source solves tic-tac-toe, which, honestly... This was four years ago. Five minutes after ResNet is published, we get the implementation in PyTorch. One year after Sam Altman says open source will never catch up with GPT-4, we get LAMAS-3. LAMAS-3 is massively harder than this, and this one we get. But for reinforcement learning, we get almost nothing. It does not have to be this way. There is no technological reason for it to be this way. There is only cultural reason. And I'm trying to kind of create a technological resource that will slightly shift the culture. That's why I'm optimistic. This will not go away. Or maybe it has a chance of going away if you start actually asking the right questions and working on the right environments and thinking about particular tricks for partially observe things which were crucial, for example, to solve my problem. But we, I mean, all of the other stuff, this can actually go away if we put our minds to it and if we develop the technology. So what can we do in the current situation? A researcher, and by researcher I mean an algorithm developer, can give up on modularity. They can just fork an existing thing and overwrite stuff there. They can give up on good code and just hack through some implementation. So all these things can be done separately. Usually after doing that, a PhD student, after, let's say, completing three papers and securing their career, will start writing their own RL framework. And usually then they will do all of these things simultaneously in the new RL framework. So what can a practitioner do? I just want to apply stuff, right? I can reduce the number of approaches to solve my problem because I will import ray 1.2 and I will be staying there forever and I will only be able to use whatever is in there and nothing more. That's what I can do. I can try really hard to extend an existing framework, but it will be difficult to get funding from my company for that because it's really hard to do. I can copy-paste non-modular code that I take from some of these left things and wrap it my own interface, and of course, eventually, I will also write my own RL framework. I will just not open source it. All right. I think the two fields need each other. I don't have much time, so I will not repeat the arguments. No, but really, presenting the right needs and helping the people on the left to solve those needs will, I think, help everybody. So I want to show some examples of giving up, so you know I'm not just completely inventing this, right? So one example of giving up from a very popular repository, CleanRL, every detail about an algorithm variant is put into a single, standalone file. That's not a library. This is telling you, you want to do RL, copy-paste the file that you like, adjust it. I'm not judging here. All of these things are good, valid approaches, because giving up is a good approach. All right, we give up on modularity. We have 90% code duplication, but at least you can understand what's going on in that file. This is from a very recent project out of Berkeley AI. I'm sure it was updated 19 seconds ago, as most of the projects that I have shown on my first slide. Berkeley AI is not a small reinforcement learning lab. It is the largest, the most important one in the world. They of course have had their own fair share of frameworks being created and dying out, So this new one started from scratch, as you do, right? So the soft-tech critic agent inherits directly from some kind of JAX structure, so just starting from zero. Okay. Also, again, a valid approach. Nothing can help you. I might as well do Greenfield. So you just add to the list that I had in the beginning. This is from Ray. So this is another form of giving up. This is one framework that contains everything, and they give up on algorithm developers contributing to that because of overwhelming complexity. So the PPO policy is assembled, the core functionality is assembled from five mixins that all have different inits that take the somehow configuration thing, which is not declared, that is then writing some default entries, and then some stuff of that is passed to the particular mixins. And then there's lots of more, more, more code here. It's enormous complexity. I'm not judging it. Maybe this is the best way to solve your problem of Ray, which is to run this thing on multiple nodes and multiple GPUs on whatever framework you want. I personally never needed this because I run it on one node, and this was always enough, I don't actually need support for various gradient back computers. But this makes it essentially impossible for external contributors to go through all of this for five years and then, ah, now I have to change this person to actually contribute. And by that time, all of these interfaces are obsolete because of the next patch release. So I think it is possible to improve on this by compromising and by creating a resource which is useful to both fields. And this resource will enable algorithm developers to contribute to it directly. It will give value to the algorithm developers. It will provide them with what they need to write papers to make meaningful comparisons to existing algorithms that don't just rely on luck, let's say. And it will also give them better code and maybe make their life easier by just relying on better code, while at the same time practitioners will have much faster access to new developments in the field. So I think it is possible to do that. Now I have five minutes, so I will go very quickly through the slides. I plan to show some of the things that we are actually doing in the library, but I think It's much less important that I actually show the solutions that I want to make you understand the problem and get this feeling that the problem can be solved. So a library of this type that would solve or address these issues needs at least the following things. It needs a large scope allowing the inclusion of many algorithms. Now RL has many different approaches to it. You have things like model-based reinforcement learning which is more akin to planning. You have multi-agent, which is still in the very infancy. You have online, offline, mixtures of that, on policy, off policy. So most frameworks choose some kind of subfield and go into that. But this is unfortunately not enough, because it will exclude too many algorithm developers. You need good performance. You need to test your algorithms on meaningful environments all the time at every release. It's very frustrating. if things stop converging after some changes. And this is hard to do and requires a lot of compute. You need reliable and representative evaluations following these guidelines that were published at NeurIPS. And you need high code quality, and this is the hardest to nail. You need high code quality without overwhelming complexity. You need to think about both fields. You need to think about the algorithm developer who does not have two years to be onboarded into your framework in order to start contributing okay so the library that I worked on this can show it was one of the entries on the first slide it's I'm going very quickly through this because essentially I want things we want to address these points in particular right so we want to address all of them in particular the library is not by the way I work for applied AI but the library was not created by applied AI was created by the Tsinghua University but just in the last six months, applied AI people. I convinced some colleagues to join me on this path. We started contributing heavily, and we have clear plans for how to move this forward. It's not a small asset. According to GitHub, at least, it has over 7,000 stars. So it already has some community. It already has some users. By stars, it's the second largest, I would say, reinforcement living library. And we selected this one because it fulfilled at least the needs that it had large scope. Many things were not there that we needed. And even now, many, many things are still not there. But I believe we can get there. All right, so I'm going to show you at least the things that I think make it useful very, very quickly. So I'm showing you now three ways. So for every single algorithm environment, we have three scripts, essentially, that show you how to do stuff. So this is a script that an algorithm developer would use. It's a low-level script. This is stuff that existed before applied AI started contributing. So here it's very low-level interfaces where you instantiate your networks, instantiate your actor and critic with whatever architecture you want. These are essentially just Torch modules. It's verbose, right? And you see that PyCharm warns me about duplication because we have lots of these scripts, and obviously these things will duplicate themselves all the time. to give you maximum flexibility. Instantiate your own optimizer, your policy, and so on. Every single thing is created by the algorithm developer. This is the kind of flexibility that an algorithm developer needs. It's still not overwhelming. Like you have these components, the algorithm developer knows these components, puts them all together into a trainer, which is also its own component, and runs them. Then the exact same thing, this is now more on the practitioner side, is done in a configurable way, so just in a declarative manner. You say, okay, I have a configuration that will tell me how my algorithm runs. I have a sampling configuration, and I have an environment factory, and now I build my experiment from what I care about. Everything here is very type-safe, and everything here has proper naming. So, for example, if I want to use a different distribution function factory, So first I can have a look that, I mean, this thing tells me which class it needs. It needs a distribution function factory. And if I ask my ID to tell me which ones I have, then from the name I can see which ones are there. So this is meant to be type-safe and simple. And then the last addition, this was just merged a week ago, is the proper evaluation where instead of building a single experiment, you build a bunch of seeded experiments. And you run all of them, and you get a proper evaluation for this random mass that reinforcement learning is, which shows you something like that with error bars and the performance profile, as recommended by the NeurIPS paper authors. All right, going back to the presentation, I'm going to go through this really, really quickly. This is almost the last slide. Again, I want to acknowledge the team at SINGHUA who has created this and is still helping to maintain it. I also want to mention that we've got really good external contributors, I'm super happy about that, and one of them, Daniel Plop, is looking for a job right now, so if you want to have some RL in your company. So what next? We are not there yet. Okay, this vision of having this resource that helps both worlds, it's incomplete, but it will be complete, I kind of promise it, you know, we want to get there. If you want to help on the way, you're welcome, but we have a team working on it anyway. And once reinforcement learning is more democratized, we will have more people here working on an industry. We will have maybe some more environments that are relevant for industry being available to researchers, and everything will get better. We get into an upward spiral instead of downward spiral. This is the last slide. So we're organizing a hackathon. If you want to contribute or you're thinking of organizing a hackathon, we'll see how it So if you want to have more information about the hackathon scan this QR code and now I'll finish and I'm open questions Sorry for I think I took one minute more

Speaker 2 [26:53]

Yes, thank you. Thank you all, and thank you for your questions. I'll start with two not-a-questions at first. So one is, not a question at all, this is the best talk so far. And then another one says, thanks for the therapeutic session. So there you go. The next most popular question to ask, so it says, of your 70% individual code, how much do you think could be abstracted into a package as Tien Shou? How much would still have to be individual code?

Speaker 1 [27:32]

Thank you very much for that question. If I had more time, I would actually include it in the presentation. So this main problem of writing environment, it cannot be part of an algorithm library. It has to be something else. It's like an augmentation library if you would think about images, or a text processing library if you think about NLP, right? It's not on the learning side. It's on the modeling side. Most of the code that I wrote could be open sourced because it's very generic. However, I will not do it, not even for privacy reasons, but because there will be no users. Because the people who will be using it will do it for their own clothes or stuff and will likely never contribute and never even tell me that they're using it. So I don't want to develop something in the dry. It's very difficult. I will not nail it. I will not find the right abstractions without users. And the people on the reinforcement learning side, like the algorithm developers, they will not care about that at all, because every paper starts given a Markov decision process, which means giving an environment, and they will never be interested in modifying the environment at all. They don't care about improving performance on one environment. They want to improve performance on all environments, so they treat the environment as given. If you want to change my mind about this and say, hey, yeah, I would love such a thing and I would use it, come see me later, because I think this has much more effect than actually the algorithms. I just have a hard time seeing it as an open source resource for these reasons.

Speaker 2 [29:03]

Thank you. We're unfortunately out of time. I will leave these questions in slide... Not one more. We have one more. Okay, so it's 11. Maybe I'll... This one seems like it could be a long one. Well, it's a yes or no question, so I'll let you answer it yes or no, and then you can get into it. So there has been a lot of progress on model-based reinforcement learning with diffusion models recently. Do you also cover these in the library?

Speaker 1 [29:28]

No, by no means maybe in two years a scope is there you want to help here But right now most definitely not and I'm not aware of any library that is doing it

Speaker 2 [29:39]

Thank you. And then again, I'll leave them in Slido until the next session.

Speaker 1 [29:42]

I'll stay here.

Speaker 2 [29:44]

People can find you.