Mastering the Hex: A Case Study in Reinforcement Learning for Strategy Games
The project addresses the challenge of developing an autonomous AI agent to play a turn-based strategy game on a hexagonal grid. The game involves capturing territory, managing income, upgrading unit strength levels, and constructing buildings like farms and towers to secure borders. The primary technical difficulty lies in the complexity of the game rules and the vast state-action space associated with hexagonal movement and placement.
The approach involves rebuilding the game logic in Python using offset coordinates to represent the hexagonal map as a 2D array. To facilitate machine learning, a Gymnasium environment was implemented to provide the agent with observations, action masks, and rewards. The observation state consists of multiple Boolean channels representing territories, units, buildings, trees, and gravestones, with normalized values for income and money. The output layer manages approximately 4,000 theoretical actions, though typically only 100 are valid per turn. The architecture utilizes a Convolutional Neural Network (CNN) to process the map channels, paired with an actor-critic model to balance action selection and evaluation. Training was monitored using MLflow.
Key takeaways highlight the difficulty of reward shaping and state complexity. The agent was tested using sparse rewards (final win/loss), dense rewards (intermediate gains), and a hybrid approach. While the model showed some success on smaller maps, it struggled on larger maps, occasionally performing worse than random play. Furthermore, training the agent against itself led to infinite loops of indecision, demonstrating that scaling down the environment and carefully tuning reward functions are critical for convergence in complex strategy games.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Machine Learning & Deep Learning & Statistics and was classified suitable for intermediate domain / novice python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Context and Motivation
This talk emerged from a year-long journey that began with a simple curiosity: could I teach a computer to play strategy games by itself? It started as a college seminar project, but the topic was chosen purely out of personal interest in reinforcement learning and game AI — this was a hobby from the start. Rather than working with pre-built environments like CartPole or Atari games, the goal was to understand the entire pipeline—from implementing game mechanics to training a neural network that actually learns to win.
The game chosen was Antiyoy, a minimalist turn-based strategy game where players control territories on hexagonal grids, build units and structures, manage resources, and compete for dominance. While the game is simple enough to understand, it presents genuine strategic depth—exactly the kind of challenge that makes reinforcement learning both difficult and rewarding.
The talk walks through the complete development process, focusing not on implementation minutiae but on the fundamental questions and design decisions that anyone building similar systems would encounter. You won't see walls of code or detailed mathematical derivations. Instead, you'll hear about the thinking process, the challenges faced, and the solutions that emerged—all with the goal of demystifying what it actually takes to build a learning agent for complex games.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:18]
Hi everyone Thank you for coming for this talk and let's give a warm welcome for Simon Hedrich with his talk on mastering the hacks a case study in reinforcement learning for strategy games
Speaker 2 [00:42]
What you can see here is AI, and I built it, but let me start from the beginning, which was about a year ago. There was a weekend where I was really bored, and I scrolled through the Google Play Store and found this really nice-looking game, which reminded me of something like Civilization, if any of you know it. It has a quite unusual name. It is called Antijoy. I call it Antijoy, but this is only my preference. Let me reset this. I want to show you just how this game works. It has a really nice campaign and all the games start with such a map and with different players with different colours. Each of the players have territories on this hexagonal beehive pattern map. Each tile gives income and we earn money with this income. For example, to buy some units like this peasant, which can in itself capture new maps. The game is turn-based, so let's go to the next turn. The other players did their turn, and now the units can again move one turn, one step, so we can capture even more tiles. There are also trees. the trees spread to our territory. The tiles with trees on them aren't earning anything, so we should probably try to remove the trees, if so possible. Then we come to the situation that we need to capture enemy territories, which we can do by upgrading our units. There four strength levels of those units, which are increasing in price and also maintenance costs. So, let's upgrade this unit, and now we can capture enemy tiles. And the goal of the game is to have all the map for ourselves. And winning, obviously. We can also increase our income by building farms on the territory, or to secure our borders, our territory against weak enemy units, we can also build towers to protect our territory. Okay, so I hope you get a sense of how this game works. The campaign gets increasingly difficult with the size and strength of enemies. I played this game for a few weeks, and then I got kind of frustrated because I had to replay those hard maps over and over again, and I lost most of the times. So I thought, why play the game myself when I could just try to program an AI to play it for myself? And with this I got also the inspiration of a YouTube channel I can see some of you know, the channel CodeBullet, which has some interesting videos in which he trains AI to play different kind of games. They are quite entertaining, and I thought, if he can do it, why shouldn't I? So I set out to build a game bot myself for the game Anti-Joy, which isn't highly explored in this arena. Then to get started, I have to get access to the game. I thought about running an Android simulator and capturing the screen or something like this, but this would be much more complicated. That's why I looked at a source code, which is gladly open source. That's the good part. The bad part is that it's 100 per cent Java, and at PyCon I can admit that I didn't understand anything. But I got at least the icons that were used in the game, and also the colours of the territories to keep it similar to the original game. Okay, then I set out to build the game from the ground up, and the first thing I saw when researching those games is that there are different coordinate systems to represent those hexagonal tiles. The first one is our cube coordinates. They have three variables to represent each tile. The benefit of this is that you can easily calculate all the neighbouring tiles by increasing one of the variables and decreasing another one. But also you have to store those three parameters, which is quite a hassle. That's why I looked at offset coordinates, which is much more intuitive to think about, because it's just a 2D array, essentially. There are also many other coordinate systems, but they have all the downsides, so I stayed with offset. Now for the game itself, we have the map, we have the coordinates, each tile needs then one entity and a territory, which I limited to a max of two players to keep it kind of simple. Then we have the map, but this is the core of the game, but not at least all of it. One of the major hurdles in implementation of such a game are the game rules. The issue with those is you can't test them with unit tests or similar testing strategies, Because there are many edge cases which you all have to represent in your tests, and when you want to do it with static testing, this would be much too great of work. So to check if my game actually works, I had to implement also a playable game, and I used Pygame for this. As you can see, there is a map drawn with coordinates on them, and with this Pygame I could test all the different scenarios and check for correct implementation. And trust me, I found a lot of errors in my initial game implementation, so this is a a really good strategy of doing such things. Now the game is finished. I can play it. But then I had to think how an agent, an ML model, they have to interact somehow with the game. For this, there is a gymnasium environment available, which is only a template for an API, an interface by which an agent can interact with some situation, a game, or other things. This originated from OpenAI, which was way smaller back then. Back then, they worked on balancing a stick on a card. So, yeah, what does this API prepare for us? There are three main functions. The first one is observe. By this, the agent asks the environment how the current state looks. And the environment returns the observation state and an action mask by which it tells the agent which actions of the possible ones are really possible in this current state. Then the agent chooses one of those actions and gives them, by the step function, to the environment, and the environment acts this action and returns the next observation state and the next action mask and also a reward to reinforce or penalize the agent for a good move or a bad one. Also there is the reset, but this is self-explanatory. Okay. Now, what does the observation state look like? Obviously, you can't just render an image and give this to the agent. You have somehow put this in a format for the agent to reuse it. And there the question is asked, what does the model actually see? And first we look at the observations. There we start with our hexagonal map, I created a scenario which is quite interesting. First, we use our offset coordinates to convert this hexagonal map to our 2D array, which is also implemented in the background. And with this 2D array, we obviously cannot put a pickle function with tile objects into an ML model, so we have to somehow separate the information in a way that is suitable for such a task. And for this, I split up the map into multiple Each of the channels is made of Boolean values in a matrix style. And for example, this channel here only represents where on the map a gravestone exists. Oh, I forgot. The gravestone, if you can't pay the units and have no money left in your turn, then all the units die and turn into gravestones, which is not a thing you should do in this game. And the gravestones themselves convert to trees later on. At this point, I would also apologise to all the people who have a red-green colour blindness. Just in the last hour, I thought that those colours aren't that suited for this visualisation. But anyways, we have multiple channels, and those are all the channels that comprise the overall map. The top three are the territories, and then for each player there's a channel for each units and building and also on the bottom there are the channels for trees or gravestones where they are located. This is not all the information an agent should get, but also the income and the money for each player. To keep the format of those channels consistent, I implemented other channels that represent this income and money, and there I took the number, like $5, and normalised them between 0 and 1, so that all of the channels, all of the elements in the channels have the same number. Okay, so we now know what the agent sees, and now we have to think about what the result of the model is, the output. This This model needs to output what action should be taken next, and there we also start with the map, and one of the actions I showed is to place a unit, buy a unit, and place it on the map. There are those four unit types, and the environment has to think about where those units may be placed on the map. Those are all the tiles, so each of the units have a limited number of tiles where they can be placed. The same goes for the buildings. This visualisation is not entirely accurate because those channels shouldn't be the same, but I was too lazy to change it up and make it accurate. So there needs to be multiple channels to represent each possibility of placing an entity. But placement is not the only action to be taken. So we can move a unit, and let's focus on this knight. The maximum possible movement is four tiles, which equates to 60 tiles to move to. But this is only in theory possible. The tiles in practice are much more limited. Those are the example tiles for this night. And we take those tiles with the indices in the spiral pattern and map it onto a list. Not only for this tile, but for all the tiles on the whole map. Because anywhere on the map a unit could be, and so there are all the possibilities. Then we keep the same format of the movement and also have the placements in a list and also at the end turn as an action. Now you can imagine how this output would look like if we concatenate all these lists to one long so that we can use it as an output node for the machine learning model. When we sum up all the possibilities together, they equate to about 4,000 actions in theory. In practice they are maximum of around 100 actions per turn. Now we looked at how the model gets its input and output. Also I had to test if this implementation worked, so I prepared another UI. This time, I skipped Pygame, because it's too complicated. So I just let Cloud wipe code a React app, and this worked surprisingly well. Okay, now we have all this, and we come to the core of the operation, the model, of course a deep learning model. As you can imagine, when we have the input channels, these are well suited for convolutional layers. So this is why the first major part of the model is a CNN, to capture all the maps, all the channels which represent the map. And then I used a technique, an actor-critic model to, well, I won't go into much technical diesel detail, but shortly the actor does the action or learns what action to take, and the critic learns if this action of the actor is really a good one, and so they balance each other out, balance each other, and learn in parallel, and hopefully give a better result. Then after the model, we also have to look at the reward. There are different methods. The first one is sparse, where we only look at the final result, so we let the model do its thing, and if it wins, it gets a reward of one, and if it loses, it gets a penalty of one. And this final reward is divided over all the steps it took to get to the final result. This has the benefit that it's really clean to implement and easy, but it's also not quite precise for the model to learn which of those actions resulted in the win. That's why there's sparse, there's dense rewards. Here we also look at wins and losses, but we only account the last step for this result. And we also have intermediate rewards for every single action. So for example, if the model takes over an enemy territory, it gets a reward for it. And if it loses a territory, it gets penalised. This helps in fast early training, but the agent may game the game and optimise this for rewards and not for the output we desire. That's why I chose a hybrid approach to mix them up and get the best of both worlds. But this is not quite easy to implement, and also you have to really look at how the model reacts to those rewards that you set at the beginning. It's more an art than a science. So now we come to the training. I did the training and let it run for, I think, two weeks. And in the middle of it, I looked at it and did some small adjustments. But after two weeks, I came to the conclusion that it didn't train really well. The green one is the ML model, the red one is randomness, and as you can see, they are undistinguishable. And because of all this work of implementation and training, I had to do some other things and couldn't invest that much more time in this project. I had to either give up or lower my expectations a lot. That's why I trained on a much smaller map. And this was the result after a few more days of training. Again, the green one is the machine learning model, and the red one is randomness. And this result was... Come on, you can do it. Just another turn, yeah, and it's okay. It wins. This was a representation of quite consistent actions. That was also viewable in MLflow, which I used to log all the parameters of the training. And because we are at PyCon, I also want to have some Python code on my slides. With this, I want to show that to integrate MLflow in your project is really easy. That's pretty much all you need. import it, set experiments, start a run, and log how much you want. Okay. So, in the beginning, I hoped that I master the hex with my superhuman AI. But as you saw, it's not that exactly. But I set up another training and just let it run for, I think, a month. And I found some really curious results, which I would like to show you. This is, again, the green, my model, and the red is randomness. And as you can see, it doesn't stack up as well as I hoped, which is quite peculiar that I made the agent worse than randomness. So what I thought is, just look at what happens when I train my agent against my agent. And this is what it looks like. And I think you can imagine why the agent didn't come to such a successful win. And this loops for infinity. So what I would like you to take away from this is that it's not really quite straightforward to come to a successful model, especially if it's so complex with such a huge map, and either one must be an expert or really scale down the operation and expectation. But here at PyCon, I have all of you, and yesterday in the opening session we heard that the community is the most important thing about this process. That's why I invite you to try it yourself, to master the hacks, and go on my GitHub and send me your results or your failures. And With that, I want to close and also want to thank Innovex, which made it possible for me to attend here, and they sponsored my stay. And thanks to you for being such an attentive audience.
Speaker 1 [27:16]
Thank you, Simon, once again for wonderful, really interesting talk. And we have a couple of questions. So first of all, your animations and presentation was really gorgeous. Can you share a little bit more what you used for it?
Speaker 2 [27:34]
Maybe some of you know the YouTube channel 3Blue1Brown, if you're interested in mathematics. Those animations look really similar to mine. It's because we use the same engine in the background. His engine is open source, it's called Manim. And it's also Python, so yeah, I just use this to create those animations
Speaker 1 [28:05]
Great. Thank you. So now we know how to do it properly. The next one is your current model seems hard-coded for a two-player game. How would you adjust the architecture for arbitrary amount of players?
Speaker 2 [28:22]
scale it up, yeah I think that's really the answer. For more players you can just use even more channels and even more, I think the outputs would stay the same, but yeah the input channels would have to increase by the number of players.
Speaker 1 [28:46]
The next one is you mentioned that you used the convolutional neural networks and the task seems to be like suitable for attention mechanism maybe. Have you tried to experiment maybe with something like used attention mechanisms?
Speaker 2 [29:05]
No, I think, as you can see, this was complicated enough and I used quite small of a model, so I think attention mechanism would be even more complicated on another dimension.
Speaker 1 [29:26]
Okay, thank you Another one thing is have you tried to change the reward for function? So you mentioned one is like for the whole game one for the steps one the hybrid any more experiments that you can share or
Speaker 2 [29:44]
You know throughout the training the changing the rewards is 90% of the work and I did a lot of reconfiguring Resetting the rewards adjusting them
