Escape the Hype: Teaching LLM Concepts Through an Interactive AI Factory Game

Speaker 1 [00:02]

Hello everybody. You're up for the next talk. So I'm Lisa. I'm very happy to chair this session and to introduce you to my colleagues Vadim and Erik. They will tell you more about our AI factory. Maybe you've already played downstairs and everything they learned when developing.

Speaker 2 [00:24]

All right, are we good to go? Okay. Hello, everyone. I hope you had a wonderful lunch break and welcome to our presentation called Escape the Hype, where we would like to talk about our journey and lessons learned while teaching LLM concepts through an interactive game called the AI Factory. So what's on the menu for today? We will start by a brief introduction, then we will talk about the AI literacy gap and how we try to close it. After that, we will talk about specific design choices we made in the game and observations we were able to gather during real-world testing. And finally, we will talk about a tricky AI functionality we had to overcome to make it actually teachable. Okay, I'm Eric. I'm one of the main developers of the AI Factory and with me today on stage is Vadim. He's also one of the main developers and unfortunately Lisa is not presenting today, but we would like to give her a shout out for her contribution to this project. Together we work as data scientists at StatForce and StatForce is a Munich-based digital engineering and consulting company where we build scalable enterprise solutions for our clients in diverse fields and diverse topics ranging over data and AI, cloud and platform engineering. We came across a study from RedHunt Labs which revealed that around 1 in 5 wipe-coded websites leak secrets, such as API keys for example. This was pretty surprising to us because obviously and ideally a secure software system should not leak any secrets at all, but maybe this is an indicator that wipe-coding is not for everyone and you might need a little bit of expertise to back it up. Also in our opinion, this would be one example of the AI literacy gap. People are adopting this new and shining technology without truly understanding how it works and more importantly, how it can fail and where it can fail. The gen AI adoption is everywhere right now. Many organizations include LLM into their daily processes. Even the EU expects some kind of AI literacy within organizations and puts pressure on them with their auai act and yeah without proper user training this gap will just continue to widen and some risks are easy to overlook there are plenty of traditional user training formats out there for example reading the documentation attending talks and tutorials or reading the provided vendor materials and even though they are great sometimes they still fall short and some gaps still remain. For example, traditionally they struggle to show the unexpected behavior and system limitations of LLMs. And in addition, they also don't really show how they actually work under real constraints, such as cost or latency. And yeah, for example, the documentation can be super detailed and super extensive, but it always assumes some kind of background knowledge. so it's not for everyone to begin with and in addition it's impossible to document every behavior of the llm because there are just so many unexpected outcomes depending on the input and we also noticed this a little bit internally when we try to teach ai concepts and through a standard format like a lightning talk we had the impression it was not really sticking that much so we came together and thought about a different approach and this is where gamification spiked our interest and yeah gamification is not a new pattern at all duolingo is probably one of the most famous examples of a successful adaptation of gamification to the platform and the key key takeaway here is that a well-designed game keeps us coming back and we would like to practice more so in turn we spend more time with the material and are more engaged by it but what do we mean by gamification we are talking about game elements such as challenges progressing feedbacks or a story in a non-game settings and the purpose of those game elements is to drive up the engagement for the learning objective so the player stays active and curious while they study. But it's also important to mention that those mechanics have to be aligned with the learning objective and not just be there for decorations, because otherwise they could even be a hindrance. So, for example, it's probably more reasonable if you would like to teach a concept of probability that you would use a dice game and not a racing game. Now, how do we think that gamification helped us close the gap we were discussing earlier through gaming through gamification we created a learning by doing scenario where the players can actually experience the unexpected behavior of the llm in a realistic setting in addition we incorporated a tight feedback loop so the player can improve right then and there in the next playthrough and also we created a safe space to fail so you are definitely encouraged to try and break the system in our game so you don't have to break it in production and yeah that's why we built the game welcome to the ai factory it's completely python based we built the ui completely in streamlit and here the players can play through level to level they start really easy with mastering their prompting techniques and go all the way up to orchestrating a multi-agent workflow and yeah you can definitely try it out at our booth we are still here today so we would be happy to see you and now vadim will take over and talk about the specific design choices we made thank you

Speaker 3 [06:27]

Thank you, Eric. So we have a game that teaches AI concepts. But how do you actually design a game part? And this is basically where the most educational tools fail. They know how to nail the content, but they forget that nobody wants to see it through them. So on the left, you can see what a typical AI course looks like. You have something like configure a rack pipeline with managing chunking and a lot of other foreign words. And this is technically accurate, but it's completely forgettable because you lose the half of your audience within a second sentence. And on the right is the same content wrapped in a story. Imagine you work at the Magic Potion Factory and some mystical character gives you tasks like find a secret recipe on the shelves in the old library, and suddenly the player doesn't think that, oh, I'm learning about reg. They think that I need a search tool to find this potion recipe. And this shift in framing changes everything about players' engagement. So we decided to map every technical component or concept to the Magic Factory world. For example, prompt engineering becomes quality control through the clear instructions. Guardrails are just the rules that protect the factory secrets. And maybe those metaphors are not the perfect one, but at least they spark creativity and interest before the technical details. I think any gamer knows this principle. You don't hand every weapon to the player in the first minute. But applying it in education is a bit trickier, because teachers naturally tend to give everything up front, explain everything in thousands of slides but students forgot most of it within a few slides and we decided to focus on progressive disclosure so players master small pieces of concepts while being constantly challenged and levels built from prompt engineering to multi-agent orchestration as eric mentioned before and for example if we take a prompt engineering level we also break it down to the smaller stages, like what is needed for the good prompt. We need to define the role. It's also recommended to have a task, output format, and everything combines into the final prompt. And this inside happens because the players build the full picture by themselves. In traditional education, you submit an assignment and you wait days for a grade. And in the game, you just press a button and you see instantly if it worked. and this tight loop of action and consequences mirrors how we humans learn the complex systems. And it's something we also heavily invested in. So on the right you can see the outputs from our game. The player prompt is analyzed within seconds and we check for example whether the role is present or maybe a task is missing and below that we also show sometime the similarity score that reveals how close the player got to the ideal or expected answer. We also have some special characters like Lady Vermillion, who serves as an LLM as a judge. And they provide the feedback about what went wrong and why. And the key is that this feedback is specific enough to act on, and it's not just like, it's incorrect, try again. Then constraints drive creativity. I think when you hear the word constraints, you probably think of limits and restrictions. But used wisely, they spark creativity. So players cannot just brute force solutions. They must think strategically and weigh trade-offs. For example, in our MCP level, six tool servers are available. But connecting all of them together exceeds the budget. That's why the player should choose wisely which servers to use. Just like us, AI engineers deciding which APIs to integrate in the system. In the RAC level, we have a storage that also costs coins per chunk. And we have a budget system that turns every AI call into a strategic choice. And those in-game constraints prepare the players for the real world and for the real use cases. Now let me share some things we only learned by watching PyCon attendees and also our colleagues playing the game. So this is something we got completely wrong on the first try. Our first version of the game gave players some limited budget and hitting zero ended up the level and the players needed to restart the whole level again. Originally, we thought that it will teach players resource management, but it basically taught fear. Because when someone ran out of the budget, they just didn't think strategically, they just thought that the game was super unfair and they closed the tab immediately. So we decided to flip this model. Now players start at zero, costs accumulate and a soft benchmark provides guidance. And nothing stops players from going over. The bar turns yellow, then red, and it basically gives the visual feedback that we placed the old game over screen while keeping players engaged. And we use so-called golf scoring where low is better, but players always finish the round and keep learning. Here's also a small confession from our side. Our rack level where players learn something about chunking, embeddings, and retrieval was originally tied to the real vector database. And when the player changed the chunking size, the system re-indexed every document. And this is technically correct, but educationally super terrible because players lost focus watching a spinner for like 30 seconds, sometimes even minutes. So what we decided to do, we just simulated it. We predefined a parameter combination, we run ChromaDB in memory, and players see a realistic retrieval pipeline, but with outcomes that was already decided. And once this worked for rack level, we applied it everywhere. For example, in MCP tool service, they just returned the simulated data. We also have a multi-agent orchestration level with mocked probabilistic outcomes. And each level fakes or simulates a layer of the stack, but it keeps the decision-making part of the game as real. And this is where the real learning happens. So the question was asking for your potential education project is that does this feature needs a real infrastructure or does a well-crafted illusion teaches the same lesson? And one thing we didn't design for, but noticed immediately, is that people from different backgrounds gravitated to different parts of the game. This, we also observed it during our tests at PyCon. So, for example, our colleagues from compliance and finance team, they went straight to guardrails. First of all, they played with prompt engineering, and they said to us that, yeah, that's fine, we know how 2GPT works, but then they spent 40 minutes just with guardrails. And I think this is just because that's where the real risks live for them. The EU AI Act is already on their desk, and this level becomes some kind of rehearsal for them. On the opposite, data engineers, they went to Rack, testing checking strategies and beddings. I guess they just wanted to see if the game matched their production intuition. And they just discovered that usual 512 tokens approach wasn't optimal for every query. And then we also tested our game a lot with business people and managers, and they were just chasing agentic hype. But then they learned that aiding more agents just doesn't make the system better, but also increases the costs and coordination complexity. And the main takeaway is that you don't need to have a special or separate training per team. Build one deep and broad enough educational game and each player finds the level that mirrors their world. And the real engagement comes not from the assigned or like a special levels, but from the one that they choose. And so far, we have talked about the game design and the learning principles. And now let's get into the technical reality of building such an educational game on top of the LLM or large language model. Because it turns out that the things that makes LLMs powerful are also the things that make them terrible for games. And here's a fundamental tension. On the left, you can see what any games need. It needs deterministic scoring, a structured output, and a fair experience. And on the right, what actually LLMs give us. Just different outputs every round. Occasionally unparsable, at times brilliant, at other times complete nonsense. And not just slightly different. The output might change. The AI might invent some facts or even hallucinate. And even a perfectly good prompt can return a mediocre response simply due to the randomness in the model. And maybe the first instinct that you might have is to eliminate this randomness, lock it down. But we thought that this non-determinism is not always the enemy. When testers played our game, they told us that the whole game feels fresh because AI said something different this time. And they were learning something new on every attempt. And this is a more realistic experience than any tutorial can provide you. because in production, LLMs are also not like vending machines. So the question wasn't how to kill the randomness, but it was where do we need to control it and where is the chaos actually helping us to teach? So we just decided to develop five strategies ranging from complete control on the left to embarrassing chaos on the right. So far left, we just skip the AI entirely. Some levels show pre-written or pre-generated outputs and the player actually doesn't know that the AI didn't generate them. And it doesn't matter at this point because the learning happens in the reasoning about the output and not watching just an API called spin. Then we also used deterministic validation. So when the AI produced structured outputs like JSON, we just compared field by field, we parse it, and the scoring is reproducible without using AI. Then I think temperature. Most of you know this parameter. it's the most intuitive control that we can use and its impact is dramatic because we can adjust the value to increase or decrease the AI creativity depending on the level or the task. Then we also use the weighted scoring and I would say this is probably our most important lesson and I will go into details on the next slide and on the far right we also had or we are also using dynamic prompts where we inject some context like expected answer or like complexity into the system prompt and this narrows down the AI range without eliminating the randomness also helps us to teach and the key is that this the different stages also use different strategies and sometimes even multiple at once so here you can see the approach we reached after extensive trial and error. Early versions of our game just used a pure LLM as a judge, where we just sent the player's output to AI, asking whether it was good. And from our experience, it worked like for 75-80% of time. But for the other 20%, the judge contradicted itself on the same or identical answers. And for an educational game, this is totally unacceptable, because players lose trust immediately. So what we decided to do in the end, we combined three methods. First of all, we perform structural checks. So we verify the format, we check whether the header is present, whether there is a final verdict, maybe whether the role was defined. And for that, we use a fast and deterministic regex. Then we also have a content layer that has the most weight. And here we just use keyword checklist to ensure that the player includes the right facts and terms. And this can be just bypassed by copying the task because the player must understand the level to go further. But we also still use LLM as a judge that evaluates the tone, overall coherence, reasoning, but it's only 30%. Because if it fails, we still have the deterministic layers that carry the grade. And the weights, they usually arrive by the tasks, but the principle stays the same. Trust the AI, but verify with something it cannot fool. So, if you take away six things from this talk. First of all, align your game mechanics to what you are actually trying to teach. A budget system, for example, it just doesn't add fun. It also teaches cost awareness. Then, wrap technical components into some narrative. Nobody remembers something like configure your rack pipeline, but almost everyone will remember talking to a mystical or super magical cat, so you need to find a secret potion. Then, when the real infrastructure gets in the way of learning, just simulate it. The players don't need a real vector database to understand the concept of chunking. Also don't forget about constraints because they can be your friends as long as they teach instead of punish. Don't forget to integrate the feedback loop. It should be immediate, actionable and specific enough so this turns the whole wrong answer into the learning moment. And finally, non-determinism is not a bag. In our view, it's actually the most realistic thing about our game and if you design your product around it properly it becomes the lesson by itself so here we come to the end of our presentation thank you for listening feel free to visit our booth at the end we're still here till the evening and we're ready to answer your questions as well thank you

Speaker 1 [20:58]

Okay, thank you. So we already got some questions. So the first one is, is there a plan to open source this game and have community go it?

Speaker 3 [21:08]

Yeah, this is also our long-term goal. We are planning to make a part of this game as an open source, but it's still being in development. We are making a beta test, so feel free to come and visit us, and we will try to make it open source as well. And we already have one open source game in another area. We have a so-called Kubernetes escape room where you can try to break the Kubernetes pipeline. So visit our website, and I think there is a link to this Kubernetes escape room where you can try to break it. Thank you.

Speaker 1 [21:39]

Okay, the next one. Have you looked into how much the three layers of the rating engine correlate? Is it really necessary to have all three?

Speaker 3 [21:54]

Could you repeat the question?

Speaker 1 [21:56]

So in your three-layer engine, did you check how they are correlated and if it is really necessary to have three of them?

Speaker 3 [22:02]

sometimes we just use one of the layers but it's actually a good point maybe we also need to research like define this the correlation between the levels and the weights it's actually a good idea but we haven't checked it so far thank you for the idea

Speaker 1 [22:20]

How did you convince your bosses to invest so much time in a game just for internal training it sounds like it would have a poor ROI

Speaker 3 [22:28]

So, I have to say it's our side project that we developed internally for our colleagues because we observed this AI literacy gap and I think our management was also observing the same things in the company and yeah, we just had fun a few evenings developing the game and our management just saw the potential. Maybe I can add for you.

Speaker 2 [22:53]

Yeah, we also have something like a COC in our company, so a center of competences. And we wanted to get more deeper into like how LLMs work and all that stuff. So we took basically this motivation to study and improve our LLM knowledge as well. And then we tried to pack it into a game so our colleagues can benefit from that as well.

Speaker 1 [23:15]

What I wanted to add is we also have this Kubernetes escape room which really draws attention to our booth if you're on other fairs. And so it's really nice to have something with AI because if there's the word AI standing there, much more people are coming. So the next question is, how will you make sure you will stay up to date with the very rapidly changing world of AI? How quickly are you able and wanting to make new levels?

Speaker 3 [23:47]

This is a good question because every week we have something new, but still now we're focusing on the basics of AI and LLMs. And I think we will still work with prompt engineering. Rack is also with us for like, it's going to stay with us for a couple of years. So far we're going to concentrate on the basics, but maybe in the future, yeah, it's going to be a challenge for us to update such a tool.

Speaker 1 [24:14]

Did you wipe coat parts of it?

Speaker 3 [24:16]

So it was partially developed with the help of AI, but not fully.

Speaker 1 [24:22]

Have you measured how well the information translates to the real use cases?

Speaker 2 [24:34]

I wouldn't say we have measured it directly, we always try to make at least some kind of parallels from the potion factory to like a real world scenario, but obviously it's still a little bit abstract, and yeah, we haven't measured it directly.

Speaker 1 [24:50]

Okay, how happy are you with Streamlit as engine? Wouldn't have seen it as production ready toolset for games Good question

Speaker 2 [25:02]

incredibly surprised how well it works actually because as you probably know Streamlit is here that we can push out of concept really quickly and build like a nice dashboard or a nice UI so we actually had to yeah really try but I think it looks pretty nice and it works actually super well so yeah we were super happy with Streamlit

Speaker 1 [25:26]

I found one last one. What other concepts to teach are you planning to add?

Speaker 3 [25:38]

We already have some plans. In the first version of our game that we developed internally in November, we also had a memory level. But we're also planning to integrate something about the deployment of LLMs or something in the direction of LLM ops because it's also a topic that's being hyped now and we also see this literacy gap there as well. Erik, you have something to add?

Speaker 2 [26:04]

We also are thinking about adding a level where you have to add some kind of images, for example, so people also get to know that an LLM can work with images.

Speaker 1 [26:14]

okay so i think i have all of them yes so thank you so much for coming thank you too thank you

Speaker 3 [26:21]

Thank you