Going Global: Taking code from research to operational open ecosystem for AI weather forecasting

Speaker 1 [00:08]

Hello everyone. So, AI weather forecasting is a bit of a niche topic, so I'm happy that all of you are here, but I think the main idea is how do we take code from like a research project to operational open ecosystem. This work obviously wasn't done by me alone. There's a bunch of names on here, the slides will be available after, and there's a bunch of meteorology centers across europe that have contributed to this work as well so just if there's anything bad in here that was me if there's anything good in here that was them just you know the drill um i don't really believe in agendas because if i tell you the agenda it will make no sense to you but we have a couple of goals for this talk so i want to talk about how we grow software projects um how we anticipate user needs even if we don't know the user yet that well um learn a little bit about weather forecasting because after all it's kind of cool in my opinion i hope after this talk you agree um then some ai cautionary tales we hear a lot about ai agents now and i can't even get an llm to reliably give me code that works, so maybe some cautionary tales along the way that are like, how would an AI agent do that? Then upskilling from coder to software architect because that's essentially what happened to me and I'm not even a coder by default, I'm a geophysicist by background. Then planning for features you can't even know about. the real fun in this entire thing. Growing a team to a significant size, we started with four. We're now 40. And then have a little fun along the way. Who am I? Why am I standing here? Also, I made a little website for this where you can have slides and links and all that if you want to check it out. So that's the QR code. I'm a scientist for machine learning. So technically, I'm something machine learning engineer, researcher, something like that. But my background is actually in oil and gas. So my PhD was actually paid for by Maersk back in the day. So I did a full 180 in my career, which I'm very proud of, to be fair. I do Python data AI education. That's why I'll be sitting on a panel later this evening as well. So if you want to talk careers, that's also going to happen today. And I maintain Python deadlines, ML recipes, data science guide, and I'm generally loud online. So you'll probably see me on LinkedIn or YouTube and stuff like that. I also apparently have too much free time. So why is weather relevant? I know this surprises everyone, but I sometimes listen to metal music. And if you're German, you have heard of Wacken. If you're into metal, you have heard of Wacken as well. And the little subtext of Wacken is rain or shine. So I've been to this muddy festival, and it would be great to know when you have to bring your umbrella, right? So how do we do that classically? Where does weather forecasting come from? Basically, what happens is we have a bunch of observation data. We somehow have to get this observation data into something that a physical model understands, so basically grid it, harmonize all of this, then throw that into a physical model. So what we do there is take the physical equations for Navier-Stokes, essentially, calculate those forward, and then make predictions. pretty good at getting great initial conditions we have a really good model and we get a very good forecast but it turns out if you slightly wiggle your initial conditions and then get multiple outputs you get an ensemble of outputs and you actually get a better estimate of your weather because the weather is very complicated things wiggle around all the time you maybe heard of the butterfly effect and all that stuff so usually what we do is we forecast the weather 50 times per forecast now observations is a word where everyone's like yeah kind of but like these are all the observations that we take in this is radio sounds that people send out aircrafts flying around we have sign-up stations we have ships and you can already see a plane isn't always on the same path in 2020 planes stop flying so sometimes they're just not there and yeah radars are very stationary satellites are great because they have a huge swath of information but they're usually very one-dimensional because they just get the reflectivity so you have a bunch of different observations and this data assimilation that you saw is somehow making sense of this and getting all of that into one and usually when I talk to like normal machine learning people the 800 million daily observations give them a heart attack because somehow processing that and getting that fed into a model is I think on a level that is quite up there obviously there's like exascale and all that that other people do but this is already significant to model the weather physically we have all of these interactions maybe you remember that from from school where all of these cycles happen the carbon the water cycle we have irradiation from the sun and all of this complicates our weather basically it's a huge dynamic system so we calculate that on our computer this is actually not a stock photo this is our our supercomputer technically we have four of those because um the little flags that you saw in the beginning they're not just decorations those are the people that yell at us when we don't do a forecast and that's very uncomfortable when you have ministers sitting in your in your office and tell you why are you late like we need those so we have redundancy across those and a forecast is usually done in 30 minutes on 6400 cpus that's the single one of those members we make 50 of those simultaneously so you can see the scale of this is quite quite something our data archive now is over one exabyte we recently relocated everything to bologna we were in reading before. We are still in Reading, but people are in Reading. The data, the compute is in Bologna. And, yeah, getting one exabyte there, you do not have the cables to do this. What you do is, you put all your servers on a truck and you drive it over. That's the fastest way to get that kind of data, actually, to a different location. And you really, really hope your truck doesn't fall over. So, why am I telling you all of this? Well, this was basically a slide that we used to present. What will machine learning look like ten years from now? And you can see observation screening. So, looking which observations are good is kind of outside this central part where we say, oh, this is very likely. And learning the equations of motion is also just outside of this. That was two years ago. What we do today is we have all that data. We don't quite use the full exabyte because that's a lot of data. But we use a couple terabytes of data. We train our graph neural network on this. And we get a pretty good approximation of the physical model. How that happens is we still rely on the physical data assimilation for now. And we get the initial conditions, our fancy model that we've trained with those forecasting steps is now doing the predictions for us. And, yeah, you may wonder what kind of computer do we need for this? We still have the same supercomputer. It has GPUs attached for us which is really nice the training usually is about 64 gpus in eight days and the forecast itself is one gpu in two minutes so you can see the appeal of this there's first of all the whole cost of the entire thing but also this is near real time like this is is much closer to real time than a physical forecast and the well the ecological cost of this is also minimal right running four supercomputers is a very different thing than having a gaming pc somewhere to do a forecast for you so somehow finding the middle between this would be quite interesting and i know people have overused the word paradigm shift a lot But I think unironically, in this case, we can talk about a paradigm shift where we are literally going from thousands of CPUs in minutes to hours to a GPU in minutes once we have the trained model. And even the training, when I talk to our HPC people about, oh, yeah, our training is usually quite expensive, they look at me like, yeah, sure. You call that expensive because you have your computer, but for us, that's still nothing. This could be the talk, but I applied for a long talk because I like to talk a lot. And I like to tell you the story how we ended up here. Because there's a huge gap in the middle. And basically, how do we go from we have a lot of data, we have a lot of compute, to, well we have a model that works because experts we ourselves were saying this is not possible how do we end up here if any music people are in the room I have strewn lyrics all across please enjoy them the initial commit is of course the typical initial commit if you ever look at any open source thing suddenly a bunch of code that magically works appears and it is called the initial commit this is also two years ago um within this i would say this is typical research machine learning code as someone that has done their phd in machine learning research applied machine learning research i would say you have a bunch of architectures you're trying out you have some inference code somewhere you use functions because you're taught as a mathematical person as a physical person so functions come natural to you and yeah um basically when when i got into this project as well i was number four i would say ish it always depends how you count um i saw this cool wish list that they already started um don't worry if you can't read it that well um it's more about I saw we need configurations, so make it usable. I saw we need refactors, so maybe move some things around, make things more robust, make things reusable, modular. And I saw we need an inference mode that actually works because we have the whole infrastructure already. We are essentially doing a full model that somewhere gets its data. This data has to go into the model, and then it has to go somewhere. so we have all the infrastructure for our classical model and how do we make all of this fit so a little bit like we don't there's no reason for us to go to amazon and go into the cloud because we have it all there we have all that specialized infrastructure verification all the tests and all that running already so slotting that in there and being able to reuse as much as possible which also means you're not losing the whole skill set of the people that you already have there, is really important. So seeing what's possible. And, yeah, I defined these as three users in my head and then formulated that into three users. It's researchers, it's developers, and it's operations. What does that mean? Researchers usually modify configs, want to experiment, want to add new weather variables in there. Maybe they want to try out if adding population maps helps because you have a warming effect over cities and really try out different things to actively improve the model beyond the code. So, experiments. That's already what happens with the physical model as well. The physical model is constantly being improved upon. People find new schemes where, for example, the irradiation of the sun is really complicated because it goes through a bunch of layers. Sometimes there's clouds, sometimes there's gases, and sometimes there's snow. So it gets really, really complicated, and people get better and better at making this faster and making this more accurate. So we already have a use case for this. The developers, they modify the code base. So, make this more extensible, make this faster, make this more reliable. And operations, it runs our model on a common interface on reliable infrastructure so that all those flags don't have to yell at us because we missed the forecast. What this means in total is we want quick switching of experimental values. These people shouldn't have to go in and dig into the code to find the magic variable that they then set to 5 or 42 or whichever is your favorite magic number. Less interaction with the core machine learning code because we have a bunch of parallelization in there and it gets really, really complicated really quick. And the experiments should be tracked. Essentially if you run this from the config files, if you run this from the terminal, anyone that has done any model training knows your best run is usually the run that you You sent off Friday evening, ran over the weekend, had some weird changes on your terminal, maybe, I don't know, a muon hit your data and flipped a bit or whatever, and you have no idea why this is the best one, and it's really hard to reproduce. So we kind of also wanted to look at that, have full reproducibility, as much as full reproducibility means anything in machine learning. And for the developers, modularity, extensibility, code quality. So if you have your bingo cards, PEP 8 is really important because suddenly the code is kind of readable and kind of nice. And we want a separation of concerns. Like your inference code shouldn't be in your training code. Those should be at two different places. And for operations, minimal dependencies. Researchers and also machine learning people really like to try out stuff. You've seen all the fancy new stuff that you can do here, the MCP and PyTorch Lightning and all those things, but what do you deep down really need to run the model? If you have a checkpoint, do you just need PyTorch maybe? So you can get rid of that two gigabyte dependency, which is not a joke. I was actually digging into the code and basically ripping CartoPy out that on certain architectures is two gigabytes to install and the people look at me like we cannot run this in operations and i'm like i agree but like we need the plots for training so that goes back into a separation of concerns and also yeah trying to trying to dig deep down and get the minimal set that you can get and consistent interfaces you don't want the model to change from from each run you want basically a config that goes in and the config can change but if you have your your containers that run somewhere or anything that gets sent off this should be should have one common entry point essentially so yeah how do we facilitate this kind of collaboration um because like i said we had to ship our service to bologna something happened in the uk um a couple years back and for some reason all those flags really wanted to have some representation in the eu so i am actually in bonn if you're ever around um shoot me a message because like bonn is surprisingly nice and yeah reading still has a lot of people reading is our headquarters still um but we even if the pandemic hadn't happened we had to facilitate remote collaboration already so it's kind of how do you make this work? Well, I'll get into this, basically. So, what I've done first is I rewrote a bunch of the trainer code. You can see right here, we have hierarchical configs with Hydra, so this is already working for some of the experimenters, some of the researchers, so you can switch values in there and easily switch those out. I'll give you an example of that in a bit. Then tidy your separation. You can see right here there's no formal architecture in this anymore. The architecture is now formulated over the configs and there is no inference in here anymore. We have a separate inference package that takes your checkpoints and runs them. And, yeah, the trainer class itself, it's a class now. We have nice attributes in there that can be reused, can be switched out. We already did this. We used weights and biases in the beginning, switched it out for MLflow because weights and biases gets really, really expensive if you have a growing team, just to keep in mind. And, yeah, we also grew this, well, basically into a package with external dependencies. And this is the second part of the talk that I want you to keep in mind, how to structure repositories. So this is really convenient to develop. We could go really, really fast. Everything is in one place. You just have to define, oh, I want that dependency. It goes in for the inference. You just define it as a dependency, or the inference defines this as a dependency. So this is a really nice way to just go very quick in the beginning. It's very quick to release. You don't have to do anything. You can just do it on GitHub and push it to PyPI. And, yeah, it's kind of weird to install. So if you want to just install the training portion, then you do the brackets with training. If you want the data portion, you do the brackets with data. And you have to do a lot of, like, defining of dependencies and your PyProject Humble gets really long. Not all code is in the ecosystem, so different repositories have then the inference portion, different repositories have the data portion, so just it can be difficult to then keep those in sync, and it gets complex really quick, and there's no unified testing or infrastructure around this because it's all there and at some point it just gets really unwieldy so when we focus on configurability you can see right here this is a gnn that's a graph transformer that's a transformer and you can literally define let me see i should have a pointer here there we go so you should be able to define that you want the gnn here and this is just a config file. And because we're using Hydra, you can actually switch these out really easily because you can even do this on the terminal. And because we're using Hydra, all of this gets harmonized into one config object that you can then store in the checkpoints. So if you grow your machine learning projects, I highly recommend having a look at Hydra. It's quite useful. And these are just different files in your config in your package so the full tracking even if your terminal overrides it you don't have any of this funny oh i just want to switch this on the on the terminal and then you never know which value you actually used if you don't have it in your bash history so yeah and it's very very easy to extend these are three examples we now have a fourth example that is just being introduced in the PR where someone tried a different like attention mechanism in the transformer and needed a different model so then coders and the machine learning researchers can add on new things that get switched out super quickly and you can basically have these config defined factories yeah so this is making making our collaboration a lot easier because suddenly you can keep all of this, you can track all of this, and it makes it very, very easy to work with each other because someone isn't overriding your code changes that you push to the repository. You don't have to keep your branch, and it just makes work a lot nicer. Then, why do we use pre-commit hooks? You can see, committed on April 7th, scaling loss in proportion to pressure, and on the same day, one day I'll remember to black. I obviously anonymized this, but yeah, if you use pre-commit hooks, all of this happens automatically before you even commit or can commit. You can go overboard with pre-commit hooks, so be careful with it. They should run fast. They should not have all your testing suite in there. But, yeah, to make, like, collaboration really easy, something like this is extremely useful. So if you're starting a collaborative project, highly recommend looking at pre-commits or a similar tool that automatically lints your code, blacks your code, or whichever you prefer. Then why do we squash commits now? You can see, well, forgot the type hints. I had to do the merge, had to remove some compile somewhere, did a typo. And I like to hide when I do embarrassing things. So now we have a bunch of PR stuff, so we do everything over PRs. We basically have started the open source development before it was even open source, which also perfectly set us up for the next kind of steps where open source was necessary, Because member states started to use these and wanted to contribute as well. And you can see right here, then when you have nice CIs as well, you get your green check marks or red crosses, depending on how well you're doing. Then one recommendation from me, do code tours. It's a shortcut to learning. I have sat down with everyone starting new, basically had a one hour meeting every week with a new person, and I was able to show them the common entry points, how we do things in the code, what a pre-commit hook is, and, yeah, basically show them, like, the most important parts in the code. They could ask questions, and, yeah, it puts your design decisions in context. It can be tailored to specific needs. i had one colleague that is more working on gpu optimization then i looked more into gpu optimization with him and another colleague who was looking into making new earth components which meant we looked more into how can you change your data that you're training on so this is really really useful and it basically shortcuts shortcuts it from a multi-week process of digging into the code to one hour maybe two depending on your code base then it establishes collaboration, you're sitting on a call already, they're getting used to it. This is great to already establish that this is how we work. And optionally use tools, like VS Code has the extension code tours. I used those in the beginning. It can be a bit janky sometimes where everything is. But this is basically a code tour that you can click through. Once I set this one up, I actually used this to do the code tours. I would skip through a step, but essentially it's a step-by-step through your repository, super useful. I'm a huge fan of this. How do we grow beyond a single repo? So basically, AFS is our model, Animoid is the ecosystem. Oh, wrong direction. How does it look before? We have train with a trainer forecaster and all that stuff in there with the diagnostics, and you can see that the colors already are a little bit mixed. Because essentially in Anamoia itself, in our ecosystem, we want a data generation and data handling facility. We want the model. The model should be the core part that has the minimal dependencies, then the trainer around that can have plotting and all that stuff, and utils where you throw everything where you don't know where it is. You always have a utils. Yeah, so basically, how does all this migrate into that, and why? Well, it looks much nicer because now you can import from animoi.models, animoi.training, so you can share namespaces for this. If you do, wait, yesterday in the lightning talk, someone said, import this. One of the paradigms, namespaces are great. You should use them more. It's a great use of namespaces to have the common namespace and then be able to go into the different parts. So, ecosystem. We can do a multi-repo. That means we have full separation of concerns actually in the repos. We can do the unit test there very specifically. The structure is really, really pretty. It's intuitive to release individually. And the complexity is contained within repos. And you can delegate responsibility within your teams. Someone can work on the graph portion. and someone can work on the model portion. So that really helps. And if there are people that work differently, that have like a faster development style, that are more classically trained computer scientists, or you have people that are more machine learning and are more on that, you can actually give them a bit of space within the repo to have different collaboration styles. Cons, PRs across repos for changes are really difficult. So if you change the models and the training, You have to somehow sync them to go in together. And that can be really difficult if there are changes and merge conflicts and all that stuff. And then keeping the versions in sync is also a little bit tricky because you have to then pin, of course, the version and make sure everything is right. So releases then get a little bit tricky. working on develop or main your whatever you work on can break because one pr might already be in in one of the repos and in the other it is still waiting for approval can get out of sync the releases like i said is difficult and cicd is hard to set up like we've lost a lot of mental health on this, like, genuinely. I wish it wasn't so, so if you want to talk about mental health after, we can have a chat about that, too, but it didn't fit into this talk. But, yeah, unfortunately, I genuinely burned out on this. And, yeah, end-to-end tests are really difficult because you have to get your model and the training and then this is over there and how do you do you put like a like a integration test uh repo at the end so doing all that gets really really tiresome um yeah and different styles are possible but also knowledge is contained in each part so you have siloing kind of built in a little bit so you have to facilitate exchange of ideas. The alternative is a monorepo. And I just want to skip through this because we're running a little bit low on time and I want to show you cool stuff at the end as well. But basically, they're much easier to refactor and all that stuff. You have consistent coding standards across everything, a common state of all mains. So monorepos have some great properties, but the release cycle is really complex. Usually when we talk about Google or Meta having, like, a huge monorepo, they have specialized tools, right? They have a full team that develops I think it's called Brazier that is especially for handling monorepos because at some point this gets really difficult. Yeah, branching strategy is really complex, and it's also really easy to break everything. Like if you mess up a thing that is central to the monorepo, suddenly you break everything for everyone, and then you have everyone yelling at you, which is not great. But we decided for multirepo, as you might have guessed. This was a multi-week process. You can see I wrote like a full issue. You can actually find that online as well. We did work in the open, and we got a release, so that was really cool. A lot of work of getting the mono package into the mono repo, getting that over. But yeah, so that was a huge milestone for us. All the member states were also really happy because now they could contribute to this and work together on this, and it wasn't just our model, the ARFS, but it was an ecosystem that everyone could contribute to. So they weren't improving necessarily our model, but they were now contributing to something greater where they could build their model with as well. That's the part of going global. So we have a bunch of member states. We have a bunch of international collaborations that we use. And if you saw the small, oh, this goes over there, this is how the ecosystem looks now. This is just firing for effect. You can see all the complicated interactions, the silos of, like, artifacts down there. So it's a lot. And maintaining this is also a lot. And we figured out that we actually need a partial monorepo. Because in my analysis before doing the split, I already saw that training models and graphs are highly interdependent, because our graph neural network, the models depend on the graph, the training depends on models and graphs, and if you change one, you want to change all of them. And it got really, really difficult to make improvements, to build new models in there, because you have this problem of how do I do PRs, when do we merge them, so doing a partial monorepo now meant we had a monorepo for the core part of our training and model. The pros are we have some separation of concerns. It puts the tightly coupled code together. So, now, you still have the different styles of collaborations. If your data set people are differently oriented, they can work in a different style. The testing gets very, very simple, and the configuration management also improves, because now you, again, have everything in one place. The CI is partially solved, but not end-to-end solved because your data set is still out there. And I'll skip through a bunch of this. The slides will be available. They're also at the link, actually. I just uploaded them before. But, yeah, consider this as well, especially for machine learning code. You have this really tightly coupled code where you can't separate everything as neatly. And yeah, here's a point where AI actually failed me. So it was suggesting that I do this git branch, git checkout, git ls tree name only whatever. And we quickly realized that we could just do this with git subtrees. So when you do your AI agent shenanigans, consider that LLM specifically write really, really complicated code, and sometimes they just write silly complicated code for no apparent reason. Like, Git subtrees are in Git itself since version 2.1 or something like that. This is not obscure. This is out there. You just have to use the right tool. So yeah, when you do these things, be careful, especially when it's this fancy stuff and especially with Git, LLMs have failed me quite often. So that's one of my cautionary tales, essentially. Then with Hydra, you can extend this because we're going global. So we maybe want some organization configs. Maybe all your data is in a certain spot, you want to put that in the config, but you You don't want every one of your employees to have to change this in the config. So you can actually put different parts of the configs in different places. Just to consider that this is something that you can do with Hydra, so it is really nice to extend this. You can have your defaults in Anamoia itself, in your repo yourself, and then you can escalate this up. write the configs on an organization level and on a local user level. It's not perfect, but it's also a really difficult problem to solve. Then automate your releases. We still did releases across all our nine packages by hand. My Lord, that sucked. Now we use release please, which is from Google, but there are others like semantic release and several Python packages, we decided on this one because it handles monorepos. And the cool part is you also have like a change log in here. So people can keep up to date what's going on. So once you get to a certain stage, have a look for automated release tools that save you so much work because otherwise you're going to go crazy. Like if you have to do like multiple releases across multiple packages, your life is going to get really, really bad because one person will have to do it. They're like the release manager. And if you do frequent releases, it's so many clicks. And now we just merge this PR and this PR automatically triggers the release process. The change log is updated automatically. And yeah, just something to consider. This is the solution that I landed on with deliberation with some of the member states that have worked in other places. But, yeah, release, please. Difficult to set up the documentation, as most things Google is not that great. But once you get it working, it's a delight, because with a click of a button, you have a full release, and you have automatic increases of your semantic versioning and all that stuff. So, yeah, huge fan. Then external and global collaboration what you think of Again personal touch kick things off with a hackathon. You can even do a code to add the hackathon So what you've already been doing with your colleagues you can now do as a presentation Maintain roadmaps and Kanban's and all that stuff. I mean the project management Inspired people I'm sure they're like yes, we do that anyways, but that was more for us and then code reviews on prs are essential think about doing dual dual reviews so one for code quality and one for in our case scientific viability but also for a business case and maintain configurability this is really important for us modularity extensibility automate what you can to reduce the mental burden and don't be afraid to jump on calls So, a couple of success stories. We're closing out now. We were able to run this remote at some point, so now we could even push this to the cloud if our entire data center loses network or loses power. Then we are starting to get fancy Earth system components in here. This is the wave height. So one of my colleagues actually added this to the model. And I just really love how this looks. The funny dots are because we're using a different grid. So this is not an artifact per se, but it's because the Earth is a globe. That's the reason why we use a graph neural network. Then our whole process is tracked through everything. So even your training checkpoint knows how to recreate the data set, because you need to know how to create the data set to create the model. And yeah, we're enabling all types of graphs. If there are any physicists in the room, this is just a very small grid in a large grid. And because we're using graphs, it's not a big, like, oh, how does this work? We just define the connectivity. But this is the boring version of this. This is the cool version of this, because Norway have actually used this to use a coarse global model. You can see these pixels are really big. To have a very fine model over Scandinavia and predict, I think this is wind. And you can even see how it perfectly shows at the fjords the wind is basically stopped a little bit. This is one of my favorite animations coming out of this entire thing. So as for the takeaways, this is a slide that you probably want to look at when you have a look. But essentially, for me, everything between multi and monorepo is on a scale. You make certain tradeoffs, yes, take pictures. As for the learnings for growing software projects, get yourself a software architect or two. Do code tools and in-person events. They're basically non-negotiable. Make decisions that are easy to reverse. Anticipate user needs and extensibility requirements by defining who your user is. And then you can see, oh, yeah, they will want to do this. So you can already leave the loose ends in your code so they can plug into that loose end really easily. Make design choices that are easy to extend, lose ends, and don't just let AI agents run through your code just yet. Repro structures are difficult. Use tools to make it easier. Keep dependencies light and collaborate and listen. And this is our core tech stack, if you're curious. And all of this is open source. So you can actually see how all of this grew, and you can try it out i put the hugging face link on the um on the link thank you so much for your attention thank you for this great talk so let's start with the q and a um how can the model be consumed and which is the best overall source for weather features to predict for example the production of a photovoltaic park i think photovoltaics are difficult because they mostly want to know about clouds and clouds are still one of the most difficult parts we do have them in the latest version of afs so our our forecasting system but essentially you want cloud cover for cloud cover and irradiation values for photovoltaics, I would say. So that is something that the model needs to be able to do already. You can't add that in the end yet. How to convince colleagues coming from non-software engineering backgrounds of the value of pre-commit, linting, formatting, and so on? I did that in the easiest way. You have the one that basically removes all debug statements for them. And once my colleagues were committing their debug statements and it said, no, get rid of those, they realized they're making less of a fool of themselves because they're not breaking the entire code base. So it's really, it's show them the convenience. Because don't put too many in the beginning that go in the way. Keep your rough fairly light. Keep it with some of the minimal ones. but make it so that it is a benefit to them. And then you hook them and then you release hell. So let's take a last question. How does the model perform in comparison to the DWD Icon D2 model? And how does the DWD use your work? So DWD is a part of the machine learning pilot project that is developing Animoid. They have already committed great things to this as well, including like a hierarchical graph, which is really fancy. I don't know how they perform specifically against their icon model, but I know that our AI model outperforms our physical model. Also, it's a very political answer, so I will be careful to say, oh, we're beating everyone. No, we're beating ourselves, and that's good enough. So let's thank the speaker again. Thank you.