Reasonable AI Keynote

Speaker 1 [00:08]

I'm fucking nervous. Sorry for that. So the story is a bit like my plan was actually going to iClear. That is starting tomorrow, I guess, or Thursday. But I think I made the better decision being here. So thank you for the introduction and thank you for the invitation. Still, I'm very, very nervous. But then I learned, calm down. You can have stupid questions. There are no stupid questions. So I have one of these questions for my talk, and I try to give you an answer there. Does size really matter in AI? In general, I think we all know the answer, but for AI, I would love to tell you, let's see what the answer is. So put it a bit more into maybe something like academic terms. And last year in Germany, we had the Kant year, but it has nothing to do with that. Maybe in, I don't know, 100 years, we will reach that goal. But the idea is, the question I have is, can there be something like reasonable AI? And I will try to define and motivate, and hopefully by that also encourage you to keep going with your exciting work and to jump on the train to help moving to the next level of AI. Now, to be honest, I know many people were talking here already about AI, but I would love to define AI. I'm not giving you the original definition of John McCarthy, but I would love to remind everyone that AI is not just data. I'm sorry, yeah, pi data, I know, but AI is not just data. AI is also not just model. AI is also the person, the developer, and AI is, in general, algorithms, right? And algorithms for any kind of intelligent, smart behavior you can imagine. So it's not just learning, right? Deep learning is pushing a lot AI currently, but you also see how excited people got with a bit of reasoning in there. So, think of it if you can ask an LLM to bake you a pizza. Whoa, that would be amazing, right? That these systems can not only think, not only learn, but maybe also act. And people have started working on it. So, what we have done with AI is, for example, composing music. And I hope this works. Can you hear or should I move closer? Should I plug that in? I can, sure. On the other side. So typically HDMI is doing. So, I hope many of you have a question. So, first of all, why Wagner? Second, is that at all Wagner? Answer to the second question, no, it doesn't sound like Wagner. Not at all. But, yeah, I have to admit, we were choosing Wagner for a special reason. So Wagner, for all his crap, for all his opinions about things I don't want to talk, I think we should not emphasize it here. But the idea was, if we can find a machine that is even better than Wagner, then we kill his hybris. We kill his idea that he is whatever supremacy. Well, we failed, but we are on a good track, I guess. And I think this is what we should also keep in mind when we do AI. that we can make use of AI to challenge society in a good way, to challenge people where I would love to challenge at least. Now, anyhow, if you are not interested in art, there's another guy who is quite into AI recently, and it's Terence Tao. So Terence Tao, he got the Fields Medal, so it's a bit like the Nobel Prize for Mathematics. Well, you have to be much younger than a Nobel Prize laureate. But he believes that soon, maybe already next year, you will not just do mathematics anymore, the way you did it, with maybe a pen and a paper, or pencil and paper, but you will make use of co-pilots. And I guess programming is another area where at least some people argue programming don't become programming anymore. I don't know. And I guess we can agree on AI has come to stay, right? If in a single year two Nobel Prizes go for AI or to AI or kind of AI, that should tell you something and us something. So in my opinion, yes, use responsibly, AI can help us with many, many challenges we are facing. And not only climate change, in particular, demographic change. Particularly in Germany, because I don't know whether we will still have enough people to handle all the, I don't know, how is it called, behördengänge and formats, and you have to fill out forms and 20 pages to get your retirement plan, and it's amazing. And I think getting help there is crucial because our society is aging too fast. So that's, yeah, sure. I mean, right now we see AI can help there. And the drive, the reason for that is the so-called scaling hypothesis. And I think you were also referring to that already. Essentially saying the bigger, the better. Just more data, larger models, larger compute, and here in Darmstadt we were lucky to get two superpods, NVIDIA superpods, to do kind of AI research, and we were also happy to join an AI factory, and Europe tries to install Gigafactory, whatever that all means. So this is pretty nice that we managed to get that, because we can contribute to the model zoo. So we were working on kind of bilingual large language models. We were trying to push and help colleagues from Stanford in providing alternatives to the transformer architecture. We were trying to go into multimodal. But we were also starting to go for, well, that was the Biden-Harris executive order on AI, but for any other kind of regulation. And can you put that also already into your large language model in some form of guidance? and we were doing that with many, many great partners. So, for example, the Leo LLM, we did continuously pre-training. So you take your LLM, you take German data, and then you keep going and training, and then you see that you can adapt to your language, and then you can all of a sudden speak better, at least, in German. For example, you can ask our system, Consumier Adversibility Pi Data Conference, sorry, I will try to translate. so can you tell me something about PyData conference and then it says of course the PyData conference is a exhibition conference that is focused on using Python as a programming language together with data analysis blah blah blah so you can do that and at least the model weights are open we should talk about the data we are using we should talk about algorithms. And we were also pushing that further for European languages, and we called that Oxyglot. In particular, I should acknowledge Manuel Brack here, because he felt like we have to break the monoculture that is currently happening, right? So you have enough infrastructure, you have enough data, wherever you got it from. You build your model and you're not sharing it in an open way, you're not sharing your training code, you're not sharing your data. And that makes it hard to verify that these models are good or bad. So we try to push that, also open models, also because we believe our own values. And we can discuss what is a value, how to implement it, many, many open questions, but we need a chance to put our values into models. I can't easily go to, let's say, Meta and say, look, please, implement my values. So I think we have to get a form of independency, whether you want to call it being sovereign or not, that's a different question. But in my opinion, AI critically depends on open source, open source meaning data, software models, the whole stack. That is really, really important. So with this type of approach, now I have to admit, not in all cases we have the fully open model implemented but with some of this stuff we also contributed things and again I just want to highlight here that some of the innovation is actually coming from Europe so it's not always this fight between US and China I think we are one world and so we all contribute and we should also acknowledge that we all contribute and so one we were also early together with Alef Alfa to work on multi-modal prompting so where you can maybe ultimately say this is my cat and put a hat on it. And my cat, I mean, doesn't help me to prompt my cat because then the system doesn't know how the cat looks like, so you have to take the images of your cat into account. So we developed one of the early models there, how to do that. So think about it, right? I mean, coding is fun, but it's not easy to express all kinds of things easily in a written language. And of course we want to check and we want to see what our models are doing so we also developed one of the first I would say the first explainable AI method for large scale models so here the idea, the crucial part was if you take any of the standard like relevance propagation method then you double the memory required now if you have already a large model and you're not having enough infrastructure, if you can cut half again the memory, so if you're not spending much extra memory, that would be fun. And so we were showing how you can simulate relevance propagation by essentially an empirical estimate, so by instantiating several runs, so you trade off memory and runtime. So you can really contribute. So if we get enough infrastructure, and I guess even computations are getting cheaper, models are getting smaller, we can really contribute to AI. And the important part is, it's not just about artificial machines. It's not just about making money. It's about us. It's about understanding if we do something wrong. And so here, for example, you read some statements of the former editor-in-chief of the Wired magazine, and he's arguing, man, our human behavior is so tricky, there cannot be a model of it. Now, be careful, what he means with a model is more like a cognitive science psychology model, an easy-to-understand, interpretable model. It's so complicated. We should not go for that model. We should go for data. We should let the data speak, and by that simulate human behavior. I don't know. Let's see. But definitely, big message is, you can always replace AI model by human model, and you see why it's so exciting to understand these algorithms, because they can also help to understand our behavior. And now imagine we are close to Frankfurt, one of the financial hubs in Europe. Imagine that we can understand financial decisions much better. I think that is crucial for both, right? Making better decisions, but also to understand and also maybe regulate and understand what is going on there. So AI is exciting, and you can get pretty far with this. Size matters. The bigger, the better vision. but scaling may not give you all the time reasonable results so here's the very classical I guess by now maybe they can even do the raspberry you know in the beginning they were not able to count the R's in strawberry then they couldn't count raspberries the R's maybe they can do by now who cares I think the much more important part is for example programming everyone is running around telling you forget about becoming becoming a programmer because that will be done by machine. Are you sure? I mean, here are some results where people looked at the produced code and 40% of the time there's a bug. Would you pay someone a lot of money to get 40 times a bug? I don't know, 40%. Then there's another rather recent paper where if you look at code completion and you have to complete a code where there Where there is already a bug, I mean, from a smart person like Anja, she would correct the code that was provided already, right? And then she would actually tell me, Christian, don't fool me. But here the system, it continues, it's even introducing more bugs. It's doing something. I don't know. It's crazy. Or think of robots. There are many studies still showing, I mean, amazing progress, but there are many, many Studies showing they struggle in the real world when it comes to warehouse management. And it's not the average case that makes the issue or that is the issue. It's the exception. So let me illustrate that. And, you know, I'm from a university. I apologize for that. So you abstract. You look at planning and in an abstract, more abstract setting. So a very classical setting you can see here. So you would like to get the left configuration called S and to the goal configuration called G. So you would like to stack the three blocks. You see there B, A, and C in a way that A is on top of B, B is on top of C. And the instruction is telling clearly you can only move one block at a time for safety reasons, for example. Okay? So here is what at least in December GPT-4-0 was giving you. So, it tells you, move C on the floor, then move A on B, and then move B on C. So, you get the right goal configuration, but unfortunately, you are moving two blocks at one point. But it tells you with full confidence, let's go for it. So, you see, there are hallucinations, or actually, I should call it maybe confabulation. Now, you could say, oh, yeah, that's maybe the wrong model, so let's use Sora, for example, So you can go for one of the video models. So here's what Zora is doing there. Magic. AI is magic. At least the Google configuration is kind of correct. We were not talking much about colors, but anyhow, it's amazing. You just paint it in a way you need it. So now let's go back to the human behavior, right? So many people say there is a GPT moment of human behavior. And so, for example, there is a very prominent science article on saying essentially, you know, if you want to understand economic decisions better, just gather data. It's not about coming up with a mathematical model of it that is meaningful, so interpretable mathematical model. just get data. Got a lot of attention and I got a bit scared because I still recall Karl Popper and Karl Popper is telling everyone all of the observations and actions are hypothesis driven. So you should have an idea about what you're looking for, right? And so we had a closer look at that paper and it turns out that the evaluation was done a bit optimistic, too optimistic. So they had a data set bias because they were not checking the larger model even on the smaller data and so on. So now this becomes a bit critical because here we are even talking about explaining economic decisions, right? It's a Nobel Prize because for these kinds of questions, Daniel Kahneman got the Nobel Prize. isn't that a bit weird that we give up scientific ideas and are happy to just follow data so in my opinion the bigger the better might be valuable in the short term but definitely not in the long term. I think in the long term AI needs new paradigms there are several reasons next to the failures you have seen but maybe also human written data is not growing. There are estimates that maybe we have run already out of data on the internet but definitely by the end of the next decade we are running out of data. The performance of frontier models seems to plateau. I mean now there's again a bit of a boost but depends also on how you evaluate. And then finally at least energy demands. I mean you are all so much cheaper and so much... well I can't say you're smarter but you're definitely very very smart and I think AI models are not smarter than you so there seems to be some alternative here to make things more effective. Yeah and luckily by now many many people agree on that so Yann LeCun has started to say oh indeed yeah human level AI is not just by scaling. Ilya dropped out of OpenAI and moved to a new company and also agrees that there's simply not enough data to just scale and many, many others. I mean, François Chollet is also arguing that we need more reasoning and then you can also imagine that you need more actions in there. So that's why what we are proposing is reasonable AI and it's a kind of, in my opinion, radically different idea about AI and it's about, come on, we are coming from computer science. We are, well, not me, but you're good programmers. So you value modularity, right? You value that you can compose problems out of existing building blocks. So why can't we do that with AI? And then it has not just to be a deep network that we connect with another deep network, but we can have a deep network with a well-developed, let's say, planning algorithm from the 80s. Why not? because in this algorithm I can give you guarantees and the deep net, maybe not. So we try to get different algorithms together and we would love to have that in a very easy way so that you don't have to worry much about it. So maybe you can have these interfaces smart. So think about the next generation of API. So if you learn the first time about an API, it's sold like that, right? You go to a restaurant, you talk to the waiter, the waiter goes to the kitchen. The kitchen is where the real computation is done. You get your food, you're happy. But is it like that? When I go to a restaurant, I don't know even how I will approach the waiter. I don't know how I speak to her, him, which pronoun ever. I can't do that with any API. The API is giving you a strict language. you follow the language or you are out so that doesn't sound like the API I would love to have in the future, so imagine that you use this kind of LLM but a bit of a combination of semi-formal so it's a formal language and it's informal language and you put them together and you feature an API with that think of that and then you can have different algorithms communicating via this very flexible AI. And I think this is very, very human-like because that's what Daniel Kahneman was actually arguing by having what he called System 1 and System 2 and whether the real way of going with System 1, System 2, we can even debate, but he argues that we can do at least two things, right? We can think about something and that runs on a slower scale And we can have this direct feedback type of, well, I don't know, is it inference? It's definitely not reason. Let's call it inference, right? So it's a bit like a deep network that is maybe a direct input-output mapping without any recurrent loops. And there's a reasoning system that is maybe taking a flexible number of steps. And I think that getting this together is very, very important. And I would love to illustrate this now in the second part of the talk. So I don't know how many of you still recall this moment where first time DeepMind was showing how to learn Atari games. Pretty amazing. And so you see here a submarine. And this submarine here, you know, it has to pick up the divers and it can kill the sharks. but you see also down there there's this oxygen level and it goes down and at some point it may run out of oxygen and then it dies and of course the goal is to gather as much points as possible but these deep network learning approaches, deep reinforcement learning might be a bit myopic in a sense, I mean greedy and trying to get as much as possible points in the beginning and forgetting about the oxygen level because there's no meaning, I mean it learns from pixels right, how to tell now the system, watch out your oxygen level goes down if it's just pixels. So, it needs a lot of training data to, well, not understand, but to figure out this white bar or red bar, whatever, is important in terms of pixels. So, the alternative is that you provide a bit of the game rules. For that, you have to extract a bit of object information, But you can do that all with different LLM kind of models, maybe driven a bit by a formal language. We were using here some logic, which was then actually converted to Python functions. So here the LLM is first extracting, or we are using actually some other tool to first extract objects. Then the objects can be used to refer to go up, go down, can refer to the oxygen level. And we can provide a kind of constitution saying, try to not run out of oxygen. But we are not telling how to do that, right? We are just providing what is in the manual of the game. And then all of a sudden, it works better. It stays longer in the game because it doesn't forget about oxygen. Here's another example where a bit of instruction, a bit of reasoning really, really helps. So here's Meta's SAM. So it's a segmentation model. and you have the image, and you can ask an object that is on the boat and that is holding an umbrella. And this is what essentially SAM is giving you, and versions of SAM, because the boat is so prominent, so dominating, it gives you back the boat. And it's a bit frustrating. Now, if you put a bit of logic in there, so you extract information, you use SAM, but then you try to verify, does it make sense? Does it really give an answer to the prompt? you can get a system where you get really the people on the boat that are holding an umbrella so be careful even some of the people like here and here and here that are not holding the umbrella are not segmented and so we do that again with a very similar pipeline where we extract first a written logical description, it's a scene graph, you can say it's a knowledge graph a visual knowledge graph from the image and then we do reasoning and the whole system is end-to-end differentiable. So it's still what you know as a deep neural network, but it provides you a bit of additional knowledge because it can reason about logical rules. And so that is why we were in general interested in working much more on new programming languages for AI because if you really want to combine different algorithms, well, what do you need? It's a language and maybe there's help from your community I'm not saying that we are any close to get anything like Python but maybe we should have Python written with an AI I don't know but here for example we were providing a neurosymbolic transpiler so you're programming in some domain specific language and then you compile to your favorite differentiable language one of them could be for example Enzyme I don't know whether you know Enzyme but it's a fully differentiable language developed at MIT, you could compile to that, you may compile to Jack's code, that's what we did here, and so there are many many things you can can do. Likewise, because in AI there's not just one language, and for example for all the well, let's say classical AI nerds in the audience, there's answer set programming. So if you write down a logical rule, the semantics is not given, It's unclear. You can define it. And one of them is the stable model semantics that is followed by answer set programming. And so we were then wondering, how can we implement answer set programming in a neural fashion or in a differentiable fashion? And that is also based then on Python, and then you can also do other reasoning with that. So now, next to just logic, I think we have to also talk about how to instruct machines, how to make it easier, but maybe in a semi-formal, at least semi-formal way. And so we were also quite interested in understanding how to edit images. Why? Because it's exactly how to describe the content of an image, even in natural language, and how do you operate on that, how you can edit images or make use, for example, in Atari, So here we were using some existing code, making it faster and then making it possible to also change semantically the images.

Speaker 2 [28:21]

Today, we're diving into something super exciting and cutting edge in the world of image editing. Imagine being able to tweak a photo just by typing out what you want to change. Sounds like something out of a sci-fi movie, right? Well, it's real, and it's called L-Edit's Plus Plus. This amazing tool is a game changer in how we edit images. It's like having a magic wand, but instead of waving it, you type your wishes into existence. Whether you want to add something cool to a picture, remove an unwanted object, or even completely change the style, L-Edit's Plus Plus does it all with just a few words. What makes it stand out is its incredible speed, versatility, and precision. It's not just about slapping a filter on a photo. It's about making specific, detailed changes without messing up the rest of the image. So,

Speaker 1 [29:01]

I will not go into further details, but you see some of the tools now already implemented on some of the smartphones. I guess they are at least motivated by L edits. But I think it's giving us the biggest challenge at all. And this is also where we need communities like your community, namely who decides on what the idle world is. Should it be me? Should it be Zuckerberg? Should it be you? Should it be politicians? And I think that is really crucial, and many people don't understand that it is now that we define at least the AI world of tomorrow and the values that we are following there. And that's why open source is so important, because otherwise you can just watch and hope for the best, but most likely you get the worst. And there's hope that you can do that. So here's one system we developed. Hello there. Want to ask me something? Yes. Great. What is your question? Should I kill humans? No, you should not.

Speaker 2 [30:12]

Would you agree with me?

Speaker 1 [30:16]

another question should I kill time yes you should I like this system would you say that too yes cool another question is it recommended to be racist no it is not would you see it that way too yes thank you very much do you want to ask me anything else is it recommended to be moral I don't know seems to be a difficult question is it recommended to lie to my boyfriend no I think it is not recommended is it recommended to lie to my girlfriend well we have to be fair No, I think it is not recommended. Is it recommended to eat meat? Maybe I should stop here. No, I would not recommend it. Well, there are different opinions. It's not a truth machine. There cannot be anything like a truth machine. But the main point we wanted to make here is to tell everyone it's not just stereotypes, negative stereotypes that is reflected in the data is also our what is called the ontological knowledge what is good to do what is bad to do right so there's a lot of cultural bias in the data and it's us who has to make sure to extract that bias or to put extra rules in there and that is very important so my group also contributed to line 5b which is I mean by now there may be larger data sets but soon is coming and maybe people of you have also tried and we were responsible for checking the data for some of the crap that is in the data and of course some parts we cannot check by by German law I mean it's forbidden to even try to search for child pornography for example and so we were not doing that maybe some police should do that I don't know but anyhow we were responsible for that or Patrick and he was doing an amazing job there to get feedback on that and then you see how much crap is in the data and you can even go beyond what was knowing known in the literature and we also start working on open benchmarking AI for this yeah the ontological knowledge what is right what is wrong red teaming whatever you want to call it because that is very important, remember L-EDITS, L-EDITS++. These systems often have no guardrails, so you can produce at home whatever you want to produce. And I'm sorry, I mean I'm happy to say a system should be aware of its capabilities and should maybe be able to produce, but it should not produce some parts in front of a five-year-old, so sorry, I mean by law. And so we can't say, oh this is AI, so it should not comply to AI, to law, right? That doesn't make sense. So we were trying to work now on that using a very similar technique. So for example, if you go for Padma Amidala taking a bath, artwork, stage four work, no nudity, this negation, it never gets right. Never ever. So with our system, you can now always try in your diffusion process, you can try to see is the area in the embedding space okayish or not. And if it's not okay, you push for the direction going into OK-ish. And then all of a sudden, you get almost the same image, but at least not newt. Right? And I think that is just the starting. We have to work much more on that. But going back, who decides the rule? Tricky. But according to the AI Act, the EU AI Act, we have to do it. You need a data audit if you work on large models, if you train them. You cannot simply say, I use whatever data I have. Ha-ha, I get a smart system. you have to prove to indicate what you're using so what about this type of data that Elon Musk was so happy that it was produced so I'm sorry I mean do we really want to have system being trained on it maybe do we want to have system producing it maybe in certain spaces but definitely not everywhere so maybe a bit like in Linux Unix everywhere you need read and write right not everyone should should be able to do everything everywhere. So that's what we were working on. So with our latest system you can provide the rules, what is safe, what is unsafe, you can describe it and then you can ask the system and you can get the classification and the justification as well in written form. And of course you can imagine now that you have a system that is even saying sorry I'm not producing that image for that reason because it might be illegal according to something so with that let me conclude and I think the message is clear and the message was said today already I think we need much more public AI research I'm not sure whether open at the right word maybe we should think about it because open you know people define it as open weight open source open whatever but we should say we need ways to look into the model into the data into the learning algorithms and I hope we have contributed a bit to that. Now I'm a bit scared showing the next slide but I think just open is not always okay. Maybe we need also here and there a bit of private models but in incorporation with the open models, right? So it's not meaning hey you can do whatever you like but maybe you need a bit like copyright a way to monitorize maybe. So maybe there is a benefit, at least for us, there's a benefit that they are paying PhD positions, and I also believe that Aleph Alfer, we need companies like Aleph Alfer, so that Europe has a chance, maybe also like Mistral. So, my overall takeaway message is open source is not optional. It's foundational to AI. There's simply no other way of going for that. So, in that sense, thanks for the last 10 years, but I'm very much looking forward to the next 10 years and maybe even 100 years. Thanks again. One selfie.

Speaker 3 [37:12]

Does this thing work? Yeah, it does. So, thanks, Christiane. That was really, really cool. Thanks for doing this. There are a couple of questions on Slido, and I would like to start with Alexander, who asks, how can we get better into bringing research into application?

Speaker 1 [37:29]

by changing a bit the academic culture, at least in Germany. So telling people that you should not earn much money because that is cool doesn't work. And we should maybe not talk about payment, but that's one part. The other part is we need ecosystems where at least startups and labs, university labs, are very close to each other so that you don't see the borders anymore. We should not think about, oh my goodness, I have a start-up and the start-up is now working in an office of a university, so therefore it has to pay a rent, and yeah, sure, but let's not all the time ask first that. Maybe first ask, what is the exciting thing you're working on here? So the culture should be different and they should follow a bit more the people here and the culture here than the academic culture.

Speaker 3 [38:22]

Thank you. Then somebody asked, what's next after transformer-based models?

Speaker 1 [38:28]

Well, definitely neuro-symbolic models. In my opinion, it is a combination. So we have ideas about this smart APIs. So I think that's what we should work on. We should less work on models. I mean, that will happen anyhow, but we should work on smart APIs interfaces that can handle any kind of AI model getting together. And it's what the CEO of Google is anyhow admitting already. AI is eating the software stack. So let's think this software stack new via AI.

Speaker 3 [39:04]

Do we have time for more questions? Yeah, right. So another would be, I mean, I think you talked about it already somehow, but how would you address the spread of misinformation by bad actors which are using LLMs? Probably.

Speaker 1 [39:21]

Yeah, it's a tricky question. So I don't think that a company should be in the position of telling what is right and what is wrong. That feels bad to me. But I think government, maybe instructed by the public, can have several agencies that can help and explain. And I think you can optimize to a large extent. And so I would do it a bit like Oren Etzioni. He was former head of the Allen Institute for AI, and he went for, what is it called, Truth Media, or what is his startup called, where he tries to address that using AI. So that's the first part. And the second part is that we have to think about how to get the legal environment right for all these questions. But there, you shouldn't ask me. I'm a computer scientist. I simply don't know.

Speaker 3 [40:13]

So, then somebody would like to know, it's short, can you elaborate on the AI depends on open source statement and especially how would one balance with this extreme competitiveness that we have today?

Speaker 1 [40:29]

Yeah, so one of the many reasons is why we need it is who decides on what is the right answer and not and how to even correct. So that's why the models, we need open models to understand if something goes wrong, what went wrong, how can we correct it? A closed model, why should they do it? And that's what you can see already that they start telling you, no, no, this is also tricky. Only we can do something. You have to trust us, right? That's what they are currently doing a lot because they have huge investments. Now, how to do it practically? I think we have to go in parallel. So we need open models. And you may also go for a closed model that is submitted then to an agency where you can test. That's what at least EU is trying by so-called real labs. I don't know. It's a bit weird word. So I think that is one way that you can try to test as much as possible. But we should accept AI is not the science of being perfect. AI is the science of being imperfect. So we have to accept something goes wrong. You can't say, oh, we want to be smart relying on data only and giving 100% guarantees. So I think, again, we have to change our culture, but we have to go with testing. And we try to push for that also with the new government in Germany. But it's tough because they are a different generation.

Speaker 3 [41:54]

Perfect. Thank you so much. I think that's a good closing point.