The Future of AI: Building the Most Impactful Technology Together Keynote

Speaker 1 [00:07]

Thanks a lot for having me. So I have the challenge this morning to give the last talk on the morning after the social party. I try to make it as entertaining as possible. And I want to talk about potentially one of the most impactful technologies that is being built at the moment, which is AI. But I want to talk about if there is a future where we can build it collaboratively together in the open. And what are the kind of things that we need to get there? So if you want to build, like, very powerful language models, you need a lot of components to get, like, to a state-of-the-art model in the end. You need data, on the one hand, like high-quality large-scale data. You need a lot of compute to train these models. You need to know how to pre-train them and select the right architecture. You need to be able to scale those models at the moment. Like we're in an era where scaling models to larger sizes still yields better models. So there is a little bit of know-how involved there. And you need to understand how do you post-train those models. In the post-training phase, usually you take a base model that has been trained on a lot of web data, and you make it really useful by making it like a chat model and also aligning it in some ways to your values. So these are kind of the key components that you need, and I want to talk a little bit about if we can build those components together in the open. And I'm going to split my talk roughly into three pieces. I'm going to start with why should we build open language models at all? Then I want to go to the other consequential question is, can we even do it? Is it possible to build state of the art language malls in the current landscape openly? And then I want to take a step back and look a little bit into the future. I put on purpose near future. I think at the moment nobody really knows where things are going. So me personally, I don't plan further than like three months into the future anymore. But I want to see like what are the things that are coming and could be interesting. So let's start with why should we build open language models at all? I think being at PyCon, it's probably a bit preaching to the choir, saying like open source is like something valuable. But I want to highlight a few things that are unique about AI models and why we should especially care about making them open since they're so persuasive. And it's going to be a little bit about values, building knowledge, trust, and also driving innovation. And I want to start by showing you a little bit like the spectrum of openness in AI, as opposed to like open source software, the term open source is like very overloaded. Some people refer to open source when actually the source is not really open. So on the one extreme, we have the closed models. The closed models are the models that you can access with an API. It's your favorite, like, cloud model or Gemini model or whatever open AI model you might be using. Currently, still, some of the most powerful models that are out there. But they're also somewhat limited. So you can't download these models. You can't run them locally. You can only access them through an API. So you only have the functionality that the model provider puts into the API. So, some of them might offer fine-tuning those models on your own custom data. Some of them won't. Another challenge that arises is that these language model providers, they deprecate their models. They can't host all the models they ever trained forever. So sometimes people get really used to one model generation, and then at some point in the morning you get an email that says, like, this is going to be deprecated in a few weeks. Then in the middle, you have the open model, and especially like in the past few months, there has been more and more talk about the open model weights, like the central group. These are the models that release the weights of the model, so you can download them, you can fine-tune them, you can run them locally. But you don't really know anything beyond that, so usually they don't really release the training code, they don't release the data set that the models have been trained on or maybe how they have been post-trained. So you can use those models in many ways, but still there's somewhat a black box. And then on the right-hand side, you have like the fully open model, and I would say that this is probably the closest to like a fully open source, in the fully open source spirit, where you get pre-training data, maybe even the pre-training data processing on how you got there. You get the training scripts, and in the end, you get also the models with, like, a permissive license. And there are a handful of them. There's Granite by IBM. Olmo is developed by the LNAI Institute. And two projects that Hanging Face has been worked on a little bit is Bloom, which was an open, multilingual GPT-3 model. And StarCoder 2 is one project I've been quite involved in, which is like a fully open model for code generation. So this is a bit the spectrum, but why should we really care going from open model weights to a fully open model? And there are a few reasons. The first of them is you want some transparency on the pre-training side. There are many questions related to pre-training data that are really important for users. So on the one hand, you want to know what kind of data was used. A lot of devices that are introduced into language models origin from the pre-training data. There's one very famous example. It's called the C4 data set that Google developed. It's used for pre-training. And they had very strict filtering on the data set. And one of the filters was they removed every website that contained the word sex because they thought that's how you remove porn. But on the other hand, it removed all Wikipedia pages, for example, that also had that term. So a model trained on that data set has no concept of sex at all. So that obviously is like something that's introduced in the pre-training data. And if you don't know how the pre-trained data looks like, you have no idea what kind of behaviors might be imprinted into the model. As someone who creates content potentially on the web or maybe writes code, you want to know if your data was used to train those models. You want to know how it was used. And potentially, you might even want to have an option to say, like, I don't want my data to be used for such a model training. So, being transparent about the pre-training and the pre-training data addresses biases, addresses attribution, and also builds up trust because you start to understand, like, based on what the model was trained on, what can it do and what can't it do. Then I mentioned a bit earlier, like, an important step in building a language model is the post-training. So you take a base model that has been pre-trained on web data, and then you fine-tune it or align it to be a more chat model that follows instructions. Usually, the big model providers, they put a lot of care into safety, so the models are not helpful to do dangerous things, for example. And they also align them to some kind of notion of human values. But it's a bit hard to know what these values are. There is zero transparency about that. And I found this interesting study recently that looked at some social and economic values that ChatGPT has, so GPT 3.5. And you can see the model shifted quite a bit between, I think that's June and November. So within five months, they released a new model version. And you can see those values have been shifting. And I'm not saying generally like this is a good or a bad thing, but I'm saying like you want to have some transparency. So you want to know what the values are of the model that you're using. And also, potentially you want values that are more fine grained. You don't want one model with one set of values to serve like the global market. Usually people have values based on countries, based on communities, based on age groups and so on. So having like a one-size-fits-all is potentially not the best thing. And lastly, also, at the moment for the API providers, it's like just a handful of people who write the kind of rules on how the model should behave, but then it's rolled out to everybody, and it's not really like a democratic process of defining the values. And so to highlight a bit more like the advantages of like the open weights model, which are like semi-transparent and the fully open source ones is that the open weights model, you can fine tune, you can run locally. These are already like really good starts, but you have no idea if the models have been contaminated. So they usually report numbers on benchmarks, but if you don't know how they've been trained and how the decontamination worked to remove the benchmarks from the pre-training data, you don't really know if they're not by accident trained on a test set or on purpose. And then if you have a fully open model, the data is available, the training code is available, and that really fosters innovation. because on day one, everybody can start building on top of whatever that team has been building, so everybody can improve their recipes, can build better filters, also builds trust in the system because you fully understand what went in there, what decisions have been made, and that just accelerates things massively. And I've been giving similar talks for the past two years, and it's always been a little bit of an uphill battle, But in the past six months, I think the very public perception has changed a bit. And part of the reason was the release of the DeepSeq model, which by accident crashed the stock market for a few days. But it got everybody talking about maybe we could actually have very powerful open models. I mean, this is also not a fully open source model, but at least it shifted the conversation a bit from we only have API models towards we also have um open open models that that are very strong um and i think people also start talking a bit about more how open source is not like a zero-sum game where like you release you either win the ai race or you lose it but you can release something and you can benefit from it as well and it has been very popular in the community and millions of people have I've downloaded it, fine-tuned it, and it's still in very heavy use. One thing that we worked a bit on in the research team at Hagniface is I said the R1 model is not really fully open. They released the model weights as well. But we wanted to rebuild and reproduce the things that they didn't release. So we started the Open R1 project, which was really targeting rebuilding the training code the training pipeline to do kind of an R1-style training, and also like building the data sets that they generated and used to do these things. And it's been very fun, right? A lot of people joined and helped us do it, and we're making a lot of progress. So we're very, I would say, in a very good state where we almost have like a whole pipeline now to build an open DeepSeq R1 model. Another quite hopeful thing is that if you look, this is the LMSS leaderboard, which measures performance on models. It's a leaderboard where people get to see two responses from two models, and they rate them, and you get an ELO score. And one thing that's really exciting is when chip D4 came out, it was so good and so far off than anything that was open. So here in purple is the closed API models, and in green are the open models. And GPT-4 was so much better than everything that exists in the open that there was a moment of desperation, or it wasn't clear if OpenAI is going to continue on that axis. Is the open community ever going to catch up? And now, two years later, we are at a point where the open models are very close to the closed models. And you can also see the time it took to catch up to GPT-4 was one year. It was like Lama 3 was the first open model that was as good as GPT-4. And now, the catch-up time has shrunk into two months. So, if you want, try to extrapolate the curve. Maybe open model is going to take over, or maybe those two curves are going to merge. But in any case, there doesn't seem to be like a big gap anymore between the two. So I hope I laid out a little bit the motivation to whether we can, we should build open models. I think it's important for values and transparency and building trust. But the elephant in the room is always a bit like, who's going to have to compute to build like fully open models? Like, even the open mall weights that I showed, like Lama and DeepSeek, those are, like, commercial companies that built those malls. And maybe tomorrow they decide it doesn't align with their goals anymore, so they stop releasing open malls. So can we do it, like, fully open? Is there, like, the infrastructure, the know-how, the tooling to do so? And I want to start by showing you a list of, like, public infrastructure that exists in Europe today. and there are like very powerful public computers supercomputers to that can be used to train those malls so there's Lumi in Finland for example that has like 10,000 GPUs based on AMD architecture Jules and Jupiter that should be coming online I think soon in in Germany that has like tens of thousands of of h100 h200 chips there is Leonardo in in Italy and also Alps in in Switzerland, which has 10,000 GPUs. To give you a sense, if you used like 10,000 GPUs to train GPT-4, it would take you roughly a month to do so. So there is public infrastructure to build like very large malls on public infrastructure these days. We started working, this is more like a sneak peek, it's something that we started working on a bit. It's the internal code name at the moment like the boom project and the idea is these public clusters are nice and gpt4 is like is already like a very impressive scale to to train a model that's that large but at some point like one cluster might not be enough anymore and the goal of the boom project is to see can we connect a few of the public clusters and train an open model collaboratively so every cluster contributes a bit of compute and then we we can together like build a very powerful model the idea is this is mostly like a proof of concept just to show that it's working and that we can do exciting things and hopefully that will catch up and more clusters are going to join the effort the other question is i mean these things are nice today but there is also we need to invest in infrastructure tomorrow in the u.s open ai has been pushing very hard to build more infrastructure to train their giant models they've been announcing the stargate model which is supposed to like invest like 500 billion dollars into into new compute clusters and also the eu uh early this year has announced that they will want to invest like 200 billion euros into public compute infrastructure. I think the nice difference between the two press releases is OpenAI's investments are going to benefit mostly OpenAI. They're going to build their model and they're going to serve their model on that infrastructure. Whereas the EU focuses more on can we build the infrastructure that then can be used openly and collaboratively. I tried to highlight it's very tiny but it's like in their mission it says allow open and collaborative development. So I think is very much in the spirit of building these open source models. The other ingredient that we need, it seems like compute seems to be something, if we can get people to work together and commit compute time on these public clusters, seems like we can do interesting things. But the other thing is also, do we actually have the know-how to do these things? Because like the big closed labs, they have a lot of engineers and a lot of know-how. build out internally. And we need to have this know-how somewhat in the open as well. And there are a few things that are necessary. So I said it's kind of like those things I mentioned in the beginning. It's like you need high quality data. So you need to know how to build high quality data sets. Sometimes building a high quality data set for language model training is a little bit counterintuitive to things that you think are of high quality personally. So there's some know-how and some intuition that needs to be built up. You need to know about what kind of model architectures are very efficient and good to train. You need good training recipes. That sometimes falls a little bit between the cracks, but it's not just like you take data and you train a big model and you get something that works well. You need to optimize these days the models along multiple dimensions. So you want a model that has very long context, potentially, especially in Europe, it should be multilingual to some degree. How do you make it very multilingual without making it very bad, for example, at certain languages? Then, as I mentioned, you need know-how on scaling the training. So a lot of people now can train a model on a GPU. I think there are lots of libraries and tutorials for that. But what it takes to train a model on thousands of GPUs is a little bit harder and that know-how needs to be built up. We need to build better benchmarks as well. I think a lot of the benchmarks that we're currently evaluating on, they're measuring very academic things, but they don't map very well to real use cases. So I think there's a lot of work that needs to go there. And in the end, we need fast inference. So we need people who figure out how we can serve those models very efficiently at scale. So if someone built these days a very large GPT-4-like model, it's not so clear who in the open community would be able to host that model and serve it to millions of people. We've been involved in some projects to build the know-how together, and there have been some other projects as well. I'll just list a few that I'm a bit familiar with. but there are more. So I mentioned LNAI. They have the Olmo family of models, and they generally release all the data sets, all the training code, and very detailed tech reports so people can read them and they can understand what it takes. We have been trying to do things in a collaborative way. So we had two projects, the Big Science and the follow-up Big Code, where we built the models in the open So people from the community can join and then you figure out how to do these things together and the knowledge kind of naturally diffuses around the community. And I think in both cases, we had like over a thousand people participating. Recently, there's the Swiss AI initiative on that Swiss cluster. They're trying to build like fully open models as well. And IBM with their granite models are doing the same as well. So this is a little bit an effort in knowledge building. At HuggingFace, specifically, the research team at HuggingFace is set up a little bit special. We don't try to compete with language model trainers. So there is people doing the large model training. If we would do that, we would need to raise more money. So we want to try to do the things that nobody else is doing but would be valuable. So one thing that nobody's working on or nobody's releasing at the moment, at least, is building large-scale, high-quality data sets. So we released FindWeb, which is a web scale data set that is kind of like Lama 3 pre-training size. We have FindMath for more math-specific stuff, or FindVideo, which is a video data set. And also recently worked on FindWeb 2, which took the FindWeb approach, which was English only, and translated it to thousands of languages. Then we also work on recipes. So we try to build transparently the kind of recipes to do the post-training. We spend a lot of time there figuring out, like, what's the best data set and the best algorithm to kind of post-train, for example, those models and then release those data sets. And we do a little bit of model training, but we try to stay a little bit in niches. So for example, one thing that we're quite excited about is, like, pushing the small models because they're very useful to a lot of people that want to run locally or on like edge devices or in your phone. And so we have like the small LM family and follow up of that is the small VLM model which is a vision model that are like between 100 and 2 billion parameters and so can run very fast on a lot of infrastructure. And we do like, we try to do very long blog posts about the things we're doing. So I think a lot of papers are nice, but they're a little bit hard for people who want to get into the field and contributing. So we try to do a little bit more like educational blog posts. So we have a very long blog post on FindWeb. So if you're interested in how you build like a web scale data and how you filter it and so on, feel free to check it out. An even longer blog that we recently released we called modestly the Ultra Scale Playbook. And the idea is to give people an idea what it takes to train models on a distributed cluster. So we trained each of those points in this galaxy is a distributed training experiment. And the color tells you kind of like how efficient a certain configuration was. So we scanned like all possible ways how to distribute models to figure out like what's the most efficient one. And we explain all the parallelism strategies, for example, or mixed precision training, things like that. So these are a little bit the things that we're currently working on. And we're also building a little bit like open source tooling. Data 12 is a data processing tool that we use to do the web scale filtering, for example. Nanotron is a lightweight training framework that we use internally to train our models with 4D to 5D parallelism, and light eval is a small, supposed to be light evaluation framework to run evaluations reproducibly. Yeah, so I think I could convince you that we can build those things in the open. We still need to figure out a few things along the way. We need to figure out how to do the distributed training, getting people working together to do it. And now I want to talk a little bit about once we're there and what are the next things that are coming. And I think there's interesting things on data, agents, using AR for science. And I want to give you a little bit of an overview of the things I think are quite exciting and coming very soon. So one trend that probably is going to continue is scaling. This is a paper from four or five years ago from OpenAI, where they analyzed a bit, like, what's the relation between scaling data set size, model size, and compute? do the models get better which is like signified by the loss and the answer is yes so if you if you increase any of those three quantities you get like a better model essentially unfortunately it's on a log curve so it's a very nice line but it's on a log curve which means like if you want to improve the models you're going to need exponentially more and more compute resources so that the recipe is kind of like if you add more data if you make your model bigger and you push in more compute you will get better models but it's going to be more and more expensive and that's why open ai is investing like hundreds of billions into infrastructure because it simply doesn't exist yet if you want to continue that that line one major constraint is also becoming that like for a single cluster uh the the energy production um is a limiting factor so that's why they're they're building like power plants together with clusters to have everything in one place. So I think that's going to continue a little bit, but there is also another interesting trend. So on the left-hand side, you can essentially see what I've just been saying, and it's like the train time compute. So this is like a math benchmark, and on the left-hand side, you can see if you add more train compute, so you add more, you make your model bigger, you make the data set bigger, you train longer. increase the overall compute to train them all, you get kind of like predictable returns. So the curve goes up, and probably if you add more compute, it would continue to rise. At some point, you're at 100%. But yeah, you can kind of see the trend. One thing that came out a little bit like last year is that you not only can increase the train time compute, you can also increase the test time compute. And that's quite interesting because you don't actually need to train the model any longer But you can use the model at inference time in a different way and If you add more compute and inference time you can get similar gains and you can see here on the right-hand side Like you add more test time compute there the performance goes up as well So what does that mean in? Practice like if you think about a transform model like one fundamental limitation is like no matter what kind of question you ask there's like constant compute used to to get to the answer so if you ask like what's 17 plus 8 the this this sequence of tokens go through the model and the model will produce just one token that's kind of like one compute unit and that's a very simple question so the compute might be sufficient to answer that question but you can ask a very hard question which is like is the Riemann hypothesis true And the model will only have the same amount of compute to answer that question, which is clearly a problem. You want the model to think longer and harder about difficult questions than it does about easy questions. And you don't want to throw a lot of compute at super easy questions either. So how can we do that? All the models these days, like the new thinking and reasoning models, they do that in some away, 01, 03, now 04. So the question is, like, how do you get there? And one straightforward way to make this happen is you let the model just think about or reason about the problem longer. So rather than just emitting the answer straight away, the model emits, like, a reasoning trace in between. And what that means is every time it produces, like, a reasoning token, that's, again, like this fixed amount of compute, but the longer the reasons, the more compute goes through the model. And there is an interesting plot in the DeepSeq report where they did their reinforcement learning training, and you can see throughout the training, the model average length, the response length increases, so the model reasons longer and longer, and it gets better and better. And you can see an example of, like, a reasoning trace on the right-hand side, where it's like There's a math question, and you can see rather than the model just giving the straight answer, it thinks about the problem longer. And apparently there is a relationship of how long you think about a problem and the chance that you get to the right conclusion. So this is very exciting, but it also means that you need more and more inference compute. So it's like if you suddenly, the average response length, I think, of DeepSeq R1 is like 8,000 tokens. So it produces like several pages of text every time you ask it a question. So that's way more expensive than just emitting like a 20-token answer to your question. So a lot of the pressure, compute pressure, goes from the pre-training to the inference, actually. Similarly related is, like, agents. So people have been talking about agents, like, the past six months or the past 12 months like crazy. And I think there's a very big, like, hype bubble around it. There's, like, very interesting use cases, but I think at the moment people are sometimes using it for things they shouldn't use it for. But like on the more like research side of things, there is an interesting problem, which is like if you want a somewhat agentic system, you need multi-step reasoning. So going back to the example before, if I ask the model like what is the answer to 17 plus 8, these days the models probably have like an accuracy of 90, 95%, which is very good and useful. But now if you go to a system that needs to make the right answer several times consecutively, suddenly the task gets much, much harder. And especially it scales, like, polynomially with, like, how many steps you need to get to the answer. And I did, like, a simple question here, like, find me a flight to go from Basel or Zurich to London Heathrow. to get to the right answer the model needs to find the right website needs to select the right date needs to select the right origins select the destination and then so on so on and if that process has 10 steps and each step has like an accuracy of like 90 percent um with 10 steps you're already at like 34 success rate so it's like probably not the the thing you want to do if you want the model to to select a flight for you if it takes even more steps like 20 steps it decreases even further to like 12 percent so that's one challenge like how do you get the model to actually like consistently um do the right choices in like these multi-step settings and we thought rather than going for like these uh flight examples which i think are are the ones that you're going to do lastly. It's kind of like the benefit of it is you save 10 minutes of looking up your flight. And the risks are the model book's the wrong flight for you. So I think this is the very last thing you want to do. It's almost no return and very high risk. But we thought one thing that would be kind of cool is if you frame data analysis as a multi-step problem. You could think of a notebook as a multi-step process. And we built a small demo we call, like, Jupiter Agent. And the idea is essentially just that. Like, you ask the model a question, and then it can write some code. Here I ask it to do the Lotka-Volterra, like, differential equation, make a plot, and then I ask it, like, ah, this is very nice, but can you make, maybe change the time horizon a little bit, increase the resolution? It's a little bit small to read, but that's the basic idea, and then the model can do things. So here it's almost like one or two steps at most that the model does something. This is like Lama 3 still running it. And we noticed like it can do things that take like two to three steps. After that, it gets very lost and confused. So we're not quite there yet where you can actually do like a full Jupyter notebook with like one of these models. But I think it's like an interesting thing where if that works to some degree, then that would give like some value and probably the risks would be somewhat limited or at least it would be a useful assistant. But one limitation we faced is like we don't really have a really good way to evaluate if a system like that works well. There are no benchmarks to measure that on a somewhat realistic benchmark. So we worked for almost a year now with ADN, which is like the stripe of the Netherlands or Europe. So they have a lot of transaction data. They're very interested in that problem because they need to do a lot of data analysis. And together with them, we built the Dapstep benchmark, like data agent multi-step reasoning benchmark. And the idea is, like, the questions and the answers are very simple. So it's very easy to evaluate if the model got it right or wrong. So here a question is, like, which card scheme had, like, the highest fraud rate in 2023? So it's going to be a name. Another type of question is, like, what's the fee a certain provider needs to pay in a certain time frame? And then we provide a list of data sets. So there's the payments data set, the first one, which is kind of like a transaction data That's like the big data frame that the model needs to analyze. But then there are a lot of auxiliary things, like there's a payments readme, there's mapping files that map categories to names, and things like that. And maybe most interestingly, there's a manual, which is for these transactions, they get for every card provider manuals that lay out how fees are calculated and so on, that are hundreds of pages. And we tried to simulate that a bit with, like, encoding actual business information in a manual. And it's, like, a few thousand tokens that lays out, like, how do you actually compute fraud? And the solution to these problems is, like, very long. It's, like, 220 lines of code. So it's, like, a very simple question, but, like, getting the answer right is very hard. And you can see here on this benchmark, even the best models, they get like maybe 14%, 15% of the answers correct. So we're very far away yet to make those models like really good at these kind of things. So I feel like unless you're somewhat higher on that thing, you don't want the model to book a flight for you just yet. Another field I just want to briefly talk about, I'm by training also a physicist, so I'm excited for using AI for science. And there are like a lot of interesting projects that have been coming up recently. This year, Demis Hassabis and John Champer got the Nobel Prize for AlphaFault, which is a model that can predict protein structure. It's computationally a very hard problem and requires a lot of simulation and experiments to get the protein structure right historically, and they found a way to do that much more efficiently with AlphaFault, and researchers are now using that. And they're also open-fold initiatives to do these things in the open. It's also, again, deep mind. They're really pioneering a bit the field of doing science and AI. They've used some models to build new algorithms or help generate the next generation of chips that Google is building. Another field that's quite cool is math. These are two of my colleagues at Hugging Face, Lewis and Ed, shaking hands with Terence Tao, who's a famous mathematician, and they won a prize where the idea was, like, can you build a model that solves Olympiad-level math questions with, like, a somewhat tiny model? And there's an interesting post by Terence Tao himself about, like, the role of AI in mathematics, and he doesn't see the models just, like, taking over mathematicians, but but see is it more like as a useful assistant or co-pilot. And there are many more things to come. So it's like, I think, just getting started in medicine, biology, physics, and so on. So I think there are a lot of exciting things to come very soon there. So I want to close this talk by throwing the ball back to you. So what can you do? It seems like you need a lot of compute. I made that meme for academics, but I think it applies just as well to the open-source community in general, is, like, Meta has, like, half a million GPUs. Hugging Face has 1,000 or a little bit less than 1,000, so we can do cool things, but we can't compete directly with Meta, and then you have, like, the lucky academic that maybe has a machine with eight GPUs, and you have the unlucky academic that has no GPUs at all. And if you, like, count the people in those three, four fields, probably most of the people are in the bottom half. So what can they do? And I think there is a lot, actually, that those people can do. And I want to, like, illustrate it a bit with a model that we built as part of the OpenR1 project, which was called, like, the Olympic Coder. It's just a 7b model here in purple, and it competes with, like, cloud or one mini deep sea car ones, like, almost at the same level, but it's just a 7b model. So the question is, like, how can you build a model that's so competitive? And the simple answer is, like, you need to build, like, very high-quality data sets. That's all it was. Like, supervised fine-tuning on a very high-quality data set gets you a very competitive model on these Olympiad informatics questions. So to give you a little bit of a summary, I think, like, you can always build high-quality data sets. They will remain, like, useful. You can fine-tune very task-specific models that run locally, very cheap, or on edge devices. And there's a lot of work that needs to happen, like, in the tooling space still, like, around data processing, fine-tuning inference, like, mapping some of the things that are maybe known in classical data processing to the kind of data processing that's needed for pre-training models. And also you can join an open collaboration and start working on training those models collaboratively. So sometimes people feel a bit like they missed the train and like everybody who's in AI is already like super far ahead, but it's like saying in, I don't know, 1970, like software is already happening. maybe we shouldn't get into it. So I feel it's very, very early days. So if you're interested and excited about these things, I don't think it's anywhere close to being too late. So I want to close with that. Thanks a lot for listening this early in the morning. I hope some people are still awake. And I think now I have like some time for it to answer some questions.

Speaker 2 [41:54]

Yeah, thanks a lot for your talk. It's very good to see that open source is catching up and that those principles also work for AI. We have really a lot of questions. I hope you're going to stay with us a little bit afterwards. So Mushahid asks, how do open source AI projects balance decentralized community innovations while preventing harmful model fragmentation and divergent incentives?

Speaker 1 [42:23]

Yeah, that's a really good question. I think one thing we noticed, like we ran big science and big code like super collaboratively, super democratic. But at the end of the day, you can only train like so many models, right? So you need to make some decisions. But I think people are actually surprisingly collaborative on these things and like willing to make compromises for the sake of actually building something. Also, if you build these open projects, you need to always balance a bit how ethical and responsible you can be and when do you actually deliver a model. So you can build a fully ethical data set, but maybe it's going to be too small to train a model at all. So you need to make some decisions on what are the things that we can do, what are the things that we shouldn't do, and still arrive at the end at something that's useful. If the open source community ties both their hands behind their back because they put so many constraints on them, closed source will just move ahead because they have so many lawyers and don't care that much.

Speaker 2 [43:27]

Yeah, so a little diversity is good. It's like in the Linux community and open source, but it's a little bit of fragmentation is good, but in the end everyone's going to the right direction. Okay, cool. So there's another question. In your honest opinion, will Europe have anything to say in AI compared to the US and of course China, or is it left out for good?

Speaker 1 [43:54]

Yeah, I think we're at the moment a little bit at the turning point for that, I would say. I think, as I showed, there's the compute is potentially available in Europe to do very interesting things. I think the know-how is also available. There are amazing academic institutions that have built the foundations for most of the things that are happening now all across Europe. So I think the know-how is there, the infrastructure is there. now it's a bit the question can we pull together and and do something and uh at least we had hanging face we're now like pushing very hard on these open collaborations in europe as well to see can we just like get something started that's exciting enough that like we we can build on top of that and get some momentum.

Speaker 2 [44:37]

How do you think, maybe related to this question, how do you think all the Europe regulations, are they rather good long term or are they holding us back in a way that we kind of rather first set a lot of rules, regulations and then innovate instead of like the US innovate and then regulate?

Speaker 1 [44:55]

Yeah, that's also a good question. I think somewhat a middle ground is healthy. I think voices from the US are typically like we don't regulate, we want to innovate only and any kind of like regulations hurts innovation. I don't think that's quite true. I think a lot of the regulations that we added is about kind of like protecting the consumer and the user. So I think we need some regulation to make sure those models are used responsibly. It's like not everybody is developing apps that give health advice or medical advice. I think so. And I don't think at the moment, especially if you're building open models, there is not that much that's holding you back. I think if you're building applications, there are regulations that say, like if you build an AI doctor, then you need to go through some process. But if you build open malls, at least then you can move quite freely still.

Speaker 2 [45:56]

And also for the data sets, I mean, there it's like a lot of copyright.

Speaker 1 [45:59]

Yeah, I mean, that's a little bit up to debate, I think, at the moment. And it could also go both ways. So far, like, we work a lot with open data sets. And I think people, they don't want to attack, like, people building open stuff. I think the companies that are being sued are the ones that are doing it in the back end. So I think you can actually do open data sets. It's like all the data sets I showed that we're building, they're open and available in Europe.

Speaker 2 [46:33]

Then another question by Radovan, don't you think the future of LLM AI is in smaller language models, so specialized one like economist LLM, lawyer LLM, teacher LLM, so rather smaller than general large language models?

Speaker 1 [46:54]

Yeah, so the way I think about it is a little bit like in software you have things that are kind of generally useful and you use for many things and you have software that's very specific for certain things. And I feel like for models it's potentially going the same way. Like ChatGPT being a very general model is very useful for many questions that you have and you don't need to decide before which model you want to use to ask a certain question. You just go there. But I think then for a lot of business applications, you want to have dedicated models that are very cheap to run, specialized for exactly the thing you want them to do. So I think it's going to be a little bit of a hybrid. You have very small models. You have large models also at some point. I don't think it's exclusive. And I feel the same way about open versus closed, actually. It's not like I'm saying every model needs to be fully open. I think it's fine if some people are selling an API. but I think it's nice if there's a nice choice that you can make.

Speaker 2 [47:53]

Okay, that's maybe related to the next question. So what did DeepSea gain from opening their A1 model then financially or in general?

Speaker 1 [48:03]

Yeah, that's always the question, right?

Speaker 2 [48:04]

Yeah.

Speaker 1 [48:05]

Like, why should a company open source something? What do they gain from it? I think, first of all, they gained a lot of recognition for it. So I think they don't have a problem anymore hiring people. So that's a good thing. I think also, like, the company value has gone up significantly if you would look at that. But more generally, like, I think one good example is, for example, Lama and Meta, they have the same questions, right? questions right and people are fixing their models or they are finding better ways to run inference and in the end that's going to benefit meta because they run those models internally and serve them to customers as well in parallel to the to the open mall so if they can set the standards and people improve those models that's that's going to benefit them even opening I wants to to open source a model like this summer and they are getting the same ways like we we're going to benefit in some way from from that it's like how you always do i feel like of

Speaker 2 [49:04]

So coming back then to being open AI actually.

Speaker 1 [49:07]

Yeah, maybe hybrid AI. All right.

Speaker 2 [49:09]

All right. And the next question is by .. What are the key challenges in creating high quality data sets? Is the bottleneck the cost for human labeling?

Speaker 1 [49:21]

We actually do almost no human labeling anymore for most of the things we're doing. I think there are still domains where, like, human labeling makes sense. Like, especially very specialized domains where those models are typically very bad at. But for many things, like synthetic labeling works surprisingly well. And, like, these models, they don't have all the issues human labelers have. Like, your data is typically quite boring, but you ask people to, like, annotate hundreds or thousands of examples. so they get bored or distracted. And some models are quite consistently good at labeling things. So I think the main challenge is, again, a little bit compute. So you need, depending on the scale of things you want to do, but you need CPU compute if you want to do large-scale data processing, especially once you move from text-only to video and image as well. It gets way more expensive. And if you're interested in, like, the fine web blog post has a lot of details about that. The way we do it is very data-driven. So, we make a hypothesis for a set of filters that we think are good. And then we train a small model quite quickly. And we look at, like, what's the performance of that model trained on that data set. And then we can iterate quite quickly. So, we don't annotate data for, like, specific criteria necessarily, but more, like, we figure out, like, heuristics data-driven. But then that also requires GPU resources.

Speaker 2 [50:52]

Okay, then it's a kind of bootstrapping, so the model labels, and then it's trained again, and then it's getting better and better? Yeah. Okay. The next question, I think we are taking two more, is, of course, the environmental impact is always an issue. So the question is, what is your take on the environmental impact of building more large-scale models?

Speaker 1 [51:18]

Yeah, I mean, clearly there is a huge impact, right, because those malls take a huge amount of compute to train. And as said, like we're moving towards more inference compute and millions of users. So the balance even shifts a bit towards like the main compute cost actually coming from serving the malls rather than training them. It seems like the malls are generally generating value and being useful. So I think it's worth building them. The question is, goes maybe back to an earlier question, like, do you always need to run the biggest model for every task? Or does it make sense to have, like, specialized smaller models that you can run locally or on your phone and so on? And that also reduces the carbon footprint, right, by a lot. Like, the smaller LEM models are one point, the biggest one is 1.7 billion parameters, which is a thousand times smaller than GPT-4. So if you can use that for like a few hundred requests that you would send to GPT-4 otherwise, then you're already saving quite a bit.

Speaker 2 [52:21]

Okay, they're choosing the right model for the right job. Okay, so maybe the last question. So where does the compute for open models come from? So you showed some of the financing, but how is the training financed other than what you showed?

Speaker 1 [52:41]

I mean, those public clusters, they're usually going to these research grants, so you apply and pitch a project and then you get compute or not, whether the committee thinks your model is exciting or not. So it's financed by the public, essentially. Those are public clusters, right? So the public paid to build them and now it's like also controlling the access to them. other than that like for example in that project I said where we want to collaborate we also want to explore if we can do a hybrid version where like for example we use the Hugging Face cluster as well as part of it so we pay some compute into the whole budget I think hopefully there is like an interesting future like companies work together with academia pull the compute and build something that's beneficial for everyone so the financing would also be hybrid in that case Yeah, cool.

Speaker 2 [53:38]

Yeah, thank you. Thank you for your extremely cool keynote, and let's give Leandro another applause, please.