The Multimodal Era of Machine Learning (and How Python Made It Possible) Keynote

Speaker 1 [00:00]

Hilde Kuehne, and Hilde is one of the, I would say, how do you say co-affair in English?

Speaker 2 [00:06]

I don't know.

Speaker 1 [00:07]

I don't know.

Speaker 2 [00:07]

No, no. That would be the wrong.

Speaker 1 [00:09]

Hilda is one of the probably most important people in multimodal learning, and she's a professor of multimodal learning at the Tübingen AI Centre and an affiliate professor at the MIT IBM Watson AI Lab. Before that, she was a professor for computer vision and multimodal learning at the University of Bonn. And now, meanwhile, her research focuses on video understanding with a particular emphasis on learning without labels and multi-modal video understanding. She has created several highly cited data sets and foundational works for analysing large collections of video data, including HDMB 51, which was awarded the ICC 2021 Helmholtz Prize. For those who do not know that, that's basically a test of time award, which is pretty cool. And she also got the PMAI Mark Everingham prize. So in her keynote today, the multimodal error of machine learning and how Python made it possible, Hilde will speak about the current state of multimodal learning and how Python shaped this trajectory. Sorry, I'm too excited for this. And so lastly, one thing I would like to add is Python, what she told me before the meeting, Hyphen was also something like the common layer across modalities and even glued research communities together. So enjoy!

Speaker 2 [01:45]

Yeah, thanks so much for the nice introduction and actually thanks so much for the Invitation and for having me As you heard my name is Hilde Kühne. I'm from Tübingen AI Center And I'm also affiliated at MIT IBM Watson AI lab, but very big disclaimer for everything I say the next 35 40 minutes all opinions in this talk are completely my own Okay, having said that just to get started I thought we'd just start with the problem of what is multi-model learning actually doing. And I didn't ask JetGPT, actually I'm old school, I went to Wikipedia. And Wikipedia defines multi-model learning as a type of deep learning that integrates and processes multiple types of data, which we refer to as modalities, but technically we just talk about things like text, audio, images, video, and so on. And we just want to process them together and mainly try to train systems on combinations of those data. And you might already have a feeling the applications that usually come from this kind of like type of learning is first everything related to representation learning, so when we want to learn better features. But what you probably experience more in real life is cross-model retrieval and zero-shot classification. And nowadays, obviously, everybody is after generative content creation. So, you might also be familiar with some of the algorithms and kind of like models that are behind those. The first probably most prominent one was CLIP, which stands for contrastive language image pre-training, and this is mainly a model that just takes text and images and tries to learn representations that actually fit. This is this magic image text representation learning that I will talk about later as well. And when you have that, you can do things like cross-model retrieval, zero-shot classification, and so on. And then next generation actually was models like LAVA and QEN, so vision language models where you actually combine some vision backbone with some generative language backbone, and nowadays they are more called multi-model LLMs, but technically it's mainly still vision text-based, and with those you can do all this kind of like fancy chatbot things like like visual question answering, they can also do optical character recognition, they can do detection, and so on and so on, and then probably the third group of models that also everybody has used at some point up to now are text-to-image or text-to-video models, like Dali from OpenAI was probably the first, but since then we had stable diffusion, we have Flux, we have Sora, Nanobanana, Genie, I guess you all know the list. Those are the things that probably have received most attention in public. There is a fourth class of models which are actually the really interesting one, the Omni models. And those are still more existing in research labs than in production settings. And there we have now models like Omni, ImageBind, LanguageBind, which really try to combine more than just text and image. So for example, image, text, and audio, image, text, video, audio. Now this is what we have at the moment, and that's impressive, but before I dive into how we got to this list of models that we have now, I thought I'd take a step back and remind everybody where this actually comes from. And the interesting story, the history of multi-model learning is actually quite old. So quite fittingly, 15 years from before now, in 2011, there was one paper coming out, it kind of like at this time probably a super-niche paper that was about learning representations from audio and video signals, and at that time they were using restricted Boltzmann machines or kind of like shallow autoencoders to actually reconstruct signals, and this paper had the title Multi-model Deep Learning, and this is at least what I found to be or what at least the internet says might be the first real mentioning of this topic. So if If you take this as the first paper that coined this phrase, we are talking 15 years before now. But the interesting thing is that the architectures that they built, you still see them in papers and systems today. So technically, they were super ahead of their time. They probably didn't have the data and not the compute to really pull it off. But architecture-wise, they knew exactly what was needed. Having said that, we also had some more kind of like history, and the second big threat that actually started pretty early was image captioning, so the idea to have an image as input and to generate text, in this case a caption as output, and here the first paper appeared actually one year later at NeurIPS 2012, which was called From Image Annotation to Image Description, but the pictures you see here are actually from a paper that appeared a bit later that's called Show and Tell from Google that appeared at CVPR 2015. And the reason why I pulled out this and not the other one is because the architecture that you see here might look a bit weird, but you will see this is also still something which was way ahead of its time, because what we are doing now is not so different from what you actually see in this paper. And probably for all the people who are old enough to remember just a bit of nostalgia, this paper was actually taking an old-school convolutional network, in this case it was an InceptionNet because it was a Google paper, and they were plugging it into an LSTM, like old-school NLP people still have PST from LSTMs, but technically they plugged it into an LSTM and the LSTM was able to give them a caption that in some times actually really reflected what was visible in the image, so pretty cool. Now the interesting thing is how did we actually get from this one to the nice models that we have nowadays. And for that, I want to talk a bit about the methods, how we developed them and why they work and technically it was multiple things happening at once somehow that kind of like drove this development. The first big bang, at least on our side, was actually the rise of contrastive learning. For everybody who is not familiar with the concept of contrastive learning, in a nutshell, we figured out that we can actually train models based on pairwise data. The cool thing is pairwise data can be anything. People do it with single modalities by, for example, taking two augmentations of a single picture. We figured actually we can train models based on pairwise data that we grab from the Internet by just downloading images and alt text, or if you would come from the video community, you would download YouTube videos and the respective subtitles, and that would give you this pairwise data, and what we would do then is actually train CNN or any kind of model based on this contrastive loss, which technically just says if you have an input that comes from the same source, make sure that the embedding space, and I explain next slide what this means, that the embedding space of those two samples are close together, so the vectors that you produce for those two samples should be pretty similar. And if they're coming from different sources, they should be far away from each other. And that was enough to start training at this time still CNNs, and now to kind of like disentangle a bit this magic about this embedding space, because this is what you get in the end when you train those models. You get a multi-model embedding space. Technically, embedding spaces sound super fancy, but it's really just vectors that have the same length. So if you want to have it simple, think of 1D array that always has 10 floats or 8 floats or any number. As soon as you have that, technically you have two sequences of floats that you can compare. You can add them, you can subtract them, you can multiply them you can do a lot of fancy stuff with them and technically any embedding space when people say embedding space it just means okay we were able to compress input from different sources to the same vector length that's all and now you can imagine we take a text and we compress it or we have a neural network and we just get an output that has a certain length let's say seven floats and then we have another network that gives us that takes another input like an image and also gives us an output that has seven floats, and then we can actually compare those two vectors to each other, and technically you can, as soon as you do this, you have a multi-model embedding space, but it's probably not going to be very meaningful, and therefore what we actually achieved with this contrastive learning was that this embedding space was organised in a way that when things are semantically related, like for example text, and semantically matching images, that those embeddings are actually close to each other versus other concepts are actually further apart. And once we have achieved this kind of like organization of our embedding space, we can do all the fancy stuff that we want. For example, for text to image or text to video retrieval, you can type a query into a text box, we can convert it into our vector, and then we can search for embeddings that are close to this vector and give you that back. if you want to do zero-shot classification, as long as you have a semantically meaningful class name. Also, it's just the text, so what you need to do, you take the text, you embed it, you look in your embedding space, which are your nearest neighbors, and you have a zero-shot classifier. And this sounds, first, probably a bit messy, but not super interesting. But for the vision community, it was something really kind of like groundbreaking, because you have to imagine, before, the main task we had was classification, which means technically we only had indexes. We never had a real textual semantic understanding of what was going on. Even ImageNet classes technically were just indices from 0 to 999. And now, with this embedding, we can actually do magic things because we no longer need to actually define classes in the first place, which never was a good idea because labels are super subjective. When you have video they depend on duration, expertise, and so on. We were always constrained by concepts that actually have a visual representation because we need to label something. And now, as we have this kind of like vast web data set, we can just take whatever was ever discussed on the internet and we can learn on top of it. And then finally, we don't need a taxonomy and we don't have a taxonomy for the world. And also, we got rid of the problem of actually having to annotate data. So, this was really kind of like a huge step in terms of what we were able to do. And as such, obviously there was a plethora of works in the field, and just to do a bit of justice to my community, we actually started this way before CLIP entered the stage. So technically when CLIP came out, I was kind of like completely underestimating the impact because I was like, okay, we are doing this since three years, what's the novelty? Technically, you see, there were a lot of people who were super excited about that concept and who built kind of like models on top of that. But before I take the next step, I need to, as someone who is actually still working in the field, I need to debunk one of the big myths of those embedding spaces and also one of the big myths of Clip. And actually, this kind of like nice visualization that I showed you is good to have a feeling of what's going on. But in reality, fair disclaimer, it's not really what's happening because when we look at what those models are actually learning for real, we see that the embedding spaces that they generate are still separate. So if you just take clip features and you just plot them in 2D with UMAP or whatever you prefer, you will actually see you will have two very nicely separated clusters, one for vision, one for text. And the interesting thing is it works anyway. So this is not a big problem for practical applications. And there are actually a lot of works analyzing this phenomenon. So figuring out what we can do with the distribution and the training data and so on. But technically there is still an understanding that each model or each backbone forms its own kind of like embedding subspace. So each model forms its own kind of like cluster somewhere. and probably on a very high level the best intuition I have at the moment on how it works anyway is we are if we are computing this we are actually computing this in a very high dimensional space so we are talking vectors of a dimension of 700 up 2000 and in high dimensional spaces weird things can happen and in this space it's absolutely possible that we for example align an outer hull or something okay so this is kind of like one of the foundations. The second foundation or the second kind of like formative event that we had that spurred this whole kind of like field of multi-model learning was actually the rise of transformer architectures and as everybody has a bit of background I'm not going to explain transformers just trying to kind of like convey what transformers meant especially for multi-model learning because you probably all know when you have a transformer and you have a text you take each word, you build a token, and then you forward all the tokens. Now, the interesting thing was that the vision community figured out this also works for images and videos. In this case, you actually just put a grid on top of an image, and you forward the single patches of your image, and you can do a similar technique for audio, and then all the modalities that you, for example, want to combine are instantly in a nice tokenized format, and we have this generalist architecture that can actually natively consume this and can natively work with this format. And this is super cool. And it's not only that we have this architecture that can deal with those tokens, it's also an architecture that allows those modalities to interact with each other via either self or cross attention, whatever mechanism you prefer. So this is super cool. And those are the two kind of like major things from the science side, but while I was preparing this talk I realized there's also kind of like another thing that was going on, and this was less of a big bang, it was more kind of like a silent revolution that happened under the surface, and for that you need to understand again it was multiple things happening at once. First we had the availability of web-scale data, which was cool. Second, we got better and better and more efficient hardware, mainly in terms of GPUs. But what was also a very important driving force in this kind of thing was we got the software stack to actually do something with the data and with the hardware, because no data and hardware without software doesn't help anybody. And And while I was researching for this talk, honestly, I didn't find a lot of stuff written about this. So I just tried to somehow convey it via my own experience and feelings. And probably you share it or not. Let me know later. So to give you a feeling of how this kind of like influenced us. This is one of the models we built around 2019, 2020, before it was published in 21. It's AVLNet. and you might see, it looks a bit complicated, but technically it's just three backbones, one vision backbone, one audio backbone, and one text backbone, and then we encode everything in kind of like this feature vector, in this case 4,000 dimensions, and then we try to learn this embedding space that you saw before. And the interesting thing is that this was technically the first model that was, in some sense, purely written in the single framework. And to understand that, you need to actually understand what we did before. So when I was doing PhD, I was always trying to bridge a bit vision and language or vision and NLP, vision and speech, whatever. But the frameworks that we built were always super hacky because some frameworks were available in C. Then you had some vision libraries and MATLAB that you needed. You always had kind of like to juggle around with kind of like different components. and usually you just dumped intermediate stuff into text files and then you tried to load them, and then you just backed and braided it somehow works. And this was the first kind of like framework that we were able to build kind of like in one big setting where we didn't have to build weird wrappers that just worked out of the box. So technically, to a certain extent, we went from something like this to something like this. And we didn't realize that at this time. It was just happening naturally. But technically, this was to the point in my, at least, researcher career where we really got away from hacky solutions to something more consistent. And I think this was, maybe it was a coincidence, but I think there is something underlying this whole story. And this is the role of Python. And what at least happened in the research community was, let's probably say, what was the role of Python in this whole kind of story? First, what we got from the research community was actually the kind of like libraries for mathematical computing. NumPy, Scikit, SciPy, you know, all of them. And this was what freed us a bit from having to rely on other kind of like tools. Then on top of that, somehow naturally, we got all those wonderful GPU support with all the frameworks like PyTorch, TensorFlow, and so on. And I have still coded CNN in cafes. so in C if someone remembers this time, so I can really appreciate that, but it's not only that it stopped there, we also got all the support for the data loading, we got the installation management, so it was this kind of like whole ecosystem that was somehow appearing from the dark or from the void, and that actually helped us to kind of like build those systems, and in Research, I think, especially in the time, let's say between 2012 to 2020 or even now, I think it created something like a virtual cycle, meaning that Python became a standard in multiple labs or probably even in multiple disciplines. And as this happened, it became actually much easier to share and adapt the code between different groups, for example, between vision people and speech people. And while this was happening, this actually allows us to merge those concepts in those multi-model models that you see now. And as more people were actually following this trend, more groups joined, we could even merge in more, and kind of like we created the cycle that now lead to what we have here. And with that, I want to jump to the present and at least give you a bit of an idea of how language models are the stuff that you probably use in JetGPT works at the moment. So obviously everybody knows LLMs are great. The standard architecture for vision language models or now multi-model language models is actually the LAVA architecture. And this architecture has been proposed in 23, so it's just three years old, in a paper called Visual Instruction Tuning at NeurIPS, actually. And technically this is how the model looks like, But to actually decode all the fancy, funny blocks, give me a minute. So if you want to have a model that actually takes an image and text as a prompt or something as input, and then gives you a nice answer or caption or OCR or whatever, the first what you actually need, obviously, is a language model. So usually we take a pre-trained language model, and usually the better the language model, the better the system. The second ingredient you need is something to encode the images, and for that we take a vision encoder, and the vision encoder is technically exactly what you see, what was trained in this contrastive manner from this few slides ago. This is what we take, and the only kind of like slight change that happens underneath the surface for the contrastive learning, usually we use one global token, and now for the language model, we actually used the token embeddings of all the patches that we kind of like sliced the image into. So you already get a feeling it's kind of like a lot of tokens. And now the next problem, as you might already have a feeling, is those vision encoder tokens, they are still living in a bit of a different embedding space than the language model tokens. So when you think about the clusters, two different clusters, they are not correlated in anyhow, and this is actually the job of the adapter, and the adapter is kind of like usually a very shallow network, like a few layers, two, three, five layers, and the sole kind of like job is to take the tokens from the vision encoder and translate them into the embedding space of the language model. And once you have done that, you can actually give this whole system an image and a prompt, and if you're lucky, it gives you a really nice answer. And this is technically the architecture that most models run on, so even, let's say, probably the most advanced is the QWEN 3VL series, at least in the open-source domain, and they actually work on exactly this way, that they have vision encoder, text encoder, and a bit of adapter in between. Now, interestingly, this also makes research life more interesting, but also more complicated, it because what we see now is that we proliferate a bit. So on the research side, we now have groups who actually deal with the representation learning part. So they only train the vision encoder, but don't care so much about adapter or LLMs. Then we have groups who are sitting in the middle, who care actually exactly about this translation from vision space to language space. This is also what my group is interested in. And then sometimes you always train the adapter, and sometimes you also want to train the LLM a bit. And then finally, there's also kind of like a new stream of people who really just look at what are the tasks that we can actually solve with those models. So what can those models do even if we don't train them? And those are approaches which are usually called training-free methods for multi-model language models. So pretty cool architecture, very versatile, has a lot of potential, but there are also some nice problems hidden underneath, and I don't want to let's say I don't want to do any kind of like too much doomsday stuff, so just for everybody to know stuff that we know works and stuff that we know it doesn't work or it might cause you problems and where those problems come from. The first one is actually the token crisis and that's an interesting problem that I don't know if a lot of people actually discuss this because everybody says, oh, long context is great but the problem is this architecture results in too many tokens and to give you a bit of a feeling, transformers and especially language models are trained on speech and are built for speech. And language, or text in this case, is actually a very compressed, efficient, discrete, and symbolic representation that has very limited rules that it can follow. When you compare that to vision data, vision data is a completely different story, because vision data first is continuous. So every patch that we encode is just a bunch of float values, but they are not kind of like quantized or something in the sense that we have a codebook of 100,000 patches, and then we capture every kind of detail. And the second thing is there are much less rules in terms of vision data can, for example, be highly redundant. Think of a nice beach scene with a blue sky, then you end up with a lot of tokens where you just have a uniform blue background, and that's it. And if you go even further into video, like my domain, visual data over time is a nightmare. It's vast. Transformers have a super hard time handling those. And, obviously, we get them to work as well, and usually the compression comes to the rescue, and even Quen and other models do already some internal compression in terms of that they usually pull over 2x2 tokens before they process them, but even then you can still end up with thousands of tokens from the image side compared to probably a few hundred tokens from the text side, so there is still this kind of super strong imbalance. And just as a fair warning, there is still the problem that the more input you put on those models, mainly in terms of the more images you give those models, or the longer the video gets, the less data they can actually process from this input, either because they need to compress it, or because they need to be selective, or because there needs to be something, because even if you would process 100,000 tokens, it would cause you other trouble because it was too far out of what you have seen in your training distribution. So this is an interesting thing that's kind of like happening underneath, and that people are actively researching, so we are trying to find solutions for that. The second thing, and this is also more from my community and not really talked a lot about, but very good to know for everybody else. It also brought us a bit of a comparability crisis. And to understand why this is a crisis, you have to understand that comparability was actually one of the main drivers that allowed us to make the progress that we have seen from 2010 to 2020. Because we always were able to tell you, okay, this is currently the best model, for example, for image classification, which means when everybody knows, okay, this is the best model, If you want to have something better, you need to improve over this number, specifically. We have, for example, nice subfields like weekly supervised action detection, whatever, who took this, had a set of benchmarks, like two, three benchmarks, and managed over the course of one year to have a series of papers where each paper was improving over the state of the art by, let's say, one to two percent. So not much. technically always a bit incremental, but at the end of the year they have seen an increase of 10% in their field. And this is exactly what comparability gives you. And now we have actually lost this comparability. And the reason why we have lost this comparability is because we are no longer comparable in training data, so all the big models that actually give you good numbers are trained on different sets of data. We have a strong proliferation in tasks. So even if you have models that are doing vision language, some are specialised for VQA, some are specialised for detection, some are specialised for captioning and so on, and you cannot compare them anymore, it wouldn't be fair. And then we also have a proliferation in benchmarks. So even if you see papers nowadays and they have those nice tables where everybody tests on, I don't know, 50, 20 benchmarks and then comes up with kind of like an average accuracy, problem is if everybody computes the average accuracy on a different set of benchmarks, you can also no longer compare this number. So the result of this is there's currently not really a lot of community wisdom left of what really works best. Everybody can tell you what works, but if you really try to drill down on, okay, but what is really the best model, you will probably not get a very crisp answer from us anymore. And that makes it actually also hard for us to choose the right model to build upon it and to actually improve it and as a result everybody is now using standard model like for example Gwen or whatever open source we have and improving over that by 5% but we don't get this cumulative effect anymore and as a result we currently or at least there is a feeling that we might waste a lot of time and resources because we actually have lost this mean of comparability now on the more practical side going away from my kind of like community, there are two more general kind of like trends which are interesting. And the first is the generalization gap. And this is nothing new. This has always been part of machine learning. It just has never been a problem because we all knew and also outside of our community, nobody cared. And to describe it in a nutshell, the problem is we have models that have great performance on benchmarks, works, but technically people will figure out they fail on random inputs that was not part of the training data, and the obvious reason for that is all those models still just learn correlations of the training data and actually have very few capabilities to really go beyond that, so you rarely have any abstractions, usually you can just somehow build mish-mashes of what you have seen in the training. And even when you train on web scale data, you need to live with the fact that even web scale is limited and it's not a true representation of the real world. And on top of that, it also comes with all the biases that you have for people posting and so on and so on. Second interesting problem, and this is now again a bit more from my work, while everybody is telling you how great this works, the truth is that this alignment, so this alignment or this kind of like adaptor layer that we see is still pretty weak compared to the backbones that we have. And the problem that we have here is even when we have fancy benchmarks to test it, the sad truth is that if you have a stronger language model, you can easily increase your numbers by 10%, versus if you take a stronger vision backbone, you will rarely move the needle by probably 1% or 2%, and that's it. so the heavy load is still on the language model and the vision kind of like plays a bit of a role but it's not kind of like the driving force and this is also a bit reflected or let's say another part of this problem is also that we focus or especially people in this area and especially people who like to brag about their great numbers on benchmark data sets usually kind of like don't tell you that those benchmarks are pretty limited So usually it's some form of VQA, so visual question answering, which means you're given an image like that one, and then you have a question like, what object is the woman in the blue shirt holding? And then you give usually something like multiple choice answers, and a very drastic example, and it's not that bad, but to give you a feeling, if you have then four options, which would be, I don't know, car, house, phone, and boat, you would probably guess okay it's probably the phone and sounds a bit ridiculous but technically we know that many of those benchmarks already reach a pretty decent accuracy even if we give them a black image or completely noisy image so they don't have to look at the image to come up with the right answer they can just rely on language statistics and if you are doing a good job you actually report this kind of like vanilla number first and then you say how much you improve If you just want to look great, you just tell them, okay, this is what I got with my multi-model model. But then there is one more, and that's the last one, and then I'm done. There is something that I realized when I was talking to some project partners on EU project on multi-model learning, and it's also something which I cannot formalize or which I don't think has been formalized. It's more something like a feeling that there is an expectation gap currently. and to kind of like try to grasp that we have the problem of that we have users that think that the model can do great stuff and then we have the reality of what the model actually can do and this is obviously also driven by marketing but technically there's the user expectation that there's one multi-model model that can do all the fancy things that they have seen on twitter or whatever. And then there's the reality that even if you have kind of like complex systems doing fancy tasks, sometimes you still have different models under the hood that are specialized for different tasks and that just appear to be kind of like one uniform model which doesn't exist in this form. And to probably somehow have a mental image of that, it's probably like the reverse elephant problem. So you might know the story of the elephant and the blind man where you have multiple blind men describing an elephant, and one touches the tail and says, oh, it's a rope, another one touches the body and says it's a wall, another one touches the leg and says, oh, it's a tree, and it feels a bit like, especially multi-model learning, we now have the reverse part, that everybody is dealing with different models, but they all say, oh, I have a multi-model model, and there's this weird feeling that multi-model models can do anything, when in reality, they're actually dealing with a lot of different things underneath. So that's another interesting one, especially because now people are coming to us, oh, can you build this? And then the answer is, yeah, but this is a different one. So it's very interesting. But to not actually end on bad terms, there's also a lot of things that we have gained from kind of like this evolution. And coming from the vision side, probably the most important is that we have now new interfaces. So compared to what we had before, we are no longer limited to indices and we can actually kind of like interact with the world and the world can interact with images in a completely new way that wasn't there before and related to that there's also something interesting changing and this is that we now have those crazily generalizable systems so before you have to imagine we had tons of different architectures but technically we were able to solve only a handful of problems usually something with classification and then we had 50 different architectures to solve ImageNet. And now we have kind of like the reverse problem. Now we are down to a few architectures, one, two, three, whatever you want. But those few architectures can now actually solve hundreds and thousands of problems, and we don't even know how many. And we still need to kind of like process this change somehow also in the community and figure out what we do with this new regime. Okay, so getting closer to the 40 minutes, let me just conclude, and I don't have fancy conclusions, just some final thoughts, probably about the field of multi-model learning. I have a feeling that we are going to proliferate even further, so this branching out is not over yet, it has just started, so we will probably see many more tasks in the future, but there will be very few concrete bars to actually jump over, and we will see more benchmarks, but as I already explained, more benchmarks also sometimes mean less comparability, not more. This is from my side, now from the more general side, and this is something that I tell my students, so I think it's a good way to end this. We have seen that big models are nowadays done by big companies with lots of GPU, lots of manpower, money, and so on, But what we have also seen, especially in the time from 2010 to 2020, is that the biggest innovations usually don't start there. They start actually very small. They start in small groups with small papers where nobody was actually giving tons of money. And keeping that in mind, even if it feels sometimes a bit overwhelming for me to this community, please continue your support for open source software. And finally, again, from the research community to Python, thanks so much for building this. We could not have this done without you. And with that, I thank you for your attention.

Speaker 1 [39:42]

Thank you so much for that talk. It was really really amazing and we have tons of questions First of all, I would like to not point out that Hilde is going to be here the rest of the day So even if you do not get answered all questions, you can reach out to her The first question is why does multi-modal embedding work if you can show that different modalities are embedded in different places for example for CLIP

Speaker 2 [40:08]

Again, can you repeat?

Speaker 1 [40:09]

So why does multimodal embedding work if you can show that different modalities are embedded in different places for example for clip

Speaker 2 [40:18]

Got it. Let's say there is the practical thing that, so probably two to three aspects to that. The visualization that you saw is always kind of like taking a high dimensional space and it's actually breaking it down to two dimensions, to 2D, so that you can visualize it. In best case, we can visualize 3D, but we still have a huge reduction in dimensionality. And this obviously takes away a lot of the expressiveness and as I said in very high dimensional spaces like 700 to 1000 dimensional spaces technically we know or you might have heard in machine learning that there is something like the curse of dimensionality so technically each point has the same distance from each other point when you get high enough in dimensions and as I said the best idea we have currently is that we have high dimensional tiers that somehow align in their outer hull so that kind of like text representations get close to image representations, but they are still closer to the next text representation. So long story short, it's an interesting area to research, but practically you can just accept that it works anyway.

Speaker 1 [41:36]

Thanks. And how do semantically similar things end up near each other in the embedding space?

Speaker 2 [41:45]

part of this magic of contrastive learning. So this is really because the embeddings that the networks produce let me slide back to the slide if we can pull it up. So it's really that we actually train the model that when they see content on the text side and content on the image side that they should produce two vectors that are as similar as possible or as close as possible in the embedding space which technically means give us to sequences of loads that are somehow very similar.

Speaker 1 [42:19]

there's a lot of hype around Yann LeCun's JEPA approach. What's your take on it? Is the future multimodal?

Speaker 2 [42:33]

Let's say I still need to go to the JEPA series in detail, so we always knew that autoregressive generation is good for representation learning, and again it's a bit tied to this problem, can I tell you if this is really the best? The answer to that would probably be no. But I have some students who will probably work on figuring this out this semester, so So at least we can give you hopefully a guess for small data regimes, but then we always have the discussion, okay, but what if you could do it on 10 times or 100 times the data? And that's always a bit hard to actually figure out. This is exactly this comparability crisis.

Speaker 1 [43:17]

What is the intuition behind the adapter, the bridge between text and image?

Speaker 2 [43:26]

The adapter is actually something pretty interesting, because let's say the problem is pretty obvious. You have two models that you need to pre-train, so you need to pre-train separately. You need to train a language model on a lot of language stuff, and then you need to pre-train the image model on a lot of image data, and as you saw, they form two different subspaces, and somehow we need to glue those two subspaces together. And technically that's the role of the adapter, to take one set of tokens and ideally just do a translation so that whatever was the meaning in the vision space or whatever was kind of like a token for a dog in the vision space, so capturing the visual features, turns into something that would be close to the embedding of dog from a language model. That's technically the role of the adapter.

Speaker 1 [44:19]

One last question, what open source vision model do you recommend?

Speaker 2 [44:25]

It depends highly on what you want to do, so for if you want to train those kind of like lava style models Any clip or any contrastively trained model works good The current thing that everybody uses off-the-shelf is usually some sick lip sick lip v2 Version, but there are also some other kind of like flavors like rice clip that some people have shown to be 2% better But yeah, so technically any off-the-shelf model will do The main performance increase actually comes from having good data.