How to Do Monolingual, Multilingual, and Cross-lingual Text Classification in April, 2024

Speaker 1 [00:05]

And before the talk starts, I want to give a few pointers. The Q&A session will be conducted via Slido, so you're encouraged to post your questions there. You can also upload a question that you find interesting. Please also mute all the devices that you have during the session, and you're encouraged to post about the talk on Twitter. Please use the hashtags PyCon.de and PyDataBerlin. Yeah. and with that yeah great thank you very much for the introduction yeah my name is and i'm postdoc and tm and i'm really very glad to see you all in my talk so yeah it's an attempt to keep up with all the rapid development in nlp uh so yeah a couple of words about myself first i am researcher in nlp and i finished my phd also in this topic so i'm doing this already for like six seven years altogether and my research is very much connected how we can apply all these very powerful models you can see they are really developing almost every year every half a year every month through many social important topics like how we can fight misinformation how we can fight toxic speech how we can fight hate speech and yes i think also It aligns with this year's thoughts and ideas of this year's conference. We also heard a lot already about AI for social good, NLP for social good. It really can cover a lot of topics, not only some textual things, but also maybe healthcare, agriculture. This year, I'm also co-organizing a workshop at YMNLP, which will be in Miami, and where we also will have a discussion about how we can, again, think about how we can apply and do some post-social impact with all these big models. Yeah, and I also want to say many thanks to all my colleagues, because this talk is very based on the work of my colleagues, PhD students, master's students, and my supervisors, and we did a lot of experiments about also different domains, about formality, about toxicity, whatever we'll talk about today, and we then published quite nice papers. You can also find them online and also read if you want to, yeah, I don't cover many more details. Yeah, so, yeah, again, so this talk is about trying to keep up with all modern NLP, and actually when I was preparing this talk, I was really very ambitious, and I was like, yeah, so I will indeed will try to keep up, and I will try to monitor all new models, and when I was going from Munich to Berlin, it was on Sunday, and I was really just preparing some slides and thinking, yeah, it was, you know, quite come two months, and I really feel that, indeed, I covered all the most popular models. And then what happened? If you also read on Twitter and read in different NLPUs, Meta introduced their new model, Lama 3. It was April 18, and just yesterday, actually, it was today's morning for me, Microsoft published also their new model, Phi 3. So what can I say? I will try my best and also with this talk we will already not cover these super modern models because they're just several hours old but yeah at least we will have a glance of how we can use and if in these models I did so capable or maybe not so capable yeah so basically what we want to find out with this talk first of all if we want to have a classifier yeah we'll talk we'll about very typical NLP task if you have ever had courses about NLP or maybe heard some courses online so really we have some data set and we have labels and we just really want to input some samples and then have this label and first to set up if we want to have such a classifier for English language which is usually the most resourceful language right now so yeah if we have some new appearances of data or models it's usually for English language but then of course English is not the only language in our world and maybe we want indeed to have other classifiers for other languages and then maybe we don't have actually so much data for this language and what we can do with this and if we can indeed have successful classifier or maybe not have but yeah hopefully we can do this and on the other hand yes like finally maybe if we have data for many languages English and somehow also for other languages so if we can combine it all together and have indeed multilingual model for all these many languages and also how we can do this Yeah, and just to have all these examples, I will be talking about just three main tasks. Just, yeah, it's very typical NLP task, which you can see also, again, maybe in some courses and just online, some popular data sets. So the first one, how we can detect toxicity or hate speech. Yeah, so toxicity, it will be about if we have a sentence with some obscene lexicon or some root intention, we consider this text as toxic, and otherwise it will be just non-toxic. So hateful is about if we want to insult someone based on their race, background, or something else, and all other sentences are counted as not hateful. Grammatically classification is about if we have maybe some sentences that have some short words or maybe some images and do not follow all the quotations, we count this text as informal, and on the other hand, if we have fully grammatically correct sentences, the sentences are formal. And also third task, I will have some examples about this as well. So it's not like binary classification task. It's actually a relation about two sentences. It's called natural language inference. So if you have one sentence called actually premise and another is hypothesis, and we want to classify a relation between these two sentences, it can be contradiction, entailment, or neutral. So this, yeah, very typical NLP task. And yes, if you will see some maybe root sentences on the slides, it's not to offend you. It's just for illustration. Yeah, and another value that maybe I wanted to also come up with this conference. Yeah, so also we talk about a lot about open source, yeah, at Piedate Berlin, and very huge resource for NLP right now in terms of open source in HagenFace. So, yeah, how many people have heard about HagenFace before? Oh, yeah, cool. Very nice. So we have a very dedicated audience, and I believe, yeah, maybe you have heard of it, but maybe how many people indeed using it every day in their job, or... Yeah, so a little bit less already, yeah. So just for a couple of people who maybe don't know, HagenFace is a very big platform, and it hosts a lot of model data sets, and actually all even these big players in IT industry are also trying to publish all their models in open source at HagenFace. and nice thing is that not only you can publish something there or download something from someone you can also for example if you don't have GPUs or you don't have a computer to hold these models you can also call APIs from HagenFace for some parts for free, for some parts for maybe not so big costs and they will also inference your models for you just also to mention so this talk is not sponsored by HagenFace but could have been but I'm just very excited about this platform, and really, as a NLP researcher, we take and we're trying to also publish everything on this platform. So let's start with our first task, and yeah, EVE was the darkest language in English, and we indeed can have some data, so indeed there are so many data sets in English language. For example, for CC detection, we have Jigsaw data set that covers several millions of samples, actually, of different granularities, and there are also some other data sets of hate speech. For example, for hate speech, a very famous one is Hate Explained. For formality classifications, there is a paper called Dear Sir or Madam, Introduced to the AGF Data Set. They have also monolingual and multilingual versions. We will talk about this later. And for NLI, there are also actually several versions of NLI data sets for English. One of the most popular ones is SNLI data set. I think it was from Stanford Group. And, yeah, so it's really, these are quite popular data sets, and you can also can see them in many papers using as benchmarks. Yeah, and today is April 2024, and what we can do, if we want to do some classifier, we actually today can either prompt some large language model, and maybe, yeah, expect that it will give us some correct answer. On the other hand, we still actually can fine-tune some mass language model additionally for our downstream task. Yeah, let's discuss indeed what should we do. Yeah, just to mention, yeah, indeed, like this prompting of LLMs, it doesn't require any fine-tuning. Indeed, you can just host LLMs somewhere. It can be your machine, or it can be your server, or it can be someone else. On the other hand, actually, indeed, you also even can fine-tune LLM, but we assume that we are not open AI, and we are more humble people, and we will just try to prompt these LLMs. Yeah, indeed, if we want to fine-tune something, so usually maybe we will go with masked language models. Yeah, how actually then indeed to choose LLM? I also will advertise one of the pages from HagenFace. It's called LLM Benchmark, and usually all LLMs that are published at HagenFace, they're also evaluated on this benchmark, so then we can understand indeed what the model is good about, what this model is not good about, and I really recommend to visit this website And you can also filter these models based on their pre-trained data on, I think, even languages, on the sizes, which is, yeah, size is important in this case, because really they can be huge models of several hundreds of millions of parameters. Or maybe we can come with, yeah, deal with several millions just maybe for this task. So, yeah, really, I recommend this page at least to understand the picture of what is going on. Yeah, and so maybe we have chosen some more or less decent models of more or less adequate size for our humble resources, and I believe you have already heard the name of Llama, and you already have heard the name of Mistral, and these are open-source LLMs. On the other hand, we have this closed-source LLMs from OpenAI here, and also there is another a company called Atropic, yes, but one of the Google models, I believe, ChargeGPT and GPT-4, but actually, I also believe that maybe not all of people heard about Flan T5, so how many people have heard about Flan T5? Yeah, actually, so not all of you, and yeah, it's actually not indeed, like, large LLM, it's also was instruction pontoon still language model, and if you can see on the workroom, I also mentioned the amount of parameters, so Flan T5, it's from 60 million. Indeed, they have like XXL model, I think this is how it's called, is 3 billion parameters. So LAMA, Mistralis, this already scale from, sorry, it's a typo, actually 7 billion to 70 billion, and even some models even more. And ChagGPT is an ensemble of several billion parameters models, really like a huge ensemble. Yeah, but why here Flanti 5 is still interesting for us, because actually this is a model with which all this power of big large language models became so nice, and I really recommend to check out the paper, so it's called Scaling Instruction of Mentioned Language Models, and indeed it was a possibility of how these models can be indeed scaled. And the secret was that they also took a lot of datasets, these typical NLP datasets from different benchmarks and say here constructed this both text classification and text generation task in text generation task so like even classification task is first constructed in the generation task so you can prompt your task you can prompt also your sample and you will indeed get the label it will be generated as a text and so that's why they yeah and like the scene is that for classification if you if we again want to have the state classification task it's it's basically for what this model was designed, that it was instructionally fine-tuned to do the classification. Yeah, and that's why we will talk about this model, and I will show you the results. But yeah, so how we can design a prompt, actually. And there are also many recipes how we can do this. We came up within our chair and within our experiments with such a way that we indeed trying to tell the models we want to classify text, and trying to explain which domain we are working specifically right now. Also, you can even maybe insert some definitions if you have some, for example, domain definitions or definitions of your task from Linguist, so you can also somehow insert it into your prompt. And also, we're trying to restrict that, yeah, please reply with only one word, because then the model generates a huge amount of text. And also try, yeah, so even names, labels, so yeah, either this word or this word or this word. Then you can have several examples. Yeah, it can be like a few shots prompting, one, two, three, as many as possible. So also actually there is a scene to find this balance between not to give too few examples, but not to give too many examples, so the model will not be over-tuned for a task, let's say. But yeah, try to showcase what you're expecting, and then you have text, your text, and then you're expecting the label. but anyway of course the model can hallucinate and indeed sometimes it can answer with one word but it also can be not a case and I'm always recommending so for example you can at least find the first keywords that appeared in the answer yeah because maybe it can be a sentence or a text or on the other hand if you have open source models that you're using on your machine and you can control the outputs you can also check out their output from the last layer because anyways, the generation is a prediction of the next token, and you can check probabilities and also to find the most probable, again, presence of some keyword in this list of candidates for the next token. Yeah, hope it's clear. Talking about transformers, actually, we have a very big transformer zoo, and this is a reminder of most also popular and very often used models in papers. BERT model, which was also like built the foundation of all this research. And then we have Roberto models, which was just a bigger version of BERT and OptiForm in many tasks. We have GPT-2 model, which is actually a decoder part of Transformer. And we have T5 and BART models. It was also both encoder and decoder. But yeah, for classification, actually, we, in the majority of cases, use encoder-based models because we want to encode text. And maybe then, yes, maybe this is a slide to talk about. We encode the text into some dimensional space. and then we even should not fine-tune the whole model for our downstream task, so we can only use the classification token and feed it into fully-connection layer and also have, for example, yes of max layer afterwards and use this for classification, and only fine-tune actually this full-connection layer and yes of max layer, so we want to do yes of fully-connection layer, and we also can design it in any, how big, how many hidden states you want to have there. So usually, actually, we don't use very big fully-connection layers. It maybe indeed can be just one hidden state, but you can add more. Maybe you can experiment. It's always a heavy parameter. But yeah, once again, because these models were designed to encode the text and also they had special tokens, so usually you have in Transformers tokenizers this token separation and also token classification. And the idea is that within this token classification, we're indeed trying to encode the whole meaning of the text. So you can only work with this specific token. Yeah, so let's talk about some numbers. The first table is about toxicity and hate speech classification. So my colleagues, they had actually very big research. I have the name of their paper, so you're very welcome to check out it. And they indeed compared all these models like GPT-3, DaVinci, so it also was a GPT-style model from OpenAI, and then FLAN-T5 on a hate speech classification task. and if we look at the numbers and also they also tried different strategies but if you look at the numbers and what we actually expect we expect the GPT-3.5 or GPT-4 models they were heavily fine-tuned with this human feedback not to behave toxically not to behave hatefully and actually detect this hate speech that was actually the idea and this model we were expecting that they should be super amazing in detecting hate speech but this is not the case So if you indeed insert some hateful speech or, yeah, some hate speech data set, and you want to indeed fairly evaluate this model, so it's actually not so amazingly good. Yeah, so it's good enough, but it's not amazingly good. And if you look at the numbers, so yeah, for GPT-3.5, it's 0.61, and for FLANTY-5, it's 0.66. It's even a little bit better, or yeah, we can argue, so maybe data sets doesn't showcase the whole hate speech in the world. But anyway, Flan T5 model, without any fine-tuning of the model, which is way less than chat GPT, behaves on the level with the model. Also, I have in comparison a classifier that we also, and I personally, fine-tuned for the task of even this dataset specifically. So, for example, the second row is the BERT model. BERT actually cased model fine-tuned on head-explained dataset, and it performs already way better than these prompted LLMs. We also prompted LAMA model and MISRAL model, and they also showed maybe not the best results. So actually MISRAL performed the best within this all LLMs. And actually, yes, it's actually kind of good model, and I will also have another information about it. And yeah, indeed, it's really super decent model. For toxicity, we also fine-tuned Roberta model. See, it's a bigger version of BERT. And yeah, it also was pretty nice, and it was not even a large model. So it's still very decent classifiers. Yeah. For formality, it's like another showcase with another task, and we firstly even prompted Lama as well, so Mistral was not so popular at that point, but we prompted Lama firstly, and we saw actually the results were not even so good, and we didn't even publish it in the paper, but we tried different classifiers, even starting with these old-fashioned classifiers with punctations, bag of words, fast text, and you can see that specifically for this task of formality, even these simple classifiers performed pretty good. Then we fine-tuned different transformer-based models, and indeed, yeah, of course, the bigger the model, it's better, and indeed, these models still are pretty decent. And, yeah, Mistral was, in the end, I would say, middle results. But, yeah, so for this task, it was the answer that it's actually nicer to fine-tune, again, some mass-language model additionally for the task than prompt LLMs. Yeah, and as this main takeaways from this part, that instruction fine-tuned LLMs can be really even more efficient and effective. Yeah, efficient because they occupy less memory and they also need less time for inference. But even if we maybe spend some compute to fine-tune language model for our downstream task, it will be even better and more also effective. Yeah, but let's go to another part. If the target language is not actually English and also the usual case for non-English languages is that we also don't have enough data. And yeah, so for me it was a case that we actually have Ukrainian language because I speak Ukrainian and it was my mother tongue and we also assume that we have a source English language, English. So what can we do then for this? And yeah, we actually have four ways and if I would name the main slide of my presentations, it will be the main slide of my presentation. Let's have a closer look for this. What, first of all, we can do? So the first step can be called back translation. So we have text in Ukrainian, and we just translate it to English because we also have right now very powerful models for translation from any language to English. And then we actually already use a fine-tuned classifier, for example, for English language, which is already English text and expects the label. On the other hand, we still maybe can try to use, to prompt LLMs, LAMA, Mistral, or any other model. What we also can do, we actually can translate English training data into our target language. And again, with the power of regular translation systems, they really allow us to do this, and then we can fine-tune to maybe LLM or just LLM for this downstream task, and then already have prompts in Ukrainian. On the other hand, so it's a little bit more efficient method how we can fine-tune is adapters. So we can use some multilingual language model for encoding our text, and then we have adapter in English language, and then we have also another adapter. Adapters, so also they are like, let's imagine, just not so big fully-connectional layers with some hidden states, and you can also stack them all together. And we freeze these first layers of encoding the language and adapter for the language, and then we fine-tune this last layer for our specific task, and then the magic can happen that we just switch the language adapter, so we have adapter in English, and then we substitute it with adapter in Ukrainian, for example, and then we can also prompt in Ukrainian and we can expect the label so I also recommend we also have the platform called Adapter Hub where you can find all these fine-tuned languages so they also try to cover as much language as they can but yeah, where you can find a lot of information yeah, another question is how we can choose the application system and we tried also three ways to do this and the first ones are open source and the other ones are commercial so DeepL I think is even a German-based company, and yeah, so what were the results? Opus, NLP, Helsinki, it's one of the open-source models, and it was actually pretty good in specifically translating toxicity. NoLanguageLandBehind is a big model for meta, and it was really also quite powerful for open-source model, and it was very good in translating some neutral synthesis, and the thing is that indeed already it was like a new generation of LLMs, and it was heavily over-tuned to be very polite, so that's why it was impossible to gain any hateful or toxic speech from this model. And DeepL actually was very good in every task, but just we stick to this open source choice, and we didn't use this model, but we actually recommend it, so yeah, it was really pretty nice model to do. And we had several thousands and hundreds of thousands of data sets, so translated from those ones that I showed you in the beginning from English data sets, and also we tried to imitate some semi-natural test set, maybe just trying to find some keywords with toxic sentences, just to imitate if we just not use translation data, but some natural Ukrainian data to test our models. Yeah, some also a couple of words about how we can design the prompts, actually it's absolutely the same structure, the main thing that anyway, all instructions, so all these key parts of prompt were still in English, so it was not possible to write everything in Ukrainian, but yeah, because the models were anyway heavily tuned for English language, so anyway, these tasks, definitions, everything was in English, and then, yeah, of course, we can have examples in our target language, and then our input in target language. Yeah, so, the results. Between LLMs, their kin and the queen was Mistraliai, and each one in every task Lama2 was not good at all, also we tried actually experimenting, translating Ukrainian data to English, and then again prompting Flunty5, but anyway didn't work so good so yeah, Mistral was really nice and we can skip this scene just to mention that if you want to find the model for your language that is somewhere included, it's very nice to go to HuggingFace, select your language and check that it was indeed participating in pre-trained data of the model yeah so the final table with results so we what we did for ukrainian language uh comparing all these different approaches what we can actually observe so yeah indeed mistral was pretty good but for toxicity fine tuning uh some uh language model was the best choice and it was their best behavior on yeah semi-natural test set for formality yeah mistral actually was quite good on both translated and semi-natural data and yeah adaptive training was actually also pretty nice And for NLI, it was actually kind of challenging task. So on one hand, if you look at the data, even in English language, those sentences are not super difficult. But if you want to emulate this data in other language, more morphologically rich language, it was actually kind of switching the domain. Yeah. Also, we did several experiments with formality classification tasks. So also there were several languages, and we're trying to combine, if we fine-tune on this language and then test on this language and et cetera, et cetera. So it was just a showcase that when you have many languages, it's also kind of heavily computing experiments trying to find the combination of language that will help you to transfer knowledge to other language. And it's still a very big room for research in our field. Yeah, but then anyway, main takeaways. So we indeed can use this approach of translating data and it can be a really good bridge for a bus pacifier for language. uh at least in our experiments for ukrainian mistral shows the best performance on a single language however yeah fine tuning language model for classification is still a strong solution and also we did the showcase of transfer knowledge for english to ukrainian but of course if we look on the family languages and the relationship between languages english is not the closest language to ukrainian so maybe if you're thinking about other language it maybe makes sense to have a look for example again at hagen face just looking for some languages that are similar to your language more than English, and maybe trying to do these manipulations together in pair with another language. And maybe also another table to how you can choose your fighter, depending on how many steps of inference you can have, and also on which external APIs you want to demand. Because of course, back-translation was a good baseline, but then you have to have these three inference steps, and you are heavily dependent on the translation system every time. With training data translation, you can translate the whole data set in once, but then you need to fine-tune, for example, yeah. With LLMs, you still also have just one inference, you can do this, but anyway, you can be dependent on API. With adapter training, it was one of the most efficient ways how you can actually fine-tune the model, but there were anyway some flaws in classifier, but yeah, if you're lucky and you have more data, maybe you can just indeed fine-tune the language model and just host on your machine. Yeah, maybe just a couple of slides. So indeed, if we have English language, another language, and more data, so if we can fine-tune some multilingual model, and we tried this for multilingual CC classification, and right now, actually, we have a shared task going on, so if you want to participate in the task of transferring style from toxic into non-toxic, you're very welcomed, but the recipe here was that we tried to have a balanced data set for training data and for test set across all languages, so yeah, per class and per language. And we have, right now, a multilingual model for classification of toxicity for nine languages. And if you look at the results, so it's more or less decent, more or less equal across all languages. Yes, of course, for some languages, it was the original quality of the data not so good. But anyway, it's possible to do this. The same I can say for formalities. Also, we tried to fine-tune models for all four languages, and we also tried different setups. But if we want to fine-tune, for example, some multilingual language model like BERT or the BERT test, it's indeed possible, and it can be really fairly enough for all languages. Yeah, so cross-lingual and multilingual classification is possible, especially right now, especially with so many models and possibilities. LLMs, you suggest you need really to cross-check if you want to do something rather than English, you need to cross-check how much data or even your language was present in pre-training data. It's really extremely important and how it's balanced or non-balanced in comparison to English data. Also you should not forget that actually it's alright to use all special classifiers, even WegeWars, even linear regressions, even LSTMs, and even just fine-tuning for models is also still valid. I want to just leave you with a couple of slides with links so you can check my Hagen phase. I was starting with a point about open source, even in my chair where I worked before and continue to work, I encourage all our research to publish online at Hagen phase, all the models for the data sets. You can find everything with all these repositories, so take pictures and QR codes. And the last QR code I want to share is that we're also trying to create a Discord server actually for all NLP developers, researchers within Germany and even maybe within Europe, so we are also very welcoming to join you, and we can discuss there all this crazy news about all these crazy models, and maybe some meetups, and maybe helping with some issues, or finding master CCCs, or whatever you need, so yeah, please join, and we will be really happy to see you. Yeah, so thank you very much, thank you very much for coming, and I hope it was useful and we will develop very fair and multilingual NLP models.