Efficient data labelling with weak supervision

Speaker 1 [00:03]

Hi, I'm Maria Mestre. Thanks for having me. So a bit about my background. I did a PhD many years ago, working with biomedical signals. And then I moved to industry, where I worked as a data scientist for more than 10 years in different companies across different domains. and recently I started a company with my co-founder Stuart where we're building an annotation tool for text and it uses Wix supervision. It's called DataQA. It has an open source library so we'll be talking about the open source app today. So let's just share. so yeah i will be talking about data labeling with quick supervision um so i'm sure you've heard about it already data centric ai so it's the new catchphrase that everybody's talking about um and i think we hear more and more about it and you know important sort of voices in the community are also uh mentioning it um and i think it's basically because there's been a recent shift in the focus from the community where people are less you know talking less about improvements of of you know model architectures and more about the you know the impact that data has on machine learning for machine learning and i think the reason for this is because um ai has you know slowly moving or i would say has moved from academia into the industry um and now we're seeing it on you know it has reached a state of maturity where we're seeing it used successfully for many different applications and people are starting to realize that you know you cannot you cannot have a successful ML project if you don't have high quality data and very often if you're trying to improve a machine learning model you will get higher gains from improving your data rather than improving the model architecture So, yeah, just put a few examples of tweets of people discussing this. Last year, there was a survey organized by O'Reilly's where they were talking to companies that had deployed AI or were planning to deploy AI. And one of the main questions was, what were the top challenges today in adopting AI? And the two main challenges were, well, struggling to hire the right skill set. and also getting the right data, like lack of data or data quality issues. And I would even argue that these two issues are sometimes even linked because what we see is very often the data scientist at a company is also in charge of creating the training data and might end up doing some annotation. So spending a lot of time doing something that might be better done by a domain expert rather than the data scientist. and so one last piece of evidence that data-centric AI is here to stay so Stanford released a report recently with you know trends in industry and one of the main takeaways was that some of the benchmarks and state-of-the-art results that were achieved this previous year were mainly due to getting more data rather than improvements on the model or the model architecture. So with this, let me talk now about why is it so hard to get label data, right? Well, labeling is difficult. So depending also on the application, there might be some intrinsic ambiguity so that multiple people looking at the same data, they might come up with different labels. So, you know, most label data sets have errors. And I think many companies, they don't even try to, you know, to assess the level of error they have, which I would say is a mistake. You know, every time you have a label data set, you should understand what are the errors that you're introducing. um and yeah there was a study last year um about about this so um it's a paper called pervasive label errors in test sets the stabilized machine learning benchmarks so in this paper they looked at some uh data sets that were used for some very you know famous uh competitions in ai and they estimated the level of label error um and for example image nets they estimated around six percent label error they even have a website which you can it's kind of fun so they create this label errors.com and you can inspect the different you know data sets and you will see some of the examples of errors that you can find um yeah so labeling data is hard and on top of this it's a manual error so sorry it's a manual labor so it's hard to scale because you need you know that person looking at the data now this is where weak supervision is uh is quite interesting so in a very simplified process like labeling process you will have a an annotator look at a bunch of documents sequentially and labeling each document one after the other so it's a linear process. It's non-scalable. You might still end up with some errors at the end, even if a person has checked all the documents. And there's also, if there's a change of label definition or in the taxonomy, then you might need to start over. And so it's quite labor intensive. And the only knowledge you get from your annotator is really those little units of knowledge, which every time labels one document, you get a little bit of information about the problem. But it's a very linear, very slow process. And so this is why I want to explain to you now how you would do this with weak supervision. So weak supervision is a way to combine other data sources together. So in this case, the annotator, if they're a domain expert and they know the data, they will know things about your data. And with Wix supervision, they're able to encode that knowledge in a way that your machine learning model can also use that knowledge. So things like, for example, if there are external data sources or knowledge bases that are similar to your problem space, You might be able to use them, you know, to pre-annotate your documents, you might be able to use simple explainable rules, so rules about the structure of the document, rules about syntax, words that you expect to see, whether there's, you know, lots of all caps, punctuation, so all sorts of things you can do. and then you can also combine machine learning models which you might have trained on a different domain or you know slightly different data domain data source and you can also use those to pre-annotate your text and the idea is that so your annotator will not only give you labels for specific documents they will also give you information about the problem you know in the in the format of rules for example and then they will label the documents in a way that you can learn which of these rules work best so you might learn for example that certain knowledge bases work really well for certain classes or that for specific documents with different you know specific structures there's a rule that works really well or that if you combine you know assemble mode you do an ensemble model you have different weak models you learn um that they work really well if one of them agrees with another one for example so it's a way of you you get your annotator to give you some domain knowledge um in the in the form of rules then you add some machine learning layer to understand which rules work well and then finally you can automatically pre-annotate text um you know with a high level of accuracy and a way to summarize this is so in a no machine learning setting where a person is looking at one document at a time um you know for zero effort you have zero labels and for max effort you have a hundred percent uh label coverage right for your documents but um if now you add machine learning or machine learning with with supervision so there are different ways that you can use machine learning here but you can achieve a high higher level of accuracy with much less effort and you might depending on the problem you might decide that you know 80 accuracy 90 accuracy is enough for your labels or you might want to have 100 accuracy but using these techniques will still speed up the labeling effort significantly. So let me now show you an example. So DataCue is an open source tool. It is written in Python. The backend is Python, the front end is React. It combines, so a library called Snorkel, which is for programmatic labeling, quite popular library. So it's built on top of Snorkel. It's built on top of Spacey. So it's a rule engine which uses regular expressions, which you can apply either on the full text or on spacey tokens. It also ships with a text search engine with Elasticsearch. And right now you can do multi-class classification, named entity recognition or named entity disambiguation. And it runs locally. Okay, so you install it with a pip install. So it's a fairly new project still, and we're still learning what works, what doesn't, but we're really keen on getting feedback from the community, which is one of the main reasons why I'm presenting today. So yeah, you can head to the, if you head to the repository, you will see, you will see different ways, yeah. So how to install it, you can install with peep or you can install it with run a Docker instead of peep install. So there are different ways you can install it. Okay, so now let me show you an example of what I mean by weak, weak labeling. So I will show you an example with a classification project. So here, I have 3000 products from amazon and i'm trying to so these are my product description and i'm trying to annotate them or assign them a class product product category out of 24 in this case for example this is talking about a sandal this would be clothes clothing shoes and jewelry okay so data query has a search engine you can search you know specific words say i'm interested in looking at all the products that have sandal in them and then you can also label from the interface so you know this just label this sandal this is sandal so it's a way to quickly generate labels for specific classes especially if you have an imbalanced data set this works really well because you might find a way to label the minority class using a search. And then you can add rules. So I'm going to show you some examples of rules first that you can use. So this is, again, in the documentation, you have some examples of rules you can use. So you have bigger expressions, which you can apply on the whole text. You can apply on tokens. you can also apply them on entities so these are spacey entities so let's say for example you want to detect all the organizations that match this regular expression so they have American in them so you will you know American Cancer Society for example would be a match and then you can also create rules that take into account the order of the matches so you might say for example, I want to match any document that talks about the economy. So that has a sentence with a word, you know, starting with this regular expression economy, and it's followed by downturn or growth. So to match, you know, the specific sentences, the economy has taken a new downturn. Or if you want to do claim detection, you might want to say, I want to add any document that the name of a person so any person followed by say or claim in the same sentence so here you would be able to match all the sentences that have this kind of structure so in our case we will create a very simple rule which says i'm gonna copy this is from the tutorial very simple which says i'm going to match any document so if the text contains this regular expression so the regular expression is if it contains the word book or author or novel or guide then i'm going to assign the label book so and the idea here is that uh these rules if they have a high level of accuracy of precision then um you know it's a way to pre-annotate your your documents and that way you can when you're manually reviewing them you should be able to go much faster so it covered 416 product descriptions and so we can see that you know this one is correct so we say yeah correct correct correct correct and this way you annotate much faster than having to you know search the 24 categories and once we add we're thinking of adding keyboard shortcuts then you know things should even go much faster than this so this is a quick example now i'm going to show you let's say that i've done this and that i've created many rules so this is the same project the same data but now i have you know created 32 rules okay um to try to have as much coverage as possible um and so my coverage here is uh you know around 2000 documents out of 3000 so two-thirds of my documents are now covered by rules um so again these rules are very simple to come up with because they're just words um so So you don't need to be an expert. I mean, you just need to understand the data to be able to come up with them. And so I'm gonna now show you, so this is a notebook that you can also look at. It's in this repository, Data QA Notebooks. And here I run some experiments. I'm not gonna show you every cell. There's a lot of code in here, but I'm just gonna focus on this. this plot. So what I did was, let's say I had two data sets, a training data set, which I used to create my rules, and a test data set, which I didn't look at, you know, with complete unseen data. So I use, here I'm comparing what's the typical approach. The typical approach is you label some documents in your training data set, you train a model, and then you go and use that model on the test data set to get some accuracy. And so that's typical approach. So here, whenever I say, so ML random manual N, so N is the number of manual labels I did on the training data set, and this is the accuracy I got on my test set. And this bar here shows you, okay, what if now I used a combination of my manual labels as well as the rules. So my training data set now would be much bigger. It will have a combination of noisy labels as well as manual labels. And what we've seen for this particular data set is that there is a significant improvement in performance with the same amount of effort in labeling. And I think there are some studies I haven't linked to the papers, but this is, you know, weak supervision does work very well across many applications. because the idea is that you give more data to your model and your model is able to understand which are the patterns that work well and which ones are the ones that don't work well based on the accuracy of each rule. So yeah, this is for classification. I can also show you an example with name and entity recognition. so in this case um okay so my data is i have a thousand forum posts so let me show you the data first um so these are uh it's user generated content and these are people posting in a forum talking about side effects of drugs okay and what i'm trying to do here is label all side effects that I find. So my class is a side effect. And in terms of rules we can apply here, I'm going to show you some examples. The rules that we support are, we can have regular expressions on the entities. If you have like a vocabulary list, so this would be, you know, you could use this um you can tag an entity that appears after specific words so if you say you know i want to tag anything that comes after married to so obama is married to michelle obama so you would match that you can match on phrases so say i want to tag all the phrases in documents that have the word ingredients say like for recipes or i want to tag any noun phrase that matches is a list of symptoms and in this case you won't just match the you know say um headache or cramp you will match the entire noun phrase so if you have a sentence i have experienced mild headaches it will match the whole noun phrase mild headache so let's just run it over our data um so i'm gonna just copy paste from the tutorial because that way i don't need to type so say i have this you a list of side effects are pretty common, like headache, nausea, weight gain, dizziness, insomnia. So this is my rigor expression. I'm gonna go on my data and I'm gonna say, I'm gonna create a rule, regX match on the entity. And I'm gonna say, this is my list of side effects and I wanna give it the class side effects, okay? So side effect word list. create my rule. So what I see is it found, you know, 220 documents with matches. So I can go and then label them, see how well it works. So it says, this sentence says, the only side effect I've ever had was a bad headache. Okay, so this is correct. I did not experience weight gain or any abnormal side effect that is incorrect so we can remove it um you know it became again so this sentence would have some mist i mean the rule would not be able to catch everything but it did catch nauseated and it found a sentence that has a bunch of side effects so you know you can then label the missing side effects okay so that is an example of using rules for named entity recognition and at the end if you go back to the summary table you can see um you know after you've labeled some examples you can see what's the accuracy um and you can go uh on this tab estimated performance it will give you the performance of an entire class across all the rules i create in this case the position is around 67 percent and if you label even more documents it will give you also a range like a confidence interval um so i think i have time for to also show you symptom mapping so this is named entity disambiguation um so here let's say that i have mapped so i have extracted phrases with all the symptoms but now i have a taxonomy of symptoms um and i would like to map it to this taxonomy okay because people might say I have a headache or I have a migraine and you might want to map those two mentions to the same knowledge base to the same entry in your taxonomy which would maybe called headache and migraines I don't know just to come up with it so here I have um i have around 7 000 you know side effects in my taxonomy um and i've got you know 57 57 documents or a few documents with 57 mentions of symptoms of sorry of side effects and i would like to map this mentions to the um you know the knowledge base so what i do is i can then i go here and i can label i can label them so say okay i found here that um okay my depression has been caused by it so that's a side effect that was reported and these are the most likely matches in my uh taxonomy so my taxonomy has 7 000 entries so it's um you know do it this manually would would take a while because there's a lot of there are a lot of entries here so we do uh here like i'm matching um using just text similarity but it's it's a way to very quickly be able to label for this problem so moody like anxious mood for example that would be your label depressed would also be so you know people might say i feel depression or i am depressed and both should match to the same entity Crying is crying, sleep disruptions. I don't know what to say here, but anyway. Decreased appetite for loss of appetite and so on. So these are the types of NLP blasts today that are supported by DataQA. And yeah, so we also have a commercial version of the product, which we're making changes to the UI. Some of these changes will also go to the open source. But yeah, the commercial version of this product is hosted. It has things like user accounts, authentication. We support PDFs. we have a machine learning module where you can train a machine learning based on your you know given your annotations so it has a lot more functionality than the open source um so yeah thank you i don't know why i did this thank you so much for uh for having me um so again the open source link is there data qa uh we're looking for any feedback we're looking for contributors as well um anything you know this is the reason why presenting today anything we we you can tell us about using this for your problem is really welcome and of course if you like it stars again very very welcome um i put my email there if you're interested in talking about anything you know about annotations for your projects uh we're really open for any discussion any way we could help you uh we're really happy to hear from you and we also have a twitter account where we are going to tweet a bit more about programmatic labeling few shot learning all you know all different ways that you can annotate today much quicker yeah thank you thank you so much maria so we have time for some questions the first one is how does it compare to other data labeling tools like light tag or prodigy um so i cannot speak so much about light tag because i haven't used it but i think the focus of this tool is really the programmatic labeling bit so it's it's uh has a very advanced rule engine which i showed on the talk so i'm not sure how you would do this with other tools so prodigy uh i know that you can configure it um to use it also like you know with advanced rules um but i think another difference is the the no code so data qa really is meant for you know quick uh data exploration has a search engine uh you don't there's today we don't really support developer APIs or any integration with notebooks or anything like this. Whereas a 2LAC prodigy has, I think, a different end user in mind. It's really meant for a Python developer. Yeah, so I think that's the main differences. Thank you. So the next question is, what types of data does Data QA support, like images or PDFs? I guess you answered that already for PDFs by now. Yeah, so, I mean, it's for text, so we don't do anything with images today, but for text, we do, we can read PDFs, and so we would understand the different sections of the PDF, and so, yeah, you would find paragraphs, and you can apply rules on a paragraph level, on a document level, yeah. Great. How do I deal with a selection bias induced by tools like search? Yeah, so selection bias. I think when you are labeling, you do have to follow certain principles. So for example, one important principle is to label. So let's say you have come up with some rules. They will not cover your entire corpus most probably. So you do need to label documents that have not been covered by the rules. you know to understand something like recall for example um or you know miss patterns so that's definitely important um so and also you need to label examples of rules so otherwise if you train a machine learning program on the on these weak labels without having annotated any examples from those weak labels it will not learn which rules are working and which ones are not so this is something that I would like to add to make it easier from the interface to annotate in a way that we do minimize the selection bias, but this is work in progress. So we have time for one last question. What are your recommendations for labeling ambiguous cases? So, I mean, ambiguous cases, I think you do need to have either you have an oracle, or I would say a person in your company that is the one that will be the final judge of these difficult cases or do a notator agreement type of approach where you have multiple people look at the same example and use majority voting. I think those are the only approaches I know of. Thank you so much. I'd say you can maybe connect on Discord and the Kuppelsaal channel afterwards if you want to. And thank you very much for the great talk. Thank you so much.