Decoding Topics: A Comparative Analysis of Python’s Leading Topic Modeling Libraries Using Climate C

Speaker 1 [00:08]

So, hi guys. My name is, yeah, you can just call me Lisa. No need for Dr. Lisa. Soon it's going to be Professor Lisa though. And before we start, just one announcement. I'm also going to say it on behalf of some other speakers. Although I'm a very intelligent woman, I have a certain like limited brain capacity. English is my third language. So if I'm going to be saying something that might trigger anybody because it's not gender neutral, please cut me some slack. Okay? let's just focus on the technical and the technical stuff so who am i i'm many things as you say in russian i'll definitely not die of humbleness however what's important today is that i have a phd in computer science from university college london which i finished a couple of years ago and the topic was computational argumentation yeah that's why i'm single because i out argue every man and the last two years I spent traveling around Southeast Asia and mainly teaching Python online to bankers. I pay a lot of money and yeah but I'm starting a full-time position as a professor at Nazarbayev University in Kazakhstan in a couple of months and yeah that's that's who I am. So the content of today's presentation is going to be first of all I'll introduce you to topic modeling very quickly I assume given that you are here you already know what it is anyway then why this talk why I decided to talk about topic modeling in the first place the different topic modeling approaches covered doing my talk for use cases that I'm going to cover based on industry experience yes I worked in industry for one single year and then I quit and off to Asia and then Then an overview of the Python topic modeling libraries, the data set that I used, why I chose this specific one about climate change policy documents, then very briefly the implementation of each approach, the evaluation, the results for each use case, then the strengths and weaknesses of top to back and bird topic, the main topic modeling libraries that I'm going to cover, and a conclusion. So what is topic modeling? It's an unsupervised machine learning technique to automatically identify themes or, well, topics, hence topic modeling, in very large data sets. So you can think about tweets, customer reviews, news articles, like literally anything in textual format. However, well, most of that corpora is not labeled, right? So hence unsupervised. So once we throw that data in a topic model, we can cluster this data, which might reveal a hidden structure in unlabeled data. So, for example, think of news categorization, social media monitoring, customer feedback analysis, and legal documents as well. And traditionally, so probably what you've covered most likely at university, if you study computer science or machine learning, most likely covered LDA, which is a probabilistic word count model. However, well, a lot has happened since the early 2000s, and today we're mainly using pre-trained language models, right, which can actually capture semantics, yeah, and take syntax into account, which LDA doesn't. So, yeah, so whether you're a researcher, a data scientist, or just, you know, dealing with large text corpora, Now, topic modeling can be of immense use to you. And today, maybe some of you will be disappointed. I've heard complaints that some were annoyed that some of the talks are very superficial. I'm going to cover out-of-the-box libraries, because I want this talk also to be useful for those who maybe don't have that much Python experience, who can't code a topic model from scratch, for whom it's much easier just to import a library and use that library and know how to tweak it. Also, yeah, I'm also not a software engineer, so this is also why I prefer out-of-the-box approaches. I know some people here, they quote stuff from scratch. So cool. So why this talk? Well, this talk was... Well, I decided to give this talk because it's based on the very short time that I worked in industry. I worked on a project in legal tech. And I was hired because I suggested to use topic modeling during the interview. and the guy liked it and hence he hired me and we achieved really, really good results. One of our clients was DEFRA, which is the Department for Environment, Food and Rural Affairs in the UK. So I used to be based in London. And they wanted to know which parts of regulations address certain topics of interest. So for example, flooding. Are there any regulations to deal with flooding? Are there any regulations to deal with air or water quality? Things like that. And yeah, we used top-to-vec, and as I said, we achieved great results, Defrag was really happy. However, Bird Topic was already around during that time as well, but it didn't support certain features that we needed. We'll get there. Now it does, kind of. However, back then, also because it's industry, we have deadlines, we needed to act fast. I never did a comparative analysis between top-to-vec and Bird Topic, and I thought, now is the time, why not do it, and why not give it as a pike and talk? So, yeah, LLM approaches using prompting will be not covered. Maybe next year. So that's already an idea for next year. Don't steal it from me. So as I said, I'll cover three topic modeling approaches. The one, well, the basic one, which is latent Dirichlet allocation, LDA, which was developed in the early 2000s, so over 20 years ago, and it assumes that documents are mixed topics and topics are distributions over words. So it works using co-occurrence statistics and outputs word clusters, like for example for topic A, government, policy, and election, and then topic B, game, score, and team. But because it uses a bag-of-words approach, it cannot capture, for example, synonyms. So instead of, I don't know, let's say government would be misspelled, that would be a completely a different vector, hence it would not probably occur in topic A. You get the point. So yeah, then top2vec came along, which was developed in 2020, and originally it used doc2vec, so it was developed before BERT, however now it also supports BERT embeddings. And it finds dense areas in the corpus using HDBSCAN, and then it identifies the most representative words in each cluster or topic. And it's fully automatic, so I forgot to mention for LDA, you need to pre-define the number of topics. So for example, it's a hyperparameter of the model, you need to say, for example, I'm looking for 10 topics, and LDA will find 10 topics for you. So what's great about top-to-back-and-bar topic is, well, because it's unlabeled data, you don't know how many topics there are. Also, as many as there are people, as many there will be opinions. There's no way of a huge language corpus, and that, you know, I give it to all of you No way you all will find the same number of topics, right? Everybody will cluster them or sort them a bit differently. So in Top2Vec, you don't need to provide this parameter. And same for BERT topics. It was developed literally just a couple of months after Top2Vec. And it uses sentence BERT embeddings also with HDBScan. However, it works a bit differently. So the clustering uses something called class-based TF-IDF to extract topic keywords. It's fully automatic. Automatic and it also comes as very interactive and nice visualizations. So it's all it's literally it's like a suite So everything is in there. You don't need to use any other libraries for example for visualization Okay. Now, let me talk about the use cases So as I said, some of them were inspired by my work that I did for Defra So for example, as I said, they wanted us to find similar documents. So let's say they already found some paragraphs so sorry for those who don't have a legal background so in the regulations in the UK can be really really long and cover many things there is no regulation which is just about to address flooding yeah most likely it's some sort of like climate change mitigation policy document blah and then there's like one chapter one paragraph which addresses flooding and so DEFRA already identified some regulations or some paragraphs within regulations that they found interesting and then they wanted us to find similar ones but not just by doing control F right but by capturing a bit more semantics so that's use case number one then classifying documents so let's say you want to build you have a website where people let's take a very banal example news news website and somebody publishes an article and you want it to be published under a certain category automatically without anybody having to label it so once you're happy with your topic model let's say you reviewed it you're happy it's about 30 topics. So basically how good are those topics that you can then use supervised machine learning to train a classifier which then can classify a new document according to those topics. Then real-time integration, so let's say you do have a topic model, you're happy with it, you got a few new documents but you don't want to retrain the whole model. Now also all those algorithms they're non-deterministic so they will always be a bit different when you retrain it. But let's say you're happy with the one that you retrained, you tuned it, you did a lot of pre-processing, you're absolutely happy, you absolutely don't want to retrain it. Is it possible to insert a new document inside that model and just put it into a certain topic? And so, by the way, Bird Topic in the past didn't allow that, and this is why we decided against using it. Now it does. And then dynamic topic modeling. So I never had to use it, but I'm going to cover it because Bird Topic supports it. Talk Octovec doesn't, so dynamic topic modelling means that we can track how topics evolved over time. And this can obviously also be very interesting to see how policies changed, how maybe the same topic, let's say in the early 2000s, used different words than we're using for the same topic now. So maybe back then we had one issue that people were interested in and now they're talking about a different one. So yeah, so these are the four use cases, which again, as I said, are industry inspired. So these are the three libraries. So for LDA, I'm using Gensim, probably many of you might have heard of it because it's not just doing LDA, it does many things. Then top2vec, it is not actively maintained. So yeah, we were talking about it yesterday, like there are libraries that are unfortunately not maintained, but however, they're still very usable. So last time someone was on my project, it was three years ago, I downloaded Top2Vec a couple of months ago when I started this research for this presentation, it still works perfectly fine. I didn't have to create some really awkward, weird virtual environment with a very dodgy version of NumPy, sorry. So yeah, everything worked out of the box perfectly fine. And then yeah, Birdtopic, Birdtopic is actively maintained. So I just checked GitHub, like the last update was literally a couple of days ago. So why did I choose that data set? So yeah, because I wanted to take something similar, which looks similar to what I've already worked with. However, I didn't want to take the regulations one, I wanted to work with something new. And this is inspired by some of my students that I supervised a couple of years ago when I was still working as a lecturer at UCL. So they used this data set by Biesberg et al., which is called Machine Learning for Research on Climate Change Adaption Policy Integration, an Explorator UK case study. And so, yeah, they created that data set. They scraped all policy documents classified as policy paper in PDF format from the website gov.uk. Then they handpicked 10 documents, each for adaptation, mitigation, and non-climate, which didn't cover anything about climate. And the final database contained around 14,000 blocks of text, right, with around 4,000 labeled as adaptation, 5,500 as mitigation, and almost 4,500 as non-climate. So that's the data set that I've used. The data set is much bigger, but this is the labeled part. So the same guys, they also, like, what my students did with their data set is they tried zero-shot learning. So there's also a lot of unlabeled data. I didn't consider that because I actually wanted to work with a smaller subset to not fry my computer, not to waste money on some virtual server somewhere, and because I actually also wanted the original labels so that I can compare, for example, LDA with three topics and see whether they're similar to the three predefined ones, pre-labeled ones. Implementation, so for LDA, I deleted stop words, that's for those maybe who are, I don't think we have any complete newbies here in NLP, but just in case, so stop words are words that don't contain any meaning, which for example, and, a, the, etc, because they will be in all sorts of documents, they don't provide us any information, so for algorithms that don't use the word order, these words are completely useless, so this is why we're deleting them, This is what I did for LDA. For top2vec, you don't necessarily need to do it if you don't use embeddings. But doc2vec, for embeddings, you can obviously keep the text as it is, because it is smart enough to not consider those. It can tell if certain words come up in all of the clusters. It just ignores them, but I still deleted it. I used the default doc2vec, so I didn't use any BERT embeddings. And then also what I did, I merged some smaller topics into bigger ones. So if there were some topics with only like 10 documents, And I merged it into the bigger one, which was similar to that one, using cosine similarity. However, you can also use top-to-back supports hierarchical clustering. So what you could also do is you can kind of like, you know, check the hierarchy and then merge models together, which basically like are together in the graph. And then they fork and become smaller topics. For bird topic, I didn't do any pre-processing because it uses bird embeddings, but I also merged smaller topics into bigger ones. So, evaluation regarding ease of use, quality, efficiency, flexibility, interpretability. So, all of them are easy to use. I said that at the beginning of the talk. That was my idea that I wanted to present you things that you can easily use without having to code a lot. a lot or at least let's say not code a lot of complicated stuff and maybe like do some pre-processing using pandas you know deleting stop words things like that but not implement topic modeling from scratch then the quality of the topics so for LDA clearly you like you really need to experiment and tune it because you just don't know how many topics there are you basically just you just guess so topics can be especially if it's a well a big corpora and you use a small number of topics, let's say two or three, they might be very, very random. Then for Top2Vec and Bird Topic, the topics are really good. They're different, but as I said earlier, if I give you guys a big corpora to label, everybody of you will come up with different topics as well. So basically what you just need to do is to sit down and manually evaluate it. But yeah, the topics that are generated, they're absolutely not random, like you can definitely see that the documents that are in one topic are very similar. Efficiency, so LDA only takes a couple of seconds to train, so it's very very fast because it uses the bag-of-wood approach and for top-to-back and bird topic they're obviously slower. However on the other hand you have to do it once and then you don't need to retrain very often. So it doesn't really matter in my personal opinion. I mean clearly if it would take days and a lot of GPU that's one thing, but with a corpus like that I could do it on my local machine without GPU, it took a couple of minutes. So yeah, we're not talking about hours or days. Efficiency, so sorry, quality of the topics, efficiency. Yeah, sorry, I got confused. So yeah, so these two take longer to train. Flexibility. So LDA is not very flexible, right? It doesn't have any embedding support. So yeah, all you can do is just choosing the type of pre-processing that you do. Top2Vec now, as I said at the beginning, the original algorithm is based on Doc2Vec, but now you can also just from Hugging Face import BERT embeddings that you want and insert them into Top2Vec. And BERT topic is super customizable. Interpretability, I think all of them are very, like when I was looking at it, at least in this data set, all the topics are very easy to interpret because all of those models they give you the top words right I mean you can check out all the words but they have like you know functions where you can just check out for example the top 50 words in top2vex case for example and then you can just see what what topics are represented with each word so with which words each topic is represented so yeah so interpretability as well and LDA is of course super easy to interpret because it's a probabilistic model. The others you could argue, well yeah but we don't know how those decisions are made, you know they use like UMAP or HDBSCAN, however if you then actually look at the topic words, in my personal opinion, they were easy to interpret. Like I wasn't surprised why those algorithms or those black boxes decided on those topics. Okay, results. So for LDA, finding similar documents works ridiculously poorly. Why? Because each vector of a topic is a vector of probabilities. So if you only have three topics, then you can see as an example, a document would be just represented with three numbers on how likely it is to be in that topic. And, well, just with three numbers, you know, using something like Euclidean distance or cosine similarity, it will not give you any super similar documents. I think it's kind of self-explanatory. Yeah, and then, as I said, it uses a bag of words, hand signals, word order, semantics, syntax, et cetera. So yeah, like, for example, the use case that I gave you for Defra, who already had some documents they were interested in, wanted to find similar ones, LDA would have been a complete disaster. not recommended. So Top2Vec works really well, so as I said I actually used that one for that client. What is also really nice with Top2Vec is that it saves the document embeddings and Bird Topic in the past didn't do it. I think maybe you could also have done it explicitly, so this is also what I've done now, Bird Topic still doesn't do it automatically, we'll get back to that, but with Top2Vec it's super nice, saves the embeddings after training, so you can very very very easily compare similarity of new documents to new documents used during training. So let's say a new policy document comes out. I want to know which ones are similar. I can easily just use Cosign Similarity and get the most similar documents. So same with Bird Topic. But as I say, Bird Topic does not save document embedding. So basically, again, as I said, if you then hosted somewhere on a server, you train it once, It doesn't matter how long it takes and then you save the embeddings also But but you need to do two things in top-to-back You don't need to do to train the model and you have the embeddings Whereas with bird topic you need to train the model and then again using that model you need to embed the documents Which is super annoying. So it takes like twice as long But it's doable. Like it's just an additional step Cool use case number two that was for classification. So with LDA just played around right? I played around with LDA 3 because there were initially three topics and And you can see then and then I just trained some out-of-the-box scikit-learn classifiers. Like I'm sorry guys I couldn't be bothered to do something super fancy. So just using scikit-learn, logistic regression, SVM, random forest, KNN, XGBoost and MLP and you can see the accuracies especially for logistic regression MLP for three topics are quite high. So I also checked how overlapping they were regarding non-climate change adaptation mitigation not so much but When I actually used those ones for classification, it also worked fine. So, either whether I used the predefined ones or LDA3, you can see we get accuracy up to 92, and I didn't do any fancy tuning. I'm sure, you know, by better pre-processing, actually bothered tuning it, et cetera, you could have gotten even better results. But then you see it starts declining. So, for LDA5, it already declined a bit, and then if you do something like LDA50, yeah, the results are really not that great, which is not super surprising. I mean, 50 topics is a lot of topics, but yeah, unfortunately, quite often in such corpora, this is how many topics there actually are. So for top-to-back and per-topic, can't remember the, no, I can't. So because it's a non-deterministic model, I always got different results for different runs. However, it was always around the number of 70. So in this specific corpus, top-to-back and per-topic, by the way, as well, they always found around 70 topics, something between 65 and 75. But as I said, if some super small topics are merged into bigger ones, especially for use case 2, right? Because if you just have a topic with, sorry, 10 documents, probably classification is not going to work really well. And yeah, you can see that it's definitely the accuracies are... Sorry, it's not accuracies, it's Evon scores. The Evon scores are higher than for LDA. They're not that great, though. But as I said before, it really, like, I don't like saying it, but it depends. So it depends on your use case. It depends on your case, on your product. If it was something like a news website, obviously 50 topics is too much. Like, you wouldn't want to classify documents into 50 different topics. However, for what I said earlier, you know, when, like, for use case one, right, where, you know, people didn't care about how many topics. They were just interested in, you know, similar documents or in which topic did that document end up with. I want to analyze that topic closer. Maybe there are other documents we haven't heard of yet. So here, maybe the more the better. Whereas for classification, I would probably recommend to reduce the number of topics. For bird topic, there was one big issue that it contained a huge class with the label minus one, which means outliers. So I included it for the classification. I included it like a class. It was massive. It was like one third of the documents. And I did a bit of research on how to fix that, basically you need to tune HDBScan. So BirdTopic seems unique, although it is well-maintained and although it uses embeddings and stuff, it uses also similar clustering algorithms like top2vec, including HDBScan. Top2vec also uses HDBScan, but we don't end up with a massive outlier class or topic, whereas in BirdTopic you do. So unfortunately, if it's about using things out of the box, I would rather go for top-to-back than bird topic, which you then still somehow need to tune. So yeah, also, which was surprising, is that I told you I didn't delete stop words, right? Because I was using embeddings. But it actually created some clusters where the most representative words were a, the, et cetera. Again, of course, it's not necessarily a fault of Bird Topic. Maybe, again, it's not my data set, and I also didn't go through 14,000 documents. Maybe there were indeed some paragraphs or some documents which really just contained a few sentences and didn't contain any interesting words. So, yeah, real-time integration, LDA doesn't support it. Top2Vec does support it without having to retrain the whole model. And Bird Topic, as I mentioned before, also supports it now. didn't in the past though. So as I said before, if you'd have a model, you're happy with it. If you want to insert a new document into it, works for top2vec and BirdTopic. And finally, dynamic topic modeling, LDA, top2vec don't support it, BirdTopic does. So again, if you have a use case where you actually want to do dynamic topic modeling, BirdTopic is the way to go. Then no need to waste your time with the other. Well, no need to waste your time with LDA in the first place. That's outdated. We're not using that anymore. That was just for demonstration purposes. So very quickly, for those who have never heard of it, what is dynamic topic modeling? So it's a collection of techniques to analyze the evolution of topics over time. And so BERT topic allows DTM by calculating the topic representation at each time step. So again, clearly you need documents that have a time step attached to it. So in my case, I did because those regulations, those policy documents were published in a certain year. So that was included in the data set, otherwise I could have easily found it online. So I did have in Pandas, I did have a column with the year. And so Bird Topic, it, as I said, calculates the topic representation at each time step, and we don't need to run the entire model several times, right? So how it works is that there is first initial model fitting without any time steps, and then it splits the data set on time steps by applying C, T, F, I, D, F to each subset of documents. And then for each time step, a specific topic representation is generated. So it's the same topic, but for each time step, the representation for that topic differs. And so this allows you to, first of all, track the frequency of topics over time. This is the graph that you saw here, which you could argue, that's not that difficult. I mean, that's just counting the year. However, it also allows you, I mean, I specifically didn't do it, but it allows you to analyze the evolution of topic representations. So in the same topic, what did people talk about in that topic in 2012 compared to, for example, 2020? So BERT topic allows you to do that. So, yeah, a very quick comparison. BERT topic might be a bit more interpretable. However, as I said, it has many outliers, but it has good visualizations. It supports DTM. It's highly modular. And it returns, in my personal opinion, from what I saw with this data set, it returns more similar documents than top2vec. top2vec. But be aware that I used doc2vec and not BERT embeddings. But BERTopic, yeah, for use case number one, it worked really, really well. And BERTopic is better maintained. So it really depends on your use case what to use. Both libraries come with their pros and cons. So conclusion, what is better? It depends. But what I want to tell you also, like from my personal experience, a lot of pre-processing is always necessary. Like, there's no way you will just have your database, you will run it through a model, and you'll You'll get amazing results under absolutely no circumstances. So I actually spent way more time, I mean, it's just data science 101. You build like more cool models, maybe 10% of the time and like 90% of the time you get data and clean it. So I had to do a lot, a lot of pre-processing. So for example, given it was, so not in this case, but for the legal tech project, right? Because it was actually actual regulations, they had a lot of legal lingo. So these are actually also stop words, right? So although there might be, like, it's not a and then, like, pronouns or whatever, but, for example, the word preamble or abstract or things like that, right? So you need to do a lot of preprocessing always. So don't think that you can just build an amazing topic model in a couple of hours. That won't work. And, of course, a lot of experimenting is necessary. So you actually need to go into each topic and check it out and, you know, and actually think about it. So you still need to manually evaluate and actually check out, like, what the models spit out. then yeah I always recommend merging topics and then again you need to experiment which way you want to how in which way you want to do it you want to do according to similarity or do you want to do it according to hierarchy again you need you need to experiment I can't give you an answer to that then of course for DTM only BERT topic because top-delec doesn't support it and so yeah here if you use case sorry use cases slash examples when I would use either one or the other. So for dashboarding and things like that because I'm pretty sure it's Plotly because they're all, you know, dynamic and pretty and interactive. So yeah, for dashboarding I would use Bird Topic. Then of course for topic evolution and dynamic insights, whereas for things like a search engine, similarity retrieval and just very fast prototyping I would use Top2Vec. So these are just my personal conclusions. And that's it. Thank you very much.

Speaker 2 [29:41]

Thank you Liza. We have some time for questions. That's good because we have many questions in Slido. Let's start with the first one. How do these methods perform on short documents such as tweets? What would you do differently when working with short documents?

Speaker 1 [30:05]

Very good question, Burr Topic is really bad with short texts, so definitely Top2Vec works quite well with short texts, Burr Topic doesn't. So you saw I had long texts and it still created a massive outlier class, so it clearly states I think even in the documentation that Burr Topic doesn't work really well with short tweets. So I would try Top2Vec, but I'll honestly say I have worked with tweets a lot, but not in a topic modeling context. So what I would recommend, what a lot of people don't know, and I also, I will admit, I found out way later than I want to admit, is that if you use something where you need pre-processing, for example, let's say you use top2vec without embeddings, you need to tokenize tweets very differently than the normal documents. And NLTK actually has a tweet tokenizer, which I only found out, like, literally towards the end of my PhD. I wish I had known that earlier. So, for example, normal tokenization would either maybe come. By the way, also, if you ever use, it's just a lot of extra knowledge now for you, so take notes. So, the way that, so if you use scikit-learn to vectorize, so if you use a bag-of-words approach. By the way, nothing bad with a bag-of-words approach. if your data is not really complex, if you use count or TF-IDF vectorizer from scikit-learn, you please go into the documentation and check out the parameters, because by default it deletes numbers, it deletes punctuation, it deletes punctuation and it deletes all words that are shorter than three token as three letters so you need to actually define a regex to tell the vectorizers to not delete that so obviously for tweets that's a massive issue so that that was regarding a scikit-learn and regarding tweet tokenization so if you use just normal word tokenized in NLTK it will maybe not that relevant these days anymore because now we're using emojis but still like you know a couple of years ago people would just write the smiley right which the colon and the bracket and that would get tokenized but actually that you want to keep together same with hashtags right you want to keep like hashtags and mentions and stuff together so yeah check out tweet tokenizer it is in the NLTK library

Speaker 2 [32:28]

Okay, so next question. This one's from Chris. What are the inputs for the classification task?

Speaker 1 [32:36]

The vectors of the... Sorry, the inputs of the... Inputs for the class.

Speaker 2 [32:43]

inputs for the classification.

Speaker 1 [32:44]

Yeah, so the vectorized documents using Bag of Words.

Speaker 2 [32:44]

Yeah.

Speaker 1 [32:52]

So because you see those are all non-deep learning methods, so it was just using a Bag of Words approach. So the sparse vectors for each word in the document.

Speaker 2 [33:06]

Okay, then. Yes, next question. How to handle freshly emerging previously unseen topics?

Speaker 1 [33:14]

That's a good question. Yeah, then you would need to retrain right if you so first Again, what I would do is I get a new policy document Let's say in this case and if I see and then you check what ends up in the model, right? You feed it into the model and then you check in which topic it ends up and then if you see that the document is Like completely different to you know to all the other documents in that topic Oops, probably you have a new topic maybe worth retraining the model Although I wouldn't risk it just with one single document that might actually just be an outlier But of course if you deal with like some let's say you wait for a year The market has changed whatever has changed whatever part, you know project slash Cooper You're working with then obviously you need to retrain the model so that this new topic is somehow captured by the model

Speaker 2 [33:57]

Okay, so could the BERT model used by BERTopic be used as a classifier for new documents?

Speaker 1 [34:06]

The what sorry can you repeat that called the

Speaker 2 [34:08]

Could the bird model used by bird topic be used as a classifier for new documents?

Speaker 1 [34:14]

I mean, Bird Topic is not a classifier, so there is no classification within Bird Topic. But as I said, so this is what I've done here, right? Like you create the topics, these are used as labels for your documents, and then you can train whatever classifier you want, right? Also more advanced ones. As I said, I just want to quickly do something out of the box. This is why I used to scikit-learn once, but obviously you could also take Bird Topic, or even freaking LDA, it doesn't matter. You just need the labels, right? And then you can build some super fancy unsupervised, sorry, some fancy deep learning model and actually use embeddings of the documents and not like I did here, back of word approach. But the topic or any of those topic models, you need to create the labels because they're unlabeled.

Speaker 2 [35:00]

Okay, so Sabina asks what measures can you use to define how distinguishable topics are?

Speaker 1 [35:08]

That's a good point and I will honestly say I didn't look into those measures. There are measures to Check like topic coherency and things like that. I decided not to do it first of all because time You don't want to sit down in Bali and analyze 50 topics and the topic coherence. I'm sorry So this is why I created those use cases and was like if any of you wants to actually evaluate those those models from a research perspective, feel free to do so. I actually wanted to do something, also because this is not an academic conference. An academic conference, I would have probably chosen exactly that approach. I would have analyzed for topic coherence and how clean they are and things like that. But because most of you guys, apart from very few exceptions in this room, work in industry and not in research, I was like, probably you guys will benefit more if I define industry use cases instead of checking how coherent topics are. So I'll leave that up to you. Whoever asked that question, you can do it.

Speaker 2 [36:07]

Okay, so the next one. What happens in DTM if the topic disappears? Will the model start filling that class with the closest document to that topic?

Speaker 1 [36:16]

That's a good question I Well first of all okay, what do you mean would disappear that the the the topic can't disappear the model was already trained So that topic is there right, but again so similar to the question a couple of Minutes ago if you believe that like time has moved on and there are new topics you need to retrain the model Or if a topic disappears in the new model well Then so it but a more but this but this model is already trained topic can't disappear topic like the frequency can drop as you can actually see here right like the frequency for here for some topics drops to zero so in that year that topic was mentioned right but it can't disappear from the model

Speaker 2 [37:01]

Okay, so one more from Chris, how do you recommend manually evaluating topics with human annotators?

Speaker 1 [37:10]

Okay, I would not do it with annotators, right? I would use it like so also what we did with my with my boss It was literally just a team of three and Only him and me we were we were coders We would literally just like look at the documents that ended up in one cluster and decide whether it makes sense or not Like I would not hire a huge team of annotators and be like, I don't know He's like what is there to annotate? Because then we're kind of like killing the whole purpose of topic modeling if you have the funds to get annotators and give it to them right and don't bother with with an unsupervised with an unsupervised method so if you once you've already done that either hire an industry expert but most likely you are already the industry expert you have it's your data set that you're interested in and then I would just deep dive into the topics and actually analyze them whether they make sense

Speaker 2 [37:59]

okay so the next one and do you have thoughts on modeling topic modeling with image video data instead of text

Speaker 1 [38:07]

instead of text. No, I absolutely don't know anything about imaging. I'm an NLP person only. The only thing I've ever done was the... What is it? The MNIST one with the... That's the only thing I ever did using scikit-learn. So with imaging, please, that's not my area of expertise.

Speaker 2 [38:30]

Okay, we still have some questions.

Speaker 1 [38:32]

Well, we still have some time, so hau raus.

Speaker 2 [38:35]

Did you use n-grams? Should you for top-to-back and bird topic and LDA?

Speaker 1 [38:44]

N-grams. So, an LDA, no. I used, well, n-grams of size one. I just used individual words. Top-to-vec, no, but you can. Actually, very good point. You can use phrases in top-to-vec. So, again, feel free to experiment. And in Bird Topic, because it's embeddings, there's no need for n-grams, because you use the whole embedding. But yeah, good point. If you use a bag-of-words model, Of course, you can use n-grams, and yeah, again, you'll need to experiment. I personally didn't, but yeah, good point. So top-to-back support style.

Speaker 2 [39:18]

Ok, so we continue. Do you think LLMs could outperform the presented methods?

Speaker 1 [39:24]

What can outperform the presented method?

Speaker 2 [39:25]

LLMs, Large Language Models, called Outperform.

Speaker 1 [39:28]

I don't know. I haven't tried.

Speaker 2 [39:32]

So, the blocks of texts are small parts of the original PDF in the Breeze book. Yes.

Speaker 1 [39:41]

Okay, yeah, in the data set. Yes, the documents were split in paragraphs.

Speaker 2 [39:48]

Cool.

Speaker 1 [39:51]

I mean, come on, it's 40 documents and we end up with 14,000 blocks of text would be a bit weird. The math doesn't add up, right? So yes, that was split.

Speaker 2 [40:06]

So yeah, we have three more questions and sometimes so if you still have questions you can still add Have you done topic modeling on multilingual data any insights or advice when doing so?

Speaker 1 [40:17]

No, I didn't. So, as I said, I did a legal tech project, which is obviously all in English, with very few exceptions, for some reason, Welsh and Scottish. But as I said, I used top2vec, so language wasn't really captured anyway, using doc2vec. Although I did also use embeddings back then, not for this project, but for the previous one. I can't remember what happened with the Welsh and the Scottish documents. Probably they all ended up in one And so, no, I didn't. And as I said, for LDA, that won't work anyway. Well, no, actually, it would. It's just that because bag of words is language neutral, right? It's just like it doesn't matter whether something is written in Cyrillic or in English. It's just a completely different token. So, sure, you can absolutely use LDA on multilingual data. and for Bird Topic, you'll have to check out one of the multilingual models from Hugging Face and plug it in and see what comes out of it.

Speaker 2 [41:20]

Okay, so for the processing, why did you not merge smaller topics into bigger one for the LDA?

Speaker 1 [41:29]

But because for LDA you choose the topics anyway, and I own it I chose And I chose either very small Numbers so for like again, we're dealing with 40,000 was 14,000 documents So LDA 3 we end up with topics that contain over a thousand documents and even an LDA 50 there wasn't Because 50 was hard coded right was predefined. There were no topics which were really small whereas for top-to-back and pro-topic there were.

Speaker 2 [42:02]

Great, so do you have any tips on dealing with documents that may contain multiple topics?

Speaker 1 [42:13]

Well, first of all, we never know like Many of those documents might have actually contained several topics, right? It's again something that I can't check in 14,000 documents So what I but what I would do then in this case, right? So this is this is what kind of use case one is about so which I don't have a Should have had a visualization. Okay, so I'll show it with my hands. So let's say you have this topic topic a Yeah, and they're like documents inside topic a right But let's say that one of the documents is kind of located here at the very outskirts of topic A Yeah, and then you have topic B here. So actually something from topic B might be actually closer to that document Right. So this is why like you can use the topic labels You know like labels from 1 to 50 or you can actually just check for similarity and then maybe actually for a new document, which Which would have maybe ended up in topic A if you check for similarity, a bit like K and N, right? So also like documents which surround it in the surrounding topics would also be extracted using a similarity score So this is how I would and then you can kind of also base You can you can set some kind of threshold and maybe based on this threshold you could even assign several topics to a document

Speaker 2 [43:32]

Okay then, what encoder model is behind BERT topic when you used it, Roberta, or did you test the new modern BERT?

Speaker 1 [43:40]

Modern birds. I didn't use Roberto. I just used the default on which is the sentence bird

Speaker 2 [43:46]

Okay, then the last question. Which method would you apply on text data containing unordered or unrelated terms? Would you expect some benefits from top-to-back or per-top-back?

Speaker 1 [43:59]

where the terms are unrelated. Say it again.

Speaker 2 [44:02]

Unordered unrelated terms. Text data containing unordered unrelated terms.

Speaker 1 [44:08]

Well, then you can try LDA, because if there is no actual human, do you understand correctly, so that it's just random terms, so it doesn't actually, so it doesn't form a human sentence, well then actually the word order doesn't really matter, right? So then you could even try LDA, well yeah, or top to back, then actually using bird topic might not be the best idea, because we kind of assume that it's a human readable language, if I understood the question correctly.

Speaker 2 [44:38]

Okay, then yeah, I think we answered all the questions and perfect timing

Speaker 1 [44:44]

Perfect. Thank you, Lisa. Thank you.

Speaker 2 [44:46]

Thank you.