Germany's next topic model

Speaker 1 [00:03]

Let's have fun. Thank you very much for the introduction and thank you all for coming to my presentation I'm very excited to be able to present some of the stuff that I've been doing at holiday check Here today at this presentation The title might be a bit misleading it's basically the reason why I took it is because first of all the main output or like the main benefit of this presentation will be if you've been working with topic modeling and and somehow you weren't really successful to get good results with the data you had at hand, because this is what happened to me, so with the kind of state-of-the-art approaches. And it's basically the main part of the topic. And the second part, because why Germanese? It's because the work that I've been doing and the kind of data that we have at HolidayCheck is basically German review texts. So the data that I'm going to present is basically in German. I will try to translate them. So it should also be possible to understand it if you don't speak German. The reason why I took this picture here is because basically this is where it all starts, you know. People going to a nice vacation spot, and then when they come back, hopefully, in our perspective, they submit a hotel review on our platform. So this is basically one of our main assets at HolidayCheck. So people write reviews such like this. And one of the tasks that we as data scientists at HolidayCheck have is to automatically extract the topics of these hotel review texts. So basically we want an automatic way of extracting what people have written about in these hotel reviews. And if you look at the German text over here, you might see that there are basically three main topics that are being discussed in here. So it's about food, it's about the room where the person stayed in, and it's about a certain kind of activity, in this case, snorkeling. And this is the kind of thing that we would like to actually extract. But the main challenge that you're facing if you're working with text is basically, I think we already heard in the previous talk, text is basically it's strings is a kind of data structure that is a bit hard to handle when it comes to determining a way of defining similarity for these strings of course you can come up with an easy way of defining word similarity in terms of a string similarity like levinstein distance or other word edit distances but this doesn't really help in terms of topic modeling and so I'm gonna also talk about two different ways of defining word similarities, contextual similarities, that will also be helpful in defining these topics. The overall structure of the presentation is basically threefold. The first part will be about word embeddings. We already probably most of you heard about word embeddings and one particular implementation which is Word2Vec, which I'm going to present also for our data here. And the second part is probably more interesting because it's defining contextual similarity which you can also have with word embeddings with a different method which I think is not so popular or not so well known in the deep learning community it basically comes from a text mining community as far as I'm concerned and it's basically it's called higher order word order co-occurrences and this is also what I used in order to define the topic so I come up with the topic modeling. And then the last part is probably the most interesting part. If you come up with the word similarities, how can you define topics? And there is one particular way how I did it or how we did it for the data that we have at hand at HolidayCheck. We define the topics as communities in a network of word similarities. So I'll show you in a minute what that means. So what is topic modeling to begin with basically what you can see as we saw before in the hotel review data you have a text and what you want to identify is basically a latent variable here so you have two variables that you can easily observe one is documents because this is the review text in my case could be like Wikipedia articles newsgroup articles and you have the words so if you just tokenize all the strings or the whole text in the documents in the reviews you end up having a list of words in the first place, or a tokenized set of words. And what you don't have is this latent variable, which is the topics. And this is what you want to have, because you can define it as each document consists of a mixture of topics, and each topic consists of a collection of words. So topics are defined by words, and each document can be defined as a collection of topics. And this, I usually see it as a way of dimensionality reduction in texts, because if you look at it this way, first if you have a large text, a large corpus, or in this case a large review, what you have first is the large dimension would be the number of words that could range from a few hundred words, in the small example that we have for the whole corpus, it's like thousands or even millions of words but you can actually reduce it to a much smaller number of topics like what we saw before the larger number of words in the review text could be reduced to three topics basically and then you can identify every document in terms of the topics but coming to word embeddings first in the first part actually what I said before is what do you want to have is first if you have all the strings all the words in your corpus what you want to have you want to define some sort of similarity in words and one very easy example of transferring one easy way of transferring strings into numbers is actually the one on the left if you have a one hot encoding then what you can do is you can just identify every word as a one position in a very high dimensional vector. So let's say you have a vocabulary size of 10,000 or 100,000. The vector would be of this size, like 10,000, 100,000. And every word in your corpus could be identified by having a one at this position and all zeros everywhere else. So in every column, you only have one one. Also, there is usually one position for out of vocabulary words for all the other words that are not in your vocabulary list but all the rest is zero so this representation is very sparse very high dimensional depending on the vocabulary size that you have and it's just hard coded which basically means that the words don't have any inherent similarity in here so even the position here doesn't show any similarity so independent of how similar the words in terms of their meaning are they just have a different position in there so you cannot just grab anything else in there and what you want to end up having and I will show in a minute how you can get to this is something on the right which we usually call in the deep learning community word embeddings which is much denser so here the differentiatings actually represent floating-point numbers so there is not only zeros and ones but every position in there has a floating-point number so it's lower dimensional so usually you don't need thousands of positions here but usually so I usually work with 100 dimensions for the embeddings in our data sometimes people use a bit more but usually there are not more than a couple of thousands and the most important aspect of these embeddings they are learned from data so as you can see here the position 2 3 & 4 these are words for the German equivalents of elevator and lift, basically synonyms Aufzug, Fahrstuhl, lift and as you can see in the shadings here they are more similar than the others in here so they have a rather white shade here and a very dark one here so you can already see that they are sort of more similar than for example the vector on position 4 and this is exactly what we want to have also in terms when we are working with topics now the question of course is how do you come up with such a representation and one this is kind of an investigation that is quite old in the linguistics literature going back at least to the 1950s where this guy John Rupert Firth coined a very famous phrase you shall know the meaning of a word by the company it keeps sometimes also like if you have Ludwig Wittgenstein, who was already mentioned in the keynote this morning, an Austrian philosopher, who said the meaning of a word is it's used in the language. So how does this help us? It's actually the kind of definition of distributional similarity. And as you can see over here, so these are two neural network architectures. Usually they are subsumed under deep learning, but actually there are shallow neural networks. On the left-hand side you see the continuous bag of word architecture on the right inside the skip gram Architecture and what these do is actually implement what these? People said because what they do is you have here you have a one at hot encoded input for three words And you have one output again 100 coded for one word And these are the surrounding words like two words before and one word after and this is the word in the middle of these three words and what you're doing is actually going through the text and trying to predict on the basis of the surrounding words to what the one in the middle or the other way around you're trying to predict the words surrounding the one here given the input word like the middle word in this case and if you go through this text which is sort of not supervised not unsupervised because it's you don't need any training to get data directly but it's supervised in the sense that you just go through all the texts through every word window at a time what you end up having is exactly this word embedding that I showed you before so these activations here in the hidden layer are actually the similarity things that we saw before so this is an actual I am as I said minus 100 dimensional and vectors but this is just 15 the first 15 dimensions of this and as you can see it automatically learned that elevator off to or off to faster lift are kind of more similar than to other words so what you end up having then if you have all these dimensions you can of course use a direct similarity on the basis of these 100 dimensional vectors and this you can do with like one approach it is with the annoy approximate nearest neighbor library from Spotify by the way I trained the word to back also with the fabulous Jensen library in Python and then you can visualize these 100 dimensions in three dimensions and this is what you can see here so all the different all the different elements in here all the notes in here actually our are words in our corpus, in our review text. So every word that occurs at least 20 times in our review text is shown here. So there are quite a lot of words in there, like 350,000 altogether. And if you include the edges, then you can see the closer they are in the space, the closer they are in meaning. So if you look at some of the words, I can maybe zoom in and show you something here. like maybe I just search for one word like if you look at the word croissant or croissant in German and you see there are similar words around marmelade you always have to take into account that this is user-generated content so there are also misspellings news please everything that is you can find at a breakfast table in a hotel and if you zoom out a bit more you can see that this is actually the kind of um the kind of breakfast topic or breakfast cluster that you end up having in this whole space you also find other things like all the different uh uh entertainment hosts for the or animateur in german then you get all the names of these people that people usually thank in the hotel reviews yeah thanks a lot we had a really great time at the hotel and And then they mentioned the people that did the entertainment part in there. And then you see all the different names. By the way, I always normalized everything to lowercase. That's why you have lowercase names in here. But you see all the different names in here. And again, as I said, if you zoom out, you see all these different clusters in here. And this is exactly the kind of thing that we want to extract. We want to identify these topics or these communities in our data. one fun fact of course as i said before having these vectors you can reduce them by taking for example a cosine similarity and just look if you have a word like shuttle spelled with le what are the most similar words to this and you see that shuttle spelled with el is the most similar one but then you get all other things like what people call a shuttle shuttle bus bus shuttle and all sorts of misspelling, like my favorite one is over here. So this is like somebody who heard the word shuttle and then tries to put a German spelling on this word. But, of course, it's nice to identify this also in the texts. So once you have this information, so what can you do with all that information? And I'll just put it here. You can use not only the different words, but you can summarize them in documents. As I said, they are reviews. But even if you take on a higher level, each review belongs to a hotel. So you can even compare hotels on the basis of the similarities of the words that are contained in here. And this is actually a Disney visualization of the 10,000 most important hotels in terms of the number of reviews that we have in our database. And the colors actually indicate the country where the hotel is located. And you can already see that the countries are actually quite nicely clustered in here. There's one big blob over here, which is basically Germany. And then if you look at it in more detail, I think I cannot do it here, is that this is basically on the right you have like the biggest cities in Germany, like Munich and Berlin, Hamburg. and then on the left you have like the smaller places where you have where you do your wellness vacation basically and this is also the same for here if you go for Italy this is the north of Italy where you go for your wellness or for hiking and this is like the south of Italy where you go for a beach vacation and then it's close to know with other beach vacation spots so this is one something where we actually use it in our database so that you can get a similarity of hotels on the basis of what people have written about in their hotel reviews. Another thing that you can do is actually you can also use these kinds of similarities that you have for something like question answering. If somebody has a question about the hotel, you can also use the similarities of words to get an answer in the review text. like is, what I have here, is something like, if you ask, gibt es deutsche Fernsehprogramme, so are there any German channels in the hotels, or something that is very important for some of the people going on vacation, and then you see this is actually a snippet from a review that could be identified according to this question, it says, es gab auch deutsche TV-Programme, es gibt auch zu wenig deutsche Fernsehsender, and as you can see, it doesn't need to be the same German word. So here I have Fernsehprogramme, it's tokenized like word by word, but still it got TVprogramme or Fernsehsender as the synonyms from the answers. So this is also one application where we can actually use this sort of information. All right, so as I said, this is one way of defining the contextual similarity, But there is another way which I said is not so popular as far as I'm concerned and which actually worked quite nicely with regard to our topic modeling approach. And this is called higher order co-occurrences. For this purpose, what I do, I usually do or use sparse matrices for that using, again, the great SciPy sparse matrix module in the SciPy library. Again, Python has you covered if you want to do something like this. And what I do, I first take window counts, again, this time with Sison, and go through all the texts. And what I extract is a count, a huge matrix, how many times did word X co-occur with word Y in a certain window. So the window size is a parameter, let's say five words. And then you can say, okay, the word hotel co-occurred with the word room in this window size like 398,000 times. And these absolute frequencies are the first step in the processing, but of course you cannot just take them directly because otherwise if you just take them directly, you would always get the high-frequency items just dominating all the results. You would always get the most frequent co-occurring item as UND or IST, so the very high-frequency items. So what I do, I use an association measure called Poisson directly in this matrix multiplication. So this is just a sparse matrix representation. I think I don't have time to go into this. The reason why I have to do all this stuff is, as I said, it's a sparse matrix. It's sparse because many words just don't co-occur with other words. And if I just use the normal Poisson formula here, it includes a logarithm. So it would automatically make the sparse matrix into a dense matrix because all the zeros get into, I think, minus infinity. So what you have to do is you only have to take those values in there which are non-zero and calculate the Poisson value on the basis of this. And what you end up having then is what people call first-order co-occurrence. It's actually how many times or which word mostly co-occurs with the other one, taking this association measure. And as you can see with the example Aufzug, again, this is one word for elevator or lift, and the other one is Fahrstuhl, which is basically a synonym. Then you see the words that mostly co-occur with it is floor, available, stairs. Again, another word for first floor, second floor, staircase, and so on. And you have a similar thing for Fahrstuhl. You have similar words that co-occur with this synonym of Aufzug. And then what you can do, and this is basically the idea of higher-order co-occurrences, you can just maybe take a threshold or just take the whole list, and you can just compare how many of these words, if you just look at this pair of words, how many words co-occur in these lists. So how many overlaps do we have? because the idea would be the more overlaps we have in the first-order co-occurrence, the more similar these words are. And this can easily be done with a matrix approach. And what you end up having is basically this. So this is the second-order co-occurrence, and they have the most similar word for Aufzug. Apart from Aufzug, I should have left this out, is Fahrstuhl. The second one is Lift, Treppe, which is also a way of going to the next floor, and then some plural forms of these and then some other words with a lower lower personal value and again you have the same thing for fast rule and as you can see it can it's kind of similar to what we get with the embeddings but it's just a different way of getting these similarities in there okay once you have defined these similarities then the last part is actually about identifying topics in your texts and one way you can do it is of course like this using the state-of-the-art approaches which is one of them is LDA latent Dirichlet allocation and what you usually people do is work with texts like Wikipedia articles news group articles and there is a whole variety of topics in the different articles that you have and then what you end up having is something like this so you have one topic i don't know which one is this so i think this is about computers and everything and this is about the bible so you have different topics in these different clusters here but the problem is if i use the same approach and i also played around a lot with the parameters settings that you can have the hyperparameters if you do the same thing with our data what you end up having is this very big topic which basically includes all words that I would like to have in different topics. So it has staff, it has beach, it has breakfast, it has hotel, it has room. So food, everything in there actually belongs to this first big topic, but this is actually something that I would like to have as separate topics individually. And this happens all over the place if I use some matrix factorization methods, some other methods that people use for topic modeling with our data. So the idea would be, could there be something else? And I think one of the reasons why it doesn't work with the kind of data that we have is that it has these basic assumptions that I put in here, like the normal methods of topic modeling. And one of them is, as I said, they assume, like with the Dirichlet prize, they assume that topics are distributed across different documents. So you have one document where you have topic one or two, and then you have other documents We have topics three and eight. But what happens a lot with our data is basically more like this. You have like sort of almost every topic in every review because it's like a very close domain, a very narrow domain. And people just talk like about everything within one sentence. So in this case, they talk about the resort, about the food, about the service, just in one sentence, just separated by commas. And this is different from Wikipedia articles or even Amazon reviews where you have a whole range of different products in there. And with our data, it's just basically hotels and everything that you can find in there. Again, I already mentioned these topics are rather huge and not so fine-grained. In our case, it's really like different fine-grained stuff that we are interested in. And of course one also one of the assumptions with all the techniques is also that you already know the number of topics in advance So you have to give it as a parameter for LDA How many topics do you want to extract and this is of course very hard to come up first? So how many topics do you think? Is are contained in your corpus? So just to come back to the visualization that we have so we just have this huge similarity network and one of the reasons why i think the network is actually a nice implementation is this because there is something in linguistics where people say okay a language is actually a system where everything is connected and this is also how i understand it so actually this whole network is the kind of connections that you have between i mean only on the semantics of words there is also so leaving out grammar altogether but what can we do as i said we would like to extract these clusters from the topics and there is one approach in the graph theory or in the network community which is called info map community detection which basically does that so a community detection algorithm actually tries to extract parts of these network of the network which are highly connected and which are less densely connected to the rest of the network so it's basically just like a clustering algorithm, but especially tailored to networks. And again, if you're in Python, then this is basically included in the iGraph implementation, which is also available in R, and also parts of it in NetworkX, of course, which is a great network library, which I used for this purpose. And what you end up getting, if you just apply this InfoMap algorithm on on this huge network so these are the kind of topics i always give the first 10 words here some of the topics that i could extract so the first one is breakfast basically the third one in here is um yeah as i said before the entertainment guys and thanks and greetings to them and the whole team then you have also some more fine-grained topics like deck chairs and parasols and stuff like this or for those people who travel with toddlers or with babies you have like baby bed and everything that is connected to this and people also want to know about this in our review text so there's a list like it goes on like i said before so this is the first thing that we have but you can get like this is the bad cluster so to say where people talk about like all the stains and mold in the room and you also get some of the things like here we have all the German TV channels this is one topic is very interesting then of course which month to travel then there is one topic which is pretty strange it's actually all capitalized words because sometimes people just write in all caps for some reason and they get a different than some names, Spanish names and so on and so forth so this is something which you could easily extract with this and just to get to the last point because i'm running out of time um so this is a hard clustering basically so every word in here just belongs to one topic but of course there are words like lift for example which can belong to different topics and in order to get there what you can do is you can again use these similarities that you could extract from the contextual similarities and you can just look how similar is this word to topic one two three four five and then you can normalize it and what you end up having then is some kind of a list of topics to which the word lift for example belongs and as you can see here lift belongs to topic 58 48 and one and all the rest is more or small numbers if you look at it 58 is actually skiing lift and everything is connected to if you go to skiing area the 48 is basically if you're handicapped or if you have a stroller and want to know if your room is on the second floor if there is an elevator on there and the last one is basically connected to the first we saw it's basically for those hotels that are in a skiing area where where it is important to know how far is it to the next lift. So this is actually the topic of where is the hotel located in this area. So just briefly, these are the individual steps. I'm not going through them again because I already mentioned some of the libraries that I used. But just to come to an end here, I want to say that I briefly mentioned that in passing. But if I were to write a review about the Python ecosystem, I would definitely write it's amazing because these are some of the libraries that I used in order to get all these things done. And I would also seize the opportunity to thank all the people who contribute to these libraries or who write some other interesting stuff that we as a Python community can use. And this is just a short thanks to all the people who make the stuff that I can work on all these things in Python. Because except for the visualization library, I actually also put his name in there. I forgot to mention it. Everything else was done in Python. So there was some node for the visualization, but everything was done in Python. And I think it's very, very cool that we have all these opportunities in the Python community. So thank you very much for your attention. so thank you so normally we are completely out of time so no time for more questions but as in five minutes it's only lunch and not another talk if you are if you are okay with it uh i would allow maybe one or two questions if there are that 3d illustration looked great what package that you use for that one yeah I just said that I forgot to mention the guy that's actually by Andre Kastja so I put in here actually if you want to use it here's the link to the visualization it also looks quite nice on a mobile phone unfortunately I cannot show it now yeah because it's like you have to show it to different direction but you can use it in there I put the link in there by the way I will also put it on my I will post a link to the slides on my on my Twitter account later on so if you're interested it's basically yeah I think maybe if I have internet it's implemented in with react and yeah just some I forget the name of the JavaScript library to render all these points but But, yeah, if you look it up, you can easily find it. And he also has a tutorial how to transform, if you have, like, the word embeddings, for example, if you just took them from Gensim, how to convert it into the format that he uses for the visualization. So, last question. Thanks for a great presentation. And my question is regarding the number of topics. How do you determine it? and then like what the approaches are there to do it thanks sorry what was the second part what's the available approaches to tackle this problem then how to know the number of two topics yeah so basically because the community detection algorithm you don't need to use a parameter for the number of communities that it want to extract it just determines for itself how many communities it will find so you don't need to specify this upfront but what we usually do we just go through so normally what the algorithm in a first step produces is a list of depending on the different parameters that you set for the words it's like it can be a couple of thousand a couple of hundred topics unfortunately I don't have the full list here because it would be too long but as you can see if you go further down to the topics you get first of all smaller and smaller in the number of words so sometimes you just end up with two words or at the end one word so you can just safely ignore these topics some of them are just synonym list like one which is actually a topic for us is honeymoon flit of a home host size which are basically three synonyms for honeymoon mere synonyms for honeymoon this is one topic that comes at the very end so what we do we decide we manually go through this and at some point make a cutoff for the way we say okay now it no longer makes sense to treat this or the rest of the topics as different topics so this is the kind of approach that I do in order to determine the last number of topics in there and usually as I said the topics are really good it works very well for our purposes I have to say but but usually when we implement this for anything, for a live feature, I would look at them manually before we just take them in there, but it's a very good starting point to have. Okay, sorry, so we are out of time, so let's thank him again for the nice talk.