Your first NLP project: peaks and pitfalls of unstructured data
Natural Language Processing improves the quality of your text data for future analysis and increases the accuracy of your machine learning model. It’s important to know what goes into the bag of words and what are some potential do's and don'ts of text pre-processing. Which text normalization steps are necessary and which ones are “nice-to-have”? Why is classic NLP still relevant in the age of Deep Learning? What metrics can be used to compare word frequencies and what can machine learning algorithms do with those numbers? This NLP talk provides answers to these questions and more! You'll see three examples of NLP pipelines using spaCy: sentiment analysis and emoji in tweets, named entity recognition in Yelp reviews, and multilingual topic modeling for news articles.
This session was classified suitable for some domain / basic python by the speaker.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:02]
So hi everybody, I'm not sure how much fun is it going to be unless you're really interested.
Speaker 2 [00:06]
interested in natural language processing. And this is the first disambiguation of the
Speaker 1 [00:10]
of this talk. If you are waiting for the talk on neuro-linguistic
Speaker 2 [00:13]
neuro-linguistic programming.
Speaker 1 [00:14]
programming, which is also NLP, you're probably in the wrong room. I'm going to be talking about natural language processing.
Speaker 2 [00:20]
And the idea of this talk
Speaker 1 [00:22]
idea of this talk is to introduce some concept of NLP and just some examples that could
Speaker 2 [00:27]
examples that could be useful for your first
Speaker 1 [00:28]
your first nature study.
Speaker 2 [00:29]
natural language processing project.
Speaker 1 [00:30]
project. Okay, a little bit about me. I am Anna Vidiga, I'm a computational linguist.
Speaker 2 [00:37]
traditional linguist.
Speaker 1 [00:37]
I speak three languages
Speaker 2 [00:39]
three languages most of the time.
Speaker 1 [00:41]
originally from Russia. I got my degree from the University of Tübingen. And this XKCD comic is how I was introduced
Speaker 2 [00:49]
how I was introduced to XKCD.
Speaker 1 [00:52]
It's about computational linguistics, and it's ill-defined.
Speaker 2 [00:52]
It's about... ill-defined field.
Speaker 1 [00:55]
field, just like NLP. You can think of computational linguistics as some parts of
Speaker 2 [01:01]
Computer science, some fun.
Speaker 1 [01:02]
some fun parts of computer science, and all the headaches of linguistics.
Speaker 2 [01:07]
okay so what this talk is about I'm
Speaker 1 [01:12]
about I'm going to start with
Speaker 2 [01:13]
going to start with introducing some NLP
Speaker 1 [01:15]
some NLP concepts.
Speaker 2 [01:16]
Mostly as it relates to machine learning, but also some rule-based stuff.
Speaker 1 [01:21]
The main part of the talk are some examples of processing
Speaker 2 [01:24]
processing text, real-life text.
Speaker 1 [01:26]
live text examples and those examples would be you can think of it as recipes first as an appetizer I'll show some NLP pipelines with the tweet
Speaker 2 [01:38]
with a tweet, and then as a main course named entity recognition,
Speaker 1 [01:41]
named entity recognition, an example of a Yelp review, and as a dessert, multilingual topic modeling with a toy example. So all of these examples are on this.
Speaker 2 [01:51]
on the smallest text possible so you can see all the fun.
Speaker 1 [01:56]
all the fun stuff, but it's easily scalable. And I will end the talk with...
Speaker 2 [02:01]
end the talk with some takeaways of some do's and don'ts.
Speaker 1 [02:03]
do's and don'ts of NLP, hopefully some of it will be useful for your NLP project. Okay, show of hands, how many of you have done something with NLP already?
Speaker 2 [02:14]
Oh, wow. That's great.
Speaker 1 [02:15]
That's great. So chances are you know more about it than me because I've only tried some of the things. And another disappointment of the talk is probably going to be I'm not going to talk about...
Speaker 2 [02:27]
deep learning other than
Speaker 1 [02:28]
other than introducing some of the concepts, because my opinion is for your first NLP project, deep learning is probably a little bit too complicated, unless you already know a lot about deep learning. All right, so let's start with NLP.
Speaker 2 [02:44]
with NLP and machine learning. Linguists like to think of languages as
Speaker 1 [02:49]
of languages as
Speaker 2 [02:50]
living organisms ever
Speaker 1 [02:51]
ever evolving. For data scientists, unstructured text is more of a headache. It's something
Speaker 2 [02:58]
It's something that they can...
Speaker 1 [02:59]
that they can apply
Speaker 2 [03:00]
apply tools to
Speaker 1 [03:01]
tools to extract some useful information. This is not always an easy task.
Speaker 2 [03:05]
not always an easy process and some familiar tools like pandas
Speaker 1 [03:10]
like pandas are not really applicable to
Speaker 2 [03:11]
applicable to unstructured text.
Speaker 1 [03:12]
structured text.
Speaker 2 [03:16]
When you organize text into some meaningful chunks of information,
Speaker 1 [03:19]
chunks of information, meaningful nuggets of information, you end up with tokens.
Speaker 2 [03:23]
tokens and something that you have to discard.
Speaker 1 [03:24]
you have to discard and filter out. So you have some tokens and some garbage.
Speaker 2 [03:31]
Once you have those tokens,
Speaker 1 [03:32]
have those tokens, what you can do with them is
Speaker 2 [03:35]
is apply some meaningful labels to them. Those labels can be
Speaker 1 [03:40]
can be linguistic, like in this case POS.
Speaker 2 [03:44]
stands for part of speech tagging and
Speaker 1 [03:47]
not the other meaning, but you can also have some custom tags. For example, for named entity recognition you have tags like
Speaker 2 [03:53]
You have tags like organization,
Speaker 1 [03:54]
like organizations and geopolitical
Speaker 2 [03:55]
geopolitical entities.
Speaker 1 [03:59]
Once you've collected
Speaker 2 [04:00]
collected the tokens, labeled
Speaker 1 [04:01]
tokens, label them, you can count them, but the raw count is not particularly useful for machine learning applications.
Speaker 2 [04:07]
you have to vectorize them.
Speaker 1 [04:08]
vectorize them. You have to transform the count into something else. And in the next step, you're going to have to weigh some tokens. Some of them are going to be more important than the others. It depends on your source material and it depends on the...
Speaker 2 [04:23]
information you're trying to extract.
Speaker 1 [04:24]
trying to extract from the text.
Speaker 2 [04:29]
That's the deep learning slide that I promised. Deep learning is a fantastic...
Speaker 1 [04:35]
is a fantastic tool in
Speaker 2 [04:36]
tool in machine learning, and it's very
Speaker 1 [04:37]
and it's very much applicable to NLP. In fact, most of the novel research, both in academia and in industry, is done using deep learning.
Speaker 2 [04:47]
And it's very lucrative because
Speaker 1 [04:48]
lucrative because all you need is a huge collection of data, but you can start with some
Speaker 2 [04:55]
You can start with something very simple.
Speaker 1 [04:57]
when you have no labels whatsoever, you can still take
Speaker 2 [05:01]
You can still take your text blob
Speaker 1 [05:02]
your text blobs and combine them to bigger, meaningful text blobs. This is the process
Speaker 2 [05:06]
This is the process known as clustering.
Speaker 1 [05:08]
clustering.
Speaker 2 [05:09]
This process doesn't require
Speaker 1 [05:10]
doesn't require anything other than your data. It's completely unsupervised.
Speaker 2 [05:15]
Once you have some labels,
Speaker 1 [05:16]
some labels.
Speaker 2 [05:17]
For example, for email.
Speaker 1 [05:18]
For emails, you have spam and ham.
Speaker 2 [05:21]
The tasks become something of a
Speaker 1 [05:22]
something of a classification. And there's a variety of machine learning algorithms to
Speaker 2 [05:26]
algorithms to choose from for that.
Speaker 1 [05:27]
for that.
Speaker 2 [05:28]
And deep learning is.
Speaker 1 [05:29]
can be done
Speaker 2 [05:30]
can be done with both
Speaker 1 [05:31]
with both labeled and unlabeled data.
Speaker 2 [05:34]
It basically can be applied as
Speaker 1 [05:35]
be applied as a black box algorithm, it picks up some linguistic patterns along the way, but you don't necessarily know how it comes up with those patterns.
Speaker 2 [05:47]
All right, so this follows.
Speaker 1 [05:50]
This follows the example by
Speaker 2 [05:51]
by Josh Wills.
Speaker 1 [05:52]
Josh Wills who introduced the
Speaker 2 [05:54]
data science infinite loop of sadness.
Speaker 1 [05:57]
I'm introducing the NLP Pyramid of Sadness, and it's based on how you view
Speaker 2 [06:02]
how you view linguistics as a discipline.
Speaker 1 [06:04]
as a discipline.
Speaker 2 [06:05]
For linguists, it's a nice...
Speaker 1 [06:05]
For linguists, It's a nice organized world where each level is built on top of each other and each
Speaker 2 [06:10]
each other and each level deals with different aspects of languages like for example phonetics and phonology
Speaker 1 [06:15]
And phonology deals with the sound of a human language. Morphology deals with some components of the word.
Speaker 2 [06:24]
like word roots
Speaker 1 [06:25]
roots and stems
Speaker 2 [06:26]
and stems, and suffixes and prefixes.
Speaker 1 [06:28]
and prefixes.
Speaker 2 [06:30]
Syntax...
Speaker 1 [06:31]
helps combine
Speaker 2 [06:32]
helps combine those words into sentences.
Speaker 1 [06:33]
words into sentences.
Speaker 2 [06:35]
It deals with rules and constraints.
Speaker 1 [06:37]
constraints of how you organize.
Speaker 2 [06:39]
words into something more meaningful.
Speaker 1 [06:40]
to something more meaningful as a phrase.
Speaker 2 [06:43]
And syntax provides
Speaker 1 [06:44]
provides help and scaffolding for meaning.
Speaker 2 [06:48]
Meaning is something that's...
Speaker 1 [06:49]
something that semantics and pragmatics deal with. And the difference is semantics, it's something you can look up in the dictionary, the basic
Speaker 2 [06:55]
the basic meaning of the word.
Speaker 1 [06:56]
meaning. of the word, pragmatics is the highest level of
Speaker 2 [06:59]
level of abstraction.
Speaker 1 [07:00]
It's something that's not necessarily written but understood by context. It's a world.
Speaker 2 [07:06]
It's world view.
Speaker 1 [07:07]
It's something that you understand out of the context.
Speaker 2 [07:11]
So, and machine learning kind of...
Speaker 1 [07:13]
learning, kind of.
Speaker 2 [07:14]
tips this pyramid.
Speaker 1 [07:16]
and makes it a less structurally sound structure. But machine learning likes to tackle really hard problems.
Speaker 2 [07:25]
problems and in this case
Speaker 1 [07:25]
And in this case, it is all about the meaning, it's all about the semantics. This is where a lot of research is, and this is where the money is.
Speaker 2 [07:32]
This is where the money is.
Speaker 1 [07:34]
about natural language understanding.
Speaker 2 [07:34]
language understanding systems.
Speaker 1 [07:36]
is where you have to combine
Speaker 2 [07:38]
to combine linguistic knowledge and a lot of
Speaker 1 [07:40]
and a lot of sophisticated machine learning algorithms, but also some rule-based stuff.
Speaker 2 [07:40]
sophistication. heuristics.
Speaker 1 [07:46]
An example of NLU would be...
Speaker 2 [07:50]
systems like Alexa.
Speaker 1 [07:51]
Alexa, but the problem is that
Speaker 2 [07:57]
The problem is that semantics
Speaker 1 [07:58]
that semantics can be really hard.
Speaker 2 [08:00]
really hard and it's very language specific.
Speaker 1 [08:01]
language-specific okay I'm going back to
Speaker 2 [08:05]
one level down, which is syntax. If you've never done syntactic parsing before, this
Speaker 1 [08:11]
before, this may be a little bit confusing. I tried to pick a more contemporary example that you don't find.
Speaker 2 [08:16]
that you don't find in usual linguistic textbooks.
Speaker 1 [08:18]
textbooks. So I'm analyzing the phrase this part is going to be lit
Speaker 2 [08:23]
And I'll start.
Speaker 1 [08:24]
I'll start with the verb phrase, going to be.
Speaker 2 [08:27]
and has two dependencies.
Speaker 1 [08:28]
independency this party
Speaker 2 [08:29]
This party, which is this
Speaker 1 [08:30]
which is the subject of the sentence.
Speaker 2 [08:32]
and the complementary phrase
Speaker 1 [08:33]
phrase be lit and you see lit in this case is classified as a noun which is not necessarily correct but it works
Speaker 2 [08:42]
but it works in the overall system.
Speaker 1 [08:42]
in the overall structure of the sample.
Speaker 2 [08:43]
structure of the sentence. Those dependencies are
Speaker 1 [08:46]
Tendencies are important when you organize.
Speaker 2 [08:47]
when you organize words into moments.
Speaker 1 [08:49]
into more meaningful chunks.
Speaker 2 [08:54]
There are some pitfalls.
Speaker 1 [08:55]
pitfalls when you're dealing with natural languages.
Speaker 2 [08:58]
Data produced by humans
Speaker 1 [08:59]
by humans is naturally messy.
Speaker 2 [09:01]
really messy. You have a lot of
Speaker 1 [09:02]
lot of special characters, you have typos, it's not necessarily correct.
Speaker 2 [09:03]
special correct English or other languages.
Speaker 1 [09:07]
or other languages.
Speaker 2 [09:08]
So you have to come up with some
Speaker 1 [09:10]
come up with some strategies how you
Speaker 2 [09:10]
strategies how you're going to deal with
Speaker 1 [09:11]
you're going to deal with that. Even when you're dealing with correct and grammatically
Speaker 2 [09:16]
and grammatically correct language,
Speaker 1 [09:16]
the correct language.
Speaker 2 [09:18]
you still have some ambiguity.
Speaker 1 [09:19]
ambiguity that makes it difficult to
Speaker 2 [09:20]
difficult to process meaning.
Speaker 1 [09:21]
process. Like, for example, homonyms in English.
Speaker 2 [09:24]
in English. Tear and tear are two different words that
Speaker 1 [09:27]
words that are spelled completely identically. When you look at words on the level of morphology,
Speaker 2 [09:38]
different word forms.
Speaker 1 [09:39]
forms
Speaker 2 [09:40]
and noun phrases, the same word can suddenly change
Speaker 1 [09:43]
can suddenly change the meaning. Like in this example, this agreement is about the principle
Speaker 2 [09:48]
about a principled compromise, not a principle compromised. So just by changing something,
Speaker 1 [09:53]
By changing some suffixes, the noun phrase
Speaker 2 [09:55]
One phrase completely changed the meaning.
Speaker 1 [09:58]
And this is something to think about.
Speaker 2 [10:01]
those corner cases
Speaker 1 [10:02]
cases, it's not necessarily something you'll find in your
Speaker 2 [10:04]
you'll find in your own natural
Speaker 1 [10:05]
your own natural language processing project, but it's something to think about before you
Speaker 2 [10:10]
before you start the process,
Speaker 1 [10:11]
start the processing and it's also something important to keep in mind.
Speaker 2 [10:14]
to keep in mind when you
Speaker 1 [10:15]
mind when you
Speaker 2 [10:16]
interpret the results. Okay.
Speaker 1 [10:22]
And it all depends on how much data you're going to have. If it's a huge collection of text or if it's just a relatively small collection.
Speaker 2 [10:30]
relatively small corpus. If you have a big data
Speaker 1 [10:33]
data, by the
Speaker 2 [10:34]
by the law of large numbers.
Speaker 1 [10:35]
large numbers.
Speaker 2 [10:37]
All the errors, all the ambiguity,
Speaker 1 [10:38]
all the ambiguities in special cases are going to be evened out.
Speaker 2 [10:42]
and a lot of time and research
Speaker 1 [10:43]
time and research is invested in annotation schemes.
Speaker 2 [10:46]
annotation schemas and tools.
Speaker 1 [10:48]
to provide
Speaker 2 [10:49]
correct and more meaningful label.
Speaker 1 [10:50]
more meaningful labels.
Speaker 2 [10:52]
But for the first...
Speaker 1 [10:53]
the first NLP project, chances are that your data is relatively small. And then every word counts.
Speaker 2 [11:00]
You'll have to invest time into preprocessing.
Speaker 1 [11:05]
And in this case, natural language processing library.
Speaker 2 [11:08]
processing libraries to the rest
Speaker 1 [11:09]
to the rescue.
Speaker 2 [11:10]
You don't have to reinvent the wheel.
Speaker 1 [11:11]
reinvent the wheel and the library that I'm going to talk about in this in this talk is called spacing
Speaker 2 [11:20]
It's an open source project maintained by Matthew Honnebel and
Speaker 1 [11:23]
Hannibal and Ines Montani.
Speaker 2 [11:25]
The idea is to get from
Speaker 1 [11:26]
is to get from an unstructured text blob
Speaker 2 [11:29]
to something that has
Speaker 1 [11:30]
that has a lot of
Speaker 2 [11:31]
a lot of meaningful labels that you
Speaker 1 [11:32]
label that you can use for machine learning. And it's an NLP pipeline. It's
Speaker 2 [11:37]
applying. It starts with the top
Speaker 1 [11:37]
It starts with the tokenizer, and then you have various taggers, part of
Speaker 2 [11:42]
part of speech taggers, parsers.
Speaker 1 [11:43]
parsers named
Speaker 2 [11:44]
named entity recognition and some custom plugins.
Speaker 1 [11:46]
some custom plugins.
Speaker 2 [11:47]
that you can combine in a way
Speaker 1 [11:48]
in a way that makes sense for your task.
Speaker 2 [11:55]
I'm going to show some very simplistic
Speaker 1 [11:58]
stick NLP pipe
Speaker 2 [11:58]
NLP pipelines using the examples of very short sentences.
Speaker 1 [12:03]
And my first example, my
Speaker 2 [12:04]
First example, my first recipe is a twig.
Speaker 1 [12:06]
is a tweet.
Speaker 2 [12:08]
In case you don't know her, Mara Everick is a data scientist and she describes data science tools on
Speaker 1 [12:12]
tools on Twitter.
Speaker 2 [12:14]
using her own very visual language.
Speaker 1 [12:16]
language.
Speaker 2 [12:16]
She uses a lot of...
Speaker 1 [12:18]
A lot of GIFs, emojis, popcorn.
Speaker 2 [12:19]
which is pop culture references.
Speaker 1 [12:21]
And the challenge here is that it's not a lot of text.
Speaker 2 [12:25]
It's a lot of other characters.
Speaker 1 [12:27]
characters. But what I want to see
Speaker 2 [12:30]
want to see is
Speaker 1 [12:30]
is how does an NLP pipeline
Speaker 2 [12:31]
um pipeline deal with that. I'm just going to read out the tweet.
Speaker 1 [12:36]
out the tweet star intro into
Speaker 2 [12:40]
Intro into a Glowing Star Tool, Detective RegExplained,
Speaker 1 [12:44]
RegExplain by at.
Speaker 2 [12:47]
Twitter handle?
Speaker 1 [12:48]
Twitter handle, and then you have a link and hashtag.
Speaker 2 [12:51]
And hashtag rstats.
Speaker 1 [12:52]
Our stats, hashtag Regex. All right. What parts of the pipeline...
Speaker 2 [12:58]
pipeline are important, what parts are nice to have. It always starts with, for
Speaker 1 [13:04]
starts with, first of all, the
Speaker 2 [13:05]
The step zero is identity.
Speaker 1 [13:06]
is identifying the language that you're going to deal with. In this case, it's English. This will determine what kind of language model you're using.
Speaker 2 [13:13]
which model you are using.
Speaker 1 [13:14]
using. The next step that is important is tokenization.
Speaker 2 [13:17]
organization.
Speaker 1 [13:18]
And this is basically splitting the sensors.
Speaker 2 [13:20]
the sentence that you have into words or phrases.
Speaker 1 [13:25]
The simplest case is splitting by y.
Speaker 2 [13:27]
splitting by white space or
Speaker 1 [13:28]
white space or punctuation.
Speaker 2 [13:29]
punctuation, but it doesn't always work this way.
Speaker 1 [13:32]
this way. Some languages, like Japanese, don't even have white spaces.
Speaker 2 [13:35]
white space so you have your own chat
Speaker 1 [13:35]
so you have your own challenge. And in German, with long compounds,
Speaker 2 [13:39]
long compound words.
Speaker 1 [13:41]
Sometimes you have to split something in the middle of the work.
Speaker 2 [13:43]
in the middle of the world.
Speaker 1 [13:44]
Like the three English
Speaker 2 [13:45]
Three English words in the phrase income.
Speaker 1 [13:47]
phrase income tax return.
Speaker 2 [13:47]
tax return.
Speaker 1 [13:48]
Einkommenssteuererklärung in German. Next step that is very important for things like organizing work
Speaker 2 [13:57]
Organizing words in a sentence, it's part of speech tagging.
Speaker 1 [13:59]
tagging and syntactic parsing.
Speaker 2 [14:01]
Without it, you cannot really
Speaker 1 [14:02]
you cannot really combine
Speaker 2 [14:04]
words into noun chunks.
Speaker 1 [14:05]
noun chunks.
Speaker 2 [14:07]
Chunking is really
Speaker 1 [14:08]
really nice to have.
Speaker 2 [14:10]
even concentrating on
Speaker 1 [14:11]
concentrating on not one word,
Speaker 2 [14:12]
not one word, but two words, increasing the window size too.
Speaker 1 [14:15]
size too, it improves the accuracy of any machine learning model. And it's all about the context. So as you can see in the examples,
Speaker 2 [14:24]
Chocolate bar, bar exam, wine bar, and space bar are very different kinds of bars.
Speaker 1 [14:30]
And in order to understand what you are talking about, you have to look at
Speaker 2 [14:32]
You have to look at the words surrounding your target.
Speaker 1 [14:34]
your target word. Alright, I will make the Jupyter Notebook available on GitHub after this talk, but I'm going to show some
Speaker 2 [14:47]
small parts of the code.
Speaker 1 [14:48]
of the code. So you can see
Speaker 2 [14:51]
The simplistic NLP pipeline
Speaker 1 [14:52]
NLP pipeline. What I'm doing here after loading
Speaker 2 [14:55]
after loading spaCy, I'm choosing the
Speaker 1 [14:59]
Using the...
Speaker 2 [15:00]
language model, in this case English, and then I'm loading the
Speaker 1 [15:03]
the example
Speaker 2 [15:04]
example sentence into a document object.
Speaker 1 [15:06]
document object and what I'm going to look at are different groups of tokens.
Speaker 2 [15:12]
tokens, in this case very simple
Speaker 1 [15:14]
very simple divide between alpha tokens and not alpha tokens. I'm only looking at the
Speaker 2 [15:19]
You're looking at words and non-words.
Speaker 1 [15:19]
words and non-words.
Speaker 2 [15:21]
And then looking at noun chunks.
Speaker 1 [15:23]
And what I'm doing here is
Speaker 2 [15:24]
doing here is a very simplistic sentiment analysis.
Speaker 1 [15:27]
analysis. I have created a custom dictionary
Speaker 2 [15:30]
custom dictionary for
Speaker 1 [15:31]
for emoji, and I'm using emoji count as sentiments.
Speaker 2 [15:37]
sentiment score.
Speaker 1 [15:41]
So you can see this is my very
Speaker 2 [15:42]
This is my very simplistic
Speaker 1 [15:43]
simplistic emoji dictionary that can be expanded.
Speaker 2 [15:47]
and the nice thing about
Speaker 1 [15:48]
about Space-C pipelines is this is the
Speaker 2 [15:51]
This is the customizable part. This is something you can expand.
Speaker 1 [15:53]
and you can expand depending on your goal.
Speaker 2 [15:55]
on your goals.
Speaker 1 [15:56]
They have special containers.
Speaker 2 [15:58]
containers for
Speaker 1 [15:59]
for things like that, and you can either flash
Speaker 2 [16:02]
either flag something,
Speaker 1 [16:03]
something as a
Speaker 2 [16:05]
token of interest, or you can create
Speaker 1 [16:06]
create your own vocabulary list.
Speaker 2 [16:08]
list. In this case, I'm flagging it in the vocabulary list. The tokenization is
Speaker 1 [16:15]
is done
Speaker 2 [16:16]
done behind the scenes by Spaceman.
Speaker 1 [16:17]
scenes by Spacey, it's basically one line.
Speaker 2 [16:19]
one line for
Speaker 1 [16:20]
for token in document one and then all you do is use built-in attributes
Speaker 2 [16:26]
in attributes to filter tokens.
Speaker 1 [16:28]
In this case, I am interested in war.
Speaker 2 [16:29]
interested in words, which is, is alpha.
Speaker 1 [16:31]
alpha and then I collect it in a list.
Speaker 2 [16:36]
Right, and then using the
Speaker 1 [16:37]
using the previously defined emoji dictionary for sentiment analysis, I'm just going to create
Speaker 2 [16:42]
create a sentiment score. And the second part of it
Speaker 1 [16:47]
part of it is, I'm going to show you.
Speaker 2 [16:49]
It's a very simple example of
Speaker 1 [16:50]
example of syntactic tree
Speaker 2 [16:52]
tree dependency.
Speaker 1 [16:53]
And you can see, I was interested.
Speaker 2 [16:55]
I was interested in...
Speaker 1 [16:56]
in emoji and how they're integrated? Are they part of the noun?
Speaker 2 [16:59]
part of the noun phrase or they're completely separate entities.
Speaker 1 [17:01]
entities. And this example shows that
Speaker 2 [17:03]
shows that they are indeed integrated, so you can see the
Speaker 1 [17:07]
the first star is identified as a pronoun, which is
Speaker 2 [17:11]
which is not correct, but it's still possible.
Speaker 1 [17:12]
but it's still part of the sentence. And I was luckier with the second emoji, the
Speaker 2 [17:18]
emoji, the glowing star.
Speaker 1 [17:18]
Glowing Star, which is part of
Speaker 2 [17:19]
which is part of the noun phrase
Speaker 1 [17:20]
of the noun phrase
Speaker 2 [17:21]
A Glowing Star Tool.
Speaker 1 [17:24]
And this kid is, it's identified as a pro.
Speaker 2 [17:26]
as a proper noun.
Speaker 1 [17:26]
proper noun, but you can use that example as something, this is the prerequisite for the chunking.
Speaker 2 [17:34]
and my noun chunk is
Speaker 1 [17:36]
Noun chunk is going to be a glowing star tool.
Speaker 2 [17:41]
And this is how you access the noun chunk in spaCy.
Speaker 1 [17:45]
So some of the noun chunks I'm not going to be particularly interested in, but this is something you can use for further.
Speaker 2 [17:53]
further filtering.
Speaker 1 [17:54]
filtering. If you're only interested in nouns or only in
Speaker 2 [17:56]
or only in adjectives for example. Alright, so some pitfalls that you can
Speaker 1 [18:02]
that you can already see with examples like tweets or social media posts.
Speaker 2 [18:03]
No. you're going to have to deal with a lot of
Speaker 1 [18:07]
with a lot of garbage, some encoding issues.
Speaker 2 [18:09]
encoding issue.
Speaker 1 [18:10]
And even with emoji, they can be UTF-8 or they can be Unicode, but you have to decide.
Speaker 2 [18:14]
but you have to decide what to do with them.
Speaker 1 [18:15]
what to do with them. Do you just ignore them, filter them out?
Speaker 2 [18:18]
or do you process them as
Speaker 1 [18:19]
them as special entities.
Speaker 2 [18:20]
special entities.
Speaker 1 [18:23]
And punctuation is normally filtered out, but in case of emoji you have to decide early on are you going to treat emoji as a special case of punctuation or are you just going to ignore them. And not all part of speech tags automatically identified are going to be correct.
Speaker 2 [18:42]
relying on built-in models
Speaker 1 [18:44]
models it's something that is not necessarily a good idea always have to go back and double-check like in my example that the emojis were completely misclassified best practices I'm just going through them really quickly but please take it with a grain of salt especially if you've already have some experience with NLP what works for my projects might not work for yours
Speaker 2 [19:09]
In any case, identifying
Speaker 1 [19:10]
identifying the language before any preprocess.
Speaker 2 [19:11]
any pre-processing is important.
Speaker 1 [19:12]
processing is important.
Speaker 2 [19:13]
And then this decision to filter.
Speaker 1 [19:14]
to filter or remove special characters?
Speaker 2 [19:16]
characters. Nice to have as a custom dictionary, like a vocabulary list, but it's odd.
Speaker 1 [19:20]
but it's time consuming.
Speaker 2 [19:23]
Again, nice to have his chunking.
Speaker 1 [19:24]
with chunking and noun phrases.
Speaker 2 [19:25]
noun phrases that you
Speaker 1 [19:26]
that just makes the output and the interpretation.
Speaker 2 [19:27]
interpretation lay
Speaker 1 [19:28]
later much, much neater.
Speaker 2 [19:30]
neater. Some not-to-dos
Speaker 1 [19:35]
to-dos is, please don't assume that everything is going to be simple.
Speaker 2 [19:39]
going to be simple and well-organized.
Speaker 1 [19:41]
newspaper articles are on a completely different level.
Speaker 2 [19:44]
completely different level then.
Speaker 1 [19:45]
level than social media posts. It may be very tempting to write your own tokenizer, but
Speaker 2 [19:53]
Why reinvent the wheel?
Speaker 1 [19:54]
invent the wheel when libraries like Spacey do
Speaker 2 [19:56]
Pacey do it for you in one line.
Speaker 1 [19:57]
it for you in one line of code. And just like I mentioned,
Speaker 2 [20:00]
like I mentioned, don't rely on
Speaker 1 [20:00]
Don't rely on off-the-shelf, pre-built, pre-trained models. Okay. The second recipe is a Yelp review. It's a one-star negative review.
Speaker 2 [20:14]
I would like to find out why was this
Speaker 1 [20:16]
Why was this review negative? Is it something?
Speaker 2 [20:19]
tangible, like the name of
Speaker 1 [20:21]
the name of an employee, maybe it's about the price, maybe it's the location.
Speaker 2 [20:25]
Maybe it's the location.
Speaker 1 [20:26]
So I'm going to be interested, I'm going to be focused on very specific tokens. Okay, the text of the review. We drove...
Speaker 2 [20:33]
30 minutes out of our way to get some dessert.
Speaker 1 [20:35]
to get some dessert and ordered a ton to go, only to find that they didn't give us what we ordered. What we did get was not good.
Speaker 2 [20:42]
We called, and they refunded us.
Speaker 1 [20:44]
us $4. Ridiculous. Don't waste your time or money here.
Speaker 2 [20:48]
money here sixth course right around the corner is
Speaker 1 [20:49]
around the corner is way better. I should probably mention this is a review of my favorite chocolate shop in San Francisco, so it is very surprising that it's such a negative review. Alright, so what's the typical pipeline for processing something like that?
Speaker 2 [21:06]
It's a longer text, but it also has some challenges.
Speaker 1 [21:08]
some challenges.
Speaker 2 [21:10]
What you can do with your tokens is to create a new token.
Speaker 1 [21:11]
with your tokens is normalize them get them to some basic form that you can look up in the dictionary
Speaker 2 [21:18]
One way to deal with it is
Speaker 1 [21:19]
with it is for example
Speaker 2 [21:19]
For example,
Speaker 1 [21:21]
Expand contractions.
Speaker 2 [21:23]
and expands.
Speaker 1 [21:24]
expand some units of measurements, like $5, spell it out as $5. Again, tokenization is filtering, is going to be part of the process.
Speaker 2 [21:37]
In this case, I'm just going to
Speaker 1 [21:38]
I'm just going to remove puncture.
Speaker 2 [21:39]
punctuation as well as stop words. Stop words
Speaker 1 [21:42]
stop words are very common
Speaker 2 [21:44]
very common words
Speaker 1 [21:45]
words that
Speaker 2 [21:46]
carry no special meaning.
Speaker 1 [21:47]
meaning, like the and of. Those stop words are language-specific, and every language has a list of those
Speaker 2 [21:54]
list of those stop words.
Speaker 1 [21:54]
stop words. And the main part of what I'm going to do with this Yelp review is going to be named entity recognition.
Speaker 2 [22:03]
And like I mentioned, it's going to be...
Speaker 1 [22:05]
I mentioned it's going to be something special.
Speaker 2 [22:06]
specific
Speaker 1 [22:07]
like location, date, quantity.
Speaker 2 [22:11]
or organization.
Speaker 1 [22:12]
I'm going to have a lot of different tags.
Speaker 2 [22:19]
side about normalization.
Speaker 1 [22:19]
normalization, you have two choices when you're trying to normalize.
Speaker 2 [22:22]
trying to normalize words, and
Speaker 1 [22:23]
and it's stemming or lemmatization.
Speaker 2 [22:26]
They both have advantages and disadvantages.
Speaker 1 [22:27]
and disadvantages. Stemming is basically removing cutting-off suffixes. Sometimes it creates nonsense
Speaker 2 [22:34]
creates nonsensical words
Speaker 1 [22:34]
words. Lemmatization is doing things
Speaker 2 [22:37]
is doing things properly,
Speaker 1 [22:38]
properly.
Speaker 2 [22:39]
by looking up words
Speaker 1 [22:40]
up words in a dictionary like WordNet.
Speaker 2 [22:42]
It produces better results.
Speaker 1 [22:43]
better results and it's usually more accurate, but it's also, it can be more time consuming. So it's always the difference between precision and recall.
Speaker 2 [22:54]
When using spaCy, you don't
Speaker 1 [22:55]
In spaCy you don't get a choice, spaCy only does lemmatization, so in this case it's going to be more by the book.
Speaker 2 [23:04]
So this is my fiber.
Speaker 1 [23:05]
This is my pipeline.
Speaker 2 [23:06]
I'm removing stopwatch.
Speaker 1 [23:07]
moving stop words and punctuation, which is again just
Speaker 2 [23:09]
again just using some built-in
Speaker 1 [23:10]
some built-in attributes, which is very nice, provided by spaCy. I'm not particularly interested in one
Speaker 2 [23:15]
interested in one specific part of speech.
Speaker 1 [23:17]
part of speech tag, which is pronouns. So I'm just going to filter them out outright.
Speaker 2 [23:21]
What I am interested in is
Speaker 1 [23:22]
in is I'm
Speaker 2 [23:23]
I'm tracking named entities.
Speaker 1 [23:24]
named entities like person, organization, geopolitical entity, and so on. And, yeah. So it's just going to provide me a list of those.
Speaker 2 [23:34]
list of those entities. And this is the result.
Speaker 1 [23:39]
we can just look at what are the named entities that are identified and what was missed.
Speaker 2 [23:48]
30 minutes was identified.
Speaker 1 [23:49]
was identified a ton as a quantity.
Speaker 2 [23:50]
is a quantity.
Speaker 1 [23:52]
money in $4.
Speaker 2 [23:55]
And here are the...
Speaker 1 [23:56]
Here are the mistakes.
Speaker 2 [23:58]
ridiculous was identified as a
Speaker 1 [23:58]
Ridiculous. as a geopolitical entity.
Speaker 2 [24:02]
The reason being its capital.
Speaker 1 [24:03]
it's capitalized, and it's a single word in the sentence, so it's an easy mistake to make. Way, because of the way it was spelled, all caps, was identified as organization.
Speaker 2 [24:15]
So you can see this named entity is very
Speaker 1 [24:17]
is very case sensitive. This is why it produces some
Speaker 2 [24:19]
produces some results that are not
Speaker 1 [24:20]
results that are not accurate. And there is also something that was missing. Sixth course.
Speaker 2 [24:26]
should be identified, but it wasn't. The reason for that is
Speaker 1 [24:29]
is both part
Speaker 2 [24:31]
parts of this named entity sixth and
Speaker 1 [24:32]
and coarse, a very common English word. And even though it's capitalized, it's not going to be identified.
Speaker 2 [24:33]
quarter Okay, just very quickly expanding.
Speaker 1 [24:42]
expanding contraction, you have to be really careful. You can customize the tokenizer to include
Speaker 2 [24:47]
to include expanding contractions.
Speaker 1 [24:48]
contractions, but
Speaker 2 [24:49]
but you're going to have some ambiguity with some.
Speaker 1 [24:51]
with some contractions.
Speaker 2 [24:52]
contractions that are the same.
Speaker 1 [24:54]
It's very tempting to clean up the text using spellchecks,
Speaker 2 [24:57]
using spell checker.
Speaker 1 [24:59]
But before you do that, you have to make sure that it's worth it, because the spell checker will try to correct every word in the sentence.
Speaker 2 [25:07]
whether it was necessary
Speaker 1 [25:08]
necessary or not. And then again, lowercase is a real problem with named entity recognition. Some people don't even capitalize their own name, so it makes it very, very hard to identify.
Speaker 2 [25:21]
And some names,
Speaker 1 [25:23]
when they are not capitalized are just very common.
Speaker 2 [25:25]
very common English noun.
Speaker 1 [25:26]
English nouns like a rose
Speaker 2 [25:31]
So best practices is the
Speaker 1 [25:32]
is the order of operations becomes very important.
Speaker 2 [25:36]
Expand contractions.
Speaker 1 [25:37]
contractions before tokenization, spell checker before limitization and normalization. If you are going to convert everything to
Speaker 2 [25:45]
everything to
Speaker 1 [25:46]
lowercase, make sure that you identify your name.
Speaker 2 [25:46]
lower case make identify your named entities
Speaker 1 [25:48]
entities before that and sometimes it's a very good idea to just create custom lists for named entities it makes it more accurate okay
Speaker 2 [25:59]
So, very quickly about those.
Speaker 1 [26:01]
about the last recipe it's topic modeling I'm not going to show a lot of pipeline code just what I did here I took two texts one translation of another and it's a text about natural language processing with spaCy one in English and one in Spanish
Speaker 2 [26:18]
And I...
Speaker 1 [26:20]
made those two texts a parallel corp.
Speaker 2 [26:22]
parallel corpus by splitting them
Speaker 1 [26:23]
splitting them into paragraphs. And by topic modeling, what I want to find out is what are the most important
Speaker 2 [26:30]
important words in those texts.
Speaker 1 [26:32]
in those texts. The pipeline is basically connecting everything I talked about before, including tokenization and name-dating recognition. And the results are, I have identified some topics.
Speaker 2 [26:45]
topics using the library called Gensim and the pre-processing step
Speaker 1 [26:48]
processing steps were done.
Speaker 2 [26:49]
were done by Spacey.
Speaker 1 [26:51]
And I don't really speak Spanish, but I know that PLN in Spanish,
Speaker 2 [26:55]
N in Spanish is the same as NLP in English.
Speaker 1 [26:58]
So I have some parallels already, some topics that were identified.
Speaker 2 [27:02]
Even though the corpus was really small,
Speaker 1 [27:03]
corpus was really tiny okay right I'm gonna
Speaker 2 [27:09]
I'm going to skip the best practice.
Speaker 1 [27:11]
the best practices for that. It doesn't really make sense.
Speaker 2 [27:13]
sense if you don't see the code.
Speaker 1 [27:15]
just two words about the approaches that you can use.
Speaker 2 [27:20]
can use one is bag of words.
Speaker 1 [27:20]
One is, bag of words which is basically collect all the
Speaker 2 [27:22]
collect all the words regardless of where they are in the sentence.
Speaker 1 [27:25]
The order is not important.
Speaker 2 [27:27]
when you convert the whole document to a bag of words.
Speaker 1 [27:29]
a bag of word. And you can also have word embeddings with the
Speaker 2 [27:32]
where the context and the order of words is important.
Speaker 1 [27:34]
is important. An example is doc2vec.
Speaker 2 [27:40]
and you can think of it as
Speaker 1 [27:41]
as
Speaker 2 [27:42]
Scrabble tiles.
Speaker 1 [27:43]
Kyle's bag of words is
Speaker 2 [27:44]
bag of words is just the whole bag
Speaker 1 [27:45]
just the whole bag of them and once you put them on the board and the order becomes important then you can do
Speaker 2 [27:51]
do more with this.
Speaker 1 [27:52]
with these styles. Some takeaways for NLP projects. Don't forget that natural language is very messy and it's harder to deal with words than numbers. Looking at your input and output in any stage is very helpful, then you can identify the mistakes.
Speaker 2 [28:11]
And then you can customize your
Speaker 1 [28:12]
Customize your NLP pipelines, like I did with Emoji. And just don't forget, in general, for natural language processing,
Speaker 2 [28:20]
processing projects.
Speaker 1 [28:21]
projects, domain knowledge,
Speaker 2 [28:21]
Domain knowledge is very important. It's better to ask the experts.
Speaker 1 [28:24]
Ask the expert.
Speaker 2 [28:25]
or a native speaker.
Speaker 1 [28:26]
speaker okay so like I promised the the code will be on github along with the slides so I'm happy to answer any questions I think we ran out of time but I'm gonna be around if you want to have some one-on-one discussions about NLP and I'll be happy about it
Speaker 3 [28:54]
Test, test. Oh, that's great. Yeah, I hope there are many, many questions. We have a whole bunch of minutes for all the questions. Questions? Yes, over there. Yeah, thank you for the talk. You mentioned the language that is getting chosen, so you had English and German. Is there also a split between like formal English and social media English at that position or is it really just the high level?
Speaker 1 [29:30]
The model is independent on the context, so it works for...
Speaker 2 [29:35]
It works for formal language and informal language.
Speaker 1 [29:36]
and informal language. Anything you do to customize for your specific case, you'll have to do it on your own. Unfortunately, there is no social media English or Twitter English. We don't have that yet.
Speaker 3 [29:50]
Over there.
Speaker 4 [29:53]
Hi, thanks for the talk. I'm just a little confused about your position on chunking. That first example you had, which was star intro to glowing star tool, that's just a nominal group, right? So, I mean, it was analyzed as a clause with a verb, but it's not actually a clause. It's just a nominal group, and therefore it would just be one chunk.
Speaker 1 [30:23]
It is a one chunk, it's a noun phrase.
Speaker 4 [30:25]
phrase yeah so what would the value of chunking be when in your own example the whole the entire text would just be a single chunk
Speaker 1 [30:35]
Not exactly. The chunks were
Speaker 2 [30:36]
The chunks would be, for example, names, first name, last name.
Speaker 1 [30:41]
or the whole phrase that belongs together as one.
Speaker 2 [30:43]
there is one concept.
Speaker 1 [30:44]
concept, like United Nations would
Speaker 2 [30:46]
that nations would be one chunk.
Speaker 1 [30:47]
So the important part is you're not dealing with words.
Speaker 2 [30:50]
towards a single element,
Speaker 1 [30:51]
single elements, but
Speaker 2 [30:53]
collect all the words that belong to the same
Speaker 1 [30:55]
to the same noun.
Speaker 4 [30:56]
chunk but in that example a star like a good review for a great tool this is a single noun phrase with embedded noun phrases yes and because chunking is flat rather than nested then yeah it just seems kind of self-defeating to apply chunking to this rather than using the dependency parse or something instead so yeah I'm just I mean you could see the tool produced a miss analysis where it said the intro was a verb where it's actually introduction and noun so yeah I'm confused why I'm like I don't think Spacey and Matt Honnable really recommend chunking either
Speaker 1 [31:42]
There are more sophisticated ways of producing better chunks. It becomes more important for topic modeling when you want to have meaning.
Speaker 2 [31:49]
to have meaningful.
Speaker 1 [31:50]
Meaningful noun chunks and you want to glue the words together It's the built-in chunking may not be as useful. You're right. Thank you. So When you have to customize it for the purposes of interpretation just to make it easier to Map the right chunks to the right concepts. This is
Speaker 2 [32:10]
This is something you may have to
Speaker 1 [32:11]
may have to do as a customized
Speaker 2 [32:13]
code.
Speaker 4 [32:15]
Okay.
Speaker 3 [32:18]
Thank you. More questions?
Speaker 1 [32:19]
Questions?
Speaker 3 [32:22]
Okay, over there, so I take the chance of going there. You know this conference only works if there are volunteers, so you all know the signup.zone page, and you visited already, and all of you already put your name in the right slots for session chairs, video recordings, conference setup and teardown, hopefully. So you have to do it. It's not a joke. The conference will not go on if there are not enough volunteers. So here's the next question.
Speaker 4 [32:55]
Thank you again also for introducing spaCy. I didn't know it so far and he is also quite questionable
Speaker 3 [33:01]
is also quite a question about how is the support for the German language for packages
Speaker 4 [33:06]
So especially regarding
Speaker 3 [33:07]
regarding Compunt.
Speaker 4 [33:08]
compound splitting and
Speaker 3 [33:09]
and,
Speaker 4 [33:10]
uh,
Speaker 3 [33:10]
uh, dramatizing.
Speaker 2 [33:13]
Sorry, I'm not sure I...
Speaker 1 [33:14]
I'm not sure I understood the question. Is it about limb disease?
Speaker 2 [33:17]
Limitization?
Speaker 4 [33:18]
um so um spacey is um the support for the german language so um in particular for the
Speaker 3 [33:24]
particularly for the compounds worth splitting.
Speaker 4 [33:25]
splitting and the lemmatizing
Speaker 3 [33:28]
How well is it?
Speaker 1 [33:31]
The language model for German is very good.
Speaker 2 [33:33]
So it's been tested many times.
Speaker 1 [33:35]
tested many times the tokenizer
Speaker 2 [33:37]
the tokenizer has been
Speaker 1 [33:38]
has been improved continually, so it's something you can find out when you run space
Speaker 2 [33:42]
on spaCy over German.
Speaker 1 [33:44]
I would say it's as
Speaker 2 [33:45]
as good as English, which is really good.
Speaker 1 [33:48]
Really good for language like German. Okay, it's not as good for I think finish was the one where they really struggled But the compound German words you might think it's a bigger problem, but it really isn't
Speaker 4 [34:03]
Okay, thank you.
Speaker 3 [34:06]
So more questions over there so as you see the most Annoying part of being a session chair is to run around But the rest is easy you can do it
Speaker 1 [34:19]
Thank you for your talk. It was very interesting. I just came across a library. It's called Vader. Do you know that? I just wanted to ask about your experiences. Are they supposed to analyze also the emojis and stuff like that? How are your experiences? Yes, so Vader.
Speaker 2 [34:41]
Vader um
Speaker 1 [34:42]
Just like Darth Vader, it's part of NLTK.
Speaker 2 [34:44]
of NLTK library nowadays.
Speaker 1 [34:45]
Nowadays, it was integrated.
Speaker 2 [34:46]
It was integrated, and it's used for sentiment analysis. It's completely rule-based. It has lists of things.
Speaker 1 [34:51]
lists of things that is like an emoji dictionary, but it's a more expanded way. It's unfortunately not very accurate.
Speaker 2 [34:59]
I have tested different sentiment and
Speaker 1 [35:00]
different sentiment analysis libraries, and Veda was one of the least accurate, surprisingly. But it's always the disadvantage of rule-based verses.
Speaker 2 [35:09]
based versus machine learning.
Speaker 1 [35:10]
versus machine learning. Even naive base algorithms were more accurate than VADR.
Speaker 2 [35:14]
So just to give you an idea.
Speaker 3 [35:18]
Okay, more questions? No? Not at all? So then, thanks again, Anna, for this great talk.
