Hidden Markov Models for Chord Recognition - Intuition and Applications

Speaker 1 [00:04]

Thank you very much. So thank you, everyone, for coming to see me. It's pretty happy to see everyone interested on this topic. Well, just one minute for myself. My name is Caio, and I work for FreeNow. But it's basically this mobility app calling taxes. But as you can see, there is nothing related to my actual project or my actual work. This is a side project. And my main intuition today is to not come up with a new technique, machine learning models or deep learning for performance-wise, but I wanted to share with you all the steps that I've been to when I wanted to use a case that I had interested. So I play music. I wanted to try to predict some chords for my music. And I also got interested in how I could learn this technique called Hidemakov models. So I just used combining these two interests. and came up with this project as a side project and do not expect super high performances but my expectations is for you is to just follow up in the learning process and maybe you can even use the concepts that you learned today for the projects when you go back to your work maybe you can see that this can be applied in the same fashion way so as i found out the internet might be so good, so meanwhile you download. You can have the PDF on that link on GitHub. I will start out in the notebook here, but as soon as everything's finished, everyone can follow along. So let's just start here loading. So I'll do a lot of talking. Because of this project, it's a little bit new for me. I think it's a little bit complex as well. I didn't want to try to have so many risks of live coding. So I'll try to talk a lot, but if everyone is interested to go a step deeper and try to understand the functions that I have written. All the functions that I have abstracted, because I just wanted to show the logic or the line of thoughts, all the functions I wrote are on this utils folder. So you can see how they were built there, or all the more technical details. Today I'll go more in a conceptual wise, or how to think about hidden Markov models, how to think about modeling a problem with that. So, let's start out. So, basically, as I said, this problem started with me. I play guitar and sometimes I see that I see a music on YouTube, I hear the music on YouTube, or I see a music, someone playing music somewhere, and I really liked it. I really wanted to play with my friends or at least for myself. And even I search on the internet, I cannot find the chords. I cannot take the guitar for myself and play the music. So, why not think about, okay, maybe I can join a few of the techniques that I want to learn, or I would like, or I want, or I know. And what if in a hypothetical scenario, we could just send the YouTube link to somewhere, or I could just upload this sound music, my MP3 file, or YouTube link to a website, and it would return to me all the chords, or mainly the main chords that has been played in that music, so I could just play along, I could follow along. So this is the main motivation. This is the main context that I started with in order to build this project. And I ended up coming with just a small pipeline, and this is what I want to go through today. So basically, we'll talk about what is sound here for computers. So if I want to start out as input data having a music signal, an mp3 file or whatever, and at the end have some good chords predicted so I could play with it, this is more or less the pipeline that we go. So we'll see how music is represented, we'll see the main technique that we are going to use in order to be able to extract features that are relevant for my problem on chords, recognition. It goes a little bit on Fourier transforms, and at the end, the main feature that we're going to use is called chromagrams. I will go over them. And this one, they will go as inputs for the way I model my Hiedemarkov model. So when I go here, this is the main chord that I want to explain about today, because on the internet, if you want to find out how this tool works, you can actually search on media, papers, archive, or somehow. But there's a few things that I didn't find easily because they were hidden behind some really crazy mathematics. So my main idea today is to speak about these probabilities in an easy way so everyone can understand the logic, try to understand the logic behind it, and try to understand why was it used. And finally, just a small step again, is about just evaluating what the Hidemarkov model has produced and how can we evaluate it somehow. Can we appreciate already in terms of sounds? Can we have an understanding in a statistic way what is the quality of the sound that has been produced? And this will be the final block. So as we start, we always have to make the questions. Okay, so what is the kind of input data that I'm working with? And here I'm showing, I'm just reading a sound and it's basically a key piano being pressed. So let's just take a look how the sound looks like here. The sound is louder than that. So it seems I don't have sound here, so I would ask Alisa to make the sound for me, please. No, just upload to the maximum value. thank you for your work so what does the sound does look like in my computer so this is basically the sound when the key was pressed and this is the sound going on through time so x-axis we have the time and the y-axis we have the amplitudes it's a kind of a mapping way to represent sound in terms of sound wave pressures so if we take the signal and we give a quick zoom inside so more specifically for this 0.5 to 0.55 seconds you're going to see that how this sounds actually quite different from the beginning so this is basically the same signal that i showed before but zoomed in in a 0.05 seconds and we can see that even though it doesn't seem inside it's pretty periodic. So it's basically a periodic signal that over time, of course, it changes its dynamics. But if you look in a quite small time window, it's a quite periodic signal. So the reason that I'm saying that is because I need to come up with some technique somehow that will bring me closer to my objective of producing relevant features to understand which chord has been played in that song. Or in this case, it's a pretty simple sound. it doesn't apply but it's just an introduction so the technique that we're going to use is called fourier transformation we're going to step by step so the first thing that we want to understand on this transformation is it basically tries to understand what its main fundamental components that were used to build the sound that we are seeing here so when what do i mean about main components what the most basic components whenever we speak about sound or signal processing we speak about sinusoids. So the main Fourier transformation will try to understand what are the all sinusoids, what are the most simple frequencies that I could add up in order to have this signal that you're all seeing here. It basically is a transformation. So it will take the signal that we have. We can take another example here. Alisa, can you play the... Again, sorry. So this is an example to show what happens with the Fourier transform. This is the sound of a whistle. And this is the signal that I have if I upload the sound to my computer. And this whistle is on a specific note in terms of the pitch, in terms of the frequency. So this A5, if we search on the internet, this note A5 corresponds to a specific frequency that corresponds to roughly 880 Hertz. So when I go and take the signal and apply my Fourier transformation, this is what we have. So it will tell me now the x and y-axis will have changed. So before it was time and amplitude, and now we will have frequency and energy. So the frequency is what our main goal is, try to understand what is the main components that builds to that signal, and the energy would be the loudness, the amount of that frequency that we have present in that signal. And because it's a pretty simple signal, I'm doing it with a simple signal first, we can see that the main components, the peak here, is around the component that we expect, around 880 Hz. So this will be our main approach for the technique. We'll try to understand what the frequency that makes up to that sound, and with that frequencies we'll try to understand what chord has been played. But we have one specific problem that falls into one of the technicalities of the Fourier transformation. So when I take this transformation on the mathematical way, the Fourier transformation will look at the whole signal, doesn't matter if it's one second, two seconds, or even 30 minutes, and try to say, okay, given the whole signal, what are the main components? What are the main frequencies that are used in order to build that? And for us, in our case, because we work with music, I don't want to do that. I don't want to lose this kind of information of, I want to understand what are the frequencies that were in the beginning, in the middle, and at the end of the song. So what is the problem? This is an example of the problem for the Fourier transformation, is that if I have a sound which changes over time, listen. So this is a person whistling different notes over time. And this is the plot if I take the whole signal and do the Fourier transformation with that kind of sound. We know that the person has whistled different frequencies, different notes, but can I say that this note happened first, or this note happened second, third, and fourth? I cannot. So if I wanted my music, I don't want to throw my signal to the computer to predict the chords, and the computer throws me a list of chords, and I have no idea which one to play first. So a way to come up and to take advantage and also have this time domain information on a signal, maybe we can take the whistle that we have done, instead of calculating the Fourier transformation in the whole signal, we can maybe split it up. So make small cuts on it, and in each one of these windows, we can call windows each one of these splits, we can apply the Fourier transformation. So with that, we will have the frequencies that make up to the first window, second window, third window, until we come up with the whole signal. And just this technique, just this change to make Fourier transformations on small time windows is where the name of the technique comes up. So this is just one step in advance of the Fourier transformation, and it's called short time Fourier transformation. So it's the Fourier, but applied to each one of the windows. So if we look at this plot here now, you can see that we have the time dimension and we have the frequencies and the color basically means the energy, how much of that frequency is there. And then we are able to see the frequency of whistles that has been played. Can you play again, Alyssa, please. And you can kind of have the intuition of what were the notes that were played, and we actually can see the dynamics of the note going up, going down, and going up again for a longer time. So, this will be a great advantage for us if you want to see the sequence of notes that has been played because that's what we want, right? And but again everything here that has been shown is on notes i don't want to predict whistle notes i want to predict piano keys press i want to give a whole song so on the song we have guitarists playing the whole all the strings of instruments you have drummers we have bass players playing everything so we need to understand not only the frequency that is being played we need to understand the whole possible notes. So if my guitar is playing a chord, which is called C major chords, and this C major chord is made up with a lot of notes. Not a lot, but with more than one note. So this is the step that we have to understand just a little bit before. So if you want to do the short time for your transform, I want to do it in a way that's relevant for my chord recognition system. and how can i understand which chord has been played so basically chords uh any musician here okay you guys can help me if i say anything wrong but chords are basically a set of notes it's not a random set of notes but it's a specific sequence of notes that can help me understand or can have a pleasure pleasurable sounds for everyone that's hearing or not they have different functions but in the end it's basically a set of notes so when i do my short time for your transform i will get all the frequencies that happens that happens in that short time window all the possible frequencies so has has in this small window do i have an asa an a note or a b note c notes and more than that in in the way we uh in the way we created the the system notation for musics, I don't have only one A note. I can have one A note, I can have an A note in a higher pitch, A note in an even higher pitch, and this system goes forever. But if we think on the chords and I want to say, okay, this guy on this moment has been playing C major chords. I really don't care if they're playing this lower pitch A or this mid-level pitch A or the higher pitch A. what I care is, okay, how much of the note A is inside this window. So, this is the last transformation that we are going to do to the music signal before using it as features for a Hedermakov model. And because we don't care which A has been played, I can basically take the output of the short time for here transform. Let me go back. I can basically take the output of this. And whatever this frequency means, an A, higher pitch, mid-level, lower level, I can sum them up. Because for me, what cares is how much A there is on it. And if I join all these frequencies, what I can have as outputs, what I can have as outputs is this kind of plot. So this is what we are going to use from the music signal. And this is called a chromogram. The chromogram is similar to the time period transform we have the x axis that's been the time and on the y axis instead of having the frequencies i add everything up and instead of having frequencies i have the notes names so basically saying how much of c notes i have on this how much of g d e and for each one of the 12 notes that we have at least in western notation so we can have an idea how that works can you play this song, Alisa? This is just the intro of the music from The Beatles. The Beatles will be the main goal that I'll be working with here today because they are good and I like them. And if if you don't like it, well, sorry. But can you play it, Alisa, please? Okay. So, beautiful song, an intro, everything clean, just a piano, should be super perfect for at least for this scenario. So, what happens is that I took that sound and I calculated the chromogram at least for that intro. And we can see that the dynamics are pretty much well detected. So we can see there are hard thresholds, basically as barriers here on the chromogram, and that means that when the person changes the chords. So we can see that on this first moment from 0 to 1.5 seconds roughly, there was a lot of C notes. And if you go to website, if you go to try what is the chord that is being played for that 1.5 seconds, it's a C major chord. chords. There's some of the E notes, there's a little bit of G, there's a little bit of B, and on this case, trust me, these are a few good set of notes for a C major chords, or C major 7 minor in this case. Okay, this I put as an extra section at the end of the complete notebook, because I thought was a more advanced topic to discuss with but i can discuss at the q a for sure of the of my intuition or doing the time resolution if that's fine yeah there is some theory so a bit in short i want to detect the smallest step from a semitone on my full year transform so i have to understand what is the minimum time window for my short time for a transform that can detect a half semitone of time of frequency resolution. We can go after, for sure. So, we see here, there's a C note, E, G, B, and afterwards there's a change of chords. So, you might be thinking, okay, maybe I don't even need the Hayden-Markov model for my chord recognition problem. Maybe that is easy, that is solved. Just look at the bottom, whoever is the lowest pitch note, take it. But right after this first chord, the C chord, if I look, I just searched for it, the second chord is not a D chord, actually. It's a G major chord. So that happens because chords don't have to be played... The notes of the chords don't have to be played in sequence. And in music, we actually have a name for it, which is chord inversion. So instead, I play the chords, and instead of playing the root notes as the lowest pitch, I just play one of the other chords. So it's actually not that easy to just make an assumption that the note that has more energy is actually the representative of the chords. So we need something more to understand and try to predict a little bit more smarter, in a smarter way, what is the chord that has been played. And actually, one of the things that makes Western, at least in Western music notation, is that each chord that I play, at least in my music, they have a certain function. They work in a certain way to make you feel relaxed, to make you feel a little bit more tense, to make you feel like, oh, the music has finished. Oh, no, the music's not going to finish now, something's going to happen. And that's actually really good and a really good input as a feature for us if we try to predict. So as a motivation to try to use Markov chain, we have to think about that. So if I play the C chords and then I play the G major chords, there's a really, really big chance that the follow-up chords will be either C major or A minor. And that's because of music theory. That's because of harmony and how western music works. And we can use this information. So if I'm trying to predict these chords of this time window, what has happened in the past window? What has happened in the previous chords? And we can use that together with the chromagram and then at the end predict what the chord is being played now. So this is the main main entry for us to talk about Markov chains. So this is all about music signal processing. That's what I want just to skim over. And now we'll talk about Markov chains. But does someone have any questions until now? So if no one has any questions, I can just try to answer your question. But can you repeat just to make sure I understood? I just asked what was the time resolution you were using in the example I brought okay so the time resolution for that just to repeat for the recording the question was what is the time resolution that I use for my short time Fourier transform and after a few derivations I showed a notebook but I used a 0.24 milliseconds time window so it's a pretty short time window because I want to be as loose as possible in terms of changing in dynamics, in terms of changing of chords, and I want to understand that, okay, this has changed. So, does this answer? Yeah, because in Fourier transform, we have a frequency resolution problem. So, if I use a too small time window, I am not able to detect changes in variations in frequency. So, the reason I chose this 0.24 is because the minimum frequency resolution that I have is able to detect a semitone in terms of frequency. If I use a time window smaller than that, I wouldn't be able to capture a change from C to C sharp, let's say. Cool? Okay. So enough about... Go on. If I have to make a window for the frequencies. No. So, in order to choose what is the lowest frequency resolution that I wanted, I took the lowest notes that a human here could hear. So, from the lowest C note that someone can hear, for a C-sharp, there's a small change in frequency there. And this is how I try to define my window resolution. Last question, just to make sure that we have time yes I didn't use it because I don't know it's I'm honest I'm interested there's actually interest if you have time afterwards but yes I thought about that but okay so enough about signal processing so just a summary I am interested of on the chromagram and I want to know what has been played in the recent past, because this can help me, together with my chromogram, to do a better prediction on the chords. So here the best case scenario that we can use at least as a first thought for me is about Markov chains, or Markov models, so to say. I use both names. So what are they? This is a really good way to represent sequences. So as I have practiced before, we can say that Markov chains is a state transition system, And when I say states, we represent each one of the chords that I have. So let's say if I look at some music. Let's say I look at Let It Be, the whole music, and I look at someone who already created the chords. And I can just take a look. Okay, Paul McCartney has played the C major chords. One note he has played afterwards. We can see that, okay, after C major, he played a few times F major, sometimes A minor, sometimes G major. And if I count every single time that he went from one chord to another, and then afterwards just normalize, so we can have probabilities, this is what you see here. This is what you would see here. We can see that if you're in C major, if I just throw a coin, I could go for each one of these states, E minor, D minor, G major, F major, or even go to the end of the music. and I would see what is the probability of going to a C major, to an A minor, C major, to a D minor, and so on. One thing interesting, for example, we can see, so if in the music I see a G major chord and the next chromagram looks like a D minor, because my Markov chain says, no, there's no way to play a G major and afterwards a D minor, my Markov chain would say, okay, probably even though it looks like the next chromagram looks like D minor, the Markov chain will say no, that's not possible because I know from the transition from my state machine, that's not possible. So that's why the sequence it captures really well what is the possible sequence that I can have. But, as we have talked until now, we have discussed we can look at the past data, we can look at some old songs from Beatles and all the annotations they have, all the chords that have been played in order to build something like this. But on this project, the main input data is not about the text of the chords itself. I have a music signal. I have the chromagram. I don't have this kind of infrastructure here. So basically what we say in terms of concept is I have a state machine running in the background, stating what is the probability of going from one chord to another in the music, but I actually cannot see it, because the only thing that I can see is the signal, is the music signal, or after all, the pre-processing is the chromagram. So because of this matter, this is what I want to predict, the sequence of chords, and this is what I actually can observe. And because of the fact that I cannot see this, This is the only change from a Markov chain to form a Markov model to a hidden Markov model. It's literally because it's hidden. The only thing that I can see is my observations, it's my chromagram or my music signal. So okay, we went from the features, we went to want to use a Markov model for sequence, to understand sequence problems. Yes? Yeah, it's literally built from historical data. So if my historical data, for some bad luck, doesn't show that, that's how my Markov chain is going to be. And exactly I'm going to talk about that in the last step, because I had some errors on my system and more or less because of that. So okay, we have a Markov model, we kind of understand what is the probability of being from one core to another, but the only thing we see is the observation. So how do I connect? How do I work with this? So Markov model is a good, hidden Markov model is one tool which is already quite established, and unfortunately is hidden behind some heavy mathematics. But here, what do you want to know, or I think what you need to know after trying a lot to understand what is actually useful or not, we need to understand three kinds of probabilities. So what we're going to do here, which is the biggest part, the heaviest part, is trying to understand the three tools that a Hidemarkov model needs in order for it to work, in order for it to be able to predict, or in this my case, new chords, but it can be whatever you're working with. And it's actually good because I don't need to go into the details of how the prediction is done. can discuss afterwards, because these three tools is what we need to call a framework. So we are going to use a framework from Python called HMM Learn. And this is the parameters that the package needs in order to be able to produce a prediction. So first one, Chord's transition probability matrix. This one we already have from the Markov model. This one is what we can obtain from past historical data. I look at a bunch of music, I look at all the chord transitions that happen in the music, I normalize it, and then I have my transition probability matrix. I'll go there in this next section. And this is what we have. Then we need to come up in a way to connect my observations with my Markov chain, because I need some way somehow to connect, okay, I saw this chromagram, I saw this time window, and I need to connect that somehow to my Markov model. And these are called emission probabilities, and we will go over that. Least but not least, we need to calculate just because of the mathematical formulation, we also need a small probability matrix that says, okay, the music hasn't even started, what is the probability that I start with a C major, with a G major? So this is called literally initial state probability matrix. The first one we have taken a look. The last one is quite intuitive and literally is just for that purpose to start the prediction. And we will take a look mostly at this emission probability. But then that's a valid question. In theory, everything works but again i have a music signal the music signal doesn't have labels doesn't have which chord has been played so how do i connect how do i how am i able to first search which chord has been played and second how am i able to connect my observations with whatever chord has been played in that time window so for that luckily there are some people who spend a lot of time producing a few data sets. And in this case, it was not different. So this is a work from this guy, from Harten. And he did a PhD work in trying to understand a few things about music and music information retrieval. And all his work, he evaluated together with all the musicians and together with all the scientists, 120 musics from the Beatles. They evaluated the song, the version, right, the studio version, which year, which album that was, and also labeled from each second to each second which chord has been played. So at least this can work for us as a really good gold standard, because we can use that as the annotation, we can have the music files, somehow work with them to connect, and then luckily for us, we have just supervised a machine learning problem. So let's go through it. Let's see if I executed everything before. So the reason why the Dockerfile is actually quite big, because I tried to make it easy to download the musics from the repo that Harte has done, but I also have to do a lot of pre-processing in terms of naming of the musics, matching the song with the annotations files. Then I decided to upload this file to the GitHub. why the github is actually so big and my sorry for that but there's a folder called labs and music in the repo and it contains pairs for all the music so I have around 50 musics all the 50 musics containing the signal mp3 file and the lab which is the annotation so the annotation is literally this it will tell from which moments to which moments in terms of seconds what chord has been plates. If you look at it, there's some hidden secrets here going on behind. So there's a read and simplify chords. Because two things, chords are actually a little bit subjective. It can go a little bit on emotion of who is listening to it. And maybe the guitar has played one chord and the bass has played, oh, and the second guitar played another. So there's a lot of interpretation of what is actually the chord that's being played there. And chords can be really complex. I showed just two examples on C major or C minor, for instance, but it can be really, really complex. And what I've done here on the reads and simplifier, I read the annotations and whatever the complex chords that Hart said the Beatles was playing, I converted them to the simplest form as major or minor chords. It's because I want to see my Markov changes working as a first. So feel free to take a look at this function. But at the end, we're going to work with major and minor chords here. So here I have the annotated start, end. And because I wanted to see, again, the Markov model working, everything that I'm doing here is just with one song so i'm trying to do all the pre-processing with one song and at the end if everything goes right at the end i do with everything so i get the song i get the annotation i get the song i calculate how much of each note so from c to b how much energy of each note i had in each one of my time windows columns. So these columns is all the signal processing for the first section about calculating the chromogram. And the last column is just a join from the annotations data sets. So I could know in that time window, given this signature of energy, I know that they were playing a C major chord. So if you look at this, this is already something that we could work with because each row here can be seen as a row of features we have 12 features for each one of the notes we can ignore the starts and ends we could ignore the temporal dependency and we have the label so i haven't tried actually i don't know how much good or bad it would be but it's a matter of interest that it could be for next steps so i could use 12 features and the label would be one of the chords that they played. But we want to have fun, so we want to now work with the Hiddermarkov model as a motivation. So we have the data set and then we go for the first matrix that I want to calculate. So this is what we have seen before, state transition matrix, the probability of going from one chord to another. And what I've done here is I took each one of the windows and I calculated what is the, I counted how many transitions happening from one chord to another, normalized it so I could have probability, so I could have probabilities. And this is what I've done here. So because, as you see, as you saw, you have, we have small time windows here, we have a lot of chords, so if a chord has been played for maybe three, four seconds, this chord is going to appear a lot of times because I have just small time windows. So, because I have a lot of repetitions of the same chords, you can see these such high values here. Because there's a lot of tendency that, okay, because it's an A minor in a time window, the chord just lasts for too much time, there's a lot of A minor in the sequence. This is a decision that I've made when modeling. I could try to create transition probability matrix just looking at the annotated data sets instead of looking at time windows. This is something that can be done afterwards as a testing, but that's okay. We have looked, we have calculated the probability of going from one chord to another. This is the first matrix. I will go back to this emission because we're going to spend a little bit more time on that. and I calculate also the initial state probability matrix just let's see if it works yep the initial state probability probability matrix I ran through all the annotations files I looked at the first chord and then I normalized this so I could have initial state probability so that's pretty straightforward but now we go to the hardest parts if you search for emission probabilities. So probability of that time window, that small time window of my signal, be a C major chord, or be a D minor chord, or be whatever chord, it's called emission probability. It's a way to connect my observations with the Markov model. But if you search on the internet, if you go for hidden Markov model in median, archive, or whatever, you're going to see explanations like this so this is what i want to explain before and we are going to say that it's not exactly the problem that we have at least for code recognition problems so when we look at admission probabilities and we want to calculate i want to give an example that's a little bit more simplistic of the chords we will generalize for the chords afterwards but let's just change our minds a little bit here and trying to predict what is going what's probably can what can be the weather based on a person's reaction or based on the person's if he's happy, if he's sad. So let's say, for example, that I want to predict what is the probability that I see this person happy given that it's sunny outside? Or on the opposite, what is the probability of this person being happy given that it's rainy outside? So this is the same intuition for emission probabilities? What is the probability of seeing that observation given it is in certain states? So when we look at this, we have actually this kind of situation where my observation is a categorical one. So if I have a categorical observation, he's either sad, either happy, maybe either neutral, whatever, it's still categorical. It's still just a set of lists, a set of it items when i have this it's easy to calculate the mission probabilities because i can just count i can look at my basic i can look at my past data and i can see okay how many times i saw that this person was said and how many times it was sunny how many times it was rainy so i look he was said 100 times he was it was sunny outside 400 400 times it was rainy outside if i normalize by probabilities, divided by the total counts. This is my emission probability. So if it's sad, there's a 0.2 probability that it's actually sunny, and 0.8 probability that it's rainy. And we can do this for all the states, sorry, for all the observations, sad, neutral, or happy. And that basically counts. That's actually easy, and that's what you find everywhere on the internet when you search for hidden Markov models and emission probabilities. But if I try to do the same on my problem... Oh, let's go one step at a time actually. Let's say that instead of having happy, sad, or neutral, I have the temperature. So what is the probability of instead of being sad, what is the probability of temperature equals 20.7 given that it's sunny. If I'm trying to count all the possible distinct values, we can have this. We can have an infinite count, so probability of temperature equals 20.7, given that it's sunny, but I also can find examples where temperature equals 20.07, or find other situations where temperature equals 20.007, and so on. So when my observation is continuous, it's actually not feasible to make counts, to do simple causal categories. And that's because it's a continuous variable. I cannot just count them. So this was the key moments that how can I do that for my problem? Because if you think about it, the observation from my problem, from the chord recognition problem, is a set of numbers. It's not a category. I have the amount of energy for each one of the notes from my time window. So the count wouldn't work. So how do we do it? How do we say, what's the probability of temperature equals 20, given that it's sunny? In order to do that, we have to make assumptions. And it goes for the modeling approach that you do. But if you think about, we actually do it pretty well based on our past experience. So if I come to someone and say, what is the weather in Brazil? So I'm from Brazil, I would say the story on this. So in summer days in Brazil, it usually tends to be between 30 and 40 degrees. So I have an estimation, I have a prior assumption of what the number will look like given some criteria, given that it's summer, given that it's a sunny day outside. But if I speak about winter, not so much different unfortunately. So winter is basically a summer day that didn't work. So we have temperatures not lower than 15 and 25. So, given the key idea that I want to make here is that we make assumptions. So, we can make assumptions, given this knowledge, we can make assumptions on this kind of calculation. So, if I tell someone here from Brazil, what is the probability of temperature equals 40, given that it's sunny, he won't know the number, but he will say, I'm pretty sure this probability higher than if the weather was rainy or if it was winter. So this works just because of these assumptions and this is what we are going to do on this example and then generalize for the chord recognition problem. As humans, we cannot say probability of that equals 0.743. We just can make just a few directions. But here, for the chord recognition problem, we want to say, given this chromagram of a lot of numbers, I want to state exactly what is the probability of observing that this is a C major chord, this is a G major chord. And the way to do that, we have, besides assuming how the chords, how the notes would look like for each one, how the notes would look like for each one of the chords, we also have to assume a specific probability distribution. because I can say, okay, when I play a C major chord, I'm pretty sure that C, E, and G notes will be a super high energy because that's how the C major chord is built. And they would have an average value in terms of energy. And because I don't want to be super strict, I will say they can also vary a little bit. So if I look at my chromagram, I see the biggest notes are C, E, and G. depends on how lower or how bigger or how higher these energies are, I will have some facts in order to say even though it varies a little bit, this is probably a C major. On the other hand, if I see a G, D, E note, I will have some prior assumptions, because I have seen it before, that this is probably a G major chord. And this is how we're going to work here for the chords. We're going to make assumptions of how, on average, each one of the energy of my nodes from my chromogram looks like for a given chord. So if you remember well, we have a whole data set, here is just one music, but we will have a whole data set of where each row is a time window, and we have the label. So what we can do is, let's start with C major chords, take all the rows that have a C major chords and calculates the mean energy for each one of the notes and the variation the standard deviation of variance for each one of them given that i basically create signatures for each one of my chords for a c major chord i have some mean energy for each one of this and some tolerance so i have some variance information about each one of the energies so with this i can calculate my probability of observing a chromagram given that it's C major. I can see, oh, this is my chromagram, this is my first time window, and given this signature I can calculate the probability of this being a C major, G major, A minor, and so on. So basically we managed to create our connection between observations and chords, or in this case states from the Markov chain. We managed to calculate the emission probabilities. This was actually the hardest part. So, this calculation and, oh, let's see. So, we have, we actually, okay, we apply this for the, this is how we can apply this for all chords and chromograms. So, we calculated our three main blocks for hidden Markov models. And we also look at a little bit of signal processing. So, we only have the last part of joining the pieces together, create predictions, and try to evaluate it. But before that, does anyone have any questions? Okay, don't know if that's good or bad, but let's go with it. I hope everyone is shy just waiting for the end. Okay, so we have everything. We have the features, we have the matrices, and we can calculate. I will speed up because we have, I think, no much... Oh, no, I think we have time. I calculate my emission probabilities here. I calculate my initial state that I had calculated before. And this is the main part. So, basically, there's not much secrets. What I use here is a package from Python. It's a framework called code of HHMM. Oh, here. HMM learn. So this package abstracts a lot of things for me, even though it doesn't seem so, but when I calculate all these matrices, there's still a heavy mathematics that happens behind the tool, so I can make the prediction. But hopefully, with this tool, I don't need to worry about how the calculations are done. I basically need to create my Hedermarkov models, and this states that I'm working with normal distributed emission probabilities. So, I just state that this is an object where my emission probabilities will have an assumption of normal distribution. And what I have to say is send the transition probability matrix some parameters that this goes one step in a more advanced topic. But when I work with multi-dimension features, I can calculate the average value of all the I can calculate how much more or less the energy varies for a C node or for a D node and so on. But I can also calculate their dependency. I can also calculate how much the C node varies together with a D node, together with some other nodes. And if you think statistically, this is the term of focal variance. How much my numbers, how much my features, varies with another one. And here I say that it's full because it's the way I calculated my covariance matrix from the emission probabilities. So we create the object, we set the values, this is the initial state probability matrix, this is a transition probability matrix, this is the average values from my emission, and these are the variances or covariances from my emission probabilities as well. Calculated everything, we can build it and hope, and let's Let's execute it. And then we can have our predictions here. So what happens is this prediction, what I did in this prediction, I got the model, I got the chromograms and chords, remember from one song, from one part of the Let It Be song, which is actually super clean piano solo, which hopefully could be super good. in order to train with that data and also predict it with that data in this first case. So I wanted to see actually if the tool worked. I wanted to see if whatever I assumed in terms of distributions, emission probabilities worked as a prediction, as creating a model that creates predictions. The reason I do that is actually, I put this as a post from, it's not related to Hedermarkov models, but it's actually a good technique when you're trying understand whether a certain architecture works. So I built a hidden Markov model, but it's also valid for neural networks that you want to work with, or let's say this complex deep convolution neural networks or LSTMs. A good thing that you can see that, okay, my architecture works, is try to overfit, is try to make your model predict exactly in a small piece of datasets. If that works, I can have some strong assumptions that if I try to generalize for a bigger datasets, that will work. So that's the only reason that I try to do okay simply with the smallest piece of data with the most simple one. And okay, I predicted, and again this is a multi-class classification problem, so I won't put too much moments here because I think you're gonna hear thousands of talks on evaluating machine learning models. The way I evaluated them is through F-score. So multi-class, we can evaluate some overall F-score from all the classes. And this is just a visualization that I found at school, but we can see, given the chords, this is the time. If it's green, it's because the Markov model got it right. If it's black, it's because it said it was a C chord, but it actually wasn't. And you can see that the biggest mistakes, even though it's a simple data again, there was a few small mistakes in the chord transition. So when it transited from a C chord to a C major to a G major, the model takes a few time windows to adapt to the new data, to adapt to the chord change in the chromogram. And let's see how it stayed. So what happens here? So my model produced predictions. My model produced predictions. And remember Remember that I'm producing predictions for each one of the time windows, pretty small time windows, like 0.5 milliseconds. What I did here is just a small preprocessing step, where if I see too many predictions of the same class together, I just join everything and also say, okay, you had a bunch of C major chord predicted from this moment to this moment, so I just join everything up. make my life easier because I will throw this data to a package called SimplifyMidi. It's a way to, given that you have your data, you can create MIDI sounds, like simple sounds from the computer, and then I can try to simulate what my model has predicted, what my model has produced. So let's take a look. Alisa, so if you could press, like, the super advanced data science methods. Can you press two buttons? a little bit faster that's fine the meat is a little bit advanced yep that's fine that's fine so because we saw here because we saw the model did pretty well Well, we could have an idea of the F-score, and we could have an idea of the visual evaluation that they monitored pretty good. Of course, I completely overfitted. Probably if I did any other song, it would be super awful. But it's already a motivation thing. Yeah, it's good. It helps you to spend more time on that. So, okay. I managed to do one small overfit, a small data set. That's not a super big step. Let's try to predict the whole song. and instead of taking the intro I would take the whole song so a big question mark is what will happen when they start to sing or what will happen when the drummer explodes the how you call the plates I don't know symbols yeah so this is a big question what if what will happen and unfortunately on the... Oh no, it works here. Okay. So we can see that now I tried with one song, trained and predicted with one song, the F-score is already not that motivating. And this is what kind of spoiler that I want to show that I talked in the beginning. At the end, it's a little bit not super average, the model, but it's still kind of funny to hear what has been produced. Try as fast as possible. There's some small mistakes here from the model. super bad mistakes it will stabilize in one moment okay I will save you I'll save you some time so it gets super good in the beginning. It gets super beautiful. But the only reason is that it's a beautiful one-person piano playing. It's just one simple instrument. And whenever we have some more instruments or whatever, have some more vocals, the sound, the signal, or consequently the chromagram gets pretty messy. And I'm not even talking about noise. I'm even talking about the instrument itself. So in music, there's a lot of concepts for harmonics. I don't know if you heard about but whenever I have an instrument and I play one note, I actually don't produce the simple Fourier transform that you saw in the beginning of one simple frequency. I end up creating a whole bunch of frequencies just because how the string oscillates. And this kind of movement actually creates a lot of difficulty for the model. That's actually a good point. I haven't tried, but yes. So what I have done is actually I didn't explain. That's a good point. When I do the Markov chain, when I do the Markov model that I explained here, I took only as a relevant historical data, only the exactly last moments. So what was the chord that i've played exactly one time window ago in order to predict the next one but his question like why did i take why didn't i take more time windows before that could help me that would make me a little bit safer and say which court could that be and that's uh although it's a good point i haven't tried but i can believe in terms of complexity it will increase a little bit as well but of course that could be a next point for to try it out there's a good there's a good comment and at the end we produce, I can produce I produce the pipeline oh, I forgot to say about that so basically I created all the functions that you saw in each one of the cells above I concatenated them into a set of a little bit higher level abstractions and at the end we can just call the train pipeline with the list of songs, the path of where the song is, and it's going to do everything that I have shown you until now. So I did that for one song for the Laribi and I could do that for all the songs. I have 50 songs in this GitHub repo. I hope that's legal to be honest. But you guys, you can take please don't sell it. But you can take it and you can try to process it. You can try to understand how the transition probability matrix is. I won't do that here because it takes a little bit of moments. I didn't care about performance at all, so this is really slow. But on the PDF, I have saved the execution and at least the outputs of it. So I took the 50 songs, I trained the Hedermarkov model with 35 of them, and then I predicted with 15. It's like a validation set. For each model with 15, I calculated all the chords on the all-time windows, calculated the F1 score, put it on a list, and calculated the histogram. And as you can see, it's not as really, really, really... Oh, I'm being super hopeful here, but it's actually bad. It's actually bad. But again, I wanted to show the whole process, the whole line of thoughts here. And actually, when trying to predict with all the songs and looking at the predictions, there was a lot of cool information that I could get in order to try to do a second iteration and we go here for the last part in terms of conclusion first conclusion, yeah, it sucks now but we can do it better, so what the model has done or what are the main mistakes from my model that I could at least understand that I can make for my second iteration so first of all So we have some situations that, as you can see, when the person starts to sing, or when, specifically, when I test with other songs, when there's a lot of drums, the chord prediction stays really, really bad. So one of the things that we can do is, what if we just removed the drums? So there's a technique in signal processing that, fortunately for us, there's a package really famous for that. Does anyone know about Librosa? yep i see some yeses yeses so it's a really cool function to process signals in general and they have a framework that i can just use a function called harmonic percussive source separation or hpss for shorts i showed my signal there and given a few hyper parameters i can give i can have with just one two lines of codes a pretty good source separation among drums and the harmonics or the harmonic instruments. Alisa, can you just try to execute one last? Sorry for that. Go to sounds, play harmonic LEDB and go to the middle of the song. So this will be the version of the song where the drums or where the percussive sounds has been removed. Oh, why we do that? So, how does it do that? It's, okay, it's a little bit roughly, but it works in the same way as we worked in the beginning, that I will take the signal, I will take what are the main frequencies with a short time Fourier transform or with a Fourier transform itself. I have some, what is usually the frequencies that correspond to the instruments and hits corresponding harmonics, I can take that and filter that out. If I do that, I can actually separate from my signal the harmonic instruments that the percussive wants. Yeah, to the middle of the song. Oh, nice. Okay, so trust me on this. It is interesting, but I think you can see on the GitHub repo, I hope when you download it, you can execute it. Has someone managed to downloads the report. And you can see that there's no drum, almost no drums on that. I think that's the solo's gonna start now, right? Okay, thank you. So this is why we just want two lines of code from the Liberosa function. Can you execute the percussive if you manage to to go to the same exactly moments. Because there's a lot of drums on that. It's not a super perfect job, but considering it's one, two lines of codes, it's pretty amazing. Okay, thank you. So this is one of the things that could be done as a second step. As I know that the drums really damage my prediction, why not remove them? Vocals also would be another story, also would be another good positive point. But when I started to search for vocal source separation, it's actually way more complex. So I don't even put that as a second interaction. I would wait for some PhD developer guys to develop that for me. Second one. Oh, I actually skipped the second one. Third one, that was pretty interesting. I looked towards giving a chord. What was the, if the model made a mistake, what were the biggest mistakes on that? And I found it pretty interesting that the model would make a specific kind of error that's really interesting from a music theory perspective. So, as an example, a lot of the moments that I was playing C major chords, or the Beatles was playing C major chords, my model would say, this is an A minor. Does anyone sound any interest in that? So, basically, on music theory, you have some conditions of how the chords are separated. And because of some properties of how we build these notes, C major chords and A minor chords actually have the same set of notes. So that's why the model got really, really confused. And what happened afterwards is that, because of my transition probability matrix, when the model made the innocent mistake of predicting an A minor instead of a C major, all the follow-up sequence notes, all the follow-up chords were made for, okay, what happened when the Beatles actually play an A minor? So it would perpetuate the error. So a good interesting thing to do here that I found is because the model made a mistake of doing a C major, doing an A minor, but for some bad luck, there was no returning from A minor to C major. So I didn't actually allow the model to fix their own mistake. So what we could do here, we could make, okay, relative minors, which is the name of this property from the music, I can actually manually change my transition probability matrix, go to each one of the relative minor from each one of the chords, and allow some buffer, allow some probability from my model to go, if they make an innocent mistake, allow them to come back to the original chords, so they don't perpetuate the error. So this one will be also an interesting thing to do. C major, that's true, that's true, that's true, actually, no, no, this is the sixth degree, okay, we'll think about that, so I don't get stuck, but that's true, I'll think about that, I think that's, yeah, I think that's the point. Yeah, that's true. Thank you very much. And yeah, and let's see. And lastly, I took all the bit of songs, all the 50 ones that I had, and I threw that into my Hidemakov model in order to build transition probabilities, in order to build emission probabilities. But if you think about transition of chords, But if a song is a certain key, there's more chance that I go from one chord to another. So if I think about if my music is on a C key, it's actually quite possible that I'd transit from a C note or a C chord to a B chord. But the same thing, just because of the way the scales are built, but if my music is on is, let's see, what doesn't have yeah, if there is one key note that doesn't have these notes one key of the song that doesn't have these notes I could actually calculate different transition probabilities for each one of the keys, each one of the music keys but that would also increase my complexity, because if I wanted to predict a new song if I ever found a new song, I would have to see and check out algorithms to understand, okay, what is the key of this song? So I can use the correct transition probability for my predictions. This is also interesting. I think on the browser, they also already have this kind of small functions to estimate. I don't know how precise they are, but I think if you show a small piece of song, they can try to say, okay, this is probably a music on C, on D major, or so on. Finally, I created a list of predictions. I even created the MIDI files, but in the beginning of the story, I wanted to have an output that I could play with it myself. I could take my guitar and play with it, but this is not relevant for the hidden Markov model parts. But let's say we have the predictions. Let's try to scrape the lyrics. Let's try to scrape where the beats, or let's try to understand where the beats of the song happen, because usually chords change depending on the beats. and then try to fit the chords to the beats if I manage to do that I could create some spreadsheet not spreadsheet, some PDF that would output for me lyrics that I scraped from the internet, where the chord change in terms of beats and what is the chord that it has predicted so in the near future which is really really future we could do that and maybe all night could be saved for because of hidden Markov models. And thank you. Does anyone has any questions? Let's start here. so the question is because music is not only depending on the past the current chord doesn't depend on the past but also on the future if I could use bag of words but how could I use the bag of words in this in this case if I manage to incorporate the time dimension on this word count I think it could be a possibility but at least as a first thought when I calculate bag of words it's go more or less than the full year transform I lose what is the order of of the chords or what's the words that happen it's this I have a very uncommon configuration, which is the Markov model, which is trying to say that in that scenario you would like to reduce the impact of the Markov model, and where is it in the code? I don't understand where it's balanced, how strong the probability is from the Markov model, and how strong the probability is from the a priori estimation. Yes, that's a good question. So if I understood correctly, just to repeat, we have two sorts of information. We have past data, the past chromagram, and we have information from transitions from the code itself, from the Markov chain, Markov model. So how can I weight them somehow? Which one should be weighted more? So this goes on how the Markov model, the hidden Markov model creates the prediction itself. So this is a presentation notebook, but if you go to the complete one, there's a lot of more information that it would take a lot more time to cover. On the last section, I go through what is the intuition of how the prediction is being done behind the scenes. I threw every probability matrix to the HMM package and I said, predict it. But what is happening in the end? What is happening behind the scenes? And if you look about it, Markov chain, they don't predict sequentially. They don't predict this chromogram and then this one and then this one. It will take the whole sequence and then it's going to say, given my sequence of observations, I will try a bunch of sequence of chords. This is actually not what happened, but it's a little bit more intuitive. What is the sequence of chords that maximize my overall probability? So let's see here. So the final sequence of chords that I have is actually the sequence that maximize this probability of what is the most probable chords given observations. If you break it down, like this is just Bayes breaking down the equation, and trying to make a few assumptions, but this is the breakdown of the two different sources that you have. So this is about emission probabilities and this is about Markov transition probability. So in the end, you take the whole sequence and you try a bunch of C-chord sequence and just check which one maximizes this probability. And you can check the equation on this notebook. It will make more sense. It's a little bit more complex, but that's why I didn't cover it so completely here. honestly no but it's a good point I haven't thought about anything how dealing with inversions but what do I expect it I saw some chromagrams I saw the energy of my notes and although my chromagram would indicate like in the beginning oh this is a D even though it was a G but I expected that from the sequence, from the past sequence and from my transition probability matrix somehow my Markov chain would understand that we saw that that didn't exactly happen because of the results so then it would be one of the extra features like consider the relative minors considering everything this is actually a good idea for extra features for the hidden macabre model on the annotation annotated data set from the beatles it's a lot of cards and i made a lot of simplifications to try to see at least a few poor kids yeah yeah i can i only have 50 songs i might be arrested because I share with everyone. So I need a bigger data set and just think about how to do a workshop with that. Anyone else? On the back. level of the DIT of how you can do notes. So, for example, when you have a solo, you can have a sequence of single notes. And those notes might be, I think, I guess the model might be very competitive, right? Yes. So, the question is, how does the model deal when there's only single notes being played, for instance, in a solo? And honestly, because the model has the full objective of giving that window, giving the chromogram, predict the chords. One out of the 24, like minor, major, minor, and sharp and flat. So there are two moments where the model just doesn't understand. One is when it's single notes. The other one is when it's silent. So if it's silent, the model says, I have to predict something here. And a cool way to maybe try to deal with that, it would be interesting to deal with uncertainty. So the model can say, I think it's a C major, but I'm actually not sure. But it's actually a problem. Oh sorry, she was I have to admit I wasn't very familiar with Markov's model but I was wondering so if I understood correctly this adds a kind of state machine in the back end basically that tries to help understand the sickness, I was wondering whether the chromatograms themselves are good enough to just read off the code because I understood that there are some tricks where the market model may be pulled into. As you said, you have this C major versus a minor discrepancy that could silence them and so on. And I was wondering whether actually there's a drawback in using this model, which also, I mean, you have to do all the training and so on. You don't need that if you don't rely on the program themselves and then you have the permission to recognize So the question is, wouldn't it be enough if I only use the chromogram to predict the chords? And you are right. You're absolutely right. So when I calculated the chromograms and joined it with the annotated data sets, so we have chromogram and chords and label, I actually thought about that. So maybe I actually don't even need this tool. But my purpose here was twofold. So one, just try to join some tool that I want to learn with the interest that I have in music and try to do something I hadn't done before. But I'm really, really interested if I stopped at that moment and maybe just threw into a logistic regression, just threw into a simple thing that could classify the curves and check the F-score. And the answer, I don't know, but I'm really interested. I haven't tried. But that's a good point. You always have to try to be as simple as possible. But in this case, it was the interest of learning the tool itself. Just here. Yeah, so just... You said you couldn't deal with silence, which I found a bit surprising because you could create an artificial hamster state, but that doesn't work because of the state, because of the artificial probability, it's a general one. If all they start playing, still the chromogram would show something, just because of noise, just because of some mic problems, electricity itself. But of course, we could make maybe some smooth down and say, hey, do not predict on this chromogram. That should be maybe enough. Like, predict this chromogram? Oh, but then you have a point that if I predict this as silence, I have to embed that into my transition probability for the next chords. So I haven't thought about that. but so we would maybe it's possible but I haven't thought fully about that okay yeah I will talk to you afterwards then to just get the name of this tool but I haven't seen it that's quite interesting does someone else have any questions well okay I'll be here in the conference for the all the days if you ever have any questions or want to talk about music next steps things that could be done I'll be happy to hear happy to do it some thank you