Whispered Secrets: Building An Open-Source Tool To Live Transcribe & Summarize Conversations

Speaker 1 [00:06]

Thank you very much. I'd love to do a little bit of hands up just to see what you know. I don't want to tell you things you already know, although you're going to get what I'm going to tell you anyway. But who has used Streamlit before? Hands up. Okay, who has used O-Lama before to run LLMs locally on your computer? Okay, interesting. Who's used a different technique for running LLMs locally other than O-Lama? Okay, so like Lama CPP, other things like that. Who has used Whisper, either the API or running locally? Who has put all of those things together into a live audio streaming pipeline that also throws everything into an LLM. Yourself, sir, okay, let's have some drinks later. I will be in the bar wherever it is. So this is the slightly mad exercise I set myself because what I do in my day job is I run a company. We're an AI consultancy, Coefficient. Follow me on the Twitter and the Mastodons and the rest of it. I'm also involved in Piedasa London. I like the Piedasas and these kind of communities. But one of the things that we do is we do a lot of work for the UK government these days. And when they say, can we use this Gen AI technology, can we use OpenAI, can we use things like Fireflies to transcribe our meetings, the answer is probably not, because you're going to be sending data to servers maybe hosted in the US, and you can't do that, especially if what you're working with is what we call official sensitive in the UK or even top secret in other countries. So maybe that's your reason for coming along. You want to know, how do I actually build something that looks like a Fireflies.ai service that can process incredibly sensitive information, whether it's government or just commercially sensitive, or you'll just really like open source, and you want to just know how to build these things yourself. Because when you can build something for yourself, you can do things that those fancy Silicon Valley companies cannot do. You can make these things do exactly what you want it to do, not just summarization but so much more um just some of the other work we do at the moment uh we are involved with uh this uh so if anybody is interested in deepfake detection that's one of our live projects helping the home office and decent um and this is just launched follow me on linkedin if you want to find the sign up link to that so this is everything i'm showing you is on github it's all open source code so you can find the whispered secrets repo on my github or you can try and scan this very funky AI-generated QR code, and I tested it earlier, and it did work, but otherwise you can find it on GitHub and find it that way. So I'll leave that open just for a moment later. Now, this repo is actually hopefully quite easy to set up and run for yourself. If you want to follow along live today, good luck. Let me know how you get on. Because it adheres to the modern data cookie cutter, so there's a load of things. It's sort of got poetry and pre-commit and CI baked in. It just kind of works with a little quick start. Find the quick start and just follow the steps, and you'll be able to get everything loaded into your environment no matter which operating system you're running. So without further ado, let's get into it. Here's what we're covering today. Why are we doing this? Something about transcription, moving into how do we put this into a nice streamlet application. Some of the trials and tribulations of trying to even do this with a real-time processing pipeline. Then how do we feed it into O-Lama? What even is O-Lama for those of you who haven't seen these components? And then trying to put it all together. so the why the why is is is what i've covered it is this idea that we've got these amazing tools that are out there but it's not just about being able to put secret information into it it is being able to go far beyond the uh i guess the the uh not very imaginative uh product designers of uh of silicon valley and make things that are so much more interesting and hopefully by the end of this talk you'll get the idea of what i mean by that don't just summarize this thing like maybe just start shouting at me in a meeting and tell me to shut up if I've been speaking too long. That's an interesting product that I haven't seen someone build yet. But you can go away, take this code, and build that for me by maybe tomorrow. So that's the real genesis of the idea. So we're going to get started with some notebooks. This is going to be very code heavy, so strap yourselves in for this session. So the first notebook in the repo, So it's introducing the concept of transcription in Python. How do we even go about building a transcription pipeline? So we're using this thing called Speech Recognition. That's my library of choice today. And let's have a look about what that is. So it's a beautiful PyPI library, and it has support for loads of different engines. So some of these work offline, like CMU Sphinx, and also the OpenAI Whisper. The open source works offline. Most of them work online, but it's a general library that interfaces with lots of different speech recognition backends. So let's have a little play with that and see how that works. Make sure I've run on my cells. So first of all, we need to configure our microphone. And so I've got four microphones. Notice Microsoft Teams Audio is one of these. So you could be running this directly plugged into your Teams output, silently transcribing all of your Teams calls in case you need to go and make a cup of tea while someone else is talking and you don't want to miss anything. That's the kind of stuff you can do when you've got open source technology. So these are the available microphone devices. We do need to select which microphone we're going to use. I want to use number zero, and that's now selected as my source. With a sample rate, off we go. Let's try listening and transcribing. So very easy to get up and running, just sr.recognizer. That sets up the recorder, and then we say, go, say something. So I'm saying something right now. It is listening to me from my microphone. I really hope demigods are paying attention. And when I stop talking, it's going to complete that cell. So now it's done that. One of the reasons we're using this library is because it has built into it the whole stop talking detection. I don't have to code that up, which is quite nice. So Whisper thinks I said. This is now running this through directly into Whisper because I have Whisper installed and it's kind of built in as a nice plugin. And there we go. It says dogs instead of demigods, but I'll take that. Maybe the demigods are dogs. This is an amazing thing that we're learning today. So it's not great. It's not perfect. it's probably using the whisper-tiny.en model. Another thing that is a bit challenging about this is it's not going to be able to do this in a streaming format. It just takes some audio, loads in the audio file, transcribes it. How do we do streaming? So there's lots of different ways you can do this. There are various implementations. There's Faster Whisper. There's people that have released a paper recently that shows how you can do streaming. I've done a way more simple hacky approach just to give you an idea of how it all works rather than showing you 1,000 lines of code, I'm going to go for maybe 400. So I'm loading in the tiny.en audio model. I'm now actually using Whisper. I'm not using the Whisper that's sort of being passed through from the speech recognizer. This is Whisper Direct. So now I'm going to do this again. I'm going to start setting up the recorder. I'm going to start configuring things. So energy threshold sounds very exciting, but it's just at what point it detects silence. So if you're very quiet, then you might want to take that down. Dynamic energy threshold, we're setting that so that it doesn't stop recording. And I'm also going to have it just adjust for ambient noise, just for a moment. Very good. Now comes the tricky bits, the bits that really have been quite challenging whilst putting all of this together, which is we have to start managing threads, and we have to make sure that everything is thread-safe. So I'm going to create a thread-safe queue. This is from the queue built in Python Library. And we're going to use that to start throwing chunks of audio data into the queue. This audio data is going to come in here, and then we're going to take the raw data and throw it into a queue. It's just a thread-safe queuing system. But this is a little callback ready to go for when I start my streaming talking. So a few other things I'm going to set. What's the record timeout? So how often is it going to sort of chunk things? And you'll see this makes quite a huge difference to how it looks down the line. So I'm just going to set this to two seconds for now. It's going to start listening in the background. And when it does so, it's going to tell me this is loaded and listening. So it's actually, again, listening to me right now, but you notice the Jupyter Notebook itself is completed. And at any point, I can just say, right, what's my data queue? And we can take some audio from that queue. We can clear the queue. We can look at the audio data. It's going to run for a while because it's a load of binary nonsense. I could try and display this in IPython, but your mileage may vary. It keeps giving me zero-second files because I'm not talking for long enough. More importantly, we actually want to turn that into a NumPy array. This is all just boilerplate-y code. Don't worry, this is not the interesting bit. It's just converting audio into NumPy numbers, lists of numbers. That's all it is. Because we can then take that, and we can throw it into audio-model.transcribe. This is our Whisper model set up, ready to go with all of that configuration. So, let's see how it goes. I was talking for a bit there. there we go and it's mostly taken in gibberish but this is because it's only taken in a tiny tiny snippet to do this properly you really need to have it constantly running listening to everything i'm saying instead of just trying to infer from what is in effect actually almost silence it's transcribed just a few like very very few numbers to get to uh total hallucination so how do we improve this let's see let's scroll past this now again if this was a useful transcription i could strip out just the text bit extract it and then i could start tracking my transcriptions this is all super useful to note because we're going to use all of these components but we're just going to throw one of those bits in a big old for loop so this is what our transcription demo is going to look like i'm going to actually run a file i'm going to choose my model it's going to start listening to me this is not live it's just a gif of something i made earlier and it will it will transcribe but it will continue to transcribe this is what it's going to look like but how do we go from what you've just seen which hallucinated nothingness into this so there's a few little things that i had to do to get there and i'm just going to give you a glimpse into number one i wanted a cli so i'm using typer that adds a few a few things i can select my model i can specify the energy threshold and all of those nice parameters including my microphone up front if i want to and also because it's typer my favorite cli library it gives you out of the box dash dash help that tells you how to use this thing which I really love so props to Sebastian for diaper then I need to maybe choose the model I might want to use one of the nicer models than the tiny model which isn't very good at differentiating between demo gods and dogs and so I've got that now built in as a little feature then there's the infinite loop so before I was pulling out little snippets from my data queue I was throwing it in to my audio data then running that through transcribe all I'm doing is I'm throwing that into a big old wild crew and it's just going to keep on doing it until I force crash the entire app. Then there is another little nice bonus that I've thrown in. Not really necessary, but I was having too much fun, which is phrase detection. It just detects, it's a lot of code, but it's a very simple idea. If I've paused for X amount of seconds, then it will just add a new line. That's all it really does. It just looks nice in the output. And it was a few extra lines of code. So let's see this demo again. Hoping the demo gods are on my side today. So, here's our nice transcribe app, I'm going to go, I'm going to be bold, let's go small, take one size up. The medium is about four gigabytes, now it's loaded and listening, so it should be actively listening to me at the moment, and every few seconds it's now going to start printing this to the screen. And we'll continue doing this, I'm going to continue talking, I've done a lot of blurbing at my computer over the last week or two while putting this together and if I pause then hopefully after 10 seconds or so it would actually just insert a new line and realise there's a new paragraph there which I'm not going to do because I can't be bothered so let's cancel that, that is a great start, we've got a live streaming application it is this one file, the file is quite big let's make that big a few, but it's only the things that you've seen, so I'll give you some glimpses But you've seen it all at this point. So you've seen the typer, CLI. You've seen the configuring your microphone. We did that in the notebook. You've seen the customization of the model, setting up our speech recognizer class instance. Then we specify, we customize it out a bit. We set up our data queue from the, again, where's that coming from? From queue, import queue, very cool piece of kit. Thank you, Python Core developers, for making this. We set up our transcription. It's just a list of strings. we then adjust for ambient noise that's why it holds for a moment before getting the model ready, we've got our callback you've seen that before, puts things in a queue we've got a thing that then kicks off and when I hit control C it'll just stop listening then we've got our wild true, big old wild true and the freeze detection which honestly could all just be deleted but I was too excited so I made it this is where we then are just loading stuff into the audio data and we're turning into NumPy, transcribing it, then extracting the text-only part of the JSON that comes back, and then we stick it in a big old list called transcription, with transcription.append, unless I'm still talking, and it's just another two seconds, but of the same phrase, so I never shut up, in which case it continues just writing onto the same line, which just looks a bit nicer, and then we just print it all out. That's what we're doing here, with a little bit of keyboard interrupt at the end. So, we've got our transcription pipeline. Let's move on. The next bit we need is some Streamlet. So I'm going to give you a little glimpse of Streamlet. Let's make sure we're not running that anymore. A little glimpse. So Streamlet, if you haven't seen it before, is a very nice piece of kit. It allows you to build things that look like this, little data apps. Very cool. So one thing that I want to point out with Streamlet is we're going to build a basic scenario. we're going to load in a data frame, we're going to write that data frame out, then we're going to we're not going to plot some charts, we're not going to plot some maps check out the repo if you want to see this. The really key thing about Streamlit is we've got some sidebars, let's add some sidebars and session state. Now, if you haven't used Streamlit before then the key thing to know is that it basically just cycles through your app like a wheel, round and round and round and round and round just constantly running the thing and that's quite weird if you're used to just writing a script and it just runs once and then gets to the end if you want to constantly do something you have to loop it streamlet just starts doing that for you so if you want to do something like say running a process in a separate thread streamlet's going to get really grumpy because it's not really happy about doing that so you'll see how i've um worked around that problem in a moment but just something to note and if we want a variable saved between one cycle through and the next cycle through or across a page reload then you have to not use the variables because it will just forget it immediately when it cycles the top and you have to store it the session state. So when you see all the session state stuff, just think, fancy streamlet variable. Could have just called it that, but it would forget it immediately, like a goldfish. So let's just see this little streamlet demo. We'll see how this looks. So this is just a data frame. It's loaded in. We made it up on the fly. It's got some cool widgets that I made in a sidebar, and the session state is maintained. Let's say John, press enter to apply, and then we can access that from other parts of the streamlet app, even though it's just constantly cycling round and round. That's been stored into the Streamlet background cache until I reload it, and then it just forgets everything again. So those are the key things to know about Streamlet. Very easy to build a thing that looks really slick and really fancy with very little amount of Python code. It's basically like React, but you don't have to learn React. So let's start to build a cool little application. I'll walk you through this code. We've got our speech recognition. We've got Streamlet. We've got a function that just literally just takes in something and it prints it back out again. This is the model you're loading. But we're not doing any of that bit yet. I just want to introduce you to the Streamlet. And then we're going to escalate up step by step. We're going to add a little title. We're going to add a sidebar. It's going to have some cool little sliders on it. And then it's just going to print out the model we've loaded. It's not actually doing any model loading yet. I just want you to get the idea comfortable with Streamlet. So this is all it's doing. And as I change my model, it's just going to change and track my session state here. So that's good, but let's plug it in to something a bit more useful. So now I'm going to extend this a bit. I'm going to keep the fake transcriber for now because I want to add a little bit of functionality. I assume it gets really, really gnarly here. So I would like to have, I've got my sidebar, that's the same, but I'd like to add a button that says start, another button that says stop. And this is where Streamlite starts to really, really kill your productivity. Because we need to start tracking, has it started yet? What do I show if it's started? Or has it stopped yet? In which case, do I need to do something when it stops? And what do I show when it has stopped? So we need to track, is it transcribing or is it not transcribing? And if we haven't started tracking that yet, we need to load that like a global variable in session state. Then we need to say, right, if you're not yet transcribing, then show a button that says start and when you click that button then let's update the global variable to say we're now transcribing let's also start the transcription likewise we need to stop transcribing if it's not yet if it is already transcribing so let's see how this works let's play around with this let's take a fold, thank you very much There we go. So this has got my start transcribing button. And when I do that, it's not actually flipping to stop transcribing, or it's showing me a new button. When I click that button, then suddenly the start feels kind of janky. And I can just go through this again, start transcribing, it adds to the stop transcribing, and I can click that, and then things move around, but it's a really bad user interface. So there's a few extra things we have to do here, because Streamlit is remembering too much at this point. So the things I need to do is I need to add a streamlet.rerun. And let's see what the streamlet.rerun does. What it does is when I click start transcribing, it's going to actually reload everything, cancel out that initial button, and load in the new button. And when I rerun after stop, it will cancel out everything. But when I click stop, I don't get the message saying I've stopped transcribing anymore, so therefore I'm going to add a little sleep. And this then starts to behave much more nicely. So there's a few things we now need to do as we start to move towards our proper working demo. So in this version, I've plugged in some of my transcription code. As I start transcribing, it is listening to me right now, but it's not actually showing any of the output in here. So if I was to look at this, it will be listening. It just isn't showing it in Streamlit. Like, it is working. But Streamlit isn't listening to it. So there's a bunch of stuff that you need to do to actually say, okay, we need to pull this out. But how are we going to pull this out of here? It's just printing it to the console. So long story short, what I've done, instead of using something like Django channels or anything funky, I have literally just started dumping this information into a file. Then I pull that out of the file with Streamlit, and I display it in a container in Streamlit. So let's see how that works. in the next demo. Endless segfaults. Thank you very much. Okay. So, what's happening now is I am able to pull that information out of the file. It's going to start listening to me it's going to start pushing something into Streamlit so we do have a live transcription pipeline but it's got a few extra bugs and flaws this is again where working with this technology can just really weigh you down what it's not doing is clearing the transcription every time I start it's just going to load in whatever it was already saved into that little file buffer from last time so if I was to open up a new one someone's basically seeing the last conversation I had So that's no good. So that's another thing that we have to fix. I'm going to skip along to the final version of the application now. To show you how this can all come together. So with those fixes in place, we've now got a system that can listen to me with some live streaming. I'm going to stop seg-14. I can start transcribing. it's going to start listening to me. In fact, you know what? I'm going to stop transcribing there and let's go for a small model or maybe a medium model. What I've also done behind the scenes is we've got a O-Lama model that is going to start looking at the TXT file and every 10 seconds it's going to start throwing a summary into the summary part of this. So it's just putting together all of these building blocks. Now, if you want to see how the O-Lama model is configured and put together, have a look at the repo. There's also a little notebook that I put in there just called O-Lama, which walks you through the four functions that are required to do this. So O-Lama is kind of really easy to get set up and running. But what you're seeing here is the live demonstration. It's listening to what I'm saying. Let's make that a bit bigger because this is quite cool. But if you didn't want to listen to all of this talk, you didn't want to listen to all of these words, you don't want to see the transcription, then what you can actually just do is read the summary that's being generated live by an LLM listening to what I'm saying. So I think that's an interesting sort of state of the art. Again, three minutes. I'm just going to show you how the Olam apart is working because it is so easy. In fact, I'm going to leave this running because that's kind of what I wanted to do with all of this. So let's have a look at our Olam notebook. I just want to show you how easy this is. Hopefully that guy is going to listen to this. So O-Lama, just go there. It'll work on macOS, Windows, Linux. It's very easy to get stuff on run. You can pull Lama 3, which came out three, four days ago now, from Meta. It's just literally llm.invoke. Now, this is actually using the langchain community wrapper for O-Lama, because if you wanted to switch out to OpenAI, then you can do that with just changing your function. Makes it super easy. So very easy just to invoke a connection to a live, locally running LLM. But also, if we want to do something more interesting, We can actually say, you know what, can you just scrape all of the PyCon talks that are happening this week and just give me some cool talk recommendations. Watch out for your prompts. They will, I think Llama 3 does have a tendency to just get distracted by other things in your context window. But hopefully what this is going to do as a final demonstration is give us all some cool recommendations of other things to check out. So what's it said? Check out the talk. AI Revolution Will Not Be Monopolised Inej Montani, you've got Alexander Meyer you've got Tobias Lampert I did not code this in advance but I'm very glad that Lama likes me and Ferdinand Schenk so hopefully that gives you an idea of how easy this will to put together and let's just see how's our translation pipeline doing it's over here somewhere where have you gone here, right, there we go, so there's the summary of this talk, thank you very much for coming along today

Speaker 2 [24:23]

Thank you so much John for your talk. It was insightful. So I have a couple of questions. So the first question is, so what is the minimum system requirement for running the transcription module? So that is the first question, please.

Speaker 1 [24:40]

I don't know what the minimum system requirement would be, but the Tiny Whisper, tiny.en, really is tiny. It's about 30 megabytes, I think, or 70 megabytes to download, and it will run on a fairly small system. Again, everything you've seen just here has been running completely offline. The Wi-Fi is not that good. It is all using O-Lama. If I was to switch across to this, you can see I'm running my O-Lama here. Again, you could be using one of the smaller models. O-Lama is dead easy to switch out to. one of the really tiny Gemma models that Google's put out as well. So system requirements, you can actually go quite a long way. Our next step at Coefficient is taking all of this and it's putting it onto an Android phone. So check in with us in about six months to see how that goes.

Speaker 2 [25:21]

Thank you so much. So the next question is, is it possible to detect a speaker with this setup? Does it have speaker diarization capabilities?

Speaker 1 [25:32]

So, the answer is yes, speaker detection is a thing. I have not played around with this enough to know if it's capable to do that accurately. I think your mileage will vary, but yes, there are tools that you can use to do that.

Speaker 2 [25:45]

Okay, thank you so much. Thank you so much, Joan, for your talk. So I think maybe we might have some other questions. Some of us can reach out to Joan after this talk. So thank you so much, everyone, for your time. So have a nice time ahead. Thank you so much.