Creating an Interactive ML Conference Showcase

Speaker 1 [00:03]

Thanks for being here. I never talked in a room with such comfy chairs. I hope you stay with me until the end of this session, which is the last of the day. So if you nick away, I won't judge you. So let's go right to the topic. So my name is Harald, and I was confronted with this question. Hi, we need a showcase for machine learning for our next trade fair slash conference. So I started thinking, okay, a conference like a booth, we need something eye-catching, something visual, intuitive. So let's do something like a webcam stream image classification, but not the things that everybody does. So maybe not just using an existing ImageNet stuff or something interactive that you can have a visitor at your booth and then you can, okay, let's train to distinguish my batch from your batch or something like that. So with custom classes, this means it should learn really fast, interactively with a visitor in front of the screen. So we need probably something with transfer learning. And of course, we want to deploy this on the machines of my colleagues, small laptops, no GPU support, something really easy. And to show that it's actually our own code, we want to show this code. We want to talk about the code. So it will probably not be that polished, but interactively with Jupyter notebooks. So after thinking for two weeks or so, just kidding, I just said, OK. and luckily there is a nice example for the same problem transfer learning example teachable machine which does actually quite the same thing so I hope never do live demos but I try my luck okay so you have a webcam view here and you can just train three classes so the green glass maybe something like this with a few examples so really it's for for deep learning this is nothing right like a hand or something like thumbs down okay so we have now taken around 90 images and if I put my thumbs up I get the cat if I put it down I get the rabbit and if I open my hand I get this thing so works really good okay so we want to do something similar but we will do it a little bit differently of course we want to use Python because otherwise I wouldn't be here and we want to train arbitrary number of classes and as I said already we want to do have something interactively something with visible code it will not be this polished like the teachable machine but more something like this and it should be self-contained where you can easily deploy it locally without any internet problems at conferences and stuff like that so similar like the talk before there will be the Python code at the end So first some theory. What do I mean with transfer learning? How many do already know what transfer learning is? Okay that's maybe half of it. So this kind of classification problem you usually have some data and some labels. So maybe you have images and the label says if this image is a cat or dog. And you usually train some deep model deep learning model and with that you can then use unseen data that you did not feed into the training and predict new labels Italy in the best case you can feed new images of cats and dogs here and you get the correct label problem is in such a trade fair environment you don't have time to label it thousands of images so what makes this actually work like that you can use unseen data and this the thing answers correctly in any case is that we have here really lots of data and really small improvements on your model so that you don't learn something about one dog but about every dog so you only adapt to each individual dog a little bit and by that you learn something about dogs in general. So this will not work in our case and this takes too much time so we do something differently like transfer learning. The idea is you have an existing model that you trained on maybe for two weeks or something, invested lots of resources to to solve the ImageNet challenge. This is 1.2 million images with around or exactly 1,000 categories. And you have this model, but you also have new data and new labels. And this is the actual problem that you want to solve now. What you could do, it's maybe a stupid but obvious idea, you just feed your new data into your old model, but of course you get the old labels out. And then you maybe you are lucky and can make some mapping from the old labels to the new labels. So this will work, for example, if you have the problem cats versus dogs. Because in the ImageNet, there are several different cat races and dog races, and you just have to combine all those labels, and then you can solve this problem. But in most cases, this will not work, because your old and your new problem usually don't fit that well together. So, what usually is done more often is so-called fine-tuning. It's the same initial setup. You have the old model and the new data. And then you take a part of the whole old model and you make a copy of it. So you have all the weights, you have all the things that the model already learned, but you feed into the training process the new data and the new labels so that you get a new model that will then also predict new labels on unseen data. This works really well if you have actually quite a lot of new data. You could in theory also train your model from scratch. You're just saving two weeks of initial training to get the whole process started. You save a little bit of resources, but it will not work well for very small data sets. For instance, if you have a problem you want to distinguish your dog from other dogs maybe from automatic dog flap or something like that you can use the ImageNet data because it already learned how to look at dogs but you can fine tune it to your problem but as I said with few data sets few data points this will be very prone to overfitting so if you very complex model that potentially can distinguish thousands of classes, but you have only a few data points. So it will actually memorize those data points instead of learning something general. So, the thing that we will do for our trade fair demo, we use the old model as a feature extractor. And what this means is we take the whole old model and we rip it in two parts. Because usually those are, they have some initial layers that are more oriented towards understanding the input. And they have them later layers that are more oriented about generating the output. And we want a different output, but we work on the same input, so to say, because we are working on images. So what you can do is you take the old, the new data, and you use a part of the old model to predict features. So these are not the labels, because we removed that part that was creating the labels, but it's actually the data that would flow between these two parts. So it interprets the input and creates some intermediate data structures. We call it features here. And then you can use these features and your new labels to actually train a novel model, a new model, and you can use different estimators here. similar that's not so something simple sorry a very simple model that's not so prone for overfitting and the rest is actually quite similar you have unseen data of course you now have to feed it first to the old model again to get the features and then you can predict your new labels what what you have seen with the teachable machine example and what our trade for example will do similarly is the feature extractor is the mobile net convolutional neural network and for the classification part that actually creates the labels we do a very simple k-nearest neighbor classifier so what do these two things mean just in in brief convolution neural networks they are based on convolutions that's the name instead of matrix multiplications convolutions in in German it's phyton you probably have seen it in blurring filters or sharpening filters or edge detectors they are quite yeah frequently used in image processing anyhow so what you can see here in the animation is you have one filter function which is a rolled over another function and the convolution is to multiply these functions and have and the area under the product so the integral of the two functions combined for images in a discrete case which we have in neural networks it's quite similar you have a filter function so to say and an input function input function would then be here the individual pixels of an image and you roll this filter over this input image and you do a piecewise multiplication so this is the multiplication of the functions and then you just sum them up this would be the integral so a times J B times K and so on you just sum them up and you do this for every position in the image so what you can do with it for instance you could wait the left part of the image stronger than the right part or something like that or you could for instance detect if there is a gradient in the brightness or something like that. So is the left part darker than the right part of this part of the image? Or is there a phase in this part of the image? You can learn things like that. Compared to a classical matrix operation this would more look like this. Every output is linked to every input so each output can only compare the whole input globally and not locally so this is what what does this mean it means CNN's are better in understanding neighborhoods so if you have this image sorry of a church of my hometown the the structure of the image is actually defined by the neighborhood of the pixels right and therefore you can distinguish if this pixel even if it if it has the same value is from a surveys surface or from an edge second motivation for CNN quite similar you roll this filter kernel over the whole image right sorry I think battery of my mouse died and therefore even if there are several phases in images at different locations if you would have a classical fully connected layer it would have to learn each phase at each position individually but with a CNN structure the filter moves around the image and they share the weights there can be one kernel that learns faces independently where in the image they occur okay this is a picture from the original paper from LeCun you have seen now all of these strategies the way of an input image with pixels and we have this convolutions and here for instance in the first convolution, we create six different filter kernels. So we have six different interpretations of the image, so to say. Maybe one is blurred, maybe the other one detects edges. And then we do some downsampling just to reduce the size. And we do some more filters on that. So interpretations of the interpretation of the image. And this increases in different interpretations and the image dimensions get a small each in each step and some at some step where you have quite quite a good understanding what is in the image you do then switch to this classical fully connected layers where you use your classification and then say yes this image contains an a so we usually have two parts understanding the content using base layers convolutional layers and assigning a class to this image which is done by this fully connected so-called top layers and if you want to use a CNN for feature extraction you usually do the split here so that you remove the classification part and only keep the image understanding part okay but what do we then use for classification I already said we use k-nearest neighbor it's the most simple classifier that one can imagine you just memorize all the training data that you get and if you then have a novel piece of information that you don't know the label for you just look into the surrounding maybe the next K equals 10 neighbors and do a majority vote so this would be a blue square instead of a red star okay enough theory only two more slides to actual Python code I already said we wanted to use Jupyter notebooks it's quite easy to set up it's quite flexible if you want to host it locally or on a server and And it's really great to combine results and code. And that's what we want to do. One little problem is that we have the Python kernel running on the server. And we have the browser which does the display. And the browser has access to the webcam. But the kernel has to do the computation. So we have to get the webcam image to the Python kernel. And what we used here is actually IPython widgets and basically IPython WebRTC, which is based on IPython widgets. The nice thing is we have some representation of a widget in the Python kernel and a representation of the widget in the front end and they are synchronized. They a synchronized state. You could have multiple views, but that's not what's important here. And another nice thing, they already have some capabilities to play video and audio and stuff like that. So what IPython WebRTC is, it uses this IPython widgets, and just a little bit on top, mainly it exposes the web browser features, MediaStream API and WebRTC, and it has some nice wrappers to record a media and send it to the kernel which works like this. You could have for instance an image recorder and you tell it please record an image from any potential web media stream, for instance the webcam. This then updates the image property of this widget model. So the image recorder dot image is then updated and this is in this synchronized set so that we can access this on the Python kernel okay so I would like to switch maybe to the actual code this is about eight eight cells for the whole thing and I would like to step you through this code so the first thing is how to get the camera. As I said the IPython WebRTC has a nice wrapper around it and it just feeds it just passes on all the the constraints that you pass to it to your web browsers API. So yeah everything what the get user media of the JavaScript can do you can access here so maybe just for starter if I just show this so it just creates an image or a video from the webcam stream this is all purely in the browser but I can then define an image recorder based on this video stream and what this does is it creates a widget and whenever I click this button maybe so better it shows the image locally but that's not what this is the important thing but it updates the image recorder dot image you can see no so how do we get this image the image recorder has an property called image which has a value which is just the bytecode of the image encoded in PNG so portable network graphics which means we have to use a library to interpret this encoding you can just use the Python image library so it can just open this PNG encoded data and what links the raw byte to file like access to for the Python image library is the bytes IO so then you can see if I just print the result and this is this PNG image from the PIL you can display the image and to work with it more easily you can just wrap it into a numpy array so they can see okay this is actually exactly the resolution that we set it earlier when opening the webcam and it has four channels like three color channels and one alpha channel and the alpha is quite uninteresting because it's 255 all the time okay this solves the problem how to get the webcam image to the python kernel the next thing would be to get the pre-trained cnn this is really quite easy thanks to keras and they provide all the pre-trained a lot of pre-trained models just by the application module you say which kind of network you want and some parameters here is the yeah we want we don't want the classification part of it this is actually the default input size of this thing and so actually this is quite fixed and I adapted the webcam resolution to fit this parts but just you can see okay we only wants three color channels and no alpha channel so having this convolutional neural network we can then define a function to extract all these features. We do exactly the same as before. We take the image obtained from the webcam, wrap it into a NumPy array to get rid of the alpha channel and then just, oh yeah, we have to do also some pre-processing. This is also a function that is delivered by Keras. Just as you see here, the image is in from zero to 255, so byte size. And the model once it's scaled from like a normal distribution kind. And then we just predict this input and then we get this feature representation. So it looks like this. And from the 200,000 individual values of an image. We then extracted 1000 to 24 meaningful image content features, so to say. Okay, let's speed up a bit. So what we then want to do is we want to collect training data. This is quite easily done. To speed up this process, we can here just have an input. That's what you see. Please input a current label and then I just say maybe up and I have the same image recorder as before and the nice thing is these widgets all their properties are observable so they are trade lets and I can register a callback that is called whenever the value changes so whenever one clicks this button this function is called which does exactly the thing that we saw right now it extracts the features it memorizes the labels and it depends both input and desired output into a global variable so that was so let's do just some images with a thumbs up and let's do the same again with the label thumbs down okay so we have here in total only 32 training examples then we have to run a classifier on top of that which is thanks cycle learn really easy. You just define which classification you want. In this case as already mentioned the k-nearest neighbor classifier and you call the function fit. That's actually everything you need. As a bonus I have here just visualized what actually these 2024 values look like. So this is thumbs down and this is thumbs up. For humans it's quite hard to distinguish I think but for the k-nearest neighbor it's super easy. So last thing, of course then we want to use this classify function. It extracts the features the same way before and it just calls predict on the classification model with this feature. That's actually everything you need. The second step here is just to get also the certainty of the model so that we have a nice output. And then And then we bring everything together. This works exactly the same way as we obtained the training data. So we have another image recorder with a callback. The callback classifies the image and updates a result label. That's actually quite everything, but to have it as a live video, we didn't do one more thing. We programmatically trigger a recording of another image. So we want another image, which then again triggers another callback, which again triggers another image. So this is quite actually it's an infinity loop. Therefore, we add another abort button to set a Boolean value to get rid of this infinite loop. And with that, we have a classifier that can distinguish at least thumbs up and down and is quite certain about it. So, of course, if you showed anything, it only knows up and down. You can also nicely talk about the yeah, the limits of this approach. Well, that is here or also on slides. And with that, I want to thank you for your attention.

Speaker 2 [26:40]

And so easy, you can do a machine learning showcase for the next conference. So I think we've got time for one question. So anybody wants to...

Speaker 1 [26:50]

Want to know something more? Where can we get the notebook? Write me an email. I send it to you. And I also collect your personal information and send you marketing information.

Speaker 2 [27:11]

Okay, then we might have time for another question.

Speaker 1 [27:13]

and you used you resize that the image sizes what can you do if the image size is 2010 24 times 70 768 or something like this yeah for simplicity I left out this I just set the webcam resolution to the correct values of course you can do an intermediate function that scales the input. Without scaling? Why would you try to get it without scaling? Okay, yeah. So this is the default value. I think especially MobileNet is also available pre-trained in other resolutions, even higher resolutions. Um, it's just not that anymore, but yeah, it's, it's also available in higher resolutions or we have to use a different CNN that's maybe was trained on higher resolutions.

Speaker 2 [28:27]

Thank you for the talk, and I think you'll also be around here the next day, so if someone has some questions, he can also approach you and get more info. So thank you for being in the machine learning use case track, and I think we are done, and it's time for the final keynote for today, and well, have fun for the rest of the time.