Distributed Hyperparameter search with sklearn and kubernetes

Speaker 1 [00:02]

Yeah, hi, my name is Jakob. I'm like already mentioned. I'm going to talk about the title with a lot of buzzwords So first who am I? I'm Jakob like already mentioned I'm a lead data scientist at CodeCentric, but I also have a lot of experience in DevOps or ops I'm in ops. I've mostly focused around kubernetes. I'm also certified in kubernetes blah blah blah I work for CodeCentric CodeCentric who doesn't know it is Consulting company in Germany. We have a lot of lots of locations If you want to work off with us, we are hiring if you want to hire us Also talk to me, but that's enough for the advertising Also, you can reach me on Twitter and all this kind of stuff so I'm gonna want to Tell a bit of story how we arrived at all this stuff. So imagine yourself yourself, you have a new project or you're starting a new model, you have that machine learning algorithm. In the case of the talk, I'm mostly mentioning machine learning, not like deep learning kind of fancy stuff. But still, there's a lot of, with usual machine learning algorithms, there's a lot of hyper parameters. For example, on the right, these are all the learning parameters for light GBM, that's an XGBoost clone, whatever, and that's only the learning parameters. It still has a lot of performance parameters and all these kind of stuff. So there's a lot of stuff. And you could tune that by hand, but since that's quite boring stuff, we usually do hyperparameter optimization. The plain one is like we do a grid search and grid search in SKLearn, parallelized with JobLib, where we can do some kind of stuff in parallel, depending on your data size and you can run that on your laptop but that won't take probably forever. So unfortunately I screwed up my cluster and all so I have to fall back for demos to pictures. And so that would look in sklearn quite straightforward. You have a random forest classifier as an example. You specify your parameters grid. Just a few numbers here. You do the grid search, you do a fit, the grid search searches all possible combinations here, and when that's done, you can get the best score, the best parameters, and also the best fitted estimators, which you can then work with. So, once you try that on your local machine, you probably run like it will take forever. So the first step usually everybody does, and we did, is try to scale it vertical. If you want to use it in production at one point anyway, you need to train it on a controlled environment somewhere, so you need to put it on a server there, the server probably has a lot more power than your laptop, hopefully, and if you do it right, you can still use Jupyter and all this kind of stuff. And also if you're in the cloud, you have a nice tradeoff that you can easily buy more CPU or more RAM to do faster or to do more stuff in parallel. So that worked for us quite some time, quite nicely. But then we discovered that most of the cloud isn't endless. You can still not scale your server vertical forever. On Google, it's quite nice with 96 cores. On Amazon and Azure, it's a bit lower. But at one point, you run into the instance where you fill up your instance quite quickly and you run into problems. Or not necessarily into problems, but you wait long. That was the case for our case, and that's the reason we started looking into distributing that kind of stuff. Because if you think about that, hyperparameter optimization, at least the basic algorithms are like embarrassingly parallel. It's easy to say independent, each job is independent, at the end you're only interested in what was the scoring result and the parameter and maybe the model, but you could even retrain that. So for them, then we had like two questions. How we do the whole distribution thing and how we coordinate the data result. How we do the distribution thing, like the title already said, we used Kubernetes for that. If you don't heard of Kubernetes, I doubt it because it was quite hyped already, it schedules container onto a cluster of nodes. It's somewhat easy to use, but the main reason we use it is A, it's the most dominant scheduler currently out there. It's available in every cloud as a service currently, or at least every cloud I know. And it's the hammer I know the best, and that's the reason I'm using it. Then we have the second problem, how we coordinate that whole distributed thing. Our first sketch, we went and thought about how we could do that quite quickly and easily done. So we still would have something like a master where we run our code. Then we need to decide how we distribute the stuff. We have our search grid. The search grid is all the possibilities of hyperparameters we try. We partition that by number of nodes or something more intelligent. We then pick all the data and the search object and the partition grid and store that somewhere. S3 or any kind of object storage or even database if you fancy. We then create worker jobs, Kubernetes already has the concepts of jobs, it would be one time container, so we start a couple of workers with some parameters, each worker loads up his data, loads his search grid and all this other stuff, he runs his hyperparameter search on his smaller search grid, when he's done he pickles the best parameter, the accuracy result and the model out again to S3 or something and the master in the meantime waits till everyone is done and then he loads up all the results from the worker, compares again which wants the best result and takes that result and returns that. You can implement that quite easily by yourself. If you want this from Google, they did it for their Google Cloud Platform that's quite hard-coded into there, but you can also look, take their code and transfer it to any other cloud in at least two hours or something. So the concept is quite simple and code wise for the user it's still somewhat similar. We still have our random forest, we still define our grid search and then comes the new introduction. We now have something wrapper code which does the algorithm we specified before, takes that. We also need to give some cluster stuff. Then we do the fit. The fit function basically does all the steps I previously explained. We need to wait till we're done, and then we can get our scorings, our best parameters, or our best estimator. Though that works, that worked quite okay for us. It was quite simple to use. We actually wrote, we used a different implementation, not the one I linked here, but I can't show the open source one by Google. It's no additional framework. It's plain Python or in Kubernetes, but it all feels a bit hacky. You have to specify how you partition the parameter grid for every different algorithm. If you have a different algorithm, you maybe need to change that. So you're starting to re-implement stuff there and it didn't really felt good. So we know, we knew that all the algorithm had used joblib as a backend. So our idea at that point was, we can just write a custom joblib backend, which does that, and that was at the point where also why it's a stock proposal. So we thought, okay, we're gonna use joblib to start parallel jobs on Kubernetes, we're gonna wrap everything behind that custom backend, we do error handling, and all these kind of stuff. But as soon we did some deeper thoughts in it and tried some prototype, we felt it's like we are reinventing the wheel all over again. And it's really complex to actually do that, like abstract everything behind JobLib for the user so he doesn't have to work. And if you, the custom job link backends aren't really documented, and the reference implementation is Dusk. So we naturally started to looking into Dusk. For the people who don't know Dusk, Dusk provides advanced, analytic parallelism for analytics, enabling performance at scale for the tools you love. That's from the homepage. But Dusk is quite known in the analytics space, and we wanted to try it all the time. It's familiar for Python, it looks almost at pandas, it can scale up to clusters, it can be used as a job backend like previously, and it has a nice integration with Kubernetes. So we just tried, okay, we're not gonna write our own backend, we're just gonna try that. So we did, and code-wise, that turned out really well. On the top, we have a little bit of difference here. We now need to set up your dust cluster. That can be tricky if you want, But other than that, we still have our random forest classifier, our parameters grid, and then the only change we do is that little line with job lead parallel back end dust, and under that the search, and then dust takes all this kind of stuff automatically care for us, and that was great because all the stuff we previously wanted to implement by ourself, dust takes care of that. So the whole DUS solution, it works quite well for us. It was really simple. It's scalable. We didn't need any kind of custom code. Unfortunately, it's an additional framework, which is kind of, and the whole DUS cluster setup, it can be tricky. If you're inside Kubernetes already, it's quite easy. If you're outside, you have to do some network fiddling and all this kind of stuff that can be quite intensive for us. But in general, we always wanted to try Dust, and it was a really nice introduction to Dust because from the actual logic code, it's one line of change. It provides the scalability for us, and it's a really stable solution for us. So that's basically it. As a summary, that was just how we transfer from the last year from different kind of projects. I would always recommend everybody try to do vertical scaling always first. If you're then getting into some limits, you can try the wrapper code solution I showed. It's easy for quick and dirty deployments. If you're in Google Cloud, it's out of the box there. If you're in different clouds, it's like really quick to modify. And I wouldn't recommend to anyone going the way we did with the custom backend anymore. It turned out a lot of effort, and we could scratch that. And I would recommend everyone dust at that point. That turned out quite well for us. And that's basically our really short story. If you have questions, it's a really high-level overview. You can ask questions now or come to me later or tomorrow or Friday. That's it.

Speaker 2 [13:11]

We actually have time for questions.

Speaker 3 [13:18]

Thank you for the talk first question I Have already seen a grid sesh done with task alone without using kubernetes on a cluster What value kubernetes adds in comparison so if I use only task and a cluster instead of using? kubernetes task and cluster

Speaker 1 [13:37]

Yeah, so if you already have a dust cluster and it's fine, then Kubernetes doesn't bring any value. The reason we also here use Kubernetes is basically in every enterprise, we are consulting companies, we come into enterprise companies, and nowadays any bigger company most have a Kubernetes cluster somewhere, and that's the quickest way to set up the cluster. We use it as basically like a provisioning tool. We could also set up the cluster with Ansible, but with Kubernetes, it's easier for us.

Speaker 2 [14:14]

Any other question?

Speaker 4 [14:20]

Thank you for an interesting talk. You mentioned that you're working with geodata, and I wonder from which scale you have to deal with parallelization of hyperparameter search. Since we have quite small data, we have simple neural networks, feedforward, shallow, and you can wait a couple of minutes or dozens of minutes until the grid search finishes on your laptop. So I wanted to ask about the scale of your application. How many instances you have to join, classify?

Speaker 1 [14:56]

Okay, so the single scale, we have two reasons for hyperparameter actually, for the single model, but for business requirement reason we have at least one project only, we have to train multiple models, like think about for every country a model, and it's the same model for every country, so we need to, that's one reason we spend some effort in that, and the scale for a single model depends between, if you dump out the CSV, between 50 megabytes up to 50 gigabytes, which isn't a good idea for a CSV anymore, but that's a different story. Does that answer your question?

Speaker 2 [15:39]

Okay, unfortunately we won't have time to take any other questions, so thank you very much, the speaker.