Hyperparameter optimization for the impatient

Speaker 1 [00:02]

Hello and welcome everyone to the session on hyperparameter optimization for the inpatient. Great to see so many people show up. That's pretty cool. My name is Martin Vistuva. I'm working at AWS. Maybe the first parts will be a bit boring, so stay with me here. Machine learning pretty much is being used now everywhere. For example, you would use it for image detection. In this particular example, let's say you have an image or just a video where you would like to detect objects, in this case a car, could be also people on the street, bikes and so on. Another popular application is basically on text data, like with ChatGPT, this whole field of natural language processing become very visible in the public. So in this particular case, this would be entity recognition and some medical records where you would identify the patient's name, gender, age, medical dosages, and so on. Other cases, of course, I recommend our systems and ranking in general. This is probably a big field. For example, at Amazon, apparently, we also do that. But you also have it in video streaming platforms or so on where the system would just recommend you tools. items you would be interested in based on your history like what things you've watched what things you've clicked on and finally also pretty cool feature code completion something we already have in our ids for a long time but this one is a machine learning driven one where basically based on your context it would recommend your entire function theme and so this little example is showing pretty much what these things can do. So here you would just write a comment describing what the function is doing, and now you start writing your Python function, and you get autocompletion for the entire logic. So there are pretty cool things you can do with machine learning. So what do we need to do machine learning? So I guess the most common ingredients are the data set, obviously. Then we have a model that has typically parameters and you would need to estimate these parameters. And you estimate these parameters typically with a training algorithm. And these components are oftentimes like the core components. That's what people think about. What's oftentimes overlooked or maybe doesn't get that much attention because it's a little bit annoying to figure that out correctly is how to find the configuration for the training algorithm. Because that's typically also something you you need to set. The training algorithm is just an algorithm, right, which gets other input than just the data set in the model. And we will focus on this led up part, which is maybe not super exciting, but that's something we need to struggle with and practice a lot. So what is this thing? And these things are basically the hyperparameters. So we will talk about hyperparameter optimisation. And hyperparameters are basically omnipresent in machine learning. Whatever algorithm you use, you certainly have one or more of them, right? Depending on how complex that is. You could consider hyperparameters, also many other things that are kind of related. For example, just the question of algorithm selection, like do I use a neural network for my data, do I use a random process support vector machine or whatnot? That's kind of a decision you need to make. There would be more classical hyperparameters, let's say, like, for example, for your optimization of your neural network, you would need to set the learning rate, the batch size, the number of epochs. If you want to go a step further, like for the neural network that you want to use, you would also need to define its architecture, right? So you would need to decide how deep is it, how wide is it, and going further, you might even choose what are the layers I'm using. Do I use the convolutional layer here? Do I use max pooling? And then, of course, there's the question, how do I connect them all, right? This entire wiring. So this could be very complex, and this is typically a manual task. So this was mostly about neural networks now, but obviously other learning algorithms have this as well. Like for tabular data, these tree-based methods like random forests or XGBoost are very popular, and here you would need to set the number of trees, the depth of the trees, splitting criterion, and so on. And finally, most often these hyperparameters are also related to model complexity. So you might have some regularization constant. And so how do I know whether a hyperparameter configuration is now a good one? What we would need to do that is just try it. And trying it means I need to train with it. And that's the big problem. Because training is expensive. So just to give you an example, and we will use this as a running example to just show you what the impact of these different algorithms are. Let's say we have a ResNet18 model, that's a convolutional neural network. It's an older one, it's a smaller one, cheaper to train. And let's say we would just want to train that on ImageNet. ImageNet is a popular image classification data set with 1,000 classes, relatively large. Now let's say we just want to use a single GPU, a V100. That currently costs around $3 when we use this EC2 instance. And so a single epoch, meaning a single pass of a data set, takes 16 minutes. So now let's say we want to optimize the hyperparameter's learning rate, weight decay, momentum, and dropout, and we would do a grid search, meaning for each of these hyperparameters we use four configurations and let's say for the weight decay we just used two, just saying we want to use, maybe not the weight decay, but maybe the momentum we say we want to use or we don't want to use it, and then if we take the cross product of all these operations, that will give us basically the grid we want to optimise for, we end up having 128 configurations. If we would train all of them, we would basically spend 128 GPU days on this, and we would pay pay $9,000. And the big problem here is that the money or time you invest for hyperparameter optimisation is something you do upfront, right? This money is gone. And you don't know whether it was worth it, right? We do hyperparameter optimisation for the only reason to improve performance so that in our task, we would, you know, increase revenue in some sense. And so we get that money back. But now we spend it already up front and we have no guarantee whether getting it half a percent accuracy higher would actually mean that in our application we would see that people are spending more money or something on our platform, right? And if you do this in practice, you know better model doesn't really mean it makes more money. So yeah, that's a problem, right? And this is the entire premise in this talk, that small improvements do not necessarily will get us gains in practice. So we are fine with sacrificing a little bit of accuracy, but spending a lot of time and money here. And so we will look into some approximate methods that will make it 20 to 50 times faster. And we, in some cases, we might lose performance, but we are totally fine with that. Okay. So what will be the lessons that I hope you will all learn in this session? So first of all, we will talk a little bit about common hyperparameter optimization methods and how they work. We answer the question what are the advantages, shortcomings, and when to use which. Then of course, that's why I hope you are here. How can I tune these hyperparameters extremely fast? And while these are mostly very theoretically, I also want to give you some pointers how to do them in practice. So how we can use some open source libraries to do it, you know, by yourself without reimplementing the algorithms I will discuss. Okay. So this is basically the outline. So we start with the basics. You might have seen some of them. Maybe you don't have. So I want to lay here the foundation for the entire talk so everyone is on the same page. And then we talk about extremely fast hyperparameter optimization. Then the third one is, I think, particularly interesting. Here we have the scenario that many people face in practice, that you basically have some system where you, over time, basically gather more data and would retune regularly, right? So, and, yeah, finally some conclusions. First, I want to define the problem and the notation. I guess that's the most complicated slide in the entire talk, so bear with me here. We will have some training algorithm, a data set, and what we want to do is we want to get a model which generalizes well, meaning it would perform good on the data we will see over time that our customers will generate, for example. The entire task in hyperparameter optimization is just to find the hyperparameter configuration x in our search space calligraphic x, which achieves this goal. And if we now define basically the generalization score of a model that's trained on D with the configuration x as f of x, we will call this function f also the response function, then the problem itself, like mathematically, looks rather simple. So we want to maximize our generalization score with respect to the hyperparameters. Keep in mind, evaluating F, that's an expensive function. What we would need to do to evaluate is train our model on the data with configuration X and evaluate it on some holdout data sets. This is not differentiable. Great. So I mentioned earlier that one could use grid search. I will tell you now why you should never do that. So that was bad advice. But I think this is oftentimes used in practice. The reason why we don't want to use it, maybe as a background, grid search is this entire thing where you pick for each hyperparameter a few settings, and then you take the cross product so you get this grid of configurations that you want to try. So the problem is most algorithms have different type of hyperparameters. some are supersensitive, and some which aren't. And if we look on the response function on the left side, well I guess that's completely useless. Do you see the pointer at all? Nah. Okay. Anyway, so if you check the plot on the left side, this is basically a 3D plot of some response function that has two hyperparameters. One being sensitive, so you see if we move our configuration at that direction, the response function changes quite a bit. And the other one being insensitive, so it doesn't change a lot. So what would that mean? In grid search, these hyperparameters are always axis aligned, basically meaning that if you now check for one sensitive parameter and different settings for the insensitive parameter, basically the performance doesn't change at all. So we basically waste time in the grid search. And the smarter and basically as complicated to implement solution to this problem is to use a random search. So we just randomly sample configurations. We don't have that problem. We effectively are testing more configurations. Okay, well, let's come to Bayesian optimisation. So grid search and random search are relatively naive. So they do not adapt over time to what they've seen, right? You tested some configurations. Now you wouldn't want to pick the next one, and you would not even consider what you already know about the response function. Bayesian optimization basically tackles this by using two components. One is the surrogate model. That's some probabilistic model that basically tries to approximate the response function based on the observations we have. And then there's the acquisition function, which would basically score each hyperparameter. So it would give each hyperparameter a score based on the surrogate model's belief on how well that is doing, as well as its uncertainty. So that might sound a bit complex, let's get a bit more visual here so I think it's easier to understand. So let's say this orange line, that's the response function we want to maximize. So what we did know is we have already three hyperparameters, we optimized for here. So three different configurations. There's only one hyperparameter in this case. So we get these circles as observations. So what we do is we fit our surrogate model, which is basically predicting as a mean the solid black line, and it gives us uncertainty estimates, which are the dashed gray line. And as you see, in areas where we have data, the uncertainty is low. Where we don't have data, it's high, right? That's what we assume. And then we can compute the acquisition function. The acquisition function basically is a trade-off between exploration and exploitation, considering the mean prediction and down certainty, and basically giving us a score for each hyperparameter. So what you would do is you maximize that acquisition function, and at the maximum, the hyperparameter is the one that we would try next. And then we would update our surrogate model, we would maximize our acquisition function again, and would keep going. In that way, our surrogate model would be better at approximating this response function, but we would also find better hyperparameters. What we've seen so far is random search, spatial optimisation, and the problem with these is that they are very expensive, because both of them require you for each configuration to train until completion. And that's, yeah, well, taking some time. However, these are great solutions if compute time is no concern for you. So, if that's no limitation, just go for these and you can leave the room, I guess. If that's a concern for you, the next part will be a bit more interesting. So, what we will look into is approximate station of these optimizations and so the entire idea will be that instead of training until completion we use some other ways that approximate basically how well a training run would do so we would save you some time but since it's an approximation it won't do as well and this is this entire field of multi-fidelity optimization so here the idea is we would either train on fewer data, or we would train for fewer epochs, meaning fewer passes over the entire data, and if we do that, this would not entirely match what you would get if you train on the full data or for full epochs, but the score we observe here, that is a proxy for the true performance. So we could use this cheaper information to basically tune our hyperparameters. So the advantage here is if we compare to something like random search, if we say we fix the number of configurations, let's say 100, if you do it with some multi-fidelity optimisation technique, you would spend less compute time. If you fix the search budget, in contrary, what that means is you could try in the same time way more configurations. So if time constraint is really, you know, hard, then this will typically find better solution even though it's an approximation. As I mentioned earlier, if compute constraints are no problem, this is just an approximation, so in that case, it will not work as well. And I guess the figure there below shows you kind of the idea of this method. So what it will do is it starts training them, so you see here learning curves, and some runs are basically stopped early if they don't look promising in the beginning. So yeah, let's check the first one. This will be basically the foundation for all the algorithms we discussed. This is successive halving, and it's a relatively simple algorithm. So the idea is that you provide some minimum budget you want to spend on a configuration, So define a maximum budget you want to spend, and what you would do is you sample, like in random search, configurations at random. You train them with the minimum budget, check for the minimum budget, how well they do, and then kick out 50% of the candidates that are doing worst. Then you double the budget, train them, do the same thing again. Kick them out, double them. And of course this whole thing of doubling, halfening, that's just one way of doing it. You can also replace this factor two by some eta to make it a little bit more general. So visually, let's assume we have eight configurations sampled completely and random. We train them for the lowest budget, which is one in this case. We evaluate them, rank them, so one being the best performing one, eight being the worst performing one. In our successive halving is doing, we kick out the 50% which do the worst. Now we double the budget, train them until to the budget two, and just keep doing the same. Kick out half of them, double the budget, kick out half of them, double the budget. And what we would see here is in random search, we would have trained them all until completion. So I guess very clear why we are saving time here. Okay. Hyperband is a variation of Successive Farthing. It basically is an answer to the question, how do I know how to set my minimum budget? It's a relatively simple approach here. What Hyperband basically proposes to do is that instead of running Successive Farthing for one minimum budget, it would just run it in parallel for multiple different settings of the minimum budget, one of them being even the maximum budget. So one of these brackets of parallel runs of successive halving is equivalent to random search. That would give you the same asymptotic guarantees as random search, so there's some nice thing on the theoretical part of this algorithm. Okay, let's come back to our computation. So how far did we come so far? So the example I gave you earlier, that's basically an example for random search. Now we look into how much we would spend with successive halving. The number of configurations we try is exactly the same, but since this algorithm is a little bit smarter about when to stop configurations, we are bringing down the number of epochs by around a factor of 20, so that's also why it's much cheaper, right? And now you can ask yourself whether $400 is better than $9K if you trade off a little bit inaccurate. Okay, great. So this is like the foundation. Most of you might have seen that before. So now we are looking into some algorithm that we've developed. So the first one being Pasha. As mentioned earlier, Hyperbend already alleviates the problem of setting the minimum budget. Pasha is also taking care of the maximum budget. And similar to the minimum budget, we have here two problems. If B is basically too small, the solution we find is not a good one. If it's too large, we simply waste compute. And what Parsha does, it basically automatically chooses the B for you, and in that way will save you some budget. And the entire idea here is that you would like to train configurations as long as they have not converged. As soon as you realize configurations have converged, meaning their ranking doesn't change anymore, that's when we want to stop increasing the budget. Okay, back to the visualization. Same thing, eight configurations, we train them, rank them, kick out half of them, and whenever we go to the next budget and evaluate them, we would need to check whether the ranking has changed compared to the previous run. If that's not the case, we would stop. And in this simple example, we already reached this time and, yeah, we stop. So this is where we save the time here. This, however, is a little bit naive. So this is more the motivation behind it. Because in practice, this ranking will never be stable simply to some inherent noise in the training process. So there will be always a little bit differences in the ranking. So what we propose to do is use some kind of soft ranking. So instead of ranking the configurations, we rank sets. And each set basically contains a configuration or all the configurations that are by at most an epsilon different. And if these rank assignments no longer change, that's the point in time where we stop training. Now we've added this additional hyperparameter, epsilon, but this is something that Pasha would also set automatically in a data-driven way by basically setting it to the 19th percentile of the distances between configurations that are crisscrossing. Okay, so here are some results we got. So we use Pasha in this case for neural architecture search, meaning we would need to decide what layers to use convolutional neural network and how to wire them, so there are various options like convolutional layers with different kernel sizes, different pooling layers, skip connections and so on, and ran it for three image data sets, SIVA 10, 100 and ImageNet. And so this table is a bit big, so just pay attention to the ASHA one, this one here, which is basically successive halving, it's an asynchronous version of that, which would work a little bit better if you run in parallel, and the parser one, obviously. So what we need to pay attention to is the accuracy. So what we would like to see here is that parser basically works equally well to usher or successive halving, right? Because we want to go faster but not sacrifice any accuracy. And this is what we see. Sometimes we are a bit worse, sometimes we are a bit better. And the other interesting part is the speedup factor. So how much faster are we compared to successive halving? And here we would see that it's two times, three times faster. So that's great. However, there are also limitations to Pasha. So I don't want to claim it's working all the time, right? And obviously it only works in cases where you have a lot of budget. So if you just have very few epochs, then, yeah, Pasha can't stop earlier, right? So for that reason, we created this artificial experiment where we decreased the number of epochs from 200, which we had on the previous slide, to 50, and just check out what it's doing in that case. And what we see, while it's still a bit faster, it's not as fast anymore as before. And obviously, setting the budget in this case was set to number of epochs, but you can choose freely, right? You could also use the number of iterations you take. one, you know, budget level is just 50 iteration steps or whatever, right? And then you can artificially blow up your budget again. Great. So, as I promised, I would also show you how to use it in practice. So, this is the first time where we come to this point. So, we integrated Pasha into Scientune. Probably no one of you have ever heard about SineTune, but SineTune is an open source hyperparameter optimization library, which has a lot of algorithms, including all the ones we talk about in this talk. It has a couple of other nice features, for example, it has all these algorithms in their asynchronous and parallel way, which means in practice you can run any algorithm in parallel on multiple machines. And if you're not only caring about increasing accuracy but you also want to do some constrained and multi-objective optimization for example you want to maximize accuracy but also minimize inference time this is also something you could do with that arm library and well since we're working on that and we are AWS it's also well integrated into this entire AWS in the whole AWS products like, for example, SageMaker, so you would not need to write any code here, it would just work out of the box. So as mentioned, it's open source, it's on GitHub, the link is below there if you want to check it out. So how to use Pasha now at SignTune, it's relatively simple, so you first of all would create this Pasha object, right, where you pass the search space, which is called your config space, and then you would define something like, I want to maximise accuracy, and then And your tuner would basically kick off the training process. What you would pass here is your training script, which would, given the hyperparameters, train your model, and then at the end report back the performance you got. And then you can define for how long to work with it, and how many workers are running in parallel. And that's it. That would basically do everything you need. Okay. Let's come to the next interesting use case, I think, which is retuning hyperparameters over and over again, so the next algorithm rush is based on this entire idea of doing transfer learning for hyperparameter optimization. I think it's not super well known, that idea, but the idea here is that we define if a dataset i is similar to a dataset j, then also the response functions are similar, meaning in practice they behave similar in this hyperparameter space, so if the hyperparameter configuration is great on dataset I, it will also be a great one for J. Figuring that out is hard just based on the raw data. Still, what we could do is learn from this kind of information. To make it a bit more visual, what we see here are basically response functions plotted for two hyperparameters of an SVM with an RBF kernel. Each plot represents one dataset, basically, so we have here response functions for a couple of data sets, and just, I mean, you don't need to really understand, just look at them and you will see that some of them have a similar shape, meaning that they are similar in the hyperparameter space, and that kind of indicates that if you would have the knowledge of one of them, that would be very helpful if you would solve the HBO problem for the other one. And this is not limited to hyperparameters for an SVM. This is the same as in algorithm selection, but also in neural architecture search. For example, here we have, and I guess not super surprising, we trained, I think, 200 different convolutional neural network architectures on CIFAR-10 and 100, and just created a scatter plot, and you see they are highly correlated. While I guess for CIFAR-10-100, that's not super surprising. We did this for a couple of other data sets as well. And even though they are not super related, I mean, on the first look, the performance of architectures are still kind of related. So there's useful information there. And yeah, coming from here now to Rush. So the premise in Rush is that we now default some good default settings. And this set we call I. I will talk later about how to get that. And given this set I, what Rush does, it's basically changing successive halving with a very simple rule. We're not only dropping configurations that, you know, the worst 50%, but we're also dropping all the configurations which are not doing as well as configurations in I. So in case of a positive transfer, with positive transfer what I mean with this is that if the data set we're currently optimizing for is similar to where the configurations in I do well on the data set we're currently optimizing for, then rush will basically be super fast because what it would do is many configurations will be immediately stopped. In the case of negative transfer, this is like the worst case scenario where basically the configurations in I are useless. Then what rush would do, it would fall back to a success of halving. So it would not give you a speedup. But the important thing here is that it would also not fail. So you would still have some guarantees that it works. So we come to our visualization again. So it looks similar in the beginning. The difference is now the blue circle. This is basically our configuration from the set I. Again, we would evaluate them, rank them. This is the positive transfer case. So our set i is a good one, so our blue circle gets a good ranking, and what we would now do is drop the 50%, which are the black circles, as before, but we additionally also drop the grey ones, according to the new rule, because they are not as good as the blue one. And then we would just keep going, and again, we would drop the other one, and we're done. So as you see, we drop way more configurations now in the positive transfer case. Now we can also have a look at the negative transfers. In this case, the blue configuration isn't doing that well. In this case, what's basically done is it's dropped immediately. And so everything else that's now coming should be very familiar to you because it's effectively just doing successive halving. So in this case, we are not saving anything, but we would still find solutions which are equally good as in successive halving, obviously. Yeah, let's come back to our running example. So, yeah, I've now added rush here compared to successive halving. This is the case where we assume a positive transfer, meaning, yeah, as I said in the first example. And what we would see here is that since Rush is dropping many configurations, we reduce the number of epochs significantly. It's now more than two times fewer epochs than in successive halving. and so we bring down the cost to less than $200. So what I didn't talk about so far is how do we get this initial set of configurations i and that really depends on what experience you have with the algorithm you want to use. So in many algorithms there are default settings so you could just use the default setting as you set i. If you're familiar with the algorithm you're using on that data set and you've tried it on some other data sets, as I showed you in the transfer learning for HBO, you could just use the best configurations from the other data sets and put them in I. And I think what many people are doing is they just have the same data set which is changing over time. And in that case, what you could simply do is you just choose the best configurations you had on the very same data set in previous update runs. So it's kind of similar to the one with the other tasks, but the difference being that the data is basically, yeah, kind of similar. I mean, there might be some shifts in distribution, but yeah. So the last part is more this continual learning aspect, while the other ones are independent tasks you're solving. And speaking of continual learning, so no, wrong transition, we still have some experiments here, so these are the empirical results for the scenario where we are doing this retuning, and so we took 40 different data sets and basically retuned with Rush and compared it against Bayesian optimization and successive halving. So these plots are a bit more complicated, so I guess, and also very tiny, So I would need to explain so in each plot we basically have different bins or cells and The number within the cell and the color indicate how many data sets fall into that category Okay, and on the x-axis what we have is basically here at zero I'm not sure one can read it that means that there's no difference between rush and successive halving So they're doing equally well if it's to the right side that means rush it's doing better if it's the left side successive halving is doing better and similarly we have the same thing on the y-axis but this is with respect to time so I don't know whether one can see it but the zero is somewhere up there and higher means rush is doing better so what we see in most of these data sets for this rich tuning scenario rush is basically giving us yeah the same accuracy as successive halving but it's two or three times faster and as I mentioned Bayesian optimization can be super competitive if you spend enough budget so this is what we're seeing here some data sets are very far to the left right so these are cases where we have just sufficient tuning budget so this will give you better solutions in that case but we see that most of these data sets are somewhere in this region where you maybe are the same or you you one percent better um and now just check the the the the numbers on the y-axis right so rushes like 30 40 50 times faster and that's that's the massive difference in time and money right and this is what i mentioned earlier you would need to think about whether you really need the best cyber parameters and that Yeah, always depends on how much budget you spend and your data set as well Okay, great, so as I mentioned this this would basically work great in this continual learning setting What Apparently I'm done. Can I wrap up quickly? So we added Rush also to Cintune, but we also added it to Renata. That's some cool open source library we are currently working in. So if you have this continual learning set up, this would be a great tool. I can't go into details, but basically you just need your PyTorch module, easy data access, And then you basically just execute one function, and you indicate you would want to use rush, and it will basically do the update for you. Concluding, either my time is wrong or his time is wrong, but I hope I gave you some introduction to hyperparameter optimisation, some basic techniques. We have seen how multi-fidelity optimization can speed up things significantly. I gave a short intro to transfer learning for HBO, which is actually a pretty cool topic and there are way more interesting algorithms. And yeah, I tried to give you some pointers to how to use it in practice. Thanks a lot for your attention. So I have pointers to the papers here for the Pasha and Rush algorithm. Pasha was just recently accepted at iClear. And also pointers to the libraries, so if you check out the repo, please leave a star. We appreciate that a lot. Okay. Any questions?

Speaker 2 [38:12]

Thank you, Martin. Thank you for the interesting presentation. I see that there are quite a few questions already. We'll try to answer as much as we can. But if not, please know that Martin will be around and you can ask him, you can talk to him personally afterwards. Sorry, Martin, also for hurrying you up, but the schedule is tight. So, yeah. The first question is from Benedict. He says, from your point of view, When would you put trying out parameters over reasoning on parameter values based on their meaning?

Speaker 1 [38:47]

I'm sorry, can you?

Speaker 2 [38:48]

So I mean I guess the question means when it's better to try out parameters instead of Reasoning a bit about the meaning of the each parameter and say maybe it's better to use a different value here and not there

Speaker 1 [39:01]

Okay, I mean, this is hard. This is tricky. I mean, also you're getting back to the point that this becomes a manual process. Sure, you have some control by setting the search space by saying for this hyperparameter, I want it between this and this range. You could say I want to have it on a log scale, for example, in learning, right? You would do it on a log scale, right? But in a way, we don't really want to do it manually, I guess. so I don't know I mean I'm not a big fan of doing a manual because I'm a lazy person but I guess if you have some domain knowledge yeah maybe maybe

Speaker 2 [39:44]

Thank you. Another question is, when using Rush, if your default settings are fairly good, do you not run the danger of getting stuck in the local optimum in which you started?

Speaker 1 [40:00]

There are problems with rush. I would not see the problem that we're getting stuck in optimum because assuming our multi-fidelity scores are very good correlated with the final performance, a better configuration would still do better than the I set in I, right? the problem will be if these intermediate scores are not highly correlated with the end one like in the extreme case they are anti correlated so you would basically drop all the configurations which actually do good if you would train them under completion but that's a problem with all these multi fidelity methods because they assume that there is some stronger correlation between early performances and late performances

Speaker 2 [40:48]

And next question is how would you handle unstable configurations that perform extremely well one out of five times? How do you include stability into the HBO?

Speaker 1 [41:00]

Interesting question. I would argue we don't want those configurations in the first place, but obviously it can happen that we pick one and then we retrain and it's not working anymore. If that's really the case, it's extremely tricky and we would need to do something like cross-validation, which becomes even more expensive. but in practice I yeah I mean at least with the neural networks I've worked with they are rather stable to be fair

Speaker 2 [41:33]

And our question is evaluation is done on test data What can we do to take into account performance on training data to be close to performance and test data to avoid overfitting?

Speaker 1 [41:47]

So, two new hyper-problems in this? No, I mean, I guess here your test data needs to be representative, right? You're basically checking on the validation data generalization. But usually you would still always perform better on tests just by definition. I'm not sure that that's where the question is hinting to.

Speaker 2 [42:16]

thanks regarding multi fidelity how do I know if results in lower fidelities are a good proxy for the performance in the final fidelity

Speaker 1 [42:27]

Good question. You don't. You would hope so. That's the assumption we make in these algorithms. So if you want to be certain about that, just try it with few configurations. If you ran before, for example, a random search or grid search, and you just tracked the entire learning curve, you could check for yourself.

Speaker 2 [42:49]

Thanks. Another question is why do you drop 50% of runs on double the epochs? Aren't those hyperparameters as well?

Speaker 1 [42:58]

They are. And in practice, even though the name is successive halving, people do not halve, but they throw away two-thirds. So this is like the default hyperparameter. But I know it's confusing. But I didn't want to confuse you with that detail. So yes, it's a hyper-hyperparameter. You can set it however you want. You can make it more aggressive, like using one-third or even more. but yeah I mean we don't want to get into hyperparameter optimization of hyperoptimization methods so I hope not at least

Speaker 2 [43:34]

conception another question is actually quite useful it's will you share your slides

Speaker 1 [43:41]

Yeah, I saw in the previous talks that people are doing that and I didn't prepare that on my slide But I should do that. I'm not sure how to communicate the link

Speaker 2 [43:53]

I guess we can talk later and maybe we can put it on this court. Yeah, maybe yeah Okay, and our question is from Flavio What do you think of using mixed precision instead of full precision for hyper parameter optimization to speed up the search and lower the training? cost what Mixed precision

Speaker 1 [44:16]

I'm not sure what it refers to. I know that some people do it with the parameters of a neural network, but I guess for a hyperparameter, it doesn't really make sense. I mean, maybe I'm misunderstanding the question.

Speaker 2 [44:29]

Maybe Flavio, you can come by later and talk directly to Martin. And our question is, when would you use Rush or Pasha? When are they more beneficial?

Speaker 1 [44:45]

I mean, in Rush, you kind of need this extra information of I, right? So if you don't have that, you would not use it. And I think Rush is also more aggressive than Pasha, so meaning it's cheaper, but might also give lesser good hyperparameters. So this, again, is the question of algorithm selection on hyperparameter optimization algorithms. So, yeah, it's tricky. You would need to play a little bit around with that. Yeah, I don't have a super definite answer for that.

Speaker 2 [45:24]

Okay, thank you Martin again