What's inside the box? Building a deep learning framework from scratch.

Speaker 1 [00:07]

A few words about myself. I am Oleg. I am a data scientist. Lately I've been working on several open source projects under the umbrella of BeastByte.ai. And Mauda Desiderio is also one of those projects. But in addition to that, I am also a co-founder of Data for Solutions, which is a company specialized in data science and AI. Our today's tutorial has nothing to do with Dataforce solutions, however I will use an opportunity to just spend 15 seconds explaining what exactly we are doing there. So one of our products is Dataforce Studio, which is a really lightweight ML platform. It is currently in early beta, we have lots of interesting features like training the the model right in your browser using QuibAssembly, LLM features and we are currently working on the deployment module and the best part is that it is going to be fully open source with a release in Q3 2025, therefore if you are interested in ML we would love to talk to you because you can really help us shape the upcoming features so please feel free to contact me after after the tutorial but enough about that let's get back to our today's topic deep learning frameworks and as I said today we are going to build a very simple framework in under two hours actually under 90 minutes and you might be wondering how are we going to achieve that given that usually it takes many weeks, months or even years to build a functional deep learning framework and you would be right. Therefore we would need to make two extremely important simplifications. First of all we are not going to focus on the speed optimizations at all and it turns out is that if we don't care about speed the problem is much much easier but still not easy enough to be solved in 90 minutes, therefore we will also limit the number of the components we are going to build today to an absolute minimum to run the simplest neural networks. But you will get the overall idea about how deep learning frameworks are structured and if you want you will get an opportunity to continue with that on your own. So who is this tutorial for? First of all, it would be interesting for data scientists just to broaden the intuition about the internals of deep learning frameworks, and usually data scientists are already extremely well familiar with the mathematical foundations behind deep learning. At the same time, they have quite a lot of hands-on experience with deep learning frameworks, but more often than not, they don't really have to think that much about how those two things are interrelated. But also, if you're not a data scientist and if you're a Python developer, this tutorial should also be interesting for you just as a soft introduction to deep learning. And maybe let's have a quick poll and figure out who we have in the audience today. So please raise your hand if you're a data scientist. Okay, so maybe 30%, which is good. And usually data science doesn't necessarily imply deep learning, so are you working with deep learning regularly? Please also raise your hands if you do. Okay, not that many, maybe 10%. And is there anyone in the audience who has no experience with deep learning at all? Okay, so also not that many people. We have a nice mix today. Yeah, so anyway, even if you don't have any experience with deep learning, we will cover the basics today. And the first thing we need to figure out what a neural network is. And one could say that a neural network is a parametric function that maps inputs to predictions. However, this definition is extremely abstract and therefore we need something else. neural networks are represented in such graphical way where we have a neurons represented by circles and those neurons are connected by some weighted edges so a neuron is actually just a floating point number and we can have a more concrete example for example we want to predict the price of the house given to features the size of the house in the square meters and the number of bedrooms therefore the size and the number of bedrooms are our features which would be the values of the neurons in the input layer I can actually use the pointer and the and the house price would be the prediction of our neural network but now the question is how can we obtain this prediction from these inputs and again it's rather easy because as you can see all neurons are connected by this edges therefore we can simply calculate the value of the next neuron as a weighted sum of the of the previous neurons so we take this neuron multiply by this weight and at this neuron multiplied by this weight and so on however you might have also noticed that all neurons in a non-input layer have this weird curve, and what does this mean? Usually this is used to denote the nonlinear activation function, so one of such functions can be ReLU, for example, which transforms the values such that the negative values become zero and non-negative values are not touched and we need those activation functions so our network can learn non-linear relationships so therefore our neuron can be in two states pre-activated and activated and also one thing i forgot to mention is this small thing here called bias it's actually not that important what it is or why we need it for the purposes of this tutorial overall it's very important but basically you can think about it as another neuron with a constant value of one and its own weight so now let's try to have a more concrete example to understand how calculations work let's say that we have neurons k and l in our input layer and a neuron m in the hidden layer now in order to obtain this value m we multiply k by wmk at l multiplied by wml and also the bias term and at this point we obtain the pre-activation of m denoted as sm and as i mentioned a neuron can be in two states pre-activated and activated therefore in order to obtain the activation we simply apply some non-linear function on pre-activation now we can repeat the same thing to calculate the value of n or a n and again this is m multiplied by w and m and these neurons multiplied by their own weights when we obtain the activation of the last neuron this is the prediction of our model which is usually denoted by y hat and now we need to figure out how good our prediction is and in order to do that we can compare it with true labels from our data set. For example in the regression setting we can use the mean squared error. Once we have the value of loss our purpose is to adjust the values of all of the weights in our neural network such that this loss is minimized and the question is how can we do that obviously we could do a random search for example to just go over a huge search space of the weight values and see which one works better however i guess you would assume that it's not that optimal, and you would be right. Therefore, we have another way to do it, which is backpropagation. Basically, intuitively, what we can do is to calculate the partial derivatives of loss with respect to all weights in our neural network, which would form a a gradient and the gradient points in the direction of highest ascent of the function and therefore if we adjust the weights in the opposite direction of the gradient we should be able to minimize our loss. I won't really go into too much details how back propagation works but on a very high level we start by calculating the delta error of the last layer and once we have this delta error we calculate the delta error of the previous layer using the delta error of the next layer then we can use those delta errors to calculate the gradients and then we can use the gradients to to update the weights but But we will look into it later. Okay, and another thing that is worth mentioning is that previously, on the previous slides, we dealt with all neurons independently, however, usually we use vectorized operations, therefore our input would be a matrix of size batch by two, where the batch is the number of observations in our input, then the weight would be a matrix of two by three, the hidden layer would be a matrix of batch by three, and so on. So the previous representation was really, really nice to get the basic intuition about the mathematical background of neural networks however if we want to implement it it's much easier to think about it as a directed acyclic graph like this so those two things are exactly equivalent and here we only have a graph that calculates the prediction of the model however we can also extend it a little bit to also calculate the loss function and when we have the the loss function the loss value sorry and when we have the loss value we can invert the direction and find the derivatives derivatives with respect to all of the weights so maybe it will be much easier if we do a very simple exercise let's say we have a graph like this which is used to calculate the value of e where e is equal to b times w multiply b times w multiply by w times a. And what we want to do is to find the value of e and also the derivative of e with respect to w. So in order to make it a little bit easier let's assign some concrete values and b is going to be equal to 2, w is equal to 3 and A is equal to 1. In this case C will be 6, D will be 3 and E is going to be 18, which should be clear. Now we find the gradients and since we only have the multiplication operations this is the only derivation rule that we would need to use. We will go independently in each branch. So dE by dC is 3. In this case, dE by dW, but only in this upper branch, is 3 multiplied by 2, which is 6. now we repeat the same thing in the lower branch, dE by dD is 6 and here we have 6 again, now we have gradients coming from two branches and we need to accumulate them inside our neuron W. So the final answer is E equal to 18 and the gradient of W is 12. And this is roughly how it would look like in some of the deployment frameworks. You are probably familiar with PyTorch, so basically here we just define the values of A, B and W, then we do the forward path calculations then we call backward on e and in the end our grad value should be available in w dot grad and if you do the same thing please don't do it now but trust me you will obtain the exact same value as we did on the previous slide and you don't necessarily have to use spy torch you can use tensorflow to do the same thing it has a slightly different syntax i will not go through that, but again you would obtain the same answer and this is how it can be done in JAX, which is another less commonly known framework from Google. But enough about that, let's proceed to building our own thing. In order to do that we would need some code templates. open this link and download the whole repository. Okay, so while you're downloading, I will also explain the structure. So here you would find a folder called Tutorials. And inside there is only one folder called Foundations. And inside there, there are several important subfolders and subfiles. First of all, there is a template folder, and this will be our workspace for today. Since we don't have that much time, we are not going to type everything from scratch, but there is already a template with a lot of pre-filled things, and we would only need to find the non-implemented functions and implement them. And if at some point of the tutorial you feel a bit lost or something is not working for you, it shouldn't be a problem because there is already a solution subfolder where you can find the solutions. And finally, there are also four Markdown files. These markdown files explain in text more or less what we are going to do today, therefore if you want to repeat this tutorial at home again, you can do that just by following these text tutorials. Okay, maybe raise your hand if you're ready. Not everyone, so let's wait for another two minutes. Thank you. okay another readiness check please raise your hand if you're ready I guess almost everyone. So, let's continue. What do we need today to build a framework or why do we even need a framework? Obviously, we need a framework to define and train deep learning models and usually we say that those models consist of layers and layers might form additional blocks. However, this layer of abstraction is not really sufficient for our needs, it's a bit too high. Why? Because we want to use our framework to also define arbitrary custom layers and obtain the automatic differentiation for those layers. Therefore, we need to go a bit deeper and define two additional layers of abstractions. The first one is functions, so one could also call them operators. And those are the blocks for which we have both the forward and backward functions defined. And going even further, we can say that there are primitives, and primitives are mainly responsible for defining the exact computations on different devices. For example, we would have a set of primitives for doing the computations on the CPU, a set of primitives of doing the computations on the GPU, and usually the functions consist of those primitives under the hood. However, as I said, today we are not really interested in speed optimizations, and if we are not interested in speed optimizations, we are not interested in GPU computations, and in this case we can forget about the primitives for now and proceed to the functions directly, and we will be using NumPy as our computational backend. So as I said, a function or an operator is, let's say, a class for which we have the forward and backward functions. The forward function should take one of several inputs and calculate the output. And the backward function should take the incoming gradient and then calculate the gradients of all of the inputs. And now we can see how it works in the code. If you open the ops folder and then base, you would see the base implementation of the operator. As I said, it has the forward and backward function that need to be implemented by the concrete operators. But here we also have two utility subclasses, a differentiable operator and a non-differentiable operator. And let's start with a non-differentiable operator. Sometimes we need some computations for which we are not able to define the backward pass, and this is totally okay, but we need to make sure that we never try to calculate the gradients flowing through such operators. therefore here we just raise a runtime error if the backward is called during the runtime and then we have the differentiable operator and in the differentiable operator we have a method called maybe the broadcast rut which is rather long and in order to understand why we need it here let's talk a little little bit about broadcasting. So usually in deep learning we work with tensors and I know physicists don't really like it, but computer scientists say that tensors are just multi-dimensional arrays, which is probably true for our needs, and a tensor could be of different ranks, so rank 0 tensor is a scalar, rank 1 tensor is a vector, rank 2 tensor is a matrix, and so on. And sometimes we might have a situation where we want to have a binary element-wise operation over two tensors of mismatching shapes or even ranks. For example here, how can we can we calculate that? And an answer is that we need to broadcast our tensors to the same shape. But first we need to figure out if the shapes are even broadcastable. And in order to do that, first we are prepending the additional dimensions to the tensor of a lower rank until the number of dimensions match. So So now we have the same number of dimensions, however the dimensions themselves are not exactly equal. Therefore we need to do an additional step and check each dimension one by one. And here the tensors would be broadcastable if each dimension is rather equivalent or equal or one of them is equal to one so for example here we have two and two they're equal it's okay here we have three and one which is not okay because they're not equal but at the same time this is equal to one so it is still okay and now what we are doing is we are repeating this dimension and finally we obtained the tensors of of the same shape and if we try to do this in numpy we would obtain the exact same operation without even have to think about broadcasting and that's because numpy is doing broadcasting out of the box which is great however now we have a small problem because during the backward pass we need the gradient to match the dimension of the input and it won't be matching. Therefore what we need to do is to kind of reverse that broadcasting operation and how are we doing that is that first we are summing up over the repeated dimension and then we remove the dimensions that we added and this is exactly what is implemented here. I don't really think it's a good idea to spend too much time here, but just you need to know that we need to have it. Okay, but now let's try to implement our first operator, which is going to be the multiplication. For the multiplication we would need to implement both forward and backward pass and I will open the solution right away also probably it's better that you copy the code from the solutions instead of typing it manually because that would be much faster and also less error prone okay so the forward pass of the multiplication is very very simple we just need to multiply two elements which is here however we also have to save our inputs in the state of our operator and you might ask why because we would need those inputs again during the backward computation. So here the gradient of A is B, however we also need to multiply it by the incoming gradient to respect the chain rule maybe. So here we have the incoming gradient we have to multiply by it and also we need to maybe the broadcast to the original shape of A. And we also need to do the same to calculate the gradient of B, and we return a tuple of two gradients matching our two inputs. And also I forgot to say that this is supposed to be an interactive tutorial, so if you have any questions right away you can ask them. If something doesn't work you can also raise your hand probably i won't be able to move to your place but we will try to fix it Okay, so the template and the solution have the exact same structure. We are now in the mull.py, therefore I also copied the mull.py from the solutions folder. maybe I'll show it here so we have the template and the solution we are working in the template and if you go in the solution you would find the exact same structure and we are in firefly ops mole dot PI Thank you. Okay, so meanwhile I will say a few words about other operators. They are already implemented, for example, division, multiplication and subtraction are also element-wise binary operators. Therefore, they have a very similar structure to the multiplication operator we just implemented. Then we have matrix multiplication. The forward path of the matrix multiplication is exactly the same as in the multiplication. the backward path might seem a little bit more difficult however in practice it's almost the same thing because the gradient of A is still B multiplied by the incoming gradient and the gradient of B is A multiplied by the incoming gradient and therefore it's actually not the different but overall matrix multiplication is probably the most important operator since most of the things we're doing in deep learning are matrix multiplications under the hood. Then we would need a ReLU activation function which I already mentioned in the introduction slides. ReLU just sets the negative values to zero so it's a very simple computation we calculate the mask and multiply the input by the mask and for the backward calculation we just mask out our gradients and the final operator we would need is the summation operator and the summation operator is is also a so-called reduction operator, which means that when we apply it, we reduce some of the dimensions. And those dimensions are specified by the additional axis keyword. And here we have an additional Boolean kip dims argument, which can be, obviously, either true or false. And it does the following thing. For example, we have a matrix like this and we sum it over the first dimension and obtain a vector with values 9 and 12. So if we set keep dimensions to false, then the first dimension is completely gone. However, if we set it to true, we obtain the same result. However, the first dimension is still here. Okay, so is there anyone who had problems with the multiplication implementation? Yeah, what was the problem? Okay, so maybe I'll reiterate. We have the base classes for our operators, and now we wanted to implement some of the operators that we would need to construct the neural networks. Most of the operators are already pre-implemented. The only one we needed to do is the multiplication operator, and for the multiplication we needed the forward and backward implementations. the forward is just first input multiplied by the second one and in the backward we calculate the gradients of the inputs and for example here the grade A is B multiplied by the incoming gradient and they broadcast it to the original shape of A so here we calculate the gradients and return them as a tuple yes This one, okay, so basically to keep it consistent among the implementations, among different operators, we can have unary operators, we can have binary operators, and obviously they have different number of inputs, therefore here we just say that it's a sequence of tuples where each tuple is an umpire array. And then we unpack it basically here. Okay, so now we already have all of the operators that we would need for today. We have both forward and backward passes for those operators. However, we still don't have a nice abstraction to use those operators therefore the next thing we are going to implement is the tensor class and the tensor class is just a container for our data which stores the data itself, so the buffer, then it stores the additional boolean value called requiresGrad, which defines whether we would need to calculate the gradients for this tensor during the backward pass. We also have the storeGradNonLeave, but probably I won't explain it today, it's not that important. But we also store the operator that created our tensor and the parents that we used as input into that operator. So again, if we think about this as our operator and we are working with the tensor C, then we will store the operator itself and the reference references to these parents a and b okay so now in our tensor class we have a bunch of methods for applying the operators and as you can see all of them are rather similar. We just call the applyOp function and pass the class of the operator we want to apply and the inputs. However we don't really have the implementation of applyOp yet. It's here at the very bottom of the file and we need to do that now. So again this applyOp function receives the class of the operator, the arguments which are the tensors that need to be used as the inputs and also the additional keyword arguments if our operator needs those. And we can see how it works here, so first we instantiate the instance of our operator, then we apply the forward pass, and here is an important thing, our operators operate on the buffers, they have no idea about tensors. Therefore, we have to pass not the tensors themselves, but it's extract the row buffers from the tensors. And optionally, some keyword arguments. In the end, we receive the buffer. However, we need to obtain the tensor. Therefore, we wrap our buffer into the tensor class. And here we need to figure out whether we need the gradient for that tensor. And we need the gradient under two circumstances, which should be true at the same time. First of all, our operator should be differentiable. Because if our operator wasn't differentiable, even if we needed the gradients here, we wouldn't be able to calculate them. And, in addition to that, we need to check whether any of the parents require the gradients. So if our operator was differentiable and at least one of the parents required the gradients, then this new tensor would also require the gradient. And now, if the gradient is required for this tensor, we need to store the operator that created this tensor because at some point we would have to call backward on that operator and also we need to store the parents but this I will explain later and we return the new tensor. Does anyone have any questions about this function? Okay, and now we can just fill in the mole method in our tensor, And for that, we can use the other methods as an example. We just need to return, apply op, and pass our mol operator, the first tensor, and the second tensor. Okay, so at this point we have everything that is needed to perform the forward path and let's try to implement our exercise from the slides. I will create an x.py file and here I will create the tensors A, which was 1, B, which was 2, and W, which was 3. Okay, so for those who don't remember the exercise, we need to multiply B by W, A by W, and then multiply the results among each other. So it's going to be C equals W multiplied by B, D is equal to A multiplied by W, and E is equal to C by D. and now if I run it we have a tensor of shape 1 equal to 18 which is correct however we still have a small issue that we not only wanted to find the value of E but also we wanted to find the gradient of W and if you say W requires grad equals true and then we call backward on E and try to print W dot grad then obviously we will have an error because our backward method is not implemented and we need to do that now so now for the backward method implementation I will first copy the solution and then try to explain what's going on. Basically it's a rather long method, however the core computations are here in only four lines. Let's assume that currently we are in tensor C and we need to calculate the gradients of A A and B. Again, a reminder that we have a reference to the operator instance, which has both forward and backward implemented, and we also have the references to A and B, which are our cell dot parents. So what we are doing now is gradient of inputs are just self-created backward. So we are just calling this. And this will return directly the gradients of the inputs, the A, the B. But now we need to pass those outputs to these tensors A and B. And for that we simply go through inputs and their respective gradients which we just calculated and recursively call backward on the inputs and pass those gradients. Now as follows from this, the tensors themselves are responsible for accumulating their own gradients. Therefore if we move a bit to the top we have these several lines where we just set self.grad equal to grad output and grad output is just an additional argument to our backward method. And this would work in some cases however as we remember from our exercise sometimes the same tensors participate in the computations several times therefore when we do a backward pass we will visit those tensors several times as well which is a problem because we don't want to just override the gradients but we want to accumulate or sum them. Therefore if our tensor already has a gradient value we don't overwrite it but we add it. And finally we have this small code block which just says okay if we don't have the incoming gradient that must be the situation here at the very beginning so we can just initialize it with one so this is the backward pass and now if we go back to our exercise and try to run it again we can see that our E is a tensor equal to 18 and the gradient of W is a tensor equal to 12, which is exactly what we calculated earlier during the exercise. Can you show the example again? The source file? The source file, yes. Can you maybe explain when the output is ever called? Sorry, I didn't hear. What is it called? The parameter for the output. Okay, grid output, yes. Sorry, I didn't see it. Okay, so who got this result? Okay, five people. Does anyone have any problems, something doesn't work? Okay. So let's maybe slowly move on. Later on, there won't be that much code anymore, so people who are a bit behind will get a chance to catch up. Basically, right now, we already have our tensor implementation, which implements the operator, so we have a nice API to work with the operators, and we have a basic autograd engine. And believe it or not, this is already almost enough to build rather complex models. For example, if we were to implement a couple of additional operators, we could already implement GPT-2, for example. So that's really the core of deep learning. However, at the same time, it's not really convenient to work with right now because we still don't have the high-level APIs. And if you remember our pyramids, for now we were mostly focused on the operators, but we also want to have layers, blocks, and models. And for that, we will now implement a module class, and this module class, similarly like it works in PyTorch, would act as either layer, block, or model at the same time. So if we go to nn submodule and open module.py, we can see our module. an important thing is that we have the forward method which defines the forward flow through the model and we don't have to implement it here but rather when we create a subclass of module we need to implement the forward pass and actually that would already be it about module if they weren't for one thing. As you remember, the whole idea of neural networks training is adjusting the weights in a way that the loss is minimized. Therefore, we need to be able to find or keep track of all of the weights we want to optimize. and right now we would have to do it manually which might sound like not a difficult thing because usually a weight would be just a property in our module however the modules can be deeply nested for example imagine a situation when we have a module inside a module inside a module that module has another list of modules and those modules all have list of weights and we need to manually go through this highly nested structure and find all of the weights. This would be really suboptimal, especially given that sometimes we don't really know the internal structure of the module we're working with. For example, it's a pre-built module from some library and for us it's really kind of a black box. Therefore, instead we want our framework to automatically keep track of the of the trainable parameters. But before implementing that let's have a couple of words about the parameters itself. We have a file called parameter.py where we have the parameter class. Again this is just another container that stores some data. It has a property trainable which defines whether this parameter should be optimized during the backward pass, also the parameter can be initialized directly by a tensor or by passing just the shape and if initialized the parameter with the shape then the values are initialized randomly. You might have noticed that here we have this function called Kiemann cuneiform, which does the initial random initialization. This initialization strategy or scheme is obviously not the only one that can be used, and there is a whole theory behind that explains when you should use which initialization techniques. However, today, for the sake of simplicity, we will not go through that and just use Kiemann uniform everywhere and and just treat it as some some black box thing. So now here we also have the update function that updates our parameter with a new tensor and that's more or less it. Now we can use this parameter inside of our module. So, first of all, we would want to implement the get parameter method, and when we call this get parameter method, we want to obtain the list of all of the trainable parameters that are stored directly by our module or by any of the nested modules, for example, children modules, by brain children and so on. So for that we actually need to override the set attribute method first. And the idea is as follows, inside of our module we have two additional dictionaries, parameters and modules. And And every time when we assign the attribute with a value that is either a parameter, a list of parameters, so a tuple of parameters, we store this in our parameters dictionary. And we do the same with modules to keep track of sub-modules. Okay, so. Here, again, it's better to just copy it from the solutions because the method itself is rather long, but it's really, really simple because when we have a value, we just check the type of the value. If it's a module parameter, list of modules, list of parameters, and so on, we store it in one of those respective dictionaries. And now when we have the set attribute method, we will have our dictionaries populated automatically during the runtime and therefore when we call get parameters we don't have to look everywhere, we just have to look in those two dictionaries. So first of all we need to collect all of the parameters from the parameters dictionary and when we have them we need to go through the nested submodules and recursively call get parameter on each of those modules. So again, since we don't only have the parameters or modules directly, but we can have sequences of parameters or modules, the get parameters method is again a little bit long, but it's It's also very, very, very simple. Okay, and now to get more feeling about the usage of this module class, let's implement one module which will be the linear layer we will need a little bit later. So going back to our slides, the linear layer is this one, it's just a dense layer that does a single matrix multiplication to map the activations of the previous neuron to the reactivations of the following one. So here we would need to set two parameters. The first one is the weight matrix, which has the shape of input features by output features. Again, for example, here it was 2 by 3. And then optionally, we have this bias term, which I mentioned earlier. And intuitively, we can say that a bias term is attached to every neuron in the hidden layer. Therefore, the bias is a vector of the shape out features. and the forward path of the linear layer is again very very simple is it is just the input multiplied by the weights and if we have a bias term we additionally add the bias term to the output Okay, so now let's go back to our slides and try to implement and train a neural network like this. So the training loop usually looks the following way. calculate the predictions of our model, then we define some loss function and calculate the loss based on the predictions and true labels, then we call backward on the loss to calculate all of the necessary gradients, and then we call step on the optimizer in order to update the weights. However, we haven't really talk about optimizers and we don't have them implemented. So what is the optimizer? Basically the optimizer can be seen as a function for example that calculates the new values of the weights based on the old values of the weights, the gradients of the weights and optionally some internal state of the optimizer. In the code this base optimizer looks like this. We simply initialize it with a list of the parameters that we want to optimize. It has a zero grad method which goes over the parameters and resets the gradients because during each optimization iteration we kind of want to start from, not from scratch, but we don't want to reuse the gradients. And then we have the step, which is responsible for updating the weights, but this step is implementation specific, therefore it's not implemented in the base class, but in the concrete optimizer. And today we will implement the simplest possible optimizer, which is the stochastic gradient descent without momentum. And the update formula is really simple. We are actually already familiar with it from one of the first slides. the new value of the weight is the old value of the weight minus learning rate times the gradient of the weight and here the learning rate is just a hyperparameter which is a small floating point number Okay, so now we finally have everything and let's try to implement the model and the training For this, we will go into the train.py file, and we have the module called MLP. MLP stands for multilayer perceptron, and this is a model like this, which has several hidden layers, at least one. So now we had two layers, therefore we need to define those layers. Okay, so our first layer will map the data from the input size to the hidden size, for for example here from 2 to 3, and then the second layer will map the hidden space into the output space, for example here it was 3 to 1. In the forward pass, first we calculate the pre-activations of the hidden layer, Then we need to find the activations by applying the ReLU activation function. And then we calculate our predictions by applying the second fully connected layer. And we return our prediction. Okay, so now we have a data block from, a code block from line 21 to line 32 that can be completely ignored because the only thing it does is generates some synthetic data which we'll be using for training, but overall we are generating 1024 samples with 10 features. Then, as the next step, we would need to define our loss function. As a reminder, the loss function evaluates how good our predictions are by comparing them with the real values from the training set. And here we will be using the mean squared error. first we find the difference between the predictions and the true values then we need to square this difference however we don't have the square operator implemented anywhere luckily we still have the multiplication so we simply multiply the loss by itself and then we need to find the average but again we don't have the mean operator so what we'll do is simply first do the summation that we have and then we will divide it by the number of samples which we simply define here outside of our loss function. Now we need to initialize our optimizer again we are using the stochastic gradient descent and we need to initialize it with all of the parameters we want to optimize and how How do we find those parameters? We simply call model, get parameters, and discard non-trainable is already set to true. And as an additional argument, we can pass the learning rate, which can set to 0.01. Then we need to implement our training loop and we will be doing training for 10 epochs. Usually an epoch is not the same as an iteration because the number of epochs is the number of times the model sees the whole training data therefore if we define our training data into mini batches then the total number of iterations will be number of epochs multiplied by the number of batches however here again for the sake of simplicity we don't do mini batch training we do full batch training therefore number of epochs would be equal to the number of iterations. And at the start of each iteration we need to call zero grad on the optimizer which will go through all of the parameters and reset the gradients if any are set there. Then we calculate the predictions of our model. When we have the predictions we can calculate the loss between the predictions and the true values, then we call backward on the loss and step on the optimizer and here just for login purposes we can add an additional print statement so now that we have our training loop we can try to run it and we can see that our training loss decreases which is a good indication that some training is going on importantly we shouldn't use the train loss to define the performance of our model or assess the performance of our model usually for that we will use an additional validation or training set which is not used for the training however today we are not going to do that since we were just interested to see that the training loss decreases and our optimization must be working. So is there anyone who was able to obtain this? Okay, so does anyone have any problems at the moment? Yeah, so So what's your problem? I'm sorry, I didn't really hear that. So even my Python was showing this error before, as you were explaining, it says linear object is not callable, and now it's just showing it here, yeah, I just saw the solutions, it's also the same there, if I'm not mistaken, so line 13, okay, line 13, right? So it should be line 13, so it should be there. I mean, I can copy it and paste it to compare it. But can you show me your module, the parent class? This? No, no, no, this is the optimizer. The module is here. Sure. Yeah, but you have the code method defined, so it should be. Ah, I forgot to implement it, right? No, no, you did implement it. If you go to the linear, we did implement the forward, so I'm not really sure why. which part you want to see in linear in layers I mean the solutions I'm sorry why does it complain here can you open it just says it doesn't find it somehow I already installed it already many times maybe I can do it again Okay, maybe you should reopen PyCharm such that you are at the template as your root, so it resolves the imports correctly, because maybe when you just hit run it doesn't resolve the imports properly. So, I'm just going to go here. Okay, now I'm here right now. So, I have to start in the training dot file, right? Same error. Where are we right now, template, so if we just, it's German, Python, Linux arguments. But here you don't have the linear layer implemented, so you need to just... I have to update this and then I'll get it. I could just update it right now. Yeah, so if you just copy and paste it from here... That's why it was complaining that it didn't find it, right? Yeah, I guess. What's that? Okay, so basically we trained, it's not live yet, but we trained our neural network. Hopefully it worked for some of you. If it didn't work for someone, I remind that you can go into the GitHub repository and open the tutorial. And you will find the markdown files that explain what we did here today. So you will have a chance to try it on your own. And I can see that we don't have that much time left. So I will slowly wrap it up. And before concluding, I will say a couple of additional words about the primitives. We already established that we are not going to do anything about the primitives today, because we don't care about the optimization. However, at the same time, how could we do that? Basically, right now our operator directly calls the computational backend, and in case if we wanted to have several computational backends for CPU, CUDA, and so on, we could implement a sort of backend dispatcher, for example, and this backend dispatcher will forward the computations for the respective backend and each backend would implement a rather small set of the primitives. So, for example, here we are using NumPy as our computational backend and we are calling NumPy functions or methods directly from from our operator and if we refactor it a little bit we would have some backend dispatcher and we would only call the multiplication of the backend dispatcher and the backend dispatcher itself would determine which backend needs to be used by the type of the buffer that is being passed and call the multiplication on the respective backend and if you want to try it out this is actually implemented in the repository but not in the tutorials but in the in the main in the main folder so to say and from my side that was it thank you you can add me on LinkedIn and now I'm ready to answer your questions I don't understand where we find that the media needed to be going to DAO. Sorry I didn't hear that. Oh, actually we have a microphone for the Q&A. I forgot about that. don't have a session chair for some reason so therefore I'm curious if you can explain where in the stochastic gradient descent optimizer we're defining that we're moving in the correct direction if that makes sense so again for example like with a learning rate anything higher than or like with anything higher, we're moving in the completely wrong direction, so I'm curious where we were defining the direction to be moving. Okay, so learn and create doesn't really define the direction, it only defines the magnitude of our step, and the direction is defined by the gradients, because what we really need to do assume this is our loss function and the gradient would point in the direction of ascent therefore if we go in the opposite direction we are minimizing the loss and the learning crate just defines how large this step is going to be so does this answer your question or not really so if our point if the red point was on the other side of the further along the x-axis here for example yes where are we defining that we need to move in the left direction okay again this is defined by the gradient because it doesn't really matter where we are, the gradient would point into the direction the function ascends. And we just need to go in the opposite direction. And we define it here because we go in the opposite direction of the gradient. Right, okay. Okay, any other questions? Question? Any other question? So let's thank again for this tutorial.