PyTorch as a scientific computing library: past, present and future

Speaker 1 [00:03]

All right, hey everyone, thanks a lot for the introduction. So yeah, basically, I'm in the original group of authors, although as you can see from the long list, we gathered quite a lot of contributors along the way, which we're very proud of and very happy of. And so today, I would like to present PyTorch to you as kind of a Python library that can be used for kind of next-generation research, both in machine learning and outside of machine learning, because ultimately that's what's hot at the moment, but there are still other domains that people are dealing with. So how many of you actually know PyTorch, have used it, read something about it? Please raise your hand. Okay, so plenty of you do. That's cool. Yeah, so those of you who know it, and if you've heard something about it, you probably know that it's one of the deep learning frameworks. And really, that's how most people see it, and that was ultimately the initial goal of why we developed the library. But during this talk, I wouldn't really like to think about it this way, so there will be some kind of more machine learning specific parts of it. But during this talk, I would really like you to see PyTorch as something more like NumPy. And so probably most of you already know NumPy, just in case those of you who don't. The short introduction is that it's just a Python library that provides you with array types. So array, you have, instead of nesting lists, you have those NP array objects, and they can pack arbitrary Python objects, but if you put numbers in them, they actually won't hold them as Python objects. They will be packed in memory much more efficiently. So you can do quick computations on multidimensional data very easily. So this is an example, we create two arrays in here, we add them together, then you can do indexing, you can select two columns in this example. You can access the shape, and there's also a lot of, there's also a lot of functions for linear algebra, for random sampling, so statistical functions, all this is there, so that's kind of the backbone of, let's say, scientific computing in Python. And ultimately, Torch at its lowest level was exactly the same. So, instead of calling those things arrays, we call them tensors now. But, so as, so yeah, Torch essentially started as like a part of this Lua, Lua Torch into Python. And the Lua Torch is actually a relatively mature package, so like we kind of carried over some kind of like legacy naming of the APIs. We have been changing it slowly and kind of converging towards what the NumPy, what NumPy does, so you can see that it is kind of similar, but there are some things that we think are convenient, so as you can see at the bottom, like the sampling method is slightly shorter in Torch because that's ultimately something you will be doing a lot. Yeah, so that's kind of the backbone of what PyTorch is, but obviously just redoing NumPy would be very silly because it is at least as mature as Torch in Lua, but it has been in Python forever, it is really well integrated, it is well designed, so there is really no point in just creating a complete clone if we won't bring anything new to the table. So now I want to kind of go over multiple features that I think are pretty crucial to what PyTorch does and what is lacking today in more general libraries like NumPy, but I think it's very useful for all kinds of scientific computing needs. So just to start with and to kind of calm you down, it's not like when you download PyTorch you will kind of get sealed off from the rest of the ecosystem. So again, Lua has a pretty much inexistent scientific computing ecosystem, so once we ported a lot of this functionality into Python, we finally wanted to take advantage of all the great packages that are there. And so Torch does support very, very simple NumPy integration, so if you allocate an array, like at the top of this slide, you can just call its NumPy method to get the NumPy array with its contents, and if you want to go in the other direction, you can call Torch from NumPy, or Torch as array, and this will convert a NumPy array into a Torch tensor again with the same data. And so this might seem to be wasteful, because ultimately if this was doing some kind of copies, that would be really expensive. The arrays can be potentially, they can hold millions of elements. That's not something that's unusual today. But ultimately, if you profile this, all those calls are extremely cheap. It's on the order of microseconds, and it doesn't depend on the size of the array at all. And that's because both NumPy and PyTorch, they use pretty much the same representation of the data in memory, and so the actual data stays in the same place in memory. we're just kind of reallocating Python objects to kind of describe how we access this data and work with both APIs. And you can even see this yourself because taking the previous slides where we have X, Y, and Z, which went through conversions both ways, if we add one in place to X and print the NumPy array we got, we will see that the contents of the NumPy array have changed, and similarly, if we just change the array in NumPy and print the TorchTensor again, the contents will be different. So there's a lot of sharing, and this is great because even if you don't, if you have an existing application, and some of the things that I'll be talking about today, you'll find it applicable to what you're doing. You don't really have to take all of your code and suddenly port it to Torch and completely drop whatever you were doing. You can kind of incrementally only apply it in the functions that are relevant, that actually need this functionality, and where it's relevant, and then just have a very cheap bridge to keep the rest of the code working in NumPy. And so now, the first big thing I think that's missing in NumPy today is the accelerator support. So most of you have accelerators with you, GPUs are the simplest examples, but there is also some more specialized hardware that will probably be coming out. In one of the earlier talks, Valerio said that machine learning is basically matrix multiplication plus random sampling, and that's true. And essentially, a lot of those applications and this array-oriented programming kind of paradigm has a lot of implicit parallelism, and GPUs are very parallel machines. They're basically like simpler CPUs, except they have thousands of cores. And so they can do this kind of math very quickly and very efficiently. And so just by porting kind of your programs to run on the GPU instead of the CPU, the array manipulations, if the arrays are very large, you can easily save, let's say, you can easily speed up your program by more than like 20x or even 100x. In some cases, if you have a really good GPU and like really carefully implement this. So just to kind of give you an idea of how this works in Torch, that's a simple program that allocates two arrays, adds them together and prints the result, and that runs on the CPU. That's like the regular stuff. Now, if you wanted to run this on the GPU, that's an example of something you could do. So in the first line, the first line is not strictly necessary, but it is very convenient to have something like this in your script. It will basically try to detect if you have at least a single GPU in the system, and it will try to use it then, and otherwise it will fall back on the CPU. So generally something we try to do with PyTorch is to allow you to write device-independent code. So you can, for example, prototype on your laptop or in some like IPython notebook where you don't really have a fast GPU, you don't want to use it. Laptops heat up really quickly if you start using those, so you don't really want to hack on the train like this. But once you ship this to a bigger server, which potentially has multiple GPUs, you kind of want your script to automatically adapt and start using all this computing power. So basically, to use that, all tensor factories, like the random normal in here, they take the device argument, so this will already allocate the tensor data on that particular device, or if you obtained some result of some computation on the CPU, possibly from NumPy again, you can use the to method to actually ship it, copy the data to a completely unrelated device. And then the API stays the same, all the like cross device synchronization is handled for you exactly in the same, so like our CUDA backend basically supports well over let's say 95% of all the ops that we support, so the coverage is there, we've been working on this for multiple years already. So pretty much any kind of function you write and works on CPU, if you only pass in inputs that are on the GPU, it will like run purely all of the math using the GPU and hopefully will be a lot faster. And something else that's very important and that we like put a lot of emphasis on is very optimized automatic differentiation backend. So AD is especially crucial for machine learning, but it's also used in other domains like engineering, some simulations, finance, physics. And so the most popular use case for us, at least right now, is gradient-based optimization. So something that's really cool about AD is that if you have a function that you want to optimize and it happens to be differentiable, you can kind of use the, so AD basically lets you write out only the actual function that computes something and then you don't need to, you never need to differentiate by hand. It will basically, if you ask for it, it will give you a gradient of any value with respect to any other value in your program. So you can easily just compute the function, ask how the inputs affected the value of the output and so the gradient kind of points you in the direction as if if you moved there locally, that would increase the value of the function the fastest. So if you want to minimize something, you can start taking small steps in the opposite direction and that will hopefully take you to some minima. Of course, it doesn't have great convergence guarantees and you can reach bad minima, but at least in machine learning, at least it's basically dead simple and in machine learning, at least, it does wonders. that's one of the best, very few other methods actually work in machine learning, which is kind of a miracle, but anyway. So in Torch, it is actually very easy to use because pretty much every single array that you have in your program can be a differentiable entity. So AD is disabled by default because it does come with some costs, so it's not something you should be using always. It's not so much a performance problem, as more of a memory pressure problem. And especially if you run this on some kind of an older GPU which has like, let's say one gigabyte or two gigabytes of memory, you kind of start to feel that you will be running out of memory very, very quickly. And the reason for this is that for differentiation we actually need to like kind of stash a lot of the intermediate values that appear in your program, even though like if you were just computing it you could kind of throw them away very quickly. So you have to opt-in to this functionality, and PyTorch implements something called reverse mode automatic differentiation, so basically it is the most efficient if you have a lot of inputs, so in case of machine learning models, you have a lot of parameters, and basically you have a single, your function that you're computing is scalar value, so ultimately, at least in the case of machine learning, you kind of have your model, and then you compute a loss, and the loss is like a single scalar that tells you basically how bad your model is doing and then you're kind of optimizing it to like minimize it which says that you're starting to do better and better so so all of your all of so you will be differentiating kind of the loss with respect to the parameters which are like inputs to your computation and so everything you need to do to enable this is to say that the things you will be differentiating with respect to you need to say that they will require a gradient and that that will like propagate automatically throughout the program for you. So then you can have a function like the poly in here, which is like a very simple polynomial function. And so you can plug in both values that will require gradients and not. This is like completely irrelevant for the program. But this particular function happens to be differentiable, so then in the third line from the bottom, when we evaluate poly of x, this will just give us a new array, and now we can ask Torch what is the actual gradient of the output of this function with respect to the inputs. And so if you print it, you can see that it exactly follows the values you would get from manual differentiation of that function. So I think those are the two most important points that kind of make Torch relevant in use cases that are not extremely specific to machine learning and that you could kind of use in other cases. But there is something big that I also wanted to talk about. Some time ago we've announced that we are actually reaching the 1.0 version. So PyTorch now is really a two year old library. So we gathered already a relatively big community around it which we absolutely love. but we've been trying to really not break people's code, but of course we've been making some changes to the API to actually make it better, but at this point we feel that it's actually better to stabilize and just let people kind of use whatever is there. And actually, at the beginning of October, we've actually, so 1.0 has been announced before, but at the beginning of October, we've actually released pre-built packages that you can try out. But just keep in mind that this is a release candidate and there will come a stable version later. But it's not to say that the release candidate is unstable. It's more like some of the functionality that I'll be talking about next, it should be there mostly, but some pieces might be missing, some pieces might still kind of have some rough edges. So we'll be making it better and easing that out. And once we think it's done, we will release the stable version. So if you write something that works today, it will work in the stable version, but just the development experience and the feature completeness might be better, ultimately, in the final package. And something that's been kind of a big topic about the new release is the research to deployment cycle. So PyTorch ultimately came from the research side. it was mostly targeted towards ML researchers who wanted to implement latest models without a lot of obstacles along the way. But ultimately, a lot of those people actually eventually, once they iterate, once they hack their models, they actually reach some kind of nice state where they work and they actually want to put them into work. They want to expose it as some kind of an API and actually serve it to the world. And it is very important to kind of clarify what I mean by deployment here, because running Python in production is completely fine in some cases. If you think about it, a lot of Instagram and Dropbox, I think, runs on Python, so it is definitely possible, and there are people who are running PyTorch in Python in production, so it is fine, but ultimately there are still some scenarios where you have more restrictive environments and you still want to take your code that you spend so much time prototyping in Python and then package it such that it can run without it. So one example are mobile apps. You don't really want to embed the whole Python interpreter, which is not a very lightweight program, into a mobile app because that will just blow up its size and make it slow. And another case is if you actually have a relatively large scale business and actually have a lot of servers and then all kinds of savings you can do basically translates into huge actual savings in power and money and in the amount of servers that you actually need to run your business. So the NumPy-like model of programming that I talked before, I will be referring to it as eager mode now. So primarily it is really simple to write and debug and we still love it and this will still remain the primary interface pretty much forever, because ultimately what makes Python great is the ability to quickly experiment with your pipelines. You can very easily kind of transform your code to do very different things, and this is very important in this research cycle. But again, unfortunately the interpreter has relatively high needs. It's a very dynamic language, so it is almost impossible to compile, and therefore it is hard to deploy in the sense of deployment that I described earlier. And so something that we kind of introduced in 1.0 is called the script mode. So essentially what TorchScript is, it's kind of a new programming language, but like when I say programming language, I don't really want you to kind of be scared of it because ultimately it uses exactly the same syntax as Python and like if you were any kind of valid TorchScript program is a valid Python program and runs just fine, except that it is a subset of Python, so not all kinds of Python expressions are valid TurdScript expressions, but ultimately this removes enough kind of dynamism from the language such that we can actually do some static analysis on it, and we can rewrite your code to optimize it and make it faster, or we can package it in an independent representation and then later run in more bare metal environments. So just to kind of give you a rough idea of what kind of subset we have in mind in here. Basically, kind of as values that you can be passing around, you obviously have tensors, you have integral and floating point scalars, and you have strings. You have basic control flow constructs, like if, while, and for. You can print values, which is like the simplest debugging tool, I guess. And then as collections, you have tuples and you have lists, possibly nested if you want. And of course, function calls. And so you can see that this is a relatively, You can see that this is a relatively constrained language and will definitely be expanding its scope over time except that it takes really time to develop all those things and make sure that they actually work well. So this is kind of the scope for 1.0. This will get better in the future, but actually at least in the machine learning use cases that we've explored, this subset is actually enough to express 95% of the actual code that people have. So kind of the fundamental building blocks of your programs in Python are there, so you might still need to do some kind of small adjustments to your source, but ultimately it shouldn't become a nightmare to actually program using only those types in the important parts. And so ultimately the question is, how do you take, you did the prototyping, you kind of have your eager program, now how do you actually get to the script subset that you could use to get all the benefits I mentioned. And so to address it, there are two functions that you can use. So the first one is TorchJit trace, and the second one is TorchJit script. So we'll start with trace. So basically how this works is you supply your function, so Torch, we ultimately don't want you to kind of, to implement those in separate files and have like a completely separate import system, like we still want you to kind of feel that you're writing Python. It shouldn't be like a huge semantic, like it shouldn't be a huge difference of how you actually think of your code when you're writing it. We like really like how this feels, so we want to retain this. So Torch to Trace still runs in Python. You just give it the function you kind of want to convert to TorchScript, and you give it an example input. And basically what it does is like it will execute this function once on this particular input and will record every single Torch call that you've made along the way. And it will just put them in a list, which will ultimately become your new program. So the benefit is that it actually doesn't inspect your Python code, so it's not even restricted to this particular subset. But the downside is that it will actually drop all the Python bits from it. So if you have some kind of code to, I don't know, send data over sockets, print, and do stuff like this, this will be completely invisible to this method. So if you try to execute the TorchScript program, it will not do those things. And the other kind of downside is that control flow is actually inline. So if you have conditionals, only the branch, like if you have an if, only the single branch that was taken on this particular input will be seen in the end program. And if you have a loop that executed five times, its body will be repeated five times, which is sometimes fine. but in some cases, it can actually be a problem. If this loop semantically actually is supposed to run a different number of times over the duration of the program, this is not a correct translation. But ultimately, sometimes this is really convenient. If you take a look at one of the most popular computer vision models, this is a picture from the residual networks paper, which are one of the most popular vision models right now. uh basically this is this is kind of the like data flow graph of the program that implements uh a resonance so ultimately what do you care like if you want to actually deploy such a model you ultimately only care about applying like those building blocks uh that that that like constitute the network but you can see that that like the the program that that implements the network is pretty much fixed so even if you like no matter how it is implemented it will ultimately they always compute this single function that is kind of visually represented on this picture. And so trace works just fine in those cases. So obviously if you were to implement something like this, this is actually like a shorter version, this is like a 34 version of this network I think, but ultimately there are like 50 layer or 100 layer versions. And so obviously you wouldn't implement this by like repeating a single application this many times, you would write this out as a loop like this. So those count to D things in here, you can essentially think of them as the boxes in this picture. So that's an example of how you'd implement this. And of course, if you were to trace it, this loop would like disappear. The actual equivalent program you would get from tracing would look like this, which is completely fine in the case of this particular network. Like of course, the list of convolutions, like its length might kind of depend on some kind of command line parameters, your program, like you actually want to pick the number of layers of your network, but when you're running your program which trains the network, the length of this list will stay constant, and so this loop in here is actually only exists for your convenience, so you don't have to repeat it, but semantically, over a single run of the program, it will always look like this. So trace will be a very good choice in here. a very good choice in here. And we actually have a TorchVision package, which implements a lot of data sets and standard models for computer vision. So this is an example of how you can download already pre-trained model on ImageNet. So it has already relatively good weights. You can retrain it, like the last layer to fit your particular problem. And so no matter how this is implemented, we didn't have to change a single thing. if you trace it, you'll ultimately recover this thing and then you can export this and run this, for example, on a mobile phone. And then there is torched script. So this is the thing that actually works in this restricted subset, and this is a function decorator. So you still write your Python program as you would usually do, except you plop some of those annotations on a few function that you actually care about. So it is doing some source code analysis, so it's just reading whatever you wrote, and the control flow will stay there in the program, so it's fine, and the benefit is just like trace was kind of, you have to use your best judgment to determine if the actual transformation is valid. In here, if you use something that we don't support, we'll explicitly tell you that no, sorry, this is a language feature that we can't deal with right now. So this is always safe. And an example where this actually comes useful are recurring neural networks is one thing. So they're basically models that can, for example, translate, they can transform sequences to other sequences so you can translate sentences in one language to sentences in another language. and basically x in here is kind of the sentence encoded as like a sequence of numbers or some kind of embeddings that represent lots of like vectors that represent the meaning of individual words and the actual lengths of the sentence will change at different invocations of this function. So trace wouldn't work very well with this particular function because we have this loop over the length in here, but if you use script, you can see that this function only uses the particular steps that I mentioned, so it is perfectly fine. And of course, this requires some static analysis, so you also need to have type annotations on your functions. If you're using Python 3, you can use the new nice syntax. If for some reason you're still using Python 2, the MyPy version of the comment, so you can put a comment as the first line of your function that will be recognized too. And the most important part of it is that they both mix seamlessly, so if you call a trace thing from a scripted thing, it will get picked up as a proper TorchScript call so it won't go through some external thing. And if you have this big kind of program which actually needs control flow in some kind of small part in the middle of it, you can just implement this middle part using script and then you can still trace it. And once the tracing reaches the scripted part, it will actually like correctly just copy paste the code that the script recovered and like the control flow will stay there so it's really easy to mix them to actually um build up whatever you mean and ultimately both of and ultimately script methods also can call python function so it's not again not like we never kind of want to force you to do those big leaps uh when you're programming we wanted to like incrementally add those annotations in places where it actually makes sense um and uh so so so you can so you really can like do this and see every step along the way that your program still works you can run tests you can verify that you haven't messed anything up uh and ultimately once you even have a program that you've converted like this you can still if you want to go back to like an iteration cycle so you want to like try different things you can just remove some of those annotations and it will still work of course it might like de-optimize some things and if you have python references you can no longer export it to run without python obviously but it will still work in python and something that we have right now is a like c++ api that we've released which basically mimics everything that we did in python you can also use in c++ to kind of for those parts that are bottlenecks for you So once you export this program, this first line basically can import it in C++. Then we allocate, again, random tensor. You can see that the syntax is very similar between C++ and Python, and then you can just compute the value of this particular program exported from Python in C++. And so that was the exportability, but ultimately this also lets us apply some performance optimizations, which would generally be very annoying to implement yourself if you were to restructure your programs. So this is very much work in progress. We mostly kind of focused on the part to actually make this work nicely with Python, give you good error messages, and actually make it robust to whatever you'll be doing. So we didn't spend all that much time actually optimizing those things. So it's more kind of a future direction, but ultimately it is still something that we have in mind somewhere in our issue tracker. So basically, the point in here is that if you're doing something that maps to our built-ins very well, so for example, if you're doing recurrent neural networks, if you just use some of the standard implementations, we can use CUDI-NN implementations. CUDI-NN is the very optimized library for GPUs, for NVIDIA GPUs, essentially, which they have people writing handwritten kernels for this, so it's extremely fast. But once you want to actually change something, so you re-implement the same logic, except in Python, that no longer calls into those C kernels, suddenly your performance drops 5x or 10x. And we really don't want it like this, because that ultimately will limit kind of what kind of experiments you can do because ultimately if you have a choice of like trying out this single simple modification but that would effectively like make your experiments run 10 days instead of a single day, you probably wouldn't like, you would probably stop considering this because that would just like kill your iteration cycle. So we do want to help with that using the JIT so we don't, as I mentioned, this is very much work in progress. We don't promise you wonders right now. But even today, if you take a simple LSTM variant, which is not the standard one, and apply script to it, it will already give you a 2.4x speedup. Just because we both, if you run a scripted function, it actually runs without the Python interpreter. It is a simpler language, so it's generally faster to interpret. And this is a particular example, so of course those numbers might not generalize, but it is kind of a sneak peek, let's say, of what we want to do in the future. And I briefly mentioned this at the end. I kind of want to touch on this because that's also something that I feel is very important. Python is this like great glue that you can use in your programs to kind of compose lower level implementations in certain ways. But ultimately, there are still things that are not exposed in Python or like some things which really require very fast implementations in C++. Like, if you would be doing some kind of research and reinforcement learning, if you would be implementing some kind of game playing agents, most, like, complicated games actually only have C++ APIs, and they don't really have Python interfaces. So it would really be more natural to kind of write those things there. So extensions have been there for some time. Interfaces in beta, that's a new thing in 1.0. I just want to kind of give you a feel of, like, what you can do with this. so the first so the extensions are basically an easy way to like integrate whatever c++ code you have and expose it in python so you can compose it with the rest of the library very easily so this is an example so we do have those like tensor types in c++ which implement essentially exactly the same api that you see in python um but so so in here like there is torch.empty like in python there is torch colon colon empty like in c++ you can like multiply by two in in python in place like this and you can also do this in c++ but also in c++ you can like get the raw data pointers to the actual data of those tensors and they look just like c array and you can for example launch cuda kernels that the like triple angle brackets are the syntax for launching cuda kernels so if you have some custom operators to implement it because you really want to get to go fast uh this is one way of how you would go about integrating them with the rest of your program and so once you have this c++ code that like binds this uh the three lines at the bottom is pretty much everything you need to expose this to python so we use pybind11 which is a very convenient uh c++11 library which essentially like lets you already use a lot of c++ types and easily expose those functions so in here all the like arguments of the functions are tensors but ultimately they can be also integers they can be like floats they can be vectors of those things all of those conversions between python and c++ types are handled for you including tensors and so this basically declares a python module and like gives this function as something that will be exported as an api and so that that's you know that's really easy to write in c++ but ultimately something that's a pain point in C++ is just compiling this and so we have two helpers for this because we know this is a pain and we want to make it easy so if you want to distribute this extension you can like we have some methods to kind of integrate with setup tools to kind of build those extensions as part of the build process of your Python package but if you're just like again if you're just doing kind of some kind of hacky integration you you just have some C++ code you want to really quickly load this in a single project you don't really care about packaging this there is some kind of a like just-in-time compilation thing so you basically say something like torch utils cpp extension load you give us the list of sources that we should compile and the first time you run this this will like compile those files create a python library out of them and import it so it will return the module to you so there's like literally no setup that you have to do. And then the next invocations will basically use the cached build product. So basically in two lines, you can connect, let's say three lines in C++, three lines in Python is everything you need to kind of connect parts of the code base in C++ with Python. And then again, at the very bottom of this slide, you can see the actual invocation of this function from Python. So you just pass in two tensors, and this just works. So those are extensions. Those are stable. You can use them. The interface kind of looks like it is, again, a mirror of our Python API. I don't want to get into details. That's an example of how you would define a neural network in Python. And if you were to do the same in C++, that's how it would look like. So ultimately, you can see that those are pretty much the syntactic differences between languages. but but like the code stays almost the same really and if you went like this is a training loop of a model implementing in python if implemented in python if you were to port this to c++ again it looks like this so very similar and many things are already there this is beta so uh some things might be missing but ultimately you already have some hoppers for building neural network libraries that search and then you have torch optum which is gradient based optimizers uh you have torch data for efficient data loading so essentially this gives you kind of a single like iterator object that like spawns multiple threads and loads like a lot of batches of data in parallel so that like you're kind of the thread that consumes the data can can be in a very busy loop and like always have something to consume uh then there's torch serialized to implement efficient serialization there's torch python for with like utilities for those integration that i mentioned and finally there's searchjet which like can execute church script modules and the last new thing in pytorch 1.0 we did a complete overhaul of the distributed backend so it kind of has a bunch of new abstractions it keeps the same api that was before but it also adds some new arguments and some new functions that like reinforce asynchronous operation so you can overlap many transfers and kind of you know fully utilize your kind of network links you can now create multiple independent groups so you can for example use one back end for gpu to gpu communication and another back end for cpu communication because they might be more efficient or one of them doesn't implement all the operations you need there are some performance improvements that we've apply and something that's really that's really interesting I think the new APIs are structured in such a way such that if you use them you can actually achieve fault tolerance so previously if one machine kind of went down during the training it would it would like shut down all of your processes now you can still recover from this and also this allows you to implement elastic sizing so you can for example use on-spot instances on AWS or any other cloud provider that has this like lower cost of your training you spin up more machines when it's cheaper and you kind of you know shut some of them down when it gets more expensive and finally we are some of you might have heard cafe2 is another framework that is also developed mostly at facebook it's been always kind of geared more towards deployment so they have a lot of those mobile kernels and implementations but we're essentially we're integrating both frameworks so PyTorch will stay as the front end but a lot of the actual execution bits and kernels will be ported from cafe too because it's like multiple years of of work time to optimize them and they work very well so we're making good strides about this this is not really very user visible but this is kind of happening somewhere in the back and it is improving everything we're doing so yeah that's pretty much everything I have PyTorch I just really wanted to stress out that like we have a lot of users from both carp and and contributors from both corporate and academic institutions plus like some independent people uh and like we really love our community we have very active forums where people uh like if you ask some questions that surely someone will like come there and and and help you and um yeah we really like take great pride in our community and we want to grow it uh we've been really amazed but like by like what happened so far so yeah, thanks for coming and I hope to see you somewhere there so thank you for that great talk any questions? thank you for the talk can you maybe briefly elaborate on the automatic differentiation in PyTorch can I do like gradients of gradients and can I do second order methods? Yeah, yeah, you can, you can. So if you can differentiate, like once you compute the gradients, if your function like has well-defined higher order derivatives, this is also available for you if you want to. Like you can, and you can compute like Jacobian vector products, Hessian vector products very efficiently using like this machinery. So yeah. hi thanks a lot for the talk you mentioned several times that you could export your model train to mobile could you elaborate a bit on how you do this and what are the requirements then on the machine on the mobile device so uh ultimately this is not like we're not fully there i think like ultimately we want to make it very easy to like export this and run this on mobile and like some kind of automatic packaging system right now probably your best bet something that you can do with all uh tort script programs you can export them to onyx which is like the standard format for neural networks that like a lot of other libraries can consume including cafe2 so probably the like most recommended path right now is take your tort script program export it to onyx and then load in cafe2 which like has its own infrastructure for running running on mobile uh we are doing as i said we are doing some work to like integrate them better it is not very easy because they're like two very complicated software projects so like bringing them together takes a lot of time uh and work but like ultimately we want to be there and we want to have a very simple like path to actually package those models so right now it's mostly uh although no sorry like if if you're okay with like linking a like shared library into your mobile app Basically, the C++ API I talked about is available as a shared library to download. So you can just put it in your app, and you can load the TorchScript program and run it as I showed you. I think we have time for one more question. One common optimization in C++ is expression templates. Do you implement that already? What do you mean by expression templates? um it's basically fusing the loop of something like a plus b times c and putting that into the inner loop of your evaluation uh yeah so we don't we're not super big on c++ templates because they ultimately like basically the biggest problem with them is that error messages are really hard to interpret so we don't really want to go down this path as you saw like the code the c++ code showed hardly used any templates so we we want to apply those optimizations but we want to apply them to torch script programs where like we can easily manipulate the like representation of the program instead of relying on c++ compiler and some kind of compile time evaluation because this applies like this is something you can benefit from no matter what kind of language you use to actually define the torch script program whereas templates are very c++ specific so So yeah, let's thank the speaker again.