Outgrowing your node? Zero stress scaling with cuPyNumeric.

Speaker 1 [00:07]

Thanks everyone, thanks for the opportunity for me to talk here. My name is Bao Dong, I'm a product manager in NVIDIA in the CUDA team looking at distributed computing. And today I'm going to share something about CUPA and Numeric. Actually this submission was done by my colleagues but they were not able to come. I'm here to present. The good thing is I know the technology and on the other hand, if there are certain deep technology things that I cannot answer, I'll bring it back to our colleague. Thank you so much. Okay, get started. NumPy is very widely used for scientific computing. I know everyone here is for Python, but just show me a hand if you are using NumPy. Wow, almost a full room. Cool. That validates data point here, so it is widely used in Python community. It is used in all kind of things from astronomy to material to signal processing. So it's a very useful tool. And why people use NumPy? I think my reading to it is it's a very convenient library. And also, it's not only the front end. It's also about the back end. Many of the CPU vendors or people are building back end for NumPy. Even though Python itself normally runs on single thread only, but with these backend libraries, these libraries enables NumPy to use the full capability of the CPU, of the GPU. And go beyond just the simple NumPy on the laptop, I believe researchers are going to scale their NumPy problems. And maybe do another quick show hand. So do you use NumPy that has a limitation that makes you think, OK, I need more node, I need more GPUs? Anyone show me the hand? OK, I see around half of the room has such problem. Yes, so there are reasons of scaling the NumPy program. One is for the data size, the problem size trying to solve if the problem size is too big that doesn't fit into one node or one gpu you have to think how to how you can make it multi-node multi-gpu this is about like weak scaling many people talk about this and secondly it's about time limitations like people has to finish certain thing in certain time like if we do a weather forecast and we do the thing Using two days to compute, no use. It's already greening. That is, people need to gather things time faster with more resources. People call it strong scaling. In the real world researches, people use strong scaling together with weak scaling. And yeah, there's a couple of examples of why people are using this scaling Python. Actually, there's more. I don't have to illustrate all of them. I believe everyone knows about it. OK, let's talk about how we scale NumPy or Python program. You can see this code, very common, right? How many people use MPI for Py? OK, I see three, four hands. Not many. I see a gap here, actually. Just now I asked how many people need to scale to multi-node, GPU, half of the room raise their hand and say, how many people use MPI for Pi? It's only like three, four people. Maybe a question. I will leave it later. So this is an example code of how to use QPi. QPi is often used as a NumPy drop-in replacement for GPU-accelerated NumPy. And together is MPI for Pi. MPI4Py is a tool that people use to make multi-rank, talk with each other, exchange data, and do a multi-node, multi-gpu computation. You can see, hey, this is pretty long, right? Got the ranks, and then I got the devices. This is code I send the data to different GPUs. And I do a broadcast to send some data to somewhere. And then I compute. Oh, this is the line for compute. And then I collect all the data, I got the result in rank zero. Pretty complex. What if we do something like this? On the left-hand side, no, on the right-hand side from your view, it's the same thing. What it does is the same as left-hand side, doing a matrix multiplication over multi-GPU and multi-node, multi-GPU. It's just that simple. I know. It's literally not so same like I didn't assign m, k, and n. That is a problem with that code. But even adding that, that is one line. Add together, it's like five line of code. We did the same thing. The difference here is we are using Coupang numeric. That is a tool that we built for a couple of years. And that is really hard to read. It's simple Python code with NumPy. You can see there's no MPI, no partitioning, no communication. Everything the same. And with the command line, actually, you can use Python. And the gate is the command line we have. Just make things easier to input options. We can make it run on multi-node, multi-GPU. Basically, the same code can run from, like, our digits, the left hand one, that's a small thing we built recently, to the, like, promoter, which is a DOE supercomputer. The reason why we do it is we want our researchers to get the science done faster. But today, we see many researchers, they have to write their code to do research, to form the hypothesis, and to validate the hypothesis. And once they get that done on the laptop or workstation, whatever smaller, they want to run it on a supercomputer. Suddenly, they have to find somewhere else to write the code for them to port the code from this simple Python code to the complex Python code. People need to wait, or people have to learn how to use the complex programming skill. What we are trying to do is to get the science done faster. We are trying to make it zero code change scaling, so that you can write your code on a workstation and run it on a supercomputer. It doesn't mean like it's Kubernetes helping you to submit a job or something. Rather, it's how you write the code, what you need to learn for it. How it happens, how it works behind the scene. Just now we said that NumPy, the nice thing of it is not only the front end, but more on the back end, the back end that people build to make it running faster. This is the back end we build for NumPy that make it scale. You can see this is a very simple one. I know the number looks small. Doesn't really matter. Most of the time, we use a much bigger number. And on the right-hand side, you will see how it runs. On the left-hand side, do something very trivial. It's like I create an array, and then I make the array multiply itself, and then create a B that only take half of the array and make some simple computation. And C is another array created. I make C do some computation. And then we create a D, which is the partial B, partial C. And on the right-hand side, you will see, actually, when it runs, this is an example of it runs on four GPUs. The data at the beginning will be automatically sharded into multiple GPUs, and both A and C. And then while doing the computation, things are running in parallel and it is synchronized. And when the computation needed, the data will copy it as needed. For sure, to make it happen, we need a runtime underneath it. This is what we call a legate. This is a runtime that originally from Stanford. This is the task-based HPC. For HPC, people know that use a lot of MPI, those SPMD thing. And for this, Leagate is more task-based. So what people does is to specify tasks. Like this task A, you start with some data and then do some computation and start task B. Depends on some task A and C, so on and so forth. It specifies a computation graph. And then there's different layers, and runtime will pass it and make it running in parallel, a runtime. It's not a JIT. It's not a compiler. It's a runtime that schedule all those things running. And below, the negate is the framework. And then the runtime is the thing that's used for users. And CodePandemic is the first runtime that is Lampy, because so many people are using Lampy. And there are others. Why we write these libraries on the same runtime? The reason is for the composability. If people use the libraries in Python today, people will know that I have a memory, I have a data here. And if I want to use it in another library, they may or may not know each other, like how the data forms, where the data is. Especially when you're running on multi-GPU situations, even, like, harder. People have to copy data wrong. And if they build on the same framework, the data will be composable. The library will know where the data is while you do the computation handled by the framework. That's about the coupon numeric and the technology we built. And there are several users today using that and today i bring one of the example uh slack and slack i don't know like how many people here is doing like high energy physics or material science okay um maybe maybe just a very briefly intro and for people who knows material science if there are question help you to answer that slack is a facility owned by DOE and run by Stanford so it's something that in the Bay Area it's very long to know and what it does is it use laser to inside the material and then use like hand and energy or energy x-ray to look into the material to find new phases of the material or find new science out of it it can be used in many areas like biology and like even semiconductor chemistry so on so forth so to analyze the data from from this facility so what it does is we have this instrument take let me call it a picture maybe not picture it's instrument take the picture of the thing being inspected in the very high frequency they look at the data very quickly and after that they'll try to find something out of the each of the data to say if there's in if there's anything special not not many times special and clean the scientists talk with us and share even some embarrassing situations like they do the experiment and later on they realize that the laser didn't shoot at the material it happens and even even the laser shoot into the material there are things like not every time you'll get the signal right you guys do the science you know that it happens rarely so this is the full 2d director detector this is like 120 1024 by 1024 they gather data like this use your eye where's the signal we cannot find signal and same like the scientist what the scientists use the select facility is they have to write proposals for years to get a 60 hours of time using it's very expensive and then once the the gather gather time they use the 60 hours they put the materials here and shoot the later time and time again and trying to find something and And every time, in the past, they see it just like this. They need scientific instinctions to inspect this screen to see, hey, where there might be a signal. And when they feel there's a signal, they will go there and do some analysis using NumPy to see, hey, I found something. Oh, no, I don't have found something. and most importantly they have to make decisions like tune the parameters tune the instrument a little bit so that they can gather important information and the decision has to be made like around 10 20 minutes that they shoot the thing a couple minutes and do it again and do adjustment in the in the middle and after the 60 hours they will spend years to find insight from what they get and publish with the thing we showed just now like legate and coupon numeric we work with slack we change we change their scientific process on the left hand side is the way that they did in the past like i described this is human in the loop scientist has to inspect those things and find signal and say, hey, here might be something. And then do a plot. And then do some computation. If not, do it again. It's very slow. And also, the signal is weak. People cannot see it easily. And with the scalability, we make things on the right-hand side. As you can see, this is an automatic process. The change here is we don't have people in this loop. After the data is collected, we use the parallel computing in the supercomputers. We partition all the data. We just analyze all the data and plot those results. And we find signals. We tell researchers, hey, here might be signals. And do something here. So it's quite disruptive. It totally changed the scientific process. It's not only about the time, it's also about efficiency and about the cost. Like in the left-hand side, if scientists want to get a signal, then you shoot the laser to the material long enough so that it can see the signal. And since on the right-hand side, we're doing like computation, even if there's a weak signal, we can find it. This is what we do, and with the 1024 by 1024, we partition it into tiles. And the same code, by the way, is the same NMPI code, and the sentence can read. It's not a very complex NPI code. So that they can modify the code on the fly when needed. That is important. like. We do weather forecast. We write a code. We run it for 10 years. It's about research. Research has to modify the code on the fly. And the thing, we run it on Promoter. That's a DOE supercomputer. We also run it on DGX. What's that? Yes, we show it's yield. And The promoter is using A100, and EOS is using H100. The outcome here is, in the past, if people use NumPy to analyze the full data, 10-plus hours, no way they can analyze full data. But for now, with the full data, we've done it in, like, five minutes. Five minutes is not a key point. I talk with people around me, and they say, hey, this is not about speed up. This is about a way people can get insight. This is about people can do something that they cannot do in the past. And that was on LCLS 1, and now we are trying to make it running on LCLS 2, which is much bigger, and take the data in a much higher frequent, and the data size is much bigger. With that, I finish my talk and maybe it's time to take some questions.

Speaker 2 [19:19]

And again, if you have questions, please drop them in slido.com. The first one is, how do QPy and QPynumeric relate?

Speaker 1 [19:31]

It's two different libraries, just the name. You know, the name space is small. Koo is something like a video called everything, like koo this, koo that, koo plus, koo the end. Right? That's koo. And pi is pisar. And then with a small space search, we choose the name. And KooPi, it was actually developed by a Japan company called IPF, if I remember the name correctly and many people think that was from nvidia actually it's not it's another company built that and for sure we work with them we're trying to improve the library for coupai and coupai was designed for single gpu so basically it's a backend for numpy that runs on one gpu and coupon numeric is something like it's sim the numpy front end but the backend we make it run on multi-node, multi-GPU on the runtime. So it's different.

Speaker 2 [20:33]

Thank you. Next question is maybe a bold question. We only have access to a cluster with AMD GPUS. Is there a way to utilize the power of QPY numeric on those?

Speaker 1 [20:50]

Yeah, I got this question in other meetings too. And I actually, it depends on like AMD. So if you look at this stack, the whole stack is all open sourced. Like before March, we have the legal core which is not open sourced yet. And the rest are open sourced. And now everything is Apache to license open sourced. people available in GitHub, people go ahead and modify, clone it, whatsoever. But we have less motivation of making AMD work. But if people are passionate about it, interested in it, feel free to contribute, make it work on AMD.

Speaker 2 [21:41]

Thank you Next question is how those coupon numeric handle situations a GPUs being physically unavailable or congested automatic fallback to CPU only

Speaker 1 [21:56]

So Well, that's a dream of the p.m. You know, I'm a product manager. I can ask them do this do that The reality is like it cannot fall back automatically and to be honest even for today some of the error handler is not ideal in my opinion When the GPU is is conject stated like the memory I taken by someone else Will fail to allocate the memory and a complaint. There's not enough memory But on the other hand, Leagate itself, it runs on CPU and GPU, and you can specify for this task running on CPU or GPU. So it's less ideal, or I would say it's not ideal, but it's possible if people do that. People have to write the code to make it GPU or CPU.

Speaker 2 [22:45]

Can we use it to paralyze it on?

Speaker 1 [22:47]

on

Speaker 2 [22:50]

Paralyze it on CPU if no GPUs are available.

Speaker 1 [22:55]

It runs on both CPU and the GPU, and it runs on multi-CPU as well. It's just like, again, we are NVIDIA. We optimize for NVIDIA GPUs. The CPU works, but how much effort the engineer put on the CPU side is always something that we debate internally. Should we even drop that? I said, no, no. before we want to keep it but it's actively tested but it's it's a lower priority for us for sure

Speaker 2 [23:30]

Another question is coupon numeric on pip Do I need special setup and can I use the same code on my machine before running it on a cluster?

Speaker 1 [23:43]

yeah good question pip install it was released just in April in the past it was only on conda because for some reason there are dependency reason there's people who like conda or hate conda they choose to use conda before and then we make it available on pip in April and now you can use pip install for coupon numeric and then the question to from like your machine to the supercomputer on right now coupon numeric work on Linux and WSL to engineer told me that works on like MacBook great but you have to compile it yourself So forget about it. So yes, laptop, Linux, WSL, and then to the supercomputers like MPI Run, Slurm, those things work. And Kubernetes is working in progress. It's not there yet.

Speaker 2 [24:47]

There's another question on how will Coupon numeric and the legate framework integrate with the recent array api standardization efforts

Speaker 1 [25:02]

So I think when people ask about this that people know that they use different arrays in different libraries and then yes, people are trying to standardize the array and This is something that the team work trying to make the array one to come to to follow the standard and to to interop with other arrays and Here one thing I I want to point out which people may or may not notice For arrays, there's two type of array. One array is like logical array. The other type of array is physical array. Biological array is like you specify array, it's very big. And it can be like here or there. It can be partitioned into different devices. And physical array is like you know this is the array in this GPU, in that GPU, or in that memory. And, in the programming model, there are different ways of doing that. Actually, JAX is more like a logical array. You declare an array, and you use whatever JAX framework to make it distributed. And for KubePyNumeric array, it's a logical array. Basically, it will just tell this is a big array, and then KubePyNumeric will handle it, partition it into different ones. And in each of the tasks, there's a physical array that is in leagate, that is in leagate definition that people can handle it and compute it. So to answer the question, yes, there are works being worked on to make them work together better. Even today, we're trying to use something like CU-PI or number in the individual task. We have to transform the logical array and the local array to some CU-PI array, for example, for today's usage. And the goal is to make it like interrupt streamlining. And on the other hand, always remember, is it a physical array, is it a logical array?

Speaker 2 [27:05]

That was the last question on slido.com. Are there any questions in the room while we still have four minutes? No? I can pass the mic. All right. Well, if there's nothing more, Beau, if there's nothing more you want to share with the group, let's go.

Speaker 1 [27:26]

this one this is the barcode lead to a tutorial we have on github and please

Speaker 2 [27:26]

Oh, yeah.

Speaker 1 [27:38]

give it a try and I have to say it's not perfect and we are getting feedback from people and trying to improve it and yeah that's a belief we believe that's that's the right thing to do is to help researchers and help people don't have to write that long code and let's make it back better.