Demystifying Parallel Programming in Python: from CPU to quantum processors, including GPU and TPU

Speaker 1 [00:00]

see the three that were all seated. Just like for the other talks, if you have any questions for the Q&A, please use the talks of PyConD to ask the questions so that we have them for the Q&A at the end. And then I will introduce you to our next speaker, Gael Pegliasco, with a talk about demystifying parallel programming in Python from CPU to quantum processors, including GPU and TPU. Please welcome him with a warm applause. The stage is yours. Thank you very much. Do you hear me well? Perfect. So demystifying parallel programming in Python, this is our talk. But first of all, to introduce myself quickly, my name My name is Gaël Pégliesco. I'm French, as you can hear it. I've been developing from Python since more than 20 years. Actually, I'm doing a lot of computer science and parallel programming on supercomputers. Most of the time, I'm a Python trainer. This is my most important work. I've got a very fluent French glitch, as you can hear. In this talk, we are going to speak about parallel computing. And for this, we are going to take the time to present the hardware basics. This is very important when you do parallel computing. You need to understand how works your hardware, to understand why you get the performance you were expecting, or why you don't get these performances, and how you can change that. So we are going to have a look on processors first, then on playing with the processors. Then we'll have a look at the landscape of Python parallelization libraries, looking at concurrent programming with code examples. And finally we'll try to speak about the future of quantum processors, photonic processors and a few others. So starting with processors. If you want to perform parallel programming, you first need at least a computer. I've got one here, that's perfect. And inside the computer, at least multiple processors to run one program by processors for the old processors, and with the next generation of processors, a processor with at least multi-cores. Of course, there is a compute unit inside the processor. We are going to explain this in a few seconds. But which kind of processor can we use with Python? CPU, the processor running your operating system, GPU, your graphic card, which is used for machine learning also, TPU for machine learning, or quantum processors are also there. When speaking about processors, we usually focus on CPUs, the central processing unit that is running your personal computer operating system. But you also have GPUs, the processor running your graphics card, used for machine learning too. But there are other types of processors that also exist, like GPU, tensor processors, we are going to explain them later, NPU, quantum processors, photonics processors, and a few others. So let's start by the most common, the central processing unit. This is the processor inside your laptop that runs your operating system, your Windows operating system, your Mac, Linux, or other operating system. And there are many manufacturers for that, this is not a question of the point here so I won't present them. To use your processor efficiently, you need to understand the internal architecture and its behavior. So three concepts are very important to understand when you do parallel programming in your processor. The first one is the processor frequency. The second one are the cores inside the processors that can run your different program or function or different part of code, and the hyper-threading technology attached to them, and the processor cache and memory. And in Python, even in Python, which is high-level language, all these processor properties will have impact on your code. Fortunately, to discover this functionality, we have on Linus, for example, some useful commands to inspect the hardware. Let's start with ls cpu and sw lock minus ls. So I take my terminal, maximize it and enter the command ls cpu. This command will display your processor Loads of data are presented. We can see the brand of the processor, this is an Intel i9. We can see the number of sockets, do you know what is a socket? The socket is the place on the motherboard where you put your processor. Here I've got one socket, it means I've got only one processor in my computer. The processor owns 14 cores, it means I can run 14 different programs at the same time. And there are threads inside the core. I will present threads a bit later. We can see also the processor frequency. Also by raising the hand, do you know what is clearly the frequency of the processor? A few people, yes. The frequency is most of the time explained as the speed of the processor. This is not very accurate, but it is a good idea. In fact, you can think about the frequency as the number of operations your processor can execute per second. So it is expressed in megahertz, so this one can run from 400 megahertz up to 5000 gigahertz. So it can execute 5 billion instructions per second, which is quite a lot. This is not completely accurate. This is not in fact an instruction, it is about cycles, but we will speak about that a bit later. And then regarding the processor, we also have internal memory. This is a text description of the processor, not so easy to read, so we can focus on the graphical map of your processors that can be displayed by hwlog-lscommand. I have to be rude to execute it. I've got a beautiful picture of my processor here. Do you read it? I don't know if the back end is okay. Let me try to explain. We can see here the different cores of the processors. Each core is a compute unit in the processors that can run a different program or a different Python function. Some cores have what is called two hyperthreads there and no hyperthread there. The hyperthread has the data that we were seeing before with the two threads per core. Processor also get an internal memory that is cache memory. When you work with your processor, your processor doesn't directly work with the outside RAM that is on the motherboard. It takes the data inside the motherboard, brings it back into its internal cache and starts working on it. This has been created before this century, processors did not add cache. They were working with external memory, and the external memory was as fast as the processor to update data. But the processor has grown in frequency and memory has grown in size despite the frequency. So now the external memory is too slow for the processor. this load on your processor. So to prevent from this, your processor owns an internal memory, very slow memory, you see it's 24 megabytes for the biggest one and one megabyte here, 48 kilobytes is very very little. But when your data are stored inside this internal cache, your processor can run at its maximum speed and it is not limited by the external speed of the memory, the external memory on the mainboard, okay. Perfect, so we've got internal memories that we can see, we've got cores, and we've got hyper-threads. I'm going to speak about hyper-threads quickly and we will see that in action a bit later. So hyper-threading is a new technology to do input output more quickly on your computer. Imagine this case. When you've got an hyper thread here, I've got two hyper threads, so this is a hyper thread, sorry, let me explain. An hyper thread is a logical unit inside a physical core. A physical core can run one program at a given time. I've got 14 physical cores, so I can run physically 14 programs at a given time. If I run 100 programs at the same time, only 14 could be executed at a given time. So in some cores, I've got two hyperthreads. This is two logical cores, two logical units. I can load two different programs in each hyperthread. But only one can be executed at a given time. So let's imagine this one is running, the thread number 10, And sometimes it is doing some I.O. input output right into the hard drive, getting data from the internet. And this is slow, some milliseconds. One millisecond when you run at one gigahertz, it's one million of instructions that you do not do. It's a lot of wasting time, wasting operation. So at this moment, the core said, hey, you are doing nothing. I'm bored. I'm boring. This is boring me. I'm going to switch to the other hyperthread and running this code. This one is suspended while the data is coming, and the second one is running. When you run a server that is doing a lot of I.O., input-output with the hard drive, working with external peripherals, hyperthreading is very useful. When you do heavy computation, hyperthreading is not useful at all, and at the opposite, an issue, it's a side-back effect, it will slow down your computer and we'll explain that later. But so remember, hyper-threading is a good technology when you do a lot of I.O. because you can switch between two functions of the same core each time one function is doing some I.O., some input-output writing to the disk, speaking with the internet. So this is very efficient, you don't lose time. But when you When you do heavy computation, this has some drawbacks. You prefer to use cores without hyper-threading. So when you choose your computer, I will explain that later. But when you choose your computer or your processor, if you want to do heavy computation, take a computer with many cores with no hyper-threading. If you do an internet server with a lot of I.O. speaking with the network card, use hyper-threading. It's okay? Perfect, let's continue. I have explained what is the processor frequency. It is the number of transcriptions the processor can do in a second. If this is not entirely accurate, the exact definition is the number of cycles. The processor frequency refers to the number of clock cycles it can execute per second. That is to say, a clock cycle is the minimum time required to perform a basic operation such as changing the state of the transistor. This is to be perfectly accurate. So changing the state of a transistor means, for example, switching a bit from zero to one. You've got the minimum time to do that, this is the computer cycle. Some instructions in your computer may require only one cycle, for example inverted a byte, some may require much more cycles, for example doing an addition, multiplication or cryptographic command. We have spoken about the frequency, we can play with the frequency in your computer. There is a command in Linux, cpu-freq-utils, that allows you to switch the frequency. Let's have a look at it. I have prepared some commands. We have seen that my processor has a frequency that can go from 400 MHz up to 5000 MHz. I am going to put all the cores to 400 MHz, the lowest frequency. So I am going to slow down my speech. And now I am going to run a Python program, IPathon. Oh, sorry. I am doing a very heavy computation, 2 plus 2. A lot of time to measure the minimum time required to do this computation. To do this computation, my computer takes at this frequency 78 nanoseconds. I keep IPython, and then I'm going to change the frequency to the highest value. More than 10 times faster. Will my compute time be 10 times faster? Let's have a look. Ready, go, set, and go. And yes, yes, yes, it is more than 10 times faster, OK? So for you as a developer, you want the maximum speed. But for example, you are in the train, in the ice, and there is no power plug. Oh, and you have to finish your work. You can slow down your frequency and work a bit longer, because at a low frequency, you consume less power. OK, it's fine for everybody? I continue. So I skipped some slides that are not so interesting. There is a technical rule about frequency. Actually, the maximum frequency a processor can have is 6 GHz. If you try to go above, there are some side effects on the computer, and the data are wrong, or the computer warms too much and can burn. The computer, sorry, the processor. The processor warm is too high and it can burn. To overcome this limitation of the maximum frequency in our processors, a multiprocessor machine first emerged, then followed by multicore processors. Let's have a look. This is a multiprocessor computer. You've got two processors under the ventilator and the radiator. here you can have two processors each one can run one program at a given time if there are many cores many programs at a given time each one so they share the same memory and they consume twice energy if you want the next year have two times more power you add two more computer processors so four processors then eight 16 32 in this real have gone up to 32 times 32 processors on the same This is used, but this came with major drawbacks, 42 times higher power consumption, 42 times more physical space required, 32 times cost. This was not so efficient. So the processor manufacturer has decided to create a processor having multiple cores. A core is a compute unit inside the processor that can run a different program by itself. We have seen the different cores on my, we have got 14 cores. On each side of the cores, Intel has invented the technology, hyper-threading, known as SMT by other manufacturers, which allows to switch between two functions of the same core to gain time when your processor is doing something. Another point about the processor memory, this is the non-uniform memory access. Do you know what is a non-uniform memory access? You can raise your hand, just a few people rather than nobody, so it is very interesting to present it. Non-uniform memory access means that on a processor having four cores here, each core can be physically connected to only a part of the external memory. So it cannot access to all the memory inside the computer. Let's go back to this picture. On this picture, you see the memory sticks, and let's say this processor can access to only this memory, and this one to the two others, physically. So if this one, one data that is stored there, he has to have this one to give him the data. This loads on the process. You can have a look on your computer to your NUMA affinity. So which core is connected to which memory inside the computer. So for this, I've got another computer on the internet, and I'm going to use a command NUMA CTL minus minus hardware, which will display the NUMA affinity of my computer. So let's have a look. This says that on this computer, I've got four NUMA nodes, four CPU nodes. That implies this number of cores, over CPU nodes, under CPU nodes, so four groups of CPU. And the CPU node group zero can access to the memory node zero at a logical time of 10. This is not a physical measurement. This is the best it can do. If you want to access to the other memory node, 1, 2, 3, it costs more time. More time, sorry, 16 here. This is still a memory, a logical time. So you can try, when you want to run a program, to say, Hey, my program, I want you to run with this special memory node because I only need a few memory. 64 GB is already good. And I want to use the physical one to which you are connected to. You can do that with the command numma ctl or I've got one somewhere. cpu node-by as I use the cpu node 0 and the memory node 0 so with the best time of 10. And I run IPython again. Let's start. In IPython I'm importing numpy and then I want to do a little computation computation, the sum of a random matrix. Running it, it will take 5.8 seconds. 8 seconds, I think. I did a lot of time to prepare. 7.8 seconds, fine. So now, I want to run the same, but let's say I haven't taken care of on which memory node I'm working. So I'll use the second one. I'm not physically connected to this one, so it will be slower. Let's do that. Import Numpy. Run it again on the same computer. It's a bit slower, not 1.6 times slower, but a bit slower anyway, 8.4 seconds. So you lose a little time. On very huge computation, we can slow down a lot or accelerate your computation. If some of you have used Dask, do you know Dask? We are going to speak about Dask later. If some of you have used Dask, you can run Dask workers taking care of NUMA affinity. You can have better performances. OK, perfect. Let's go back to the slide. So we have played with the NUMA affinity, with the frequency. Let's speak about GPU quickly, and then we going to do a bit of code. GPUs are graphical processing units. Since the beginning of this century, GPUs have included dedicated computing units for floating-point operations. They have been started to be used for data science, for computational tasks, and they are massively parallelized. In a GPU you have a thousand of cores compared to a few hundreds for the best processors. So GPU have a lot of cores and they run and so they can do a massively parallel computing task. So let's take a look to understand the differences between GPU and CPU. The best CPU and GPU we can find actually on the market. If you look at the best CPU the best one is this one it's an AMD EPYC and with only 200 cores so you can run 200 tasks in parallel it's it's not so bad but if you look at the best GPU on the market actually the best one if you look at the number of core only this part. It's the NVIDIA RTX on this number, which owns 21,000 cores. It's a lot more. It's 100 times higher than the CPU. So, wow! Couldn't I just use the first 20,000 cores of my last NVIDIA graphic card for computational, and allocate the remaining 700, apparently, to replace the few cores of my CPU, only 14 here. That way I could say goodbye to the CPU entirely, and that will save money also. Can we do that? We cannot. Do you know why? We cannot. No. No Andres. In a CPU, all cores are independent and can execute different programs simultaneously at a given time. They are well fitted to run an operating system. On your operating system, you've got your web browser running, your text processors, your code editor, many different programs. On a GPU, all cores execute the same task in parallel at a given time. GPUs are well suited to do massively parallel computation. For example, on a table you want to add plus one in all table cells. Each GPU core takes one cell and adds one inside. And they do that in parallel, 20,000 by 20,000, where your CPU can do 200 by 200 times. But you cannot manage an operating system on a GPU because all cores are doing the same thing at the same time. So a GPU is fine for processing large data set or matrix on a CPU for running an operating system. That's why GPUs have not replaced CPUs. OK, I don't reply to this question. If you have it later, I will answer with pleasure to have a bit of time for the examples. Okay, we have over a processor. GPUs are very powerful for machine learning, but they consume a lot of power. If you have a look at the best NVIDIA or this one, the best AMD, it's 1,000 watts per second. It's a lot of power consumption. In 2013, Google has estimated that to keep its growing of deep learning, it has to double its number of servers. It was unsustainable, so Google decided to create a new kind of processor. This kind is named the TPU, the Tensor Processing Unit. You can think about a TPU, it's the same thing as a GPU, except that you don't have the display, the video function. Only the computational function, specialized on matrix multiplication and tensor operation. So a TPU is like a GPU, massively parallel, but with the lowest power consumption for high parallel tasks. So Google was the first one to create GPUs, but now there are many manufacturers that you can find here that are producing GPUs. And you can run your Python code with Python, for example, on any of these GPUs. TPUs, sorry, too many processors. It's okay? So I've got a few minutes to conclude this hardware part. TPU is very good for training a neural network. If you want to train a massive neural network, do not use GPU. Try to consider TPU. It may save money, save energy, and be very fast. But if you want to do machine learning inference, just predict value without training, you can use NPU. NPU is a neural processing unit specialized with very low power consumption with matrix operation but just for predicting values from machine learning models, not for training machine models. They are not so powerful for that. Let's finish this talk about the hardware basics and start the parallel paradigm. When you do parallel programming, you can use different kinds of programming. The actual programming we are using with Python is sequential programming. You write one function, then the others, then the others, as written here. First function, 3 seconds, second function, 3 seconds, last function, 3 seconds, total 9 seconds. You can also do asynchronous programming. Asynchronous programming in Python, or in general, computer science, is like hyperfreading. It's using the waiting time to do something else. Imagine that. You want to cook. You have two tasks to do, boiling potatoes and peeling carrots. You put the potatoes to boil, but you don't look at them all the time. You will say, hey, this is boring. I'm going to switch to the other task, peeling carrots. This is asynchronous cooking. The same for asynchronous programming. So if I come back to my picture here, in asynchronous programming, on one call, I run only one function at a given time, this one, three seconds. But when I start writing, this is now an IO. So my hard drive is working, and my computer is waiting for the hard drive. I can switch to the other task. And then this one is writing at its turn. I can switch to the last one, and so on. The final on the global time is seven seconds. So on the same core, we save two seconds of computation. And finally, the best one, parallel programming, the one we are going to see right now. OK, when you do, finally, parallel programming on a computer, you have two choices most of the time, multi-threading, running many functions in parallel on the same program. So this function shares the same data. And then multiprocessing, running different programs, different occurrences of the same program or different programs on your computer or many computers. If you run different programs on different computers, this is named distributed processing. OK? And this is a big picture. If you want to do parallel programming, computing, With Python, which library can I use? There are so many. In Python, you can do a lot of things. You can do compilation. You can compile Python code, like in C. You can do asynchronous programming, multi-threading, running function in parallel in a program, multiprocessing, running programming parallel on the same computer, and distributed programming, running programming parallel across different computers. On grid computing, this is distributed programming on different networks, like on different supercomputers. For the compilation with CPU, what do you have? That's 14 minutes. It's not so bad. NUKTA. Do you know NUKTA? Someone, resident, nobody? So it's a library, one people, fine. It takes your Python code, convert it to C, and generate an executable that you can run. Sometimes it's faster. Python is a fork of Python that includes a just-in-time compiler. But the project is sometimes under heavy activity, sometimes very low activity, actually. Nothing happens. Hop, this is a library. You put a decorator in front of your function, and it compiles the function on the fly. This is a German function, a library by a German research center. It is no more maintained since 10 years, so it is an old one. But I put it here because if you want to understand how you can do Python compilation, you can have a look at the code. It is very simple, and there is no C line, only Python code. To compile Python is very surprising. PyPy, this is a fork of Python in very huge activity that compiles Python on the fly. Very efficient. We are going to try it in a few seconds. Cyton, I'm not going to explain all of them, but a few. This is a mix of Python and C that is converted to C compiled. Pandas and NumPy are written in Cyton. Most of their code are written in Python. When the Cyton code is converted to C and compiled, then you can import it as a normal Python code, except that it's much faster. Python, another compiler. You can compile a slow for GPU with Numba. We are going to try that. Numba for CPU, GPU. Just a decorator in front of your function, and your code is compiled on the CPU or the GPU. It's very, very fast. JAX, it can compile on CPU, on GPU, and on TPU, especially Google, Google, GPU. QPy, QPy, do you know QPy, too? It's a Python library for GPU. It's a Numpy for GPU. You import qpy as cp and use cp as numpy. But your code is executed on the GPU. We are going to try to show that in a few moments. And you can work with Rapids AI. Rapids AI is a project on the internet. Let's have a quick, quick, quick look. That implement, try to re-implement the main Python data science library for the NVIDIA GPU. So you have a panda for GPU, this is QDIF. You have the scikit-learn for GPU, this is QML. And you have a network library, like NetworkX for GPU, too. This is very fast. This is an heavy activity and very interesting. If you want to do asynchronous programming, you have a lot of libraries. The first one is AsyncIO. I have to hurry up. Greenlight, and many others. Vast libraries are most of the time libraries for web connection. Twisties is for web connection. FastAPI is for web server. Because asynchronous programming is about using the waiting time to do something else. It's all about library doing connection with something else. OK, and if you want to do the best one, distributed programming in Python, you've got many other libraries. Joblib is the one used in Scikit-learn. Seri can run many processes on different machines, computers. Grammatic, SeriClone. MPI, this is the one used on most of all supercomputers. It can run many processes in parallel on many computers. And they can exchange data across the network very quickly. And you've got Rea for machine learning, a very good library to run, to train your model on many computers, having many graphic cards, for example. If you have a very huge model that cannot fit on only one computer, you can train it on many computers with Frey. It is very easy. Thus, this is a NumPy on Pandas parallelized on one computer or many computers. We are going to try it quickly, I hope. And JAX, this is the one we have already presented quickly, that can compile Python too. And finally, for the grid computing, we have a few libraries too. OK, these are the best I know, but there are many more. You can find many more at the Python Wiki. So let's start with a demonstration, nine minutes to do a bit of code. It will be short. I'm going to do my best to not bother you. OK, let's try with PyPy, quick. So PyPy is a Python compiler. First you install it, and you run PyPy my code, my Python script, my script.py, as you run Python your script.py. So this is a simple code, decrementing an integer and measuring the duration. I create the file, execute it with Python, Nearer than seven seconds, apparently. Three seconds, sorry. Four seconds. And I run it with PyPy. Nothing to change in the code, just PyPy, my script. And attach your belt. Only 0.03 seconds. It's 100 times faster. I haven't changed anything in my code. This is very, very fast. It was to be there, no? So we can ask why everybody is not using PyPy. I will reply later if you have a question. If you want to compile with something else, you say, oh, bye bye, I don't want. I prefer Numba. So here is a typical Numba example. I'm going to create a fractal, a mandelball fractal. The mandel function is computing the color of a pixel. And this one, with two loops nested, is calling mandel for each pixel. So creating the function, executing the code, and measuring it. Let's run it. Beautiful picture. Have a look. Perfect. And it's three seconds with pure Python, no compilation. So I import Numba. I add this only nine in front of each function, the decorator, Numba.jit. And this say, compile my function the first time you execute it. Then use the compiled version. And I run the same code. Attach your belt again. So I'm 10 times faster. Not so bad with three lines of code. But this includes the compilation time. If I execute it again, the compilation time will be removed. Let's do that on 0.05 second. OK? Much faster again. There are many things to say about Numba, but that's enough for the demonstration. Let's switch to the basics of parallel programming. I've got six minutes. I want to do Dask, and that is very short. Let's try. So if you want to do multi-threading or multi-processing in Python, you've got native libraries, threadings, and multi-processing. Let's start with multi-threading. So I've created a function that displays 10 times the same message with a little sleep. So it will be five seconds executing it. one time, two times, and then the main program, 15 seconds. Let's run it. This is sequential code. Then I switch back to my presentation. And let's try to explain how do you parallelize this with multi-threading. If you want to create a thread, remember a thread is a function that is executed in parallel of the main process or the main program in your program, in your script. So I have to import the thread class that I renamed task. I've set my functions. The functions that I want to execute is f1 with this parameter, and I want to execute a second one with this second parameter. Then this just creates the task. This doesn't execute the task. To execute the task, I say t1.start. We start the first function in parallel of the main program. This can be parallelized if I've got at least two cores on the computer. If I've got only one core, no parallelization is possible. Okay, then I start the second thread here and at this time I've got three tasks at the same time on the computer, so I need at least three cores. If I've got only two, the last one will wait its turn. And if I want that my main program waits for the task to be completed, I can use the join operation. Let's go back to the first example. sequentially the code to 15 seconds. If I run it as explained with Fright, it will be 5 seconds. We see that each function is running all together. All functions are running all together and the messages are displayed at the same time. And the final time is five seconds. Fine? Welcome to the magic world of parallel programming with Python. Four minutes. OK. I would like to speak about a concurrency. So when two functions want to update the same data. But I won't have time to explain quickly Dask and speak about the cache. So I'm going to jump on Dask. Let's have a look. Dask is a library. Who is using Dask, actually? Can you raise your hand? Okay, a bit of you. I'm going to try to show you something nice that we have learned. If you use NumPy on Pandas, NumPy and Pandas have two drawbacks. They are not parallelized. If you have ten 10 cores and you want to do the sum of 10 columns in a table, you do columns one by one. This is not so efficient. And they all work in memory. So we need to load all the data in memory. If you have one terabyte of data and only 10 gigabytes of RAM, you cannot do that with NumPy and Pandas. Dask can solve this issue. The main idea of Dask is to cut the NumPy table in many pieces, and if you want to compute the sum of the full table, Dask will compute the sum of each piece, then aggregate all the results to have the final sum. This works for most mathematical functions. The minimum is the minimum of all minimums, the maximum is the maximum of all maximums, and so on. This doesn't work for the median. For the median, you need to load all the data in memory, short them to find the middle value. But for most functions, this works fine. The main idea of Dask is that I will run the computation of each block in parallel on all the cores of the computer, or if I have a cluster, on all the cores of my cluster, so on many computers. This is possible across the network. So how does it work? You import Dask. Yes, two minutes is just the right time. Then you do a little computation as a reference measurement, a normal random matrix that I I compute, and I compute the mean. Running it, it takes 20 seconds on my computer most of the time at the highest frequencies. This is my frequency, actually. 12 seconds, perfect. And now, I want to do the same with das. So to do the same with das, you just import das.curate as da and use da as in p. So the same syntax, the same mean, just to finish, you have to say, hey, do the computation. Else, task, only store direction you have to do to process them in parallel later. So I do the computation, and I've got 14 cores. It should be 14 times faster, so one second. Let's do it. 1.5 seconds. That's good. I wanted it to be slower. I'm surprised. So this can be better. Can we do much more than NumPy? NumPy is working on one core. I'm working on 14 cores. So it should be a bit lower than once ago. Yes, we can do much more efficient. Have a look at X. X is this table. It's a table having 7 gigabytes of memory. And chunks are made of 128 megabytes. This is huge chunks. If you want to compute chunks, it doesn't fit in my memory cache, which is only one megabyte. So it is too high to be stored inside this cache. So my processor doesn't work at its full speed, and it's doing a lot of round trips with the external memory to load all the chunks. If you take some chunks lower that can fit in the cache, we can have much better performances. It will be the final cloud, the final point. So we can do that by saying that I use little lower chunks. Sorry. So now my chunks are less than 1 megabyte. So if it enter in the cache, I run the computation again on the same matrix on attach your belt, 0.5 second. This is 22 times faster than NumPy. I've got only 14 cores. How is this possible? Because of the cache. My processor can run at its maximum speed. It's OK for everybody? Perfect.