PyTorch and CPU-GPU Synchronizations
PyTorch executes GPU operations asynchronously, allowing the CPU to schedule tasks and run ahead of the GPU. Performance degradation occurs during CPU-GPU synchronization, which happens when the CPU must block and wait for data to return from the GPU to make a decision or allocate memory. This creates "bubbles" of inactivity on both the CPU and GPU, reducing overall hardware utilization.
Common triggers for synchronization include calling .item(), .cpu(), or printing tensors, as well as using GPU tensors within conditional if-else branching. More subtle synchronizations arise from operations that result in dynamic shapes, where the output size depends on the tensor data. Examples include boolean indexing, slicing with a GPU-resident integer, torch.non_zero(), and torch.unique(). Because the CPU manages memory allocation, it must synchronize to determine the output shape before the GPU can proceed.
To mitigate these issues, developers can reduce the frequency of synchronization—such as printing loss every 100 iterations instead of every one—or use padding to maintain static shapes. Some PyTorch APIs, such as torch.repeat_interleave(), provide optional parameters to specify the output size, bypassing the need for synchronization.
Profiling tools like NVIDIA Nsight Systems can visualize these delays as CUDA stream synchronize calls. For automated detection, PyTorch offers an experimental debug mode via torch.cuda.set_synchronize_debug_mode(), which can be set to warning or error. This allows for the creation of unit tests using decorators that fail if a function triggers a GPU synchronization, ensuring production code remains efficient.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track PyData & Scientific Libraries Stack and was classified suitable for advanced domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
PyTorch gets its speed from asynchronous execution: the CPU launches operations quickly while the GPU executes them later. CPU–GPU (host-device) synchronizations break this pipeline by blocking the host until the GPU reaches a specific point. The result is often counterintuitive: even if kernels are fast, the GPU develops idle gaps, throughput drops, and latency rises because the CPU can no longer run ahead and keep the GPU fed with work.
This talk builds intuition with a minimal loop that alternates a slow GPU operation with a quick “bookkeeping” operation, a pattern that resembles many inference and training pipelines. By adding a seemingly harmless action—such as printing a CUDA tensor—we’ll see how easily a synchronization can be introduced and why the slowdown can be disproportionate to what the code appears to do.
We’ll then walk through a practical profiling workflow in NVIDIA Nsight Systems. The key technique is to correlate GPU utilization gaps with long CPU-side CUDA API calls (for example cudaStreamSynchronize) that indicate the host thread is waiting. Comparing a healthy trace to a sync-heavy trace makes it clear where the pipeline stalls and which code region triggers it.
Beyond the usual suspects (.item(), printing device tensors, explicit device transfers), the talk highlights dynamic shapes as a common synchronization trigger. Patterns like boolean indexing with a GPU mask or slicing with a GPU-resident index can force PyTorch to fetch information back to the CPU to determine output sizes and allocations. We’ll discuss how to recognize these cases and how to restructure code toward shape-stable alternatives when possible.
Finally, we’ll cover how to prevent regressions. Instead of relying on profiling alone, we’ll use PyTorch’s experimental API torch.cuda.set_sync_debug_mode() in unit tests to surface synchronizations early, while keeping production code unchanged. We’ll close with guidance on when a small Triton kernel is worth considering to avoid sync-inducing patterns and to fuse multiple small ops into a single, fully asynchronous kernel.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:30]
So hi everyone, good afternoon. Welcome to the first session of the afternoon in this room. So with us we have the session PyTorch and CPU GPU synchronizations writing fast PyTorch code with Thomas Ruiz. So it's up to you now. Thank you.
Speaker 2 [00:56]
Yes, thank you very much. I'm super happy to be here. Yeah, so I'm a PhD student in Munich at the LMU University, and I'm going to talk today about the synchronizations in PyTorch. Come in. So, who is this talk for? Well, I guess it's good for anybody who is writing PyTorch in their day-to-day and is using GPUs to accelerate their workloads. In this case, this talk is for you. I want you to take away, leave the room understanding what are these synchronizations that I talk about and why they are important, namely because they silently slow down the performance of your application. And it's not that your code is incorrect. Your code is still going to be correct, but it's just going to leave a lot of speed on the table. And finally, I want to be able to spot them and then to fix them. So the overview is going to be, we're going to be looking at a basic training loop, like everybody has probably seen before. And this is going to have a synchronisation, so that's problematic. We will then dive into, look at some real profiling traces, so that you can see how If you use real profiling tools, what will you see in this case of synchronizations with NVIDIA inside systems? Then we will dive a little bit deeper into more subtle patterns that trigger synchronizations. And finally, we will have a look at unit testing to check whether your code contains synchronization so that you can be confident that your code is synchronization-free. So, let's get started. This is the familiar training loop that you might have written yourself for your machine learning project. So, basically, we have a loader where we're loading batches of data. We are calling the model on this batch of data, like, getting some output, then based on this output, computing a loss, then doing a backward, and using this optimiser and optimiser to change the model weights. Finally, you probably want to have a look at how the loss is doing. You want to see it going down, so you print the loss. We often start by writing this on CPUs to test and make sure it's working correctly. Then we move it on the GPU, and we see that indeed it gets faster, but it's not as fast as we expected it to be. And NVIDIA provides this command line tool that you can use, NVIDIA SMI, and it will show you the metrics around your GPU, and you might be somewhere around 50% utilization, more or less. It definitely is probably not 100%. And you check different things, and you see that, well, everything seems to be correct, so what is it that's happening that's slowing down your code? And the hint is obviously, well, the synchronization. Good. So that is the familiar training loop. Now I want to take a step back and think about or explain how the GPU actually is working. So how is the CPU and the GPU collaborating? Hello, hello, hello. I'm back, right? The GPU is actually asynchronous. So the principle that you need to keep in mind is the CPU is bossing around the GPU. and telling it what to do and what we see here is a timeline with GPU and CPU and the CPU is scheduling these operations these pytorch operations and to be to be skewed and executed by the GPU so you see these are the operations that we had in the loop zero grad forward loss function backward and step and it takes a little bit of time for them to be for them to move to the GPU and then start on the GPU, and then since the GPU is the one actually doing the work, it takes a longer time, each operation takes a longer time on the GPU than it takes it to be just scheduled from the CPU. And importantly, this means that the CPU finishes scheduling the operations before the GPU finishes executing the operations. And this is a healthy thing. This is what you actually want to have. And it's also very interesting to see that the CPU is done, like here at this step, it's already finished scheduling. The GPU is still in the middle of the forward pass and hasn't even started to do compute the loss function backward or optimisation step. This is why we say, or it is said that the CPU runs ahead of the GPU because it finishes earlier. And if you were to naively time how long it takes the CPU to finish all the operations, then you would be misled to think that it's very fast while the GPU is still working on the actual, getting the actual work done. And the question is, so what is a CPU-GPU sync? So it happens when the CPU gets blocked and has to wait for data to come back from the GPU. For example, when the CPU needs to take a decision based on that data. So here we have an example, so we have in our code something like if this CUDA tensor is larger than zero, then do something, otherwise do something else. So the data resides on the GPU, but the CPU needs that information to take the decision. So it has to move this information back. And this is another timeline where now I have said let's call them up one, up two, up three. And up one and up two are scheduled normally. But up three cannot be scheduled until the CPU gets this information back from the GPU. So what we see here is that the CPU blocks. It has to wait. So all this time here spent doing nothing, waiting for the GPU to send information back. Then it can schedule up three and send it back to the GPU. The GPU will be waiting in the meantime doing nothing. So these are the bubbles on both sides of the hardware. On the CPU side you have bubbles, the GPU side you have bubbles. And this is going to slow down your code. So when is it happening in our training loop? It happens precisely here when we want to print the loss that we currently have, right? This call to the item forces the CPU to fetch that information from the GPU and synchronise. And this is happening on every iteration. So you schedule these five operations, or I don't know how many they are. Schedule them, synchronise, schedule them, synchronise. And one simple fix would be just to say, well, I synchronize just every 100 iterations. That means these 100 iterations will be just batched, batch scheduled, executed tightly without bubbles. And I still want to know my loss sometimes, so I just print it out. There are also solutions where you asynchronously fetch back the data from the GPU. that's a bit more involved. All right, so let's say we understand these timelines that I showed you, but you want to actually see it with real tooling. So we can use the NVIDIA Insight Systems Profiler and I prepared a snippet of code that's simpler because it doesn't have the backward pass and optimization step which complicate the picture. So in this simpler example, it's also a loop, but it consists basically of a slow operation and a quick operation, and then an optional print, which is going to either trigger the synchronization or we leave it out and there is no synchronization. And if you have an NVIDIA GPU, then you probably have this nsys command available, with which you can create a profile. And this is basically the program that we're profiling. It's available on this GitHub gist if you want to try it out yourself, maybe later today. And what we're going to see, this is a screenshot from On the inside systems profiler, it has again two traces, so there is a CPU timeline, which is this one, and on the top you see a GPU timeline, which is the longer one. That's why I put the CPU on the bottom, even though it would have made intuitively more sense to put it on the top. But this is just how it's structured. I want you to notice what is most striking immediately is that the CPU timeline is a lot shorter. So it is running ahead. It's scheduling all these operations. Slow, fast, slow, fast, slow, fast, slow, fast. Scheduling everything quickly. And then the GPU starts working on the stuff a bit later. And the slow operation is the yellow one. I think you might be able to read it. And the quick operation is the blue one, which is shorter. And they're operating back to back. Here on the top, this light blue bar is the GPU utilization. So you see that it has a little bump here, a little bump here, but in general, it's continuous. So you're getting 100% utilization from here all the way until the end. And this is a healthy run. So this is no synchronization whatsoever the print statement is disabled and now if we enable the print statement we have a synchronization so this is the case where this is the this is the same screenshot but for the workload with synchronization and what we see here is again the the the two timelines but the lower timeline of the CPU is a lot longer and they both end up around the same finish line, so to speak. This means the CPU is not really running ahead, but rather it's being, yeah, it's being delayed. And here on the bottom you see these green bars which are completely new, and what they say, if you can read it, it says CUDA stream synchronise. So while this green operation is running, your CPU is doing nothing else. Yeah, basically that's your waiting time on the CPU. It's waiting for the information to come back from the GPU. The GPU operations look very similar, so you still have slow operations followed by quick operations, but you have these gaps in between, and importantly, on the top you see the utilisation has some gaps there, and that's precisely where the slowdown is coming from. So you have, on the top, you have the slowdowns, on the bottom you have the green ones, and this workload becomes like 400 microseconds slower. It could be a lot more. This just happens to be a lot of matrix multiplications which occupy the GPU very well. Okay. So, we say we understood this. We want to write fast code, so we're going to avoid some of the idioms that trigger synchronisations. So there is a PyTorch performance in Docs which has a section about this specifically, and it mentions these bullet points over here. So avoid .item calls, avoid these .cpu calls, don't put it in conditionals, and I think once you've seen them, it's easy to memorise them and be like, OK, those are easy to avoid, I can do that. But the more interesting part of this talk is that there are more subtle idioms or patterns that trigger synchronisations as well. And those are related to dynamic shapes. So let's go through some of these dynamic shapes and understand what's behind them. These operations may look harmless, but they create also synchronisations, without you intending to, right? So let's start with this one. This is a case where both T and mask are tensors on the GPU, torch tensors, and this is basically a boolean indexing, right? So mask is an indexing, you're selecting a subset of the T tensor. I think it's a very common operation. You do it in NumPy, you do it in Torch. This is very, very common. The problem here is that the length of X, how large X is, will depend on how many trues there is on the GPU side, on the data that's living on the GPU side. A very similar pattern is this one. This is slicing. So if K is just an integer that's living on the GPU, let's say 10, 20, it doesn't really matter, and you slice another torch tensor with it, this would be like selecting the first 20 elements of this tensor, then the CPU will also not know how long the resulting tensor will be without fetching that information back from the GPU. And more Torch APIs have the same problem. So this is non-zero. What it does is it gives you the indices of non-zero values. And how many non-zero values are there in a tensor? Well, that depends on the data. The same with unique. This is like calling set on a list in Python. So how many unique numbers are there in a tensor? Well, that depends on the tensor and its data. data. So what they all have in common is that they synchronise, because PyTorch needs to allocate this output tensor X, and it needs to know the shape in advance, and if this shape depends on GPU data, the CPU must ask the GPU. And if you think about this boss and this relationship of boss, the CPU is the boss, but if the CPU needs to ask the the GPU, then it's not the boss anymore. So you're having this inversion of who's the boss that usually both have. This is basically what I mentioned before. So rule of thumb for you or mental model should be if your tensor has a static shape that is known in in advance, then you can have async operations on it. If it definitely has a dynamic shape, then it's likely that you will have some operation that triggers our synchronisation. There are ways to fix this issue with dynamic shapes. So, for example, this function from Torch called repeat interleave. It has an optional parameter that you can pass that tells Torch how large the output will be, so you tell it the output size is total, this can be an integer, let's say 10, 20, and then, since Torch doesn't have to ask the GPU anymore for the size, it can allocate this without a synchronisation, and the code runs through without synchronisation, while this idiom does trigger a synchronisation. So Torch gives you some tools to work around this and avoid the synchronisations. Yeah. So you try to use the APIs that allow you to pass the output size. These variables saying what is the total size should definitely not be residing on the GPU, so keep that on the CPU side. Otherwise, you will have a synchronisation again. Padding is also a solution, which I didn't, so I'm not going to delve into that a lot, but you can use padding to avoid dynamic shapes. And And finally, let's talk about unit testing. So we saw that you can see the synchronisations with a profiler, that you will see these long calls called CUDA stream synchronise. But do you need to do this every time just to identify a synchronisation? The answer is no. There are tools to do this quicker without having to spin up profiling. And for that, PyTorch offers an experimental debug mode that flags every synchronisation in your code. So how it looks like is like this. Torch.cuda.setSynchroniseDebuggingMode, and you can set it to warning, such that if you take a CUDA tensor and you trigger any type of synchronisation on it, it will raise a warning for you. So you can activate this, run your code, and if you see a warning, then you know there's some synchronisation somewhere. You can set it to a strict mode with error, bypassing error, and then this same pattern will actually raise an exception. And that is actually quite useful, because then you can use this pattern in unit tests. So what you can do, here what you see in this case is a unit test that is loading some inputs and then passing those inputs to something called my PyTorch function. It can be really anything. It could be your model, it could be some function that you wrote yourself, and then there is some correctness check, and what I have done here is to add a decorator that says fail on GPU sync. So this unit test will fail if my PyTorch function has a synchronisation. And what's happening under the hood, yes, is there is a decorator. Before calling the function, I set a debug mode to error, such that it raises an exception on the function, and afterwards, you deactivate it again. And the cool thing is that you're not touching this function under test. You're only modifying unit testing codes, so the production code stays clean of this experimental API. And you can test also code that is coming from a library, right, or that is not your own code. You can still test it for synchronisations without modifying it at all. So the takeaways from this talk are the asynchronous mode is the default for operations launched on the GPU. The CPU is supposed to run ahead. That's healthy. The GPU is supposed to stay busy with work. Synchronization is when you are blocking the CPU while it waits for the GPU, inverting this bossy relationship between both, and there are some really clear obvious triggers that you should easily remember, like item, print, .cpu, if else branching, and there are more subtle triggers that you might have to reason about with dynamic shapes. And there are obviously solutions to that, which the optional parameters that I showed, but also padding, or just, you can still sync, it's fine, the world is not going to end by syncing. But you can do it just less often. And finally, obviously, if you want to unit test your code for it you can do so and PyTorch provides the APIs to test this on your own code that's it, thank you thank you very much
Speaker 1 [22:28]
Thank you very much for your presentation. Now we have some questions for you from our audience. One of them is, how much of an issue is this on unified memory architectures such as NVIDIA GB200 or Apple M-series?
Speaker 2 [22:49]
That's a very good question. So my understanding is that in unified memory, you supposedly don't care anymore where these tensors are. But I haven't actually gotten my hands on one of these systems to be able to tell how much of an issue it is. I assume it's less of an issue. But if you have one of these machines, reach out to me.
Speaker 1 [23:15]
Thank you very much. Do tools such as TensorBoard sometimes introduce an expectant synchronization?
Speaker 2 [23:25]
Good question. So I haven't looked at the code of TensorBoard. My understanding is that they are very much aware of this, and they fetch these... They fetch those... So they do this async solution, basically, where they are moving the memory asynchronously from the GPU to the CPU. But I'm not sure. I would assume they do. So, yeah, it would make sense. So you're not synchronizing on the hot loop of the training.
Speaker 1 [23:58]
Okay, thank you Why does the CPU need to know the sizes if the resulting? tensors are created slash continue to live in the on the GPU
Speaker 2 [24:11]
Yeah, that's a that's a good question. I don't know. The short question is I don't know why it necessarily needs to know the shapes and Why if you just tell it the shape it can just go merrily forward So That's a good question. I don't know I would would be necessary to look into the Into the maybe the memory allocator or something like this The boss is the CPU, and he needs to tell the GPU how much to... Yeah. The GPU can't do it by itself. Yeah. Agree.
Speaker 1 [24:54]
Thank you Do you need to run the unit test with your wrapper on? Nvidia GPU to detect the sinks or can will? PyTorch also raise an arrow on CPU
Speaker 2 [25:08]
I haven't tried on the CPU. Interesting question. I'm not sure if the concept of a synchronization even exists if you're running entirely on CPU. I don't think so. So probably it won't do anything.
Speaker 1 [25:26]
Okay. One more. Is synchronization also a problem in, for example, ONNX models? Would NVIDIA and CYT also work in this case?
Speaker 2 [25:40]
I'll win the next.
Speaker 1 [25:41]
I'm not
Speaker 2 [25:42]
I'm not familiar with the ONNX runtime, so I don't know much about it.
Speaker 1 [25:52]
Okay, thanks. Do all dynamic result operators have allocation hints or are there known exceptions?
Speaker 2 [26:03]
So, I talked about this repeat interleave, and this is, I would say, one of the few functions that offers this output size optional parameter. Let's have a look at the rest. I don't think the others have this, but the documentation is there, and you can look it it up and then try to select always the API that offers you this path to pass the output sizes.
Speaker 1 [26:39]
Thank you. The last question is do we need CUDA and PyTorch versioning?
Speaker 2 [26:46]
In PyTorch, what?
Speaker 1 [26:48]
versioning
Speaker 2 [26:50]
Do we need CUDA in PyTorch versioning?
Speaker 1 [26:54]
And, and, CUDA and PyTorch versioning.
Speaker 2 [27:00]
Yeah, so this was entirely tested on CUDA devices or GPU devices. I haven't tested it on other GPUs like AMD or whatever, but I expect similar problems if the devices are separate.
Speaker 1 [27:17]
Okay, thank you very much. Thank you for your presentation and answer.
