Cuda on icyveins7's blog

Large pinned host allocations in CUDA

Wed, 02 Sep 2026 20:00:00 +0800

Sometimes, you just can’t fit everything in VRAM.

In most CUDA projects, the advice is usually to transfer all inputs, outputs, and temporary scratch space to the device. Working completely inside VRAM is fast, and avoids both the complexity and the throughput hit that comes with over-PCIe transfers being scattered throughout your hot path.

All of this is correct, and I adhere to this myself as much as possible. Sometimes, though, you simply just can’t do this, and recently I was left with no choice and had to start migrating stuff back to host memory.

Occupancy-maxxing is just Starcraft

Wed, 12 Aug 2026 20:00:00 +0800

If you can manage minerals and gas, you can manage CUDA threads and registers.

CUDA is just resource management.

Okay, of course there’s a lot more to squeezing CUDA kernel performance than just resource management, but occupancy is very often the simplest thing to strive for, once you know how. And usually, if you’ve hit 100% occupancy, then unless you’ve done something heinous in your code, further optimizations are unlikely to budge your performance by large factors (aside from an entire algorithmic shift, but we won’t discuss that here).

Your left-right boundary search is actually a CUDA-friendly single scan

Fri, 26 Jun 2026 20:00:00 +0800

At least, most of the time it is. As long as it fits the 1-2-1 pattern..

Many segments can be boiled down to what I call a 1-2-1 pattern. For an array of input elements (I’ll just call them pixels, it’s shorter for me to type), a naive segment detection is to loop forwards until you hit an activation pixel. This can internally be any condition you’d like; that is, any if (...) is possible here as long as you operate only on the pixel you’re currently at. For simplicity, I’ll assume this is the 2 pixel:

Avoid being baited by your printf statements in CUDA kernels

Mon, 23 Mar 2026 20:00:00 +0800

Behaviour of printf on device is not the same as on host!

If you’re a printf aficionado like me, then you use printf for debugging. A lot. In fact I previously wrote a small logger that uses printf called spfLogger. I do enjoy the flexibility of tuning every single width/precision with a few characters, and it’s something I haven’t yet seen C++ be able to emulate with as little irritation.

I haven’t gotten around to writing code using newer std::format or println yet, so the jury’s out on that.

Iterative algorithms with CUDA

Sat, 29 Nov 2025 20:00:00 +0800

Because not everything can be made un-iterative..

There’s a pretty large class of algorithms that have been developed for CPUs that involve, optimally, an iterative solver.

Usually, this is defined by some notion of convergence; some variable is updated, and then at the end of each iteration it is checked to determine whether further iterations are required.

This is all fine and dandy in CPU-land, but in GPUs this almost always makes the flow awkward.

Some tips for integrating small bits of CUDA code into larger codebases

Wed, 13 Aug 2025 20:00:00 +0800

Some lessons from general C++ come in handy here..

In some recent work I integrated some CUDA code I developed into a larger existing codebase, written for the CPU. Essentially, my module(s) would accelerate and replace some existing functionality, but was only a small cog in the machine.

This is likely applicable to many others, so hopefully the lessons I document here will be concisely useful to those who chance upon this post.

Quickly sketching out a BlockQuickSelect

Sun, 20 Jul 2025 20:00:00 +0800

I wrote a blockwide quickselect. That’s the post.

As of today, NVIDIA’s cub library has a few block-wide primitives to do sorting - like BlockRadixSort - but none that do the equivalent for the $k$‘th order statistic i.e the $k$-th smallest element.

This would be functionally equivalent to the CPU’s std::nth_element, but that apparently uses Introselect, which is a tad too much for me to want to implement. I’m going to stick with the simpler quickselect.

Interesting Tidbits from GTC 2025: CUDA Graphs

Tue, 25 Mar 2025 20:00:00 +0800

In the 2nd post of this series, I give a short introduction on something that is also not particularly new, but new to me - CUDA Graphs!

As stated in my first post of this series, this topic isn’t particularly new per-se. Indeed, it looks like it has been out since 2019. But maybe CUDA graphs have increased in relevance now that GPUs are more powerful. As usual, a good starting reference is NVIDIA’s own blogpost.

Interesting Tidbits from GTC 2025: Asynchronicity Beyond Streams

Mon, 24 Mar 2025 20:00:00 +0800

Some notes for CUDA programmers who haven’t kept up with the times; this first post covers in-kernel pipelining..

I recently had the privilege of being sponsored to attend the March 2025 GTC in person. While the sessions were very largely dominated by AI-related things, I generally selected the CUDA-related ones, since they were more relevant to my work (and interests).

I’ll try to encapsulate some of the new things I learnt while I was there into a few major concepts. Note that most, if not all of these, are not shiny new CUDA features - some of them have been out for several years, but I just didn’t know about them, so I learnt about them there.

AtomicMinFloat; overloading integer-only atomics for floating-point numbers in CUDA

Sun, 16 Mar 2025 20:00:00 +0800

Convincing you (and myself) that with some minor edits, we can still use atomicMin for floats in CUDA..

CUDA has a set of atomic functions for safely updating the same memory address with different threads. There’s a whole list of them in the programming guide here.

However, for atomicMin (and also atomicMax), there isn’t an overload that works with floats (or doubles , but who uses those in CUDA anyway..). For this discussion we’ll just focus on atomicMin, but all the points we discuss can be inverted to explain atomicMax.

Gotta go (randomly) fast, thrust vs cuRAND

Mon, 10 Mar 2025 20:00:00 +0800

How fast can you generate (pseudo-)random numbers on the GPU? How fast can you generate (pseudo-)random numbers on the GPU?

I recently had to answer this question. Well, not exactly this question, but this was a key part of it.

A quick google search will bring up two common methods (libraries) when trying to do this in CUDA: thrust and cuRAND. Thrust is known to be a lot easier to set up; no need to write the kernel code and nitty gritty details, so I started with that.