icyveins7's blog

Large pinned host allocations in CUDA

Wed, 02 Sep 2026 20:00:00 +0800

Sometimes, you just can’t fit everything in VRAM.

In most CUDA projects, the advice is usually to transfer all inputs, outputs, and temporary scratch space to the device. Working completely inside VRAM is fast, and avoids both the complexity and the throughput hit that comes with over-PCIe transfers being scattered throughout your hot path.

All of this is correct, and I adhere to this myself as much as possible. Sometimes, though, you simply just can’t do this, and recently I was left with no choice and had to start migrating stuff back to host memory.

Occupancy-maxxing is just Starcraft

Wed, 12 Aug 2026 20:00:00 +0800

If you can manage minerals and gas, you can manage CUDA threads and registers.

CUDA is just resource management.

Okay, of course there’s a lot more to squeezing CUDA kernel performance than just resource management, but occupancy is very often the simplest thing to strive for, once you know how. And usually, if you’ve hit 100% occupancy, then unless you’ve done something heinous in your code, further optimizations are unlikely to budge your performance by large factors (aside from an entire algorithmic shift, but we won’t discuss that here).

Your left-right boundary search is actually a CUDA-friendly single scan

Fri, 26 Jun 2026 20:00:00 +0800

At least, most of the time it is. As long as it fits the 1-2-1 pattern..

Many segments can be boiled down to what I call a 1-2-1 pattern. For an array of input elements (I’ll just call them pixels, it’s shorter for me to type), a naive segment detection is to loop forwards until you hit an activation pixel. This can internally be any condition you’d like; that is, any if (...) is possible here as long as you operate only on the pixel you’re currently at. For simplicity, I’ll assume this is the 2 pixel:

Avoid being baited by your printf statements in CUDA kernels

Mon, 23 Mar 2026 20:00:00 +0800

Behaviour of printf on device is not the same as on host!

If you’re a printf aficionado like me, then you use printf for debugging. A lot. In fact I previously wrote a small logger that uses printf called spfLogger. I do enjoy the flexibility of tuning every single width/precision with a few characters, and it’s something I haven’t yet seen C++ be able to emulate with as little irritation.

I haven’t gotten around to writing code using newer std::format or println yet, so the jury’s out on that.

Iterative algorithms with CUDA

Sat, 29 Nov 2025 20:00:00 +0800

Because not everything can be made un-iterative..

There’s a pretty large class of algorithms that have been developed for CPUs that involve, optimally, an iterative solver.

Usually, this is defined by some notion of convergence; some variable is updated, and then at the end of each iteration it is checked to determine whether further iterations are required.

This is all fine and dandy in CPU-land, but in GPUs this almost always makes the flow awkward.

Build your own tools, gcc edition

Mon, 20 Oct 2025 20:00:00 +0800

Not complicated, but always good to know the process.

In this series (?) I’ll cover something I always wanted to write down: compiling the toolchains/compilers I use from source. This means, specifically, no using of package managers i.e. not allowed to sudo apt install dependencies. Of course, this has its limits (you need a compiler to compile gcc, and some OSes don’t come pre-installed with one). However, the premise here is to at least be familiar with the minimal required dependencies, and where to get them (at the time of writing), and then what to do to build everything.

Stumbling into a sweep line algorithm's edge case

Wed, 10 Sep 2025 20:00:00 +0800

Mistakes that you should learn from, lesson 1..

I recently wrote some code for interval merging. Every leetcoder can probably recite 15 algorithms to do this in their sleep, but I figured it out myself in this case, and in the process fell into an edge case.

Here’s to writing down your mistakes.

Interval merging

The generic interval merging problem provides you with a list of start/stop pairs which denote individual intervals. Each interval is then to be merged with any other overlapping intervals; an overlap is defined by at least 1 element being shared between 2 intervals.

Some tips for integrating small bits of CUDA code into larger codebases

Wed, 13 Aug 2025 20:00:00 +0800

Some lessons from general C++ come in handy here..

In some recent work I integrated some CUDA code I developed into a larger existing codebase, written for the CPU. Essentially, my module(s) would accelerate and replace some existing functionality, but was only a small cog in the machine.

This is likely applicable to many others, so hopefully the lessons I document here will be concisely useful to those who chance upon this post.

Quickly sketching out a BlockQuickSelect

Sun, 20 Jul 2025 20:00:00 +0800

I wrote a blockwide quickselect. That’s the post.

As of today, NVIDIA’s cub library has a few block-wide primitives to do sorting - like BlockRadixSort - but none that do the equivalent for the $k$‘th order statistic i.e the $k$-th smallest element.

This would be functionally equivalent to the CPU’s std::nth_element, but that apparently uses Introselect, which is a tad too much for me to want to implement. I’m going to stick with the simpler quickselect.

Old dog, old C struct tricks

Mon, 14 Jul 2025 20:00:00 +0800

Look at this code snippet and tell me with a straight face you didn’t think it was a memory leak at first either..

A while ago I worked on some code which contained a container that looked like this:

struct S
{
 size_t sz;
 int data[1];

 void Set(size_t _sz, int *_data){
 sz = _sz;
 memcpy(data, _data, sizeof(int)*sz);
 }
}

If you’re a modern programmer like me, you might be looking at this and thinking: there’s no way this isn’t a flagrant memory violation in almost every case. Right? Well, it turns out this is something them old boys before C99 used to do.

Interesting Tidbits from GTC 2025: CUDA Graphs

Tue, 25 Mar 2025 20:00:00 +0800

In the 2nd post of this series, I give a short introduction on something that is also not particularly new, but new to me - CUDA Graphs!

As stated in my first post of this series, this topic isn’t particularly new per-se. Indeed, it looks like it has been out since 2019. But maybe CUDA graphs have increased in relevance now that GPUs are more powerful. As usual, a good starting reference is NVIDIA’s own blogpost.

Interesting Tidbits from GTC 2025: Asynchronicity Beyond Streams

Mon, 24 Mar 2025 20:00:00 +0800

Some notes for CUDA programmers who haven’t kept up with the times; this first post covers in-kernel pipelining..

I recently had the privilege of being sponsored to attend the March 2025 GTC in person. While the sessions were very largely dominated by AI-related things, I generally selected the CUDA-related ones, since they were more relevant to my work (and interests).

I’ll try to encapsulate some of the new things I learnt while I was there into a few major concepts. Note that most, if not all of these, are not shiny new CUDA features - some of them have been out for several years, but I just didn’t know about them, so I learnt about them there.

AtomicMinFloat; overloading integer-only atomics for floating-point numbers in CUDA

Sun, 16 Mar 2025 20:00:00 +0800

Convincing you (and myself) that with some minor edits, we can still use atomicMin for floats in CUDA..

CUDA has a set of atomic functions for safely updating the same memory address with different threads. There’s a whole list of them in the programming guide here.

However, for atomicMin (and also atomicMax), there isn’t an overload that works with floats (or doubles , but who uses those in CUDA anyway..). For this discussion we’ll just focus on atomicMin, but all the points we discuss can be inverted to explain atomicMax.

Gotta go (randomly) fast, thrust vs cuRAND

Mon, 10 Mar 2025 20:00:00 +0800

How fast can you generate (pseudo-)random numbers on the GPU? How fast can you generate (pseudo-)random numbers on the GPU?

I recently had to answer this question. Well, not exactly this question, but this was a key part of it.

A quick google search will bring up two common methods (libraries) when trying to do this in CUDA: thrust and cuRAND. Thrust is known to be a lot easier to set up; no need to write the kernel code and nitty gritty details, so I started with that.

When circumstances don’t allow you to use unique_ptr

Sun, 23 Feb 2025 20:00:00 +0800

Some background on my single header file memory manager and why I made it..

Picture this. You are working on a codebase with some ~~lazy~~ ~~inept~~ questionable decisions from external forces. You are given a class to work on. For the purposes of this post let’s call it MyClass.

The class will have its methods invoked in the following order:

MyClass() i.e. the constructor.
setup(…)
run(…)
teardown()
Repeat 2-4 many times.
~MyClass() i.e. the destructor.

This class will be reused in multiple different scenarios repeatedly, with different input data.

Cross-platform trigonometric SIMD and how the C ABI confused me

Thu, 30 Jan 2025 20:00:00 +0800

My small journey in discovering libmvec functions and the C problems I encountered..

I was recently working on making an existing Windows-only library of SIMD functions cross-platform i.e. making everything work in Linux. Here I’m going to highlight some problems I encountered in the process; hopefully it will help someone if they encounter something similar too.

The Windows (MSVC) function

Intrinsics are usually tied to compiler implementations. The one that gave me issues was a bunch of trigonometric functions. Let’s use the SSE version that works on floats; in MSVC this was implemented as a simple

Esoteric Errors: 'hidden symbol is referenced by DSO'

Fri, 18 Oct 2024 20:00:00 +0800

Hidden linkage isn’t something I’d encountered until now, so maybe this will help someone else too..

A new series of posts

I recently started work on a new codebase - one that is very large and has multiple moving components, from a frontend desktop UI (not my business) to the hardware interface (also not my business) and the backend processing (this one’s my business).

The standard build process had been setup on a remote server, and most people were ok with the process of connecting to it with tangible amounts of latency, but I was not. So my stubborn ass decided to figure out how to build it myself on a local machine. In the process, I encountered a ridiculous number of build errors; they were using custom Makefiles, but they had been generated by some other tool (possibly from Windows, even though it was being built in Linux, since there was a Windows VS solution as well).

Who knew a simple logger class would be this complicated?

Mon, 09 Sep 2024 20:00:00 +0800

Writing a printf-based C++ logger class was more of a journey than I originally thought..

I recently had to work with a codebase where the build process was so convoluted it couldn’t be run and debugged from within a Visual Studio instance, despite being a Visual Studio solution.

This was primarily because it had a bunch of Java components, which were mainly used for the UI, and that prevented it from being run as a standard C++ application within the debugger (or maybe you could? I couldn’t find a way..)

Some notes on 2D real-to-complex Fourier transforms

Mon, 15 Jul 2024 20:00:00 +0800

IPP in particular has some very niche ways of packing R2C DFT output, but otherwise there’s a few pointers here to keep in mind for how they are implemented in most libraries.

It’s pretty well known that the output of a Fourier transform of real inputs has symmetric properties. This is due to the fact that real waves consist of two conjugate pairs of complex exponentials.

What may be a bit less obvious (at least to me, when examining some programming libraries) is exactly how many useful output elements there are, and under which scenarios. In particular, I looked at IPP (which has some special packed structure), and cuFFT/NumPy/SciPy (which follow the FFTW structure I think).

CRTP, method chaining, and static polymorphism

Sat, 04 May 2024 20:00:00 +0800

Yes, it’s yet another blogpost about CRTP and how it’d be useful..

The 1st issue: method chaining

Have you seen a billion other blogposts about CRTP? Yes, so have I. But maybe there’s a reason for all of them; it wasn’t really apparent when reading them previously why it would be useful and/or why I would ever need it.

But recently, while writing some simple templated code for ufl, I had the bright idea of trying to make method chaining possible for the class. That’s when my templated class and its derived friend implementation started to fall apart. I fixed this by using CRTP for the first time.

Some heuristic proofs for cyclostationary methods

Tue, 16 Apr 2024 20:00:00 +0800

Carrier offset and baud rate estimation can be done blindly using cyclostationary (cyclic moment) methods, but why?

Introduction

In ReImage, there are options to blindly estimate a signal’s baud rate or residual carrier offset. These require some working knowledge, and only work on some types of modulations like PSK. But why do they work, and how do we explain the peaks we see in the resulting spectra?

First, some terminology. Here I’ll often mention the exponents in a CM XY form: this refers to applying an exponent onto a signal with a total of $X$, where $Y$ of that is the conjugate. These refer to the cyclic moments.

Getting rid of clangd's errors on Windows

Tue, 26 Mar 2024 20:00:00 +0800

My first steps into migrating to neovim, some clangd problems and my solutions..

Kickstart.nvim brought me here

I started using vim motions in VSCode a month or two ago, and decided to actually try to see if neovim would work for me. Honestly, I probably wouldn’t have gone down this path if kickstart.nvim didn’t exist. But it does, and I tried it, and it looks like I’m here to stay.

clangd as my first LSP

In kickstart.nvim’s template, clangd is listed as an example LSP, so I decided to uncomment that line and try it out. I opened my ffs repository at the time, and was immediately greeted with a flood of warnings on my code. This brings us to the first problem.

Clang and Eigen's alternatives to complex multiplication SIMD

Mon, 11 Mar 2024 20:00:00 +0800

Clang isn’t much better than MSVC for complex number multiplication, while Eigen is equivalent to GCC but uses slightly different instructions.

Complex Number Multiplication Is Probably Not Common Enough

I think I might have bashed MSVC too much on the last post; trying to vectorise our simple 4-element complex number multiplication using clang with AVX instructions delivers similarly poor results:

#include <complex>

void cmul(
 const std::complex<float>* __restrict__ x,
 const std::complex<float>* __restrict__ y,
 std::complex<float>* __restrict__ z
){
 for (int i = 0; i < 4; ++i)
 z[i] = x[i] * y[i];
}

void mul(
 const float* __restrict__ x,
 const float* __restrict__ y,
 float* __restrict__ z
){
 for (int i = 0; i < 8; ++i)
 z[i] = x[i] * y[i];
}

cmul(std::complex<float> const*, std::complex<float> const*, std::complex<float>*): # @cmul(std::complex<float> const*, std::complex<float> const*, std::complex<float>*)
 vmovsd xmm0, qword ptr [rdi] # xmm0 = mem[0],zero
 vmovsd xmm1, qword ptr [rsi] # xmm1 = mem[0],zero
 vbroadcastss xmm2, xmm0
 vmovshdup xmm0, xmm0 # xmm0 = xmm0[1,1,3,3]
 vshufps xmm3, xmm1, xmm1, 225 # xmm3 = xmm1[1,0,2,3]
 vmulps xmm0, xmm3, xmm0
 vfmaddsub231ps xmm0, xmm1, xmm2 # xmm0 = (xmm1 * xmm2) +/- xmm0
 vmovlps qword ptr [rdx], xmm0
 vmovsd xmm0, qword ptr [rdi + 8] # xmm0 = mem[0],zero
 vmovsd xmm1, qword ptr [rsi + 8] # xmm1 = mem[0],zero
 vbroadcastss xmm2, xmm0
 vmovshdup xmm0, xmm0 # xmm0 = xmm0[1,1,3,3]
 vshufps xmm3, xmm1, xmm1, 225 # xmm3 = xmm1[1,0,2,3]
 vmulps xmm0, xmm3, xmm0
 vfmaddsub231ps xmm0, xmm1, xmm2 # xmm0 = (xmm1 * xmm2) +/- xmm0
 vmovlps qword ptr [rdx + 8], xmm0
 vmovsd xmm0, qword ptr [rdi + 16] # xmm0 = mem[0],zero
 vmovsd xmm1, qword ptr [rsi + 16] # xmm1 = mem[0],zero
 vbroadcastss xmm2, xmm0
 vmovshdup xmm0, xmm0 # xmm0 = xmm0[1,1,3,3]
 vshufps xmm3, xmm1, xmm1, 225 # xmm3 = xmm1[1,0,2,3]
 vmulps xmm0, xmm3, xmm0
 vfmaddsub231ps xmm0, xmm1, xmm2 # xmm0 = (xmm1 * xmm2) +/- xmm0
 vmovlps qword ptr [rdx + 16], xmm0
 vmovsd xmm0, qword ptr [rdi + 24] # xmm0 = mem[0],zero
 vmovsd xmm1, qword ptr [rsi + 24] # xmm1 = mem[0],zero
 vbroadcastss xmm2, xmm0
 vmovshdup xmm0, xmm0 # xmm0 = xmm0[1,1,3,3]
 vshufps xmm3, xmm1, xmm1, 225 # xmm3 = xmm1[1,0,2,3]
 vmulps xmm0, xmm3, xmm0
 vfmaddsub231ps xmm0, xmm1, xmm2 # xmm0 = (xmm1 * xmm2) +/- xmm0
 vmovlps qword ptr [rdx + 24], xmm0
 ret
mul(float const*, float const*, float*): # @mul(float const*, float const*, float*)
 vmovups ymm0, ymmword ptr [rsi]
 vmulps ymm0, ymm0, ymmword ptr [rdi]
 vmovups ymmword ptr [rdx], ymm0
 vzeroupper
 ret

Compiled with clang 18.1.0, with -O3 -ffast-math -march=x86-64-v3, this still refuses to vectorise the cmul function correctly, only using the xmm registers. I included the normal real-valued float multiplication to check that clang is indeed able to vectorise that. Note that you still need the __restrict__ keywords for the vectorisation to work. Using -ffast-math doesn’t seem to do anything for us in the real float vectorisation, but it does make the complex-valued vectorisation less verbose.

MSVC's terrible auto-vectoriser for AVX

Sat, 24 Feb 2024 20:00:00 +0800

MSVC has extremely lackluster auto-vectorisation, so I handrolled intrinsic calls by backtranslating GCC’s output.

The Motivation

I recently decided I wanted to spend some time understanding intrinsics and SIMD at a deeper level.

In developing code for my project ffs, I wanted to make sure that the code was running with at least AVX instructions (because that’s my target architecture, and honestly very few computers don’t have AVX these days..).

This led me down a path of discovery; first I discovered the amazing-ness that is godbolt.org, then I joined their discord, where I then asked for some help with understanding basic .asm compiler output.

Getting IPP to work on non-Intel chips

Mon, 19 Feb 2024 20:00:00 +0800

Intel Performance Primitives is not guaranteed to work on non-Intel chips, but there are some ways around it..

Over the last year or so I’ve written a C++ wrapper of header-only templates around a library known as Intel Performance Primitives (IPP). I often use this for its signal processing library, which is - at least by my measurements - one of the fastest, if not the fastest one around. It also has the benefit of having almost everything I need in one place: FFT/DFTs, math array processing, low-pass filtering etc. You can see my templates at the ipp_ext repository.

Is this thing on?

Wed, 14 Feb 2024 20:00:00 +0800

Testing out this template..

Are we live?

Seems like we are. Playing around with next-js for like the 2nd time in my life here..