Sometimes, you just can’t fit everything in VRAM.

In most CUDA projects, the advice is usually to transfer all inputs, outputs, and temporary scratch space to the device. Working completely inside VRAM is fast, and avoids both the complexity and the throughput hit that comes with over-PCIe transfers being scattered throughout your hot path.

All of this is correct, and I adhere to this myself as much as possible. Sometimes, though, you simply just can’t do this, and recently I was left with no choice and had to start migrating stuff back to host memory.

Pinned memory and when to use resort to it

Do less, more times

Let’s start with what is not a good reason to leave everything in pinned memory. In many processing scenarios, you can get away with slicing or tiling your data in some way:

• Process in sections of rows or columns
• Process in tiles

If your processing is element-wise, then you can effectively cut the data using any method and then treat each sub-unit as its own problem. Invoke the kernel(s) per sub-unit, and hide the latency of the H2D/D2H transfers by having a separate stream(s). This is all standard CUDA at this point.

gantt
    title Tilewise CUDA Processing
    dateFormat x
    tickInterval 1millisecond
    axisFormat %L
    section Tile 1
        H2D                :active, h2d_1, 0, 5
        Processing kernel  :crit, k_1, 5, 15
        D2H                :active, d2h_1, 15, 20
    section Tile 2
        H2D                :active, h2d_2, 10, 15
        Processing kernel  :crit, k_2, 15, 25
        D2H                :active, d2h_2, 25, 30

Until the smallest unit is simply too big

However, when data dependencies across your array/image are non-trivial, then you’re out of luck here. Maybe your input element index is just completely unknown until runtime, so for all intents and purposes it’s a random lookup. You now have no way to know which row/column/tile to copy up for that particular kernel invocation. So you must have the entire thing available in global memory, query-able at all positions.

But what if your data is huge? 1000 x 1000 images is probably not a big deal, but maybe you’re doing precise processing using double precision on 10000x10000 images? Then suddenly this is 800MB of VRAM. 20000x20000 becomes 3.2GB. That’s just 1 array. What if you have 10 of these?

My point is that it’s not a completely remote possibility if you work on data of this size, especially if the VRAM also needs to be shared with other processes.

Random but uncommon lookups

In my case, my estimates for my arrays came out to be around 40GB in total. Not viable to hold all of them in VRAM. Okay, so then the first thing I thought was to transfer the data for each step to the GPU, only at that step, and then transfer it back down when finished. Remember, the whole point was that I was memory constrained, so I couldn’t have everything on the GPU at the same time.

But this turned out to be pretty bad in my case. Each step took a very short time to compute, and only operated on a few pixels at each step. These pixels were randomly distributed (so I couldn’t carve out a tile reliably) but they were also few and far between (so the total time for computation was small). I spent more time copying the array up and down than the kernel.

Duh, you should be doing more computations (more kernels) before copying it back down. Again, remember that I was memory constrained. I could not do this because the next step would need a different set of arrays, so I would need to clear the first set out from VRAM in order to operate on the next step. It looked something like this:

1. Uses arrays A and B. H2D A and B, invoke step1_kernel, D2H B because next step still uses A.
2. Uses arrays A and C. H2D C, invoke step2_kernel, D2H A and C.
3. Uses arrays B and D. H2D B and D, invoke step3_kernel, D2H B and D.
4. …

Mapped pinned memory to the rescue

It turns out that kernels can now (and have been able to for quite some time) execute directly on pinned host memory pointers. This comes with the caveat that the pinned host memory is mapped, but from my testing on several machines with different (albeit fairly new) CUDA versions, the typical cudaHostAlloc and cudaMallocHost calls will automatically do this. In fact, I could not even turn it off when I tried to.

How this works is the kernel will pull memory from the host over PCIe as and when it is required. If a thread reads a particular address, then the kernel will transfer that memory up to the GPU (with some standard cache-line shenanigans, different from the GPU’s global memory read transaction sizes, but that’s not the main point here).

This is great for my use-case, since I don’t touch the majority of my array. I only want it to read/write the few indices that actually get processed by the kernel. These are scattered around the image, so there’s no reasonable clean way for me to do this myself unless I invoke another kernel or some host-side function to cudaMemcpy only those addresses up. Unnecessary, and in my opinion, very messy.

Instead, I just write the kernel as per normal, wrap the pinned host pointer in a cudaHostGetDevicePointer() and let the hardware do its thing. It turned out to be surprisingly effective in my case.

The kernels really just work

There’s absolutely nothing different in the way you have to write the kernel (aside from doing weird things like trying to atomically modify a mapped pinned host memory pointer, but why would you even do that).

In fact, most of my kernels were written for device memory at the beginning; when I did my testing, I used small image dimensions so everything fit in VRAM. It was only later, after I realised I didn’t have enough VRAM, that I moved the allocations to mapped pinned host memory, and simply wrapped the host pointers:

// Original
mykernel<<<grid, blk>>>(d_a, d_b, d_c);

// After
mykernel<<<grid, blk>>>(d_a, d_b, cudaHostGetDevicePointer(h_c));

That’s the beauty of this, in my opinion. The kernels work seamlessly. I can mix-and-match device and host mapped memory without having to worry about it (other than performance).

All in all, another thing to add to the CUDA toolbox.

If you can manage minerals and gas, you can manage CUDA threads and registers.

CUDA is just resource management.

Okay, of course there’s a lot more to squeezing CUDA kernel performance than just resource management, but occupancy is very often the simplest thing to strive for, once you know how. And usually, if you’ve hit 100% occupancy, then unless you’ve done something heinous in your code, further optimizations are unlikely to budge your performance by large factors (aside from an entire algorithmic shift, but we won’t discuss that here).

Starcraft and unit compositions

For those who are too uncultured to have played Starcraft before, the goal of the game is essentially to construct a base, gather resources, and then build an assortment of units to destroy your enemy.

I’m going to use Protoss units and buildings here, because Protoss master race. Also because pylon sounds better than bunker(boring) and overlord(ew). Facts, not biased at all.

There’s a few things to keep in mind during a game:

• Supply: this is the maximum army count you can have. You increase this by building more pylons.
• Minerals: this is the base resource. It’s generally the more common resource, and you have more of it than gas.
• Vespene gas: this is the secondary resource. It’s harder to get, but is used to build more advanced units.

Imagine you have some limited amount of the above 3 at some point in the game. Your goal is to maximize the use of your resources. Let’s say you have

• 16 supply
• 400 gas
• 1500 minerals

We’ll choose from a small subset of possible units to make it simple:

You could say: I just want carriers. So you derp your way over to your Stargate and just build one… and then you run out of gas. Obviously, this is a terrible army composition. You still have a lot of supply - 6/16 - along with a ton of minerals and gas left over.

Alright, so then you just want to make sure you at least use all your supply. Simple way is to mineral dump into zealots, so you max out with 5 more zealots. Total consumption is 350 + 5 * 100 = 850 minerals, and 250 gas. Obviously, this is still leaving a lot on the table.

Consider getting 2 stalkers, a colossus, an immortal and a zealot. You now use 2 * 125 + 300 + 275 + 100 = 925 minerals, and all 400 gas. This is a far better use of all your resources.

What (theoretical) occupancy is

The idea in GPU kernel occupancy is basically the same. You have the following to play with:

• Number of threads per SM
• Number of blocks per SM
• Number of registers per SM
• Shared memory per SM

https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html#features-and-technical-specifications

Technically there’s a list of things, but honestly you can just focus on these few, because they’re the ones that come up the most often.

The easiest one to think about is number of threads per SM. Let’s consider a typical CC 86 card with 1536 threads per SM. The idea is that you essentially slot your blocks onto the SM one at a time, until the SM has insufficient resources to slot in the next block.

A simple example is the common novice GPU programmer gotcha of launching very large blocks; usually the maximum block size is 1024 threads. With 1536 threads per SM, the SM cannot fit in the second block after it already accepts one. SMs do not accept fractions of blocks. This essentially means that you might have 1/3 of the compute resources sitting unused; nsys will tell you your theoretical occupancy is 66%. If there is no strict reason for needing 1024 threads per block, then you would gain this compute back simply by lowering the block size.

Occupancy is thus a statement of how many threads you are using per SM.

The converse is also true. There is a maximum number of blocks that can be in flight per SM. That number usually corresponds to something less than 128 threads per block. This is one reason why many CUDA tutorials recommend starting with 128 thread blocks..

So for our CC 86 card with 1536 threads per SM that is 16 blocks. If you were to launch your kernel with 32-thread blocks, the SM will still only hold up to 16 blocks. Thus that means the total number of threads being handled per SM falls to a measly 16*32=512 threads, and your theoretical occupancy will rest at 33%.

Adjusting your CUDA army composition

In all the scenarios above, what we are really trying to do is to simply ensure the SM is fully saturated with threads. This is very easy to do with simple kernels; just change your block size to a number like 128 or 256, in most cases.

Two things to address here:

1. Why does this even matter? Performance. Internally, although the SM is likely not processing all its threads concurrently, using more threads here allows it to hide latency between memory, compute and whatever else. I agree that this isn’t a convincing argument, so the best way to know is to try it yourself whenever possible.
2. What if changing the block size isn’t doing anything? Then you are likely dealing with the next 2: registers and shared memory. We discuss this now.

The argument against fat kernels

You have probably heard of kernel fusion. The primary reason for this is to reduce global memory pressure by doing more work within one kernel; that way, you don’t have to read and write back to global memory at what used to be the beginning/end of 2 separate kernels.

Now, I would go so far as to claim that this is essentially true all of the time. You should default to this mindset when looking for optimization avenues; however, we should always remain cognizant of what we give up by doing this. This is why I tend to write small kernels and then merge them, rather than write long ones and split them later. It’s always far more maddening to try to disentangle a fat kernel.

So why fatshame kernels? Well, in general, the longer your kernel is, the more registers per thread it will use. The compiler does an excellent job of reducing and reusing registers, but it isn’t perfect. It is unlikely that any non-trivial kernel will stay below 20 registers per thread.

The story here is the same as that of the thread count. If we exceed the total number of registers per SM, the SM will cut blocks until it fits. Consider the following configuration:

• Threads per block: 128
• Registers per thread: 50
• Maximum registers per SM: 65536
• Maximum threads per SM: 1536
• Maximum blocks per SM: 16

As we saw earlier, under normal circumstances, you would easily fit 12 blocks ($12 \times 128 = 1536$) worth of threads on a single SM. But now the SM needs to ensure all the registers your blocks need are available. This comes out to $128 \times 50 \times 12 = 76800$, which exceeds the 65536 cap. So what happens? The SM simply refuses to house blocks that would exceed its registers, so it really only houses $65536/(128\times50) = 10.24$ blocks. No such thing as a fractional block, so we get 10 blocks in flight per SM. That’s a theoretical occupancy of $\frac{10}{16} = 60\%$.

In fact, the problem is actually worse than this. CUDA actually allocates registers at a warp granularity. In most cases, this is 256 registers per warp (or 8 registers per thread). What this means is you get breakpoints; 40 registers per thread and 41 registers per thread is a gigantic jump, since 41 registers per thread is effectively ’the same as 48 registers per thread'.

The same goes for shared memory - static and dynamic alike. You may have heard that the maximum shared memory per thread block is 64KB (at least by default, yes I know you can increase it nowadays, but the point still stands). You might then think it’s ok to use as much of it as possible for each thread block; after all, we get to exploit the fast cache-like memory bandwidth right? Not quite. You have essentially the same maximum shared memory per block as maximum shared memory per SM. So if you use all 64KB of shared memory for 1 block, then the SM will have capacity to hold only that one block. If your block size is 128 threads, then your theoretical occupancy is tanking all the way down to $\frac{128}{1536}$ (or a similar calculation for your compute capability). That is an 8% to be ashamed of.

Occupancy isn’t everything

After reading all this, it might sound like occupancy is the only thing you should care about. But in reality, this is usually not the case. Kernels are complicated, and just because you increase occupancy does not guarantee better performance.

The tradeoffs are the same:

• If you cut registers you usually end up doing more computations
• If you cut shared memory you either store/load more from global memory, or you use more registers

And these are just the technical reasons. You also often lose code clarity if you force your kernel to do awkward computations just to reduce some shared memory workspace or registers. Sadly, there is no magic bullet here; at the end of the day the correct thing to do is to profile your choices.

Please profile your damn choices. With AI nowadays there’s no reason not to try a bajillion different kernel configurations and just pick the best one.

At least, most of the time it is. As long as it fits the 1-2-1 pattern..

Many segments can be boiled down to what I call a 1-2-1 pattern. For an array of input elements (I’ll just call them pixels, it’s shorter for me to type), a naive segment detection is to loop forwards until you hit an activation pixel. This can internally be any condition you’d like; that is, any if (...) is possible here as long as you operate only on the pixel you’re currently at. For simplicity, I’ll assume this is the 2 pixel:

. . . . . 2
---------> activated

The activation pixel is what defines us to be within a segment. This can happen several times within a segment, but it doesn’t matter (activating an already active segment does nothing):

. . . . . 2 2
-----------> still same segment

We now seek the boundaries of the segment, $[a,b]$ i.e. the (inclusive) pixels on the left and right where the segment is deemed to start/end. A naive next step is then to iterate from the first activation pixel leftwards and rightwards to look for the boundary conditions. Again, this can be defined by any if(...) statement, but here we will simplify it to be a 1 pixel:

. . 1 . . 2 2 . . 1 .
     <---- ------>

In this simplified scheme, anything other than 1 and 2 for the pixels are irrelevant (hence the 1-2-1 naming).

Note that under the above definition, a simple 1-1 boundary does not denote a segment, as it lacks an activation 2 pixel:

. . 1 . . . . 1
This is not a segment

Scan when there are many

Now the astute leetcoder will probably realise that this is terribly inefficient. If you have multiple segments, you will end up doing a lot of backtracking. The fundamental idea of using a scan here is to remember the last boundary. In this case, we also need to remember if we are in an active segment.

For the uninformed, a prefix scan is essentially a rolling operation with a binary functor. That is, you define an operation taking in two values - before, after - and return a new value. This new value is used in the next operation (instead of the original input). The most common variant is arguably a prefix sum, otherwise known as a cumulative sum or cumsum in many languages.

This easily boils down to a few simple operations:

• Reset: whenever we see a 1, it means the current segment (active or not) has ended, and this is also the start of a new segment.
• Extend: in any other case, we simply copy the index we currently have. Importantly, we also need to keep track of the last 2 we saw, so we know at the end of a segment whether it was activated or not.

In practice, we can couple the incrementing index with the original pixels, and perform the scan in the following way:

 In 0 1 0 1 2 0 1 0
Idx 0 1 2 3 4 5 6 7

Sdx - 1 - 3 - - 6 -

Here, Sdx is simply a transform of the original incrementing index: set to -1 if it’s not a boundary 1, otherwise keep the original index. Then, our scan operation looks something like this:

FlagAndIndex operator()(
	FlagAndIndex a,
	FlagAndIndex b
){
	if (b.flag == 1) // reset
		return b; // we 'forget' everything before
	else // extend
		// copy ('drag') the earlier index,
		// take the larger flag (so we keep 'activations')
		return {max(a.flag, b.flag), a.idx};
}

The scan then results in this:

 In 0 1 0 1 2 0 1 0
Sdx - 1 - 3 - - 6 -

Out 0 1 1 1 2 2 1 1
Odx - 1 1 3 3 3 6 6

We can walk through this simple example:

1. Index 0. We are performing an inclusive scan so the first input is just copied. No-op.
2. We hit the first 1 at index 1. This hits our Reset condition so output becomes the current index i.e., 1, and we use the input flag 1.
3. Index 2: Extend. Simply copy the same value from before; max(flagA, flagB) is still just 1.
4. Index 3 has hit our Reset again. We hence update to the current index, 3, and flag, 1.
5. Index 4: Extend. This hits the first activation. The index is dragged up (still 3) but now the flag after the max() operation is 2.
6. Index 5 is again an Extend. Everything is copied, flag is still 2.
7. Index 6 hits a Reset. The output index is updated to 6, and our flag is reset to 1.

Hopefully this toy example makes it clear how a simple custom functor with a few operations maintains the one-pass scan.

Okay, but where’s the bounds?

Readouts should now be pretty obvious. Note that you can actually do this during the scan, but for now I will just explain it with another pass.

We simply need to read the inputs for 1s, and then check the activation flag one step on the left. If it’s true, the segment on the left is valid and you can then read the scan’s output index, also one step on the left, to get the segment’s starting index. Otherwise, it’s invalid and you can skip it.

For the above example, we have three 1s:

1. Index 1. Index 0’s flag is 0, so skip.
2. Index 3. Index 2’s flag is 1, so skip again.
3. Index 6. Index 5’s flag is 2 (active), so read the scan output index: 3. Segment bounds are $[3,6]$.

CUDA really favours this

If you’ve read any of my other posts, you’ll know I really like CUB. CUB’s scans - like DeviceScan::InclusiveScan - are all really efficient, primarily because they do the scan via a tree. The result is that this is usually faster than anything I (you) will ever write, so we should just use it.

However, in my opinion, there’s another more compelling reason to use a scan in this scenario: simplicity. Consider the original naive method; find a 2, find surrounding 1s, rinse and repeat.

A blind conversion of this into CUDA would probably look something like:

1. Launch kernel to cover all pixels; simple grid-stride.
2. At every pixel, check if it’s a 2.
3. If 2, search outwards for the 1s.

To do this search in step 3 from the kernel, you have a few options, all of which are terrible for occupancy and/or simplicity:

• Use the thread that found the 2. If you think this is acceptable Jensen Huang will personally strangle you with his leather jacket.
• Use the warp/block. Still unacceptable. Also complicates things because you will have to somehow gather the candidates and then iterate on them cooperatively.
• Use a new kernel. Or use CDP from the thread that found the 2 (I will silently judge you for this).

Maybe I’m being subjective here, but all of these pale in simplicity and efficiency to the scan provided by CUB; do it once, and use all of the GPU’s resources at all (kinda) times. We trade time spent implementing for time spent thinking, and get performance in the process.

One extra requirement

Not all is fine and dandy with CUDA’s scans though. They impose one further constraint: associativity.

For those who have forgotten your math, this implies the following:

This is elegant to look at, but easy to stumble over. In the 1-2-1 scan I just described, it is possible to write a binary functor that does not obey this. By that I actually mean that the functor written above is a perfect example of one that violates associativity.

To see why, consider the following example, with the transform applied before the scan:

 In 0 1 2
Sdx - 1 -

We can define these 3 scan-pairs - (flag, index) - mathematically as

We can test exactly what the previous functor did on these 3 points pair-wise

and then test the associativity

Thus, in this case, the two are not equal, proving that the binary functor as defined above is actually not associative. This means that it is not guaranteed to produce the correct (intended) results when using cub::DeviceScan.

But what actually happened here? The fact is that using the flag to decide between the if/else paths for the functor in this case breaks associativity, because it gets mutated in one of the paths.

In the extend path, we perform the max operation and keep the larger value. In this case, that changes a 1 flag to a 2, and that switches the path from the if(reset) to the else(extend).

The solution is hence to ensure that this does not happen, by using some other method to decide between the reset/extend paths:

FlagAndIndex operator()(
	FlagAndIndex a,
	FlagAndIndex b
){
	// our transform changed all non-reset indices to 
	// negative values, so we can use the index as
	// the decider instead
	if (b.idx >= 0) // reset
		return b; // we 'forget' everything before
	else // extend
		// copy ('drag') the earlier index,
		// take the larger flag (so we keep 'activations')
		return {max(a.flag, b.flag), a.idx};
}

With this, associativity is restored! And you won’t get a nasty surprise when some scans suddenly don’t work the way you expected.

Behaviour of printf on device is not the same as on host!

If you’re a printf aficionado like me, then you use printf for debugging. A lot. In fact I previously wrote a small logger that uses printf called spfLogger. I do enjoy the flexibility of tuning every single width/precision with a few characters, and it’s something I haven’t yet seen C++ be able to emulate with as little irritation.

I haven’t gotten around to writing code using newer std::format or println yet, so the jury’s out on that.

But that’s not the topic today. I just want to highlight something that can happen when you use printf in CUDA kernels, which is what I do very often.

A simple example

Consider the following kernel and its equivalent host function:

template <typename T, typename U>
__global__ void badprintfkernel(const T *a, const U *b) {
  // i do 4 prints here, you'll see why below
  printf("device: a = %d, b = %d, a = %d, a = %d\n", *a, *b, *a, *a);
  printf("device: a = %d, b (with ld) = %ld, a = %d\n", *a, *b, *a);
}

template <typename T, typename U> void printfhost(const T *a, const U *b) {
  printf("host: a = %d, b = %d, a = %d\n", *a, *b, *a);
  printf("host: a = %d, b (with ld) = %ld, a = %d\n", *a, *b, *a);
}

We invoke the kernel with 1 thread just to see the prints, and also the host function:

thrust::device_vector<int32_t> d_32(1);
thrust::fill(d_32.begin(), d_32.end(), 0xFF332211);
thrust::device_vector<int64_t> d_64(1);
thrust::fill(d_64.begin(), d_64.end(), 0xFF332211112233FF);

thrust::host_vector<int32_t> h_32 = d_32;
thrust::host_vector<int64_t> h_64 = d_64;
badprintfkernel<<<1, 1>>>(d_32.data().get(), d_64.data().get());
printfhost(h_32.data(), h_64.data());

What do you expect to see?

Not just undefined behaviour, but unexpected behaviour

This is what gets printed on the host:

host: a = -13426159, b = 287454207, a = -13426159
host: a = -13426159, b (with ld) = -57664913528441857, a = -13426159

Now this is what gets printed on the device:

device: a = -13426159, b = 0, a = 287454207, a = -13426159
device: a = -13426159, b (with ld) = -57664913528441857, a = -13426159

For reference, 0xFF332211 is -13426159 whereas 0xFF332211112233FF is -57664913528441857.

So 2 things have happened on the device:

1. For int64_t, the ‘cast’ seems to have just completely bugged out, as it simply prints 0.
2. It appears to have corrupted the subsequent argument’s formatting. We see what we would expect to see (287454207) for b in the 3rd argument which prints a. Only on the 4th argument which prints a again is the printed value correct.

Now, it is understood that printf’s behaviour is technically undefined when the format specifier is invalid i.e. using %d for a 32-bit integer. However, in the CPU code, you can clearly see that what it prints is the least significant 32-bits of the number; 287454207 is 0x112233FF, the lower 32-bits of the 64-bit number used as the input. This is in fact, what I have come to expect, which is why it can catch me off-guard when I don’t pay attention to my printfs in kernels.

Usually, especially during development and basic unit tests, I would instantiate small numbers to test my kernels, but may possibly swap the types around to ensure things work (or to time the kernel with different types).

This quirk of printf in the kernel means that if I print some number the wrong way (format specifier), even without any risk of overflow, the number will be 0 (or gibberish - at least I think I’ve encountered gibberish before).

For example, if the result of some number is 5, since 5 is smaller than 32-bits, it would print the same way in CPU code whether I used %d or %ld or %lld, since the bits 32-63 don’t matter anyway.

Even worse, it will mess up other numbers in the same print statement, confusing me further, since it’ll start to make me second guess the calculations involving the other numbers I am printing.

All in all, this is just a short post to warn CUDA printf users that you should be very careful when interpreting the results of your print statements, especially in templated scenarios where it is (or may be) difficult to change the format specifier based on the templated types. Refer to the official docs for other limitations, like the maximum of 32 printed arguments (which I have discovered for myself before).

Because not everything can be made un-iterative..

There’s a pretty large class of algorithms that have been developed for CPUs that involve, optimally, an iterative solver.

Usually, this is defined by some notion of convergence; some variable is updated, and then at the end of each iteration it is checked to determine whether further iterations are required.

This is all fine and dandy in CPU-land, but in GPUs this almost always makes the flow awkward.

The copy-back

The easiest way to do this is usually a memcpy back to the host. We can make this as efficient as possible by of course making the memory pinned (it’s usually just a single variable after all).

So the flow would be something like

1. Launch iteration kernel.
2. Pull check back to host pinned memory.
3. Decide if iterations should continue.
4. Return to 1, or break.

If your iteration kernel takes fairly long, the pull back and CPU-side decision making probably be insignificant in comparison, so this method is actually decent.

The cooperative grid

However, there is another way to do this; if anything, the aspiring CUDA practitioner should know of all the ways, just for the sake of it.

This is via in-kernel grid synchronization. In essence, this is just another flavour of cooperative groups, but grid-wide instead of the more commonly known block-wide groups.

The idea is pretty simple, as shown in the link; slap on a grid.sync() at the end of a single iteration, and it would be just like the kernel had returned.

So the order of operations inside a cooperative kernel would now look like

1. Declare the cooperative grid.
2. Place all your previous iteration kernel’s code into a while or for loop, depending on whether you want to limit the maximum number of iterations (always a good idea).
3. At the end of the iteration while/for loop, issue a grid.sync().
4. Now the line after this is guaranteed to have every thread in the grid arrive at the same time; hence all threads of the kernel can read the check variable together, without any race conditions.
5. Just like before, each thread will decide if the iterations should continue (and they will all make the same decision, by definition).
6. Break and exit, or go again.

Now there is a major caveat to this, as also shown in the link: you cannot allocate a grid larger than the device (all SMs) can support at any given point. Ideally, this shouldn’t matter too much, since if every thread in every SM is doing work then you aren’t really going to get extra performance out of using a larger grid.

But ideal isn’t the world we live in, and it very likely isn’t the world your code lives in either. In reality, some threads may be doing nothing due to conditionals, and/or having more blocks waiting could hide some latency. This would imply that using more blocks (like allocating enough to cover all your data) can also be beneficial.

As always, profile your own kernel; for my recent use-case, I found that it could be slower in some scenarios and similar in others (but never faster, at least not tangibly).

Not complicated, but always good to know the process.

In this series (?) I’ll cover something I always wanted to write down: compiling the toolchains/compilers I use from source. This means, specifically, no using of package managers i.e. not allowed to sudo apt install dependencies. Of course, this has its limits (you need a compiler to compile gcc, and some OSes don’t come pre-installed with one). However, the premise here is to at least be familiar with the minimal required dependencies, and where to get them (at the time of writing), and then what to do to build everything.

While this seems like a pointless exercise, doing this at least once comes in handy when you have to

• install something without sudo rights (because let me compile things in my user space if I want to damn it)
• install something on an offline machine
• install a version that doesn’t (easily) exist on your system’s package manager (because of reasons like an outdated OS, which happens way more often than you think on workplace servers)
• or any other equivalent reason

For the above reasons, none of the steps described in this post or later posts will require administrator privileges; everything will be installed into your home directory (and with that you will have to set things up yourself after).

We begin, as we should, with the granddaddy of all tools: gcc.

Installing gcc-11.5 on Ubuntu 22.04

The default repositories for Ubuntu 22.04 do not contain gcc-11.5, so we begin by reading its documentation.

Here we are presented with the essential dependencies:

• GMP
• [MPFR](https://www.mpfr.org
• [MPC](https://www.multiprecision.org

After downloading all of them, we must install them in the above order. This is because they are recursively dependent i.e MPC depends on MPFR which depends on GMP.

A primer on autoconf-style builds

Feel free to skip this if you’re already familiar with it.

Every tool that is linux-based will almost certainly include an autoconf script. This comes in the form of a configurescript.

For most scenarios, running it without any extra options would suffice. Hence, other than some variations with extra steps like for testing, almost everything will build via

./configure
make
sudo make install

What each line does, very briefly, is

1. Configures the build and generates (optimized) makefiles. This usually exists to somewhat tailor the compilation to your system and/or detect missing dependencies.
2. Compiles the thing.
3. Installs the thing; read: this means it copies the compiled executables/libraries to the designated path, which by default is a system path like /usr/local, hence the requirement that you do it with sudo. It may also help you with setting up reasonable symlinks.

A bit of guidance

While the instructions on the gcc site are pretty comprehensive, I’ll tailor a few steps here to make sure we can build and use gcc without sudo.

1. Install GMP to a specific subdirectory in home. Example:

export GMPDIR=/home/username/gmp
tar -xzvf gmp-...tar.gz
cd gmp-...
./configure --prefix=$GMPDIR
make
make install

2. Install MPFR to a specific subdirectory in home, pointing to our GMP install prefix. Example:

export MPFRDIR=/home/username/mpfr
tar -xzvf mpfr-...tar.gz
cd mpfr-...
./configure --prefix=$MPFRDIR --with-gmp=$GMPDIR
make
make install

3. Install MPC to a specific subdirectory in home, pointing to the two previous install suffixes. Example:

export MPCDIR=/home/username/mpc
tar -xzvf mpc-...tar.gz
cd mpc-...
./configure --prefix=$MPCDIR --with-gmp=$GMPDIR --with-mpfr=$MPFRDIR
make
make install

4. Now we can build gcc using these. Somehow, I wrote down that I had to define the library paths explicitly before this, so I’ll include it here as well:

export LD_LIBRARY_PATH=$GMPDIR/lib:$MPFRDIR/lib:$MPCDIR/lib:$LD_LIBRARY_PATH
# gcc recommends you to
../configure --prefix=/opt/gcc-11.5.0 --with-gmp=$GMPDIR --with-mpfr=$MPFRDIR --with-mpc=$MPCDIR

5. After this it’s simply building and installing as per usual:

make # with -j whatever if you want
make install

6. The final step is to make sure it runs (instead of your system’s default version):

export PATH=/opt/gcc-11.5.0/bin:$PATH
export LD_LIBRARY_PATH=/opt/gcc-11.5.0/lib:$LD_LIBRARY_PATH
# I usually chuck these into my ~/.bashrc
# and disable it if i want to switch back to the system version.
# Kinda like a manual way of handling alternatives.

And that’s all.

Mistakes that you should learn from, lesson 1..

I recently wrote some code for interval merging. Every leetcoder can probably recite 15 algorithms to do this in their sleep, but I figured it out myself in this case, and in the process fell into an edge case.

Here’s to writing down your mistakes.

Interval merging

The generic interval merging problem provides you with a list of start/stop pairs which denote individual intervals. Each interval is then to be merged with any other overlapping intervals; an overlap is defined by at least 1 element being shared between 2 intervals.

There are several variants of this problem, depending on how the data structure of the input pairs, but I’ll focus on the one I ended up with:

struct Event {
  size_t idx;
  uint8_t flag; // 1: start, 0: stop
}

Always be sorting

The first step in all interval merge solutions is to sort the intervals. This is, of course, to put adjacent and possibly overlapping intervals next to each other so they can be compared over a single pass.

In my case, the disparate Events need to be sorted, so this is as simple as using std::sort with a defined comparison operator:

bool operator<(const Event& rhs) {
  return idx < rhs.idx;
}

Energy states determine closure

The sweep line algorithm now essentially sweeps through the sorted list of on (1) and off (0) events. At each event, it does the following:

1. If the current counter is 0 and the event flag is 1, add a new open interval with an unknown ending.
2. If the current counter is more than 0 and the event flag is 1, simply increment the counter.
3. If the event flag is 0, decrement the counter.
4. If the counter reaches 0, close the current interval.

# Evolution of counter over events
X-----X  X-----X
|   X-------X  |
|   | |  |  |  |
1   2 1  2  1  0

The physicist in me likes to see this as going up and down energy states (no? just me?).

The mistake (and the fix)

There is one special case that causes this to fall apart, and it has to do with the way we defined (or failed to properly define) the sorting order.

Because we deal only with events, and the above comparison operator only checks the index, there is no guarantee of whether an on event will come before an off event if both of them have the same index, and this is specifically what we require.

To see why, simply consider the below scenario, which should result in a single merged interval.

X----X
     X-----X

We necessarily have the following events:

• Index $a$, flag 1
• Index $b$, flag 0
• Index $b$, flag 1
• Index $c$, flag 0

Ideally, after sorting, due to the counter mechanism outlined above, we would need the two events with the flag 1 to be next to each other. This would result in the counter changing as $1 \rightarrow 2 \rightarrow 1 \rightarrow 0$ as intended, merging the two intervals.

However, since we did not check for this, the above order may remain unchanged, causing the counter to do $1 \rightarrow 0$ twice, leaving the two intervals separated, with an illogical start/stop on the same index.

In fact, this may result in very odd size-less intervals when there more overlaps occur. Consider the following 4 intervals:

X-----X
  X---X
      X------X
      X----X

If we just focus on the 4-way overlap in the centre, we have two 1s and two 0s as possible event flags. In the worst case, this may be ordered as $0,1,0,1$, which would result in an interval that starts and stops on the same index (from the centre 1 and 0).

Edit: I previously thought the above would result in a 0 size interval, but on second thought I was wrong.

The fix is then, of course, to use flags to sort when indices match:

bool operator<(const Event& rhs) {
  if (idx == rhs.idx)
    return idx.flag > rhs.flag; // 1s come before 0s
  else
    return idx < rhs.idx;
}

Some lessons from general C++ come in handy here..

In some recent work I integrated some CUDA code I developed into a larger existing codebase, written for the CPU. Essentially, my module(s) would accelerate and replace some existing functionality, but was only a small cog in the machine.

This is likely applicable to many others, so hopefully the lessons I document here will be concisely useful to those who chance upon this post.

The obvious bits

During my kernel development, I pulled out the necessary functionality and experimented on it in a small standalone project, like any sane person. Like most of my personal projects, this lets me use CMake, but the larger codebase was built on (mostly handmade) raw Makefiles.

Inspecting compile_commands.json produced by CMake will reveal two common compiler options used for all CUDA code:

1. -forward-unknown-to-host-compiler, which does what it says on the tin. You can also use the lazier form of this, -forward-unknown-opts, though I don’t usually bother until linker errors start showing up.
2. -x cu for every source file. This isn’t technically required for .cu files since nvcc will recognise them as CUDA files, but becomes important when you place your eventual code into existing .cpp files. This ensures they are compiled as CUDA.

Relocatable device code requirements

The codebase is also partitioned into many separate distinct libraries, compiled individually and then linked together at the end to create one final executable. Of course, my standalone project with its multiple small individual executables and unit tests didn’t need any extra compiler flags, but integrating the code as it was made nvcc cry at me.

This may have been partly due to the structure in the codebase itself, but the solution was fairly simple:

Add -dc to the nvcc’s compiler flags, turning on relocatable device code with the expectation of linkage later.

We will be returning to the docs for nvcc often; get used to it, because that’s how I (you) will figure out what flags you need to add.

Macro pollution

This compiler error is strictly due to existing code in the existing codebase, so you may not experience anything similar. In fact, it is not strictly a CUDA related issue, but could occur with any code.

The observed error was something like this, upon inclusion of anything that used cub, either directly or indirectly, like via thrust headers:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin/../include\cub/warp/specializations/warp_scan_shfl.cuh(408): error: identifier "input" is undefined
    __declspec(__device__) __forceinline  InclusiveScanStep( input, ScanOpT scan_op, int first_lane, int offset)    
                                                             ^

Note that this never occurred in my standalone project, so it was odd that these headers would suddenly cause problems here.

As mentioned above, this isn’t specifically a CUDA issue, but I’ll show how I investigated it in the context of using nvcc.

Read the preprocessor output

We can dump all intermediate output from nvcc via -keep. This lets you see, among many other things, the preprocessor output in a file usually suffixed with .ii.

Opening it in my case showed that the input argument’s template argument, _T, had somehow been stripped away.

// note: not the exact same function as the above one, but obvious that the first template argument is gone
template <typename , typename ScanOpT>
__declspec(__device__) __forceinline 
InclusiveScanStep( input, ScanOpT scan_op, int first_lane, int offset, ::cuda::std::true_type )
{
  return InclusiveScanStep(input, scan_op, first_lane, offset);
}

This smells of macro replacement, but where is it coming from? Searching the original preprocessor output file for #define _T didn’t seem to yield anything.

I had to enable the gcc-specific option -dD in order to enable the define directives. Then I was able to track down an include file several levels up which happened to do exactly #define _T. Welp, there goes all the cub headers I guess.

A simple #undef _T in my translation unit and I was back on track.

Hiding CUDA from the rest of the codebase

Congrats for making it this far down the post. Below lies the meat of the problems I had. Almost all my issues stemmed from having to do this, simply because it would be too gargantuan a task to refactor everything else.

I’ve already mentioned above that compiling .cpp files as CUDA requires the nvcc option -x cu; in fact, this is what CMake does overtly, even for .cu files, once you use

set_source_files_properties(sourcefile.cu PROPERTIES LANGUAGE CUDA)

as seen in many CUDA examples.

Why you should specify source file compilation explicitly

Aside from the fact that your code probably won’t compile, let’s go into exactly why that actually happens, and why using your host compiler doesn’t work:

1. Device code will not be parsable. This should be obvious, but it is important to note that this will also happen if you have a header-only template calling a kernel, having some __device__ function, or any other CUDA-specific syntax. This will become important later.
2. CUDA headers and libraries will not get automatically found. Now this is arguably fixable when the project is small by explicitly adding CUDA header and library paths to the Makefiles (or equivalent), but this quickly gets unwieldy once the project starts getting larger. There is also no guarantee that all the CUDA headers will just work; some may include some __device__ code, for example.
3. Hyper-aggressive inlining of code. This is, of course, preferable in most instances of device code, but can cause issues in some CPU-side code sometimes. Fairly rare, but this was observed for one file in my case.

The above probably isn’t a definitive list, but hopefully it’s convincing enough. They are all underscored by a general software idea though: separation of concerns. We shouldn’t use nvcc for things unless strictly necessary.

But including CUDA headers implies parsing them

Here’s the problem. You’ve already developed some CUDA code: some kernels, and some encapsulated classes to help call them and/or manage some device-side states and workspaces. Perhaps you also want some of the external code to hold on to some memory through thrust::host_vector or thrust::device_vector.

Normally, it wouldn’t matter much (unless you are extremely pedantic) where you included headers like thrust/device_vector.h. However, with our nvcc-specific constraints, this becomes a bit more nuanced.

You can no longer make these includes in the headers of your external code, or else all other translation units which transitively include the header will need to be compiled by nvcc.

This is probably not a good idea, even if you think you are okay with the maintenance.

A transitive inclusion example

I can give a MWO of my scenario. Consider the following implementation for a class in the larger codebase:

// codebaseclass.h
#include <thrust/device_vector.h>

class CodebaseClass
{
// ...

private:
  thrust::device_vector<int> m_vec;
}

// codebaseclass.cpp
#include "codebaseclass.h"
// all the implementation details for CodebaseClass in here
// ...

Now we set our build system to nvcc for codebaseclass.cpp, and we should be good to go, right?

No, but you forgot that somewhere else in the codebase, something else uses this header:

// othercodebase.h

#include "codebaseclass.h"

// ...

// othercodebase.cpp
#include "othercodebase.h"

// ...

Now when the compiler sees the translation unit for othercodebase.cpp, it pulls in codebaseclass.h as well, and doesn’t know how to deal with the thrust vector.

pImpl-ing our way through

The pImpl pattern isn’t new, nor is it technically meant for purposes like this; it is meant to hide functionality from a client interface. Here, we’re using it to hide the functionality from our compilers.

The idea is to simply forward declare our CUDA-classes (including any that we have defined that use CUDA things), and only store pointers to our objects, rather than the objects themselves.

// codebaseclass.h

// forward declaration
namespace thrust{
  template <typename T>
  class device_vector<T>;
}

class CodebaseClass{
public:
  CodebaseClass();
  ~CodebaseClass();
private:
  std::unique_ptr<thrust::device_vector> m_vec;
}

// codebaseclass.cpp
#include "codebaseclass.h"

#include <thrust/device_vector.h>

// ... use the device vector as per normal, just that it's via a pointer

Now, transitively including codebaseclass.h into othercodebase.h does not cause problems, because a pointer is simply an address to the compiler. It expects that if you actually need to use it, you will link it in yourself later.

Yes, technically pImpl is a lot more than this, and usually involves a class-internal forward declaration which gets inherited from. We don’t really need all that here, and pImpl-ing has a nicer ring to it than ‘forward-declaring’.

Why this works, and when it doesn’t

This goes back to compiler fundamentals, and essentially boils down to what information the compiler needs when a full type is specified.

When the entire class object is a direct member of another class, like I wrote originally, the compiler must know how to construct and destroy it. This is because its inclusion as a member variable implies that it must be default constructed. Without the class header/implementation like thrust/device_vector.h in this case, it will be unable to do this.

Similarly, you wouldn’t be able to call any methods directly without first having its definition e.g. having an inline function that calls m_vec->reserve(). In my above example, we just store the pointer to it, which is basically saying “store an address first, I’ll tell you what it does later, don’t worry about it”.

Now there are a few additional things to note when we implement this pattern. Notice how I did not define the destructor inline within the header. This isn’t an arbitrary decision; the code will not compile if we had done so i.e. written the innocuous-looking

~CodebaseClass(){}

even if we were going to do this anyway in the source file!

Why does this happen? Because the compiler will emit full code to destroy all objects at the point where the destructor is defined. So if you define it inline in the header, it must know how to destroy thrust::device_vector, and since we only forward declared it, it will not know how to.

The two options in this case are to:

1. Define the destructor out-of-line i.e. in the associated .cpp, by which point the compiler should have full knowledge of the thrust::device_vector type. This is particularly important for polymorphic classes since the compiler does not auto-generate a virtual destructor, so you must always declare it in the header at least, but not define it there.
2. Simply not defining it at all. The compiler will emit it after it has the full definition of the encompassing class, which occurs in the .cpp, same as the above.

Note that the second version is not equivalent to writing ~CodebaseClass() = default in the header, as the compiler will then attempt to emit it in the header again, and the same problem will arise. You can, however, declare it in the header and then write ~CodebaseClass() = default; in the source .cpp file.

If you really want the full pImpl experience..

Then you can use constructor inheritance to get you most of the way there. In these scenarios you would actually forward declare a Impl class or struct for whatever it is you want to hide.

A lot of the time, you don’t really need to care about the constructor for this hidden implementation, and you just want to use your hidden class’s constructor directly. Then something like this in the .cpp would work:

struct Impl : public RealClass{
  using RealClass::RealClass; // this inherits all the constructors
  // ...
}

Again, this isn’t a strictly CUDA thing, but it’s useful to remember it in this context.

I wrote a blockwide quickselect. That’s the post.

As of today, NVIDIA’s cub library has a few block-wide primitives to do sorting - like BlockRadixSort - but none that do the equivalent for the $k$‘th order statistic i.e the $k$-th smallest element.

This would be functionally equivalent to the CPU’s std::nth_element, but that apparently uses Introselect, which is a tad too much for me to want to implement. I’m going to stick with the simpler quickselect.

What we need

Similar to cub’s block-wide primitives, we probably want to just get some shared memory and work on the data inside it. I’m at a crossroads here: using statically allocated shared memory like cub via templates makes it easy to implement, but doesn’t let you handle differing sizes per block (at least not explicitly).

But dynamic allocation may take a bit more time to implement?

Meh, let’s just go with static for now. I ended up doing some half baked static and dynamic nonsense, but whatever.

The flow

1. Load block’s array from global to shared memory.
2. Sync threads.
3. Select pivot; just use the first element(?)
4. All threads other than pivot compare against pivot.
5. Warp-wide counters for < pivot and > pivot get warp-aggregated atomically added to the two counters in shared memory.
6. The previous warp aggregated counters let us update a second workspace with the left-side and right-side elements. The way I thought to do this would basically be writing all left-side elements from index 0, forwards. Right-side elements would then be written from index N-1 backwards.

# example input
4 5 6 3 2
|
pivot

# example output
3 2 X 6 5
    |
    not used

7. Sync threads.
8. Check counters and find which side to recurse into, or end if the pivot is the answer.
9. Go back to 3.

Sounds easy enough. I’m sure I can do this in half an hour 1 hour 2 hours, right?

Why not just swap rather than use a second workspace?

Because it’s easy. Honestly, maybe doing the swaps would be faster and more memory-efficient, but this was quicker to implement (read the title).

Here I just allocate enough shared memory for two workspaces of equal length, and swap them after every iteration; no need to care about handling races across the blocks.

How fast is it?

As it turns out, for my test with 10000 rows of randomized lengths (up to a max of 100), with each block taking one row, this was about 50 % faster (33% reduction in time) than cub::BlockRadixSort! I compared it to selecting the median element here (which is what I was concerned with):

 ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):

 Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)                                                  Name
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----------------------------------------------------------------------------------------------------
     59.2            62336          1   62336.0   62336.0     62336     62336          0.0  void blockwise_median_kernel<unsigned short, (int)128, (int)1, (bool)1>(const T1 *, int, int, const.
     38.0            40000          1   40000.0   40000.0     40000     40000          0.0  void blockwise_quickselect_kernel<unsigned short>(const T1 *, int, int, int *, int *, T1 *)

Above timings from my 5080. Also, what a timing - exactly 40000 lol.

Not too bad for 2 hours of work I guess? Code is here for those who want to nsys profile it yourself on your GPU. And check out the rest of my repo while you’re at it!

The struct works fairly similarly to the BlockRadixSort, in the sense that you instantiate it at the thread level and then pass it some shared memory. Everything else is taken care of inside the __device__ methods, and the result should be valid for the first thread, as seen in the example.

Like previously mentioned, it assumes a maximum length for a row input, but only utilises a row-specific length when performing the calculation.

Look at this code snippet and tell me with a straight face you didn’t think it was a memory leak at first either..

A while ago I worked on some code which contained a container that looked like this:

struct S
{
  size_t sz;
  int data[1];

  void Set(size_t _sz, int *_data){
    sz = _sz;
    memcpy(data, _data, sizeof(int)*sz);
  }
}

If you’re a modern programmer like me, you might be looking at this and thinking: there’s no way this isn’t a flagrant memory violation in almost every case. Right? Well, it turns out this is something them old boys before C99 used to do.

Cool kids are manual and contiguous

The above struct works because it is meant to be manually managed, and because the array is the last field of the struct.

The idea was that you would heap-allocate the struct with padded bytes according to your required array length; this was the poor man’s dynamic array allocation back then.

struct S* sptr = (struct S*)malloc(sizeof(struct S) + sizeof(int) * (N-1));
// we have -1 because the struct already has room for 1 int

Now, when you access sptr.data[i], you are simply looking farther into the memory block you have already allocated. You could easily range-protect this access with an std::vector-like method; the cool thing here is that we get to have all the memory in one contiguous block.

This is unlike std::vector (and other containers) where the parameters like the size and capacity live on the stack, whereas the data is heap-allocated somewhere else.

Nowadays, post-C99, the syntax is a bit less misleading:

struct S
{
  size_t sz;
  int data[]; // this now indicates dynamic length. Still needs to be the last member though, and still needs manual management

  void Set(size_t _sz, int *_data){
    sz = _sz;
    memcpy(data, _data, sizeof(int)*sz);
  }
}

The benefits are still the same though! Parameters and data, living happily next to each other!

Premature optimization is the..

Yeah yeah, but I looked into it anyway. A quick chat with our favourite GPT brought up a topic I expected: cache-lines. But it was somewhat opposite to my intuition, which was interesting.

I thought that having things contiguous would benefit small data the most, but it turns out that the cache changes things (surprise surprise).

The crucial idea here is the total size of all the structs. Assuming you have to iterate/read all of them at least once, there shouldn’t be any significant difference between the small struct and the large struct if all of them fit in L1, because you pull data in cache lines anyway.

The real driver of costs comes from reaching further out to L2, L3 and RAM. This is where contiguity - and the associated prefetching - plays a huge role. Having them all read one after another will assist hardware prefetching, which is present in most CPUs these days.

I will be the first to admit that I didn’t test or benchmark this myself, because this is out of my concern. I merely wanted to highlight this interesting old C trick, and hope someone else recognizes that sometimes, boomers’ code isn’t the trash you think it is.

In the 2nd post of this series, I give a short introduction on something that is also not particularly new, but new to me - CUDA Graphs!

As stated in my first post of this series, this topic isn’t particularly new per-se. Indeed, it looks like it has been out since 2019. But maybe CUDA graphs have increased in relevance now that GPUs are more powerful. As usual, a good starting reference is NVIDIA’s own blogpost.

Note that CUDA graphs require a minimum toolkit version of 12.4, with some additional features being present in the newest (as of this writing) version, 12.8.

Why Use Graphs?

Perfectly reasonable question. The problem we are trying to combat when using graphs is overhead. This is really only present when our kernels are short, repeated or both.

In those scenarios, the overhead of the kernel launch becomes comparable to the duration of the kernel itself. This is often observed in an Nsight Systems timeline where there are visible gaps between the kernels after zooming in (yes, of course there are always gaps, but its the relative size of the gap to the size of the kernel that matters).

How It Works in a Nutshell

I liken this process to the distinction between compiled and interpreted languages. Interpreted languages are usually slower - yes, I know many nowadays have JIT and can be optimized pretty well, but bear with me for the comparison - because each command or instruction has to be submitted individually. This is necessary since all code is parsed only at runtime, so to balance start-up time with actual run duration, some assumptions are made and/or not all optimizations are performed. Of course, this is a vast simplification, but the idea still stands.

Compiled languages don’t suffer from this, since you as the programmer get to choose what occurs at runtime and what occurs at compile time. This affords the compiler a lot of flexibility to optimize as much as possible.

In the same vein, CUDA graphs offer the compiler a way to ‘compile’ multiple kernels together, and also submitting the work in a more ‘compressed, single call’. This is, in fact, how it appears in a standard Nsight Systems profiler timeline if no additional settings are specified.

In plain English, CUDA graphs allows you to tell the device to

do these N kernels in a row

rather than

do this kernel, then do this kernel, then do this kernel ….

Method 1: Record Your Kernels

This is the simpler method to convert existing code to the graph format. As seen from the blog post, you can do the following:

bool graphCreated=false;
cudaGraph_t graph;
cudaGraphExec_t instance;
for(int istep=0; istep<NSTEP; istep++){
  // Create graph only on first iteration
  if(!graphCreated){
	// Start 'recording'
    cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
    for(int ikrnl=0; ikrnl<NKERNEL; ikrnl++){
      shortKernel<<<blocks, threads, 0, stream>>>(out_d, in_d);
    }
    // Stop 'recording'
    cudaStreamEndCapture(stream, &graph);
    cudaGraphInstantiate(&instance, graph, NULL, NULL, 0);
    graphCreated=true;
  }
  // Launch graph on every iteration
  cudaGraphLaunch(instance, stream);
  cudaStreamSynchronize(stream);
}

In fact, I think you can probably just do a warm-up recording before any of the actual iterations, as long as the data pointers and all other parameters are valid. This is, of course, to avoid the much more expensive graph instantiation in your actual hot loops (even if it’s only the first iteration).

Method 2: Explicit API Calls

This method requires a definition of a graph in the more traditional sense: creating and connecting nodes to define dependencies.

As seen in the documentation, this looks like the following:

/*
  A
 / \
B   C
 \ /
  D
*/

// Create the graph - it starts out empty
cudaGraphCreate(&graph, 0);

// For the purpose of this example, we'll create
// the nodes separately from the dependencies to
// demonstrate that it can be done in two stages.
// Note that dependencies can also be specified
// at node creation.
cudaGraphAddKernelNode(&a, graph, NULL, 0, &nodeParams);
cudaGraphAddKernelNode(&b, graph, NULL, 0, &nodeParams);
cudaGraphAddKernelNode(&c, graph, NULL, 0, &nodeParams);
cudaGraphAddKernelNode(&d, graph, NULL, 0, &nodeParams);
// All the parameters go into `nodeParams`, including
// kernel launch parameters, the kernel function pointer,
// kernel input arguments etc.

// Now set up dependencies on each node
cudaGraphAddDependencies(graph, &a, &b, 1);     // A->B
cudaGraphAddDependencies(graph, &a, &c, 1);     // A->C
cudaGraphAddDependencies(graph, &b, &d, 1);     // B->D
cudaGraphAddDependencies(graph, &c, &d, 1);     // C->D

You might be wondering, why would you want to do this? Well, using the API to create graphs opens up a solution to 1 particular issue I suspect every CUDA programmer will eventually reach..

Conditional Nodes

There comes a time where your overarching logic must diverge based on some condition. Common examples include: iterative loop stops after convergence, check for thresholds etc.

In these cases, you might have some chain of logic which would be different depending on (effectively) some boolean. Now you have 2 options:

1. Combine all the divergent logic into the kernel itself, then launch just 1 kernel, within which the divergent logic is executed at a thread level. This can get very unwieldy very quickly, and may not even be possible.
2. Copy the boolean out to the host (ugly, and also likely to be slow). Read the boolean in the host, and then decide which kernel or chain of kernels to execute.

Refer to this blogpost for an excellent introduction to this, but I’ll summarise here.

Using CUDA graphs, you can now define a conditional handle via

cudaGraphConditionalHandle handle;
cudaGraphConditionalHandleCreate(&handle, graph);

You would then use this inside a kernel like this:

__global__ void setHandle(cudaGraphConditionalHandle handle)
{
	// ...
	// as long as value is non-zero it is 'true'
    cudaGraphSetConditional(handle, value);
}

The graph will now execute the conditional node without returning the boolean to the host, and you have the freedom to define (potentially) an entire graph of nodes in the if true expression.

Some notes for CUDA programmers who haven’t kept up with the times; this first post covers in-kernel pipelining..

I recently had the privilege of being sponsored to attend the March 2025 GTC in person. While the sessions were very largely dominated by AI-related things, I generally selected the CUDA-related ones, since they were more relevant to my work (and interests).

I’ll try to encapsulate some of the new things I learnt while I was there into a few major concepts. Note that most, if not all of these, are not shiny new CUDA features - some of them have been out for several years, but I just didn’t know about them, so I learnt about them there.

Edit: it seems like each section is getting a bit lengthy, so I’m going to split them into separate posts. This is the first one.

You’ve Heard of Async Stream Execution, Now Get Ready for Async In-Kernel Execution

Old CUDA programmers already know about asynchronicity via memory-copy overlaps, latency hiding etc. The story is pretty simple:

• Enable and use non-default streams
• Within a stream, all commands are processed serially
• Across streams, all kernels are processed in parallel (as far as the launch is concerned, execution is still up to the SMs)
• For memory copies to and from the host, you also have to make sure the memory is pinned

CUDA stream concepts are pretty old. They are analogous (in my opinion) to CPU threads, where you simply fire off a bunch of functions/instructions that you’d like to execute in parallel, and let the scheduler handle it; if there are SMs available to service your kernels, they will do so, similar to how if there are active CPU cores to execute CPU functions, they will do so.

Bytes In-Flight and Memory Prefetching

CPUs nowadays do a pretty fantastic job of prefetching both required memory and required instructions (which has also led to hardware-level security problems but that’s another story..).

Traditionally, GPUs have not done this, but we can now quite easily perform this with CUDA’s new pipeline functions. A good resource is this blog post, but I’ll try to summarise very quickly.

Conceptual Understanding

The first concept to understand is what is known as bytes in-flight. This wasn’t in the blog post link but it was mentioned many times across several sessions in GTC. The idea is that, as much as possible, you want to maximise the uptime of global memory reads or writes.

We already know that global memory access is slow, so what we want is to ensure we can do it as quickly as possible, and hide it as much as possible. The old ideas of global memory coalesced access and vectorised loads still hold true today, and contribute to this thought process.

The new idea is to then overlap memory access inside the kernel with compute. When a warp tries to access global memory (a load instruction), it takes several cycles before the values are actually ready to be operated on. If there are no other instructions that can be performed in the meantime, this results in a stall. Effectively, the SM cannot proceed because it needs the loaded data to be ready before it can continue.

As mentioned, this isn’t new to GPUs, and is why prefetching was invented for CPUs. To solve this, CUDA introduced the pipeline interfaces. Let’s assume that a load instruction that we are executing requires 5 cycles to complete (latency). We can split our kernel’s work into batches. For each batch with index $n$, we do the following:

1. Load the data required for batch $n+1$ asynchronously using the pipeline mechanisms. This will not block and will allow the kernel to continue executing.
2. Block/wait for data loads for batch $n$ to complete; this would have started in the previous iteration.
3. Run computations for batch $n$.

A more graphical representation might look like this:

Loads   :  X---- n+1 ----> X---- n+2 ----> ...
Computes:  X---- n   -->   X---- n+1 -->   ...

I deliberately made the computes take a shorter amount of time to demonstrate the wait on the data loading. The reverse - computes take longer than loads - still warrants this method; the idea is now we are bound by whichever operation takes longer, but not by their sum.

It is important to remember that all of the above is taking place inside the kernel; although the timeline view is similar to how one might have pipelined batches previously with compute and copy streams, this is a different, deeper level of asynchronicity. Do not confuse this with streams!

libcudacxx pipelines

Since the blog post covers the C-like primitives, I’ll try to cover how you would do this via the C++ APIs provided inside libcudacxx. See the original API for more details.

We’ll look at the unified pipeline for now - this is the simpler case where all threads are producers and consumers. You can do fancy things with the partitioned pipeline where you carve out some threads to do either one, but it’s probably not necessary for most cases.

The pipeline mechanism is a template construct that uses compile time constants where you define the number of stages and the scope of cooperation, similar to CUDA’s cooperative groups. Let’s look at a more annotated version of the example code:

#include <cuda/pipeline>
#include <cooperative_groups.h>

template <typename T>
__global__ void example_kernel(T* global0, T* global1, cuda::std::size_t subset_count) {
	extern __shared__ T s[];
	auto group = cooperative_groups::this_thread_block();
	// Split our shared memory into 2, one for each stage
	// Here we have 2 inputs we want to load, and we load 1 element for each
	// thread, so we use 2 * group.size() for each stage
	// | stage0, input0 | stage0, input1 |
	// | stage1, input0 | stage1, input1 |
	T* shared[2] = { s, s + 2 * group.size() };

	// Create a 2-stage pipeline, synchronised at the block level.
	constexpr auto scope = cuda::thread_scope_block;
	constexpr auto stages_count = 2;
	__shared__ cuda::pipeline_shared_state<scope, stages_count> shared_state;
	auto pipeline = cuda::make_pipeline(group, &shared_state);
	
	// Prime the pipeline.
	// Here we are submitting things that we want to load asynchronously
	pipeline.producer_acquire();
	// NOTE: these functions DO NOT start the memcpy. Think of it like an
	// SQL transaction; we are just queueing the things we want to happen
	cuda::memcpy_async(group, shared[0],
					 &global0[0], sizeof(T) * group.size(), pipeline);
	cuda::memcpy_async(group, shared[0] + group.size(),
					 &global1[0], sizeof(T) * group.size(), pipeline);
	// This is what actually fires off the load instructions!
	// Think of it as now having 'Producer State: 0'
	pipeline.producer_commit();

	// Question: Why did we have to do the above?
	// Answer: same as with other pipeline mechanisms. We need to seed
	// the first batch before our iterations begin. Otherwise, the first
	// batch wouldn't be ready when we first need it in the loop.

	// Pipelined copy/compute.
	// Notice that we start on iteration 1 because that's the next batch
	// of data we are going to load
	for (cuda::std::size_t subset = 1; subset < subset_count; ++subset) {
		// The next section is identical to above;
		// we are loading the n'th batch of data
		pipeline.producer_acquire();
		cuda::memcpy_async(group, shared[subset % 2],
						   &global0[subset * group.size()],
						   sizeof(T) * group.size(), pipeline);
		cuda::memcpy_async(group, shared[subset % 2] + group.size(),
						   &global1[subset * group.size()],
						   sizeof(T) * group.size(), pipeline);
		// Again, loading doesn't start until we commit
		// In the first iteration, we would now have 'Producer State: 1'
		pipeline.producer_commit();

		// In the first iteration, at this point, we would potentially have
		// both Producer State: 0 and Producer State: 1 in flight.
		// Note that we haven't done anything with the consumer yet so
		// the consumer would be at State 0.

		// Here we are waiting on the PREVIOUS batch of data
		// In the first iteration, this would entail having the consumer
		// wait on State 0.
		pipeline.consumer_wait();
		// After this it is hanging on to State 0 while it computes.
		compute(shared[(subset - 1) % 2]);
		// And finally when we release, the consumer increments to State 1,
		// in preparation for the next iteration.
		pipeline.consumer_release();
	}
	
	// Drain the pipeline.
	// This is to compute the last iteration.
	pipeline.consumer_wait();
	compute(shared[(subset_count - 1) % 2]);
	pipeline.consumer_release();

}

Notes and guidelines to follow

The above makes clear that the pipeline mechanism suffers from 1 obvious downside:

Pipelining requires $n$ times the memory for $n$ stages.

This isn’t unique to this particular mechanism; all pipeline concepts usually trade extra memory for more throughput. Note that the pipelining does not necessarily have to be done via shared memory - registers (stack variables or arrays) can also be used, but these assume that the lengths required are compile-time constants as well.

As always, premature optimization is evil and should be avoided. You can and should use Nsight Compute to check whether latency hiding within the kernel will be beneficial (and the degree to which it will be beneficial).

Don’t do this at the beginning, but rather at the end when the kernel is already ‘ready’, because the pipeline mechanism can make things messy if the computations are not organised in a simple manner.

As a side-note, thrust and cub libraries are actually built to do this internally for the templates where it can. This is why if your operation can be pipelined and you write your own custom CUDA kernel where you don’t perform pipelining, you will often be slower than the equivalent thrust functor call.

If you can use thrust (or cub) to express your computation, do so, because thrust will internally pipeline the kernel for you.

For those with money: Programmatic-Dependent Launches (PDL) and Tensor Memory Accelerators (TMA)

These are both features introduced in compute capability 9.0 and above i.e. Hopper cards like H100 and newer (yes, that’s the previous generation since Blackwell just released, but I don’t have access to one either). Since I’m stuck in the Ampere days, I’ll only touch on these 2 very briefly. The interested reader can simply search these 2 keywords for more information.

PDL

This allows you to overlap kernels within a stream. Essentially, what you do is to mark points inside consequent kernels where dependence is absent. A common example is pre-loading some constant data at the start of the kernel:

| --- Kernel 1 --- |
             | xxx | -- Rest of kernel 2 -- |
             ^
             |
             Kernel 2 can start here to read data that doesn't
             depend on kernel 1's output. Then it can wait on
             kernel 1 to complete before it does the rest.

TMA

This is somewhat like a further hardware extension of the pipeline mechanism. There is now a separate unit called the TMA that performs the data copies from global memory. There are several pros to this:

• No/less registers used; since a separate unit is doing the reads, the threads don’t need to read into registers first
• Skips the entire cache hierarchy; goes straight from global to shared

But there are also cons:

• The code (in my opinion) is a lot more complex
• There are far stricter memory alignment requirements
• No option to go to registers directly (maybe not necessarily a con, depending on use-case)

Convincing you (and myself) that with some minor edits, we can still use atomicMin for floats in CUDA..

CUDA has a set of atomic functions for safely updating the same memory address with different threads. There’s a whole list of them in the programming guide here.

However, for atomicMin (and also atomicMax), there isn’t an overload that works with floats (or doubles , but who uses those in CUDA anyway..). For this discussion we’ll just focus on atomicMin, but all the points we discuss can be inverted to explain atomicMax.

int atomicMin(int* address, int val);
unsigned int atomicMin(unsigned int* address,
                       unsigned int val);
unsigned long long int atomicMin(unsigned long long int* address,
                                 unsigned long long int val);
long long int atomicMin(long long int* address,
                                long long int val);

As you can see, they all deal with integers. Okay, so how would you deal with floats? You could do some of the following:

1. Rework the code to not use atomics. This probably involves an additional kernel to do comparisons and/or some additional scratch memory. I used to do this until cooperative groups came out and the compiler started optimising warp-aggregated atomics very well.
2. Use atomicCAS to define the atomic operation with some funky do-while loop. I found this to be very inelegant.
3. Use some scaling function to turn your floating point values into an integer, preserving the order i.e. $x > y \rightarrow f(x) > f(y)$. Then just use the existing integer-based atomicMin calls, which would benefit from hardware acceleration.

The last method sounds tricky, but it doesn’t really need to be.

Comparing floats is (almost) the same as comparing ints

First off, I’m not the one that thought of this. See the answer here. I’m going to explain why this works. To do this, we need to recap how the 32 bits in a float, int or unsigned int are laid out.

IEEE-754 integers (32-bit)

This should be obvious but we’ll restate it for completeness.

Unsigned integers are simply a list of bits representing the powers of 2, where the leading bit is $2^{31}$ and the ending bit is $2^0 = 1$.

Signed integers are identical to unsigned integers, but with the leading bit used for the sign bit.

IEEE-754 single-precision floating point

Refer to this concise graphic from the wiki page:

Here you can easily observe the following:

• First (leading, leftmost) bit is the sign bit
• Next 8 bits define the exponent $2^{E - 127}$
• Final 23 bits define the mantissa or fraction (with implicit offset), $1 + x$, with $0 \leq x < 1$

Exponent and mantissa sections can be compared like unsigned integers

The above is the definition, but the first important property here is that both the exponent and the mantissa sections are still ordered, in the sense that leading bits represent larger numbers than trailing bits.

This is important, because then we can treat each section like its own unsigned integer space i.e. if we treat the exponent as an 8-bit unsigned integer and compare it like an 8-bit unsigned integer for two numbers with all other bits equal, then the float with the ’larger 8-bit unsigned integer exponent’ is indeed the larger float! The same goes for the mantissa section’s 23 bits.

This also holds for the combination of the 2 sections; to see why, simply consider whether the following is possible: given fraction $x$ and exponent $E$, can we write a number where $y_1 = (1+x_1) 2^{E_1} > (1+x_2) 2^{E_2} = y_2$ but one of the following occurs:

1. $x_1 \leq x_2$
2. $E_2 \leq E_1$

Again, this isn’t particularly complicated when you realise you only need to consider the smallest trailing bit of the exponent. Flipping this final, rightmost exponent bit engineers a factor of 2 change to the entire number. Since the other 23 bits represents a number $1 \leq 1 + x < 2$, it is impossible for this number to double and create a number that overcomes this factor of 2.

Let’s use the above number in the graphic as an example. The number is

We can increment the exponent by 1 in the trailing bit to make this

which is double the original number, as discussed above. It should be obvious by now - since we’ve already established that the mantissa number has an upper bound of 2 - but let’s finish up the example for completeness. What’s the maximum number we can make in the fraction (turning on all the bits)? Well this is just the glorious number

Clearly, this number multiplied by $2^{-3}$ is still smaller by definition than 0.3125. Thus, it is possible to conclude that we can compare both mantissa and exponent sections as if it were one long 31-bit unsigned number when we are simply concerned with magnitude of numbers.

In other words, we could simply reinterpret the trailing 31 bits of a float as an unsigned integer and then call atomicMin for our purpose! But now we need to handle the leading sign bit..

Sign bit implies a flip of min/max operations

Let’s just look at a bunch of examples of floating point values after they are reinterpreted as unsigned or signed integers (shown in row-pairs). In each pair, the smaller number will be in bold.

It is important to note that in our atomicMin case, the access pattern is not symmetric, in the sense that the val is usually on-chip/local, whereas address usually refers to some global memory which is way more expensive to access.

This means that we should consider our information to be limited to val alone - we observe what val is and then we make a decision - and we will consider both the (negative val, positive address) and (positive val, negative address) cases separately.

In the first 2 cases - where val is positive - we see that reinterpreting as a signed integer and then taking the minimum matches our floating point minimum i.e. here we take the atomicMin.

In the next 2 cases - where val is negative - the above is inconsistent, and we should instead reinterpret as unsigned integers, where the floating point minimum now corresponds to the unsigned maximum i.e. here we take the atomicMax of the integers, even though we were looking for the atomicMin of the floats!

Remember, in all our cases above, we are making the decision for the reinterpreted type based solely on the sign of val.

Why this happens

This should be apparent when considering unsigned versus signed integers. For signed integers, we compare the trailing 31 bits (magnitude) in a way identical to unsigned integers, but if the sign bit is on (negative number) then a smaller magnitude implies a larger number, and vice versa.

When we first observe the sign of val, we don’t know what the sign of the other operand at address is. But we can infer the following:

1. If val is positive and
   1. address is positive then taking the minimum of the signed or unsigned reinterpretation makes no difference. Taking the maximum in either reinterpretation gives the wrong answer.
   2. address is negative then we must take either the minimum of the signed reinterpretation or the maximum of the unsigned interpretation.
   3. The consistent logic is thus taking the minimum of the signed int reinterpretation.
2. If val is negative and
   1. address is positive then we must take either the minimum of the signed reinterpretation or the maximum of the unsigned interpretation.
   2. address is negative then taking the maximum of the signed or unsigned reinterpretation makes no difference. Taking the minimum in either reinterpretation gives the wrong answer, since we need to look for the bigger magnitude in order to be more negative.
   3. The consistent logic is thus taking the maximum of the unsigned int reinterpretation.

A final note on signed zeroes

In the StackOverflow link, there is an older answer that used a simple val >= 0 comparison to determine which logic branch to use. This works, except in some cases where val == -0.0f.

For example, if val == -0.0f and *address == -1.0f, then since val >= 0 evaluates to true, one might attempt to use the minimum of the signed int reinterpretation as discussed above, evaluating val as -2147483648. This would, however, mark -0.0f as the minimum erroneously since -1.0f is -1082130432; indeed, since val is ’negative’ we should have used the maximum of the unsigned int reinterpretation.

This is entirely due to zero being a special number in the exponent section (0.0f is just 0x00000000 and -0.0f is just 0x80000000). The correction, as stated in the link, is to just read the bit directly, either via custom bit twiddling or CUDA’s signbit().

How fast can you generate (pseudo-)random numbers on the GPU? How fast can you generate (pseudo-)random numbers on the GPU?

I recently had to answer this question. Well, not exactly this question, but this was a key part of it.

A quick google search will bring up two common methods (libraries) when trying to do this in CUDA: thrust and cuRAND. Thrust is known to be a lot easier to set up; no need to write the kernel code and nitty gritty details, so I started with that.

Before I begin though, here’s a link to my growing GPU-related codebase. Some of the following snippets come from there.

The thrust way

Generating random numbers in thrust is somewhat similar to how you would do so in std C++.

1. Initialize an RNG engine.
2. Initialize a distribution, like uniform_real_distribution.
3. Call the distribution on the engine via operator() to generate a random number.

This is usually done, as seen in the examples, via a custom struct with an overloaded operator(), which is passed to a thrust::generate call, like so:

// The following is adapted from
// cuda-samples/Samples/3_CUDA_Features/cdpQuadtree/cdpQuadtree.cu
template <typename Engine = thrust::random::default_random_engine>
struct Random_generator2d {

  int count;
  __host__ __device__ Random_generator2d() : count(0) {}
  __host__ __device__ unsigned int hash(unsigned int a) {
    a = (a + 0x7ed55d16) + (a << 12);
    a = (a ^ 0xc761c23c) ^ (a >> 19);
    a = (a + 0x165667b1) + (a << 5);
    a = (a + 0xd3a2646c) ^ (a << 9);
    a = (a + 0xfd7046c5) + (a << 3);
    a = (a ^ 0xb55a4f09) ^ (a >> 16);
    return a;
  }

  __host__ __device__ __forceinline__ thrust::tuple<float, float> operator()() {
    unsigned seed = hash(blockIdx.x * blockDim.x + threadIdx.x + count);
    // thrust::generate may call operator() more than once per thread.
    // Hence, increment count by grid size to ensure uniqueness of seed
    count += blockDim.x * gridDim.x;

    Engine rng(seed);
    thrust::random::uniform_real_distribution<float> distrib;

    return thrust::make_tuple(distrib(rng), distrib(rng));
  }
};

Random_generator2d<thrust::random::default_random_engine> rnd;
thrust::generate(
    thrust::make_zip_iterator(
        thrust::make_tuple(d_x.begin(), d_y.begin())),
    thrust::make_zip_iterator(thrust::make_tuple(d_x.end(), d_y.end())),
    rnd);
    

In the above example, the generator functor has been modified to output 2 random values per call. There is also a custom hash function in the sample.

This is all well and good, but I had several issues with this:

• thrust documentation is nearly non-existent. I had to search for stack overflow articles and look at the samples directly to really figure out what to do.
• Because of the above, I wasn’t really sure whether I could remove the engine instantiation from within the functor operator(). Not being able to do so would imply that every time the functor is called, it would restart from the beginning i.e you wouldn’t be able to use this in a loop, since the 2nd call would generate the same numbers as the 1st call. It could probably be done - by returning the engine and taking it in as an input? - but there was no reason to experiment since cuRAND already had this well-documented.
• Like all other thrust things, there was no way to finely control the grid and block dimensions.
• The thrust::generate and functor way of doing things, while simple if trivially generating straightforward data structures, started looking uglier - in my opinion - when more complicated indexing was required.

This (cuRAND) is the way

The NVIDIA mandalorians have declared it so.

Seriously though, it seems pretty obvious that NVIDIA wants you to use cuRAND for HPC implementations. And the performance looks pretty sick to be honest.

The library by design forces you to initialise a bunch of RNG states with curand_init. Each of these states are used in function calls like curand_uniform to generate random numbers, after which the states are updated so that the next call will generate the next random number in the sequence.

The beauty of this is that there is a lot of choice for how to use the states:

1. The most basic way: use one state per thread, per element. You’d save the updated states back into global memory.
2. A bit more complex: use one state per thread, per N elements. You’d load the states from global memory, generate N elements, then write back to global memory at the end.

You also get to tweak the index to start at in the subsequence, just from either the curand_init arguments (subsequence is usually the thread index, offset is what you’re looking for), or via a skipahead function call (which would be the same as if you had init-ed at that offset, but you don’t want to init again).

Since all of these calls are done using the device API - inside a normal kernel function - you get to play around with all the standard block/grid choices you would with a kernel as well; this is unlike in thrust functor calls where the grid and block dimensions are chosen for you.

I ended up writing a simple class to wrap these operations, which could be easily extended for other use-cases, either by explicit inheritance or just a simple rewrite:

1. Class constructor initialises the number of states; how the states are used is internal to the implementation, but what is important is that it is constant i.e. used in the same way for every invocation later. This essentially allocates device memory and performs the curand_init calls appropriately.
2. Simple public methods to call the kernel that does the RNG (and any other pre/post-computations). Again, this is implicitly understood to be constant every call (the shape of the data, and the grid/block dimensions operating on it are the same and part of the design for that use-case).
3. Skipahead function calls are also defined, similar to the previous point.
4. Destructor doesn’t need to be defined since RAII thrust vectors are used to store the states.

You can take a look at some of the test implementations I made here and here. I originally wanted to try out curand’s float4 generators that only work with Philox, but I didn’t bother because this part didn’t end up being a bottleneck in my final code.

Some background on my single header file memory manager and why I made it..

Picture this. You are working on a codebase with some lazy inept questionable decisions from external forces. You are given a class to work on. For the purposes of this post let’s call it MyClass.

The class will have its methods invoked in the following order:

1. MyClass() i.e. the constructor.
2. setup(…)
3. run(…)
4. teardown()
5. Repeat 2-4 many times.
6. ~MyClass() i.e. the destructor.

This class will be reused in multiple different scenarios repeatedly, with different input data.

Here’s the problem: you also need to allocate different things/sizes of arrays depending on the data. Hence, there is effectively no way to pre-allocate this in the constructor. You will have to do this in setup(), and then destroy or free these in teardown().

(Over-)dramatisation

Them: You must pre-allocate your data in setup and remember to free it in teardown.

Me: Sure, but I’d like to use a unique_ptr or similar smart pointers to do it so that my colleagues and I don’t have to spend extra time searching for memory leak bugs. I can’t do this if we don’t call the destructor for automatic cleanups.

Them: We cannot keep destroying and reinstantiating the objects. It will be too much overhead.

Me: But we are allocating gigabytes of data anyway, the constructor overhead is going to be negligible in the midst of all this.

Them: …

Me: ??? (Your brain is the overhead)

A compromise

With no other solution, I decided to write my own. Well, I had an idea of a memory manager, and then technically I prompted ChatGPT to write one for me.

Specifically, I needed to make sure it would handle the following:

1. Ability to handle arbitrary POD type heap allocations. Stuff like new arr[size] would need to be replaceable.
2. Ability to handle custom class constructors. Stuff like new SomeClass(arg1, arg2) would need to be replaceable.

A few prompts, and some manual edits later, I had what I wanted.

The premise is pretty simple: have a memory block structure and hide your allocations inside them.

Hold them all in an std::vector, and then .clear() it when finished with them.

All the references disappear when the vector is cleared, and each memory block will call its own functor to free the internal memory appropriately i.e. delete or delete[].

Why not just hold a vector of MemoryBlocks?

Great question. ChatGPT reminded me; I’ll paraphrase it here with the following important question:

What happens when the vector is resized?

If we had taken the code I linked above and simply changed vector<unique_ptr<MemoryBlock>> to vector<MemoryBlock>, the resize would have done the following:

1. Copied the MemoryBlocks over. This is just a copy of each pointer we allocated. This is fine.
2. Freed the existing memory holding the original memory blocks. This would call each memory block’s destructor, which would call each functor, which would free the internally kept pointer. This is not fine.
3. The pointers we copied over now point to nothing.
4. Freeing the new resized vector would result in double frees (or another resize would also do this).

The typical solution is to redesign the MemoryBlock so that the copy constructors, assignment operators etc are well-defined. The rule of 5 strikes again.

In this case, we really just want to make sure that the internal pointer (which implies the MemoryBlock itself) is move-only, so that when the vector resizes it will not delete the internal pointers at any point. But this is exactly what using unique_ptr achieves! Hence, no need to reinvent the wheel.

Jokes on me..

In the end, they still didn’t want to implement this. ‘Unnecessary code changes’..

My small journey in discovering libmvec functions and the C problems I encountered..

I was recently working on making an existing Windows-only library of SIMD functions cross-platform i.e. making everything work in Linux. Here I’m going to highlight some problems I encountered in the process; hopefully it will help someone if they encounter something similar too.

The Windows (MSVC) function

Intrinsics are usually tied to compiler implementations. The one that gave me issues was a bunch of trigonometric functions. Let’s use the SSE version that works on floats; in MSVC this was implemented as a simple

__m128 _mm_cos_ps(__m128 a)

This function, and others like it, simply do not exist in GCC implementations, at least not explicitly. As such, there is no way to direct a call an SIMD vector to a simple cos() function for evaluation. Why I highlighted simple will become clear in a minute.

How GCC handles it

GCC does, of course, have a way to vectorise these operations through its -ffast-math flag. Using this and a standard -O3, we can see that a simple function like

void manyCos(const float *x, float* y)
{
	for (int i = 0; i < 4000; ++i)
		y[i] = cosf(x[i]);
}

gets turned into assembly that looks like

lea rdx, [rdi+4]
mov rax, rsi
push r12
mov r12, rsi
sub rax, rdx
push rbp
mov rbp, rdi
push rbx
xor ebx, ebx
cmp rax, 8
jbe .L2

.L3:
movups xmm0, XMMWORD PTR [rbp+0+rbx]
call _ZGVbN4v_cosf
movups XMMWORD PTR [r12+rbx], xmm0
add rbx, 16
cmp rbx, 16000
jne .L3

.L1:
pop rbx
pop rbp
pop r12
ret

.L2:
movss xmm0, DWORD PTR [rbp+0+rbx]
call cosf
movss DWORD PTR [r12+rbx], xmm0
add rbx, 4
cmp rbx, 16000
jne .L2
jmp .L1

Now we can see this calls a particular intrinsic that looks like

_ZGVbN4v_cosf

This is a special routine that comes from libmvec, which is a vector library that comes coupled inside glibc. GCC automatically uses this library when it is asked to vectorise more involved mathematical functions like cosf, sinf, logf etc (and their double variants of course).

But how do we manually call libmvec functions then?

The link above should make it clear that there is no intended use-case to manually call stuff in libmvec. Indeed, the vector ABI it uses also doesn’t correspond to the typical C++ name-mangling of symbol names.

Googling it doesn’t really come up with much information either. I have found that the best way in my opinion is to simply disassemble code compiled with -ffast-math, like above, and look for the function called. I can easily do this in godbolt.org so that’s usually where I do it (and it doesn’t seem like the function name has/will change in the foreseeable future).

Okay, we have the function name, so how do we call it?

The first thing to do is to include the header. This is simply in math.h, which is where you would find the normal math functions like cosf() anyway.

Now the next thing that is required is to make it extern "C", because the library is written as a C library. If you don’t do this and you’re compiling with g++, then C++ mangling is going to mangle the name again and it will result in undefined references.

Alright, but what about the input argument types? It’s meant to be used on gcc’s in-built vector type definitions like __m128, so that’s what we will use. But here is where my problem began.

Type qualifiers? Reference?

In a typical C++ function nowadays, it is usually recommended that input functions that are not modified in the function be declared as const, and that where size may matter, references (or pointers) be used. As such, my declaration looked like this:

extern "C" __m128 _ZGVbN4v_cosf(const __m128&);

As it turns out, using a reference via & is completely incorrect, and very dangerous; this will compile without error, but will then produce nonsensical output at runtime. The const is technically harmless, but is also irrelevant to this discussion anyway (since we are not the ones writing the function implementation).

The correct declaration is simply

extern "C" __m128 _ZGVbN4v_cosf(__m128);

where the const is irrelevant.

The C ABI (and the compiler) cannot know your mistake

I’ll leave a link here to my question in StackOverflow, with the fantastic answer by the assembly master Peter Cordes. I’ll try to summarise his answer here.

Effectively, telling the compiler that a reference is input is the same as telling it to load a pointer address into the input registers of the function. Doing so, and then wrongly calling the function with the C++-like syntax for references (where you just write the variable name), would result in the compiler taking the variable’s value as the address.

I also asked this question to the experts in the Compiler Explorer discord, and @dragonmux was kind enough to respond. The explanation given to me there helped to clarify some things.

I first asked why the compiler couldn’t help to catch it as I was still in my C++ mindset. For example, having two files with

// # a.cpp
extern float func(float x){
	return x*2.0f + 0.1f;
}

// # b.cpp
// Forward declaration with 'wrong' input qualifier (added &)
extern float func(float& x);

// create a variable and explicit ref
float x = 2.5f;
float& xref = x;

// pass explicit ref to func
func(xref);

will fail to compile as g++ will correctly identify that the reference is undefined (because it has retained the type qualifier information in the name-mangling).

If I instead now qualify everything with extern "C" instead,

// # a.cpp
extern "C" float func(float x){
	return x*2.0f + 0.1f;
}

// # b.cpp
// Forward declaration with 'wrong' input qualifier (added &)
extern "C" float func(float& x);

// create a variable and explicit ref
float x = 2.5f;
float& xref = x;

// pass explicit ref to func
func(xref);

this now fully compiles, since the C ABI discards the input types, so the symbols are identical in both translation units. But then it WILL crash (or in my contrived experiment it just outputs the wrong value 0.1f), since we are effectively passing a pointer to something that expects a value.

Somehow, even though I was aware of name-mangling in C++, I didn’t stop to think how it was protecting me from things like this.

Essentially, in the C++, non-extern "C" case, we have

1. A symbol for func in a.cpp, name-mangled to indicate the pure value argument.
2. A symbol for func in b.cpp, name-mangled to indicate the reference argument.
3. Linker attempts to find definition for symbol used in (2) from (1) and fails.

In the C version, everything links, because both symbols just look like func. Then at runtime, undefined behaviour results.

Final question: is it bad/slow that the library is using SIMD register values instead of references?

In my same StackOverflow question above, Peter Cordes explained the nuances behind this pretty well, so go back to his answer directly if you want.

TL; DR, it’s important to note that it’s

cheap to pass/return by value in a single register.

Think of it just like any other POD like an int.

There are other details on how the vector registers are handled in cases like this, where the SVML (which includes things like the trigonometric functions for SIMD) aren’t really hardware intrinsics, so they have to respect the

x86-64 System V calling convention: there are no call-preserved vector registers.

and that means that

only the return-value register can be relied on to have a useful value

but I think that’s not really necessary to understand deeply from a library maintainer’s perspective. You probably just want to know that the way you’re calling the equivalent cosf intrinsic is the best way possible. The answer is yes, doing it by value is indeed the best way possible.

Hidden linkage isn’t something I’d encountered until now, so maybe this will help someone else too..

A new series of posts

I recently started work on a new codebase - one that is very large and has multiple moving components, from a frontend desktop UI (not my business) to the hardware interface (also not my business) and the backend processing (this one’s my business).

The standard build process had been setup on a remote server, and most people were ok with the process of connecting to it with tangible amounts of latency, but I was not. So my stubborn ass decided to figure out how to build it myself on a local machine. In the process, I encountered a ridiculous number of build errors; they were using custom Makefiles, but they had been generated by some other tool (possibly from Windows, even though it was being built in Linux, since there was a Windows VS solution as well).

This will be the beginning of a series where I will document these errors for myself (and hopefully for others!).

Topic for today

Alright enough backstory. The error being discussed here goes something like this:

hidden symbol X in Y is referenced by DSO

GCC (or rather, the linker) spat this out at me when it was trying to link in a Boost static library.

As a note, DSO stands for Dynamic Shared Object, which is just Linux’s name for shared libraries.

It’s not an undefined reference

You’ll be forgiven for thinking so, because that’s what came to mind at first. But no, it’s in the description; the symbol was there in the .a archive (an nm and objdump confirmed this).

This was a different kind of error, and one that I think resulted from the choice to build multiple parts of the code into separate shared library objects (.sos).

If you want to go ahead and read the stackoverflow links that helped me, go ahead and click here.

It’s also not a link order issue

You’ll again be forgiven for thinking this might have been the cause. This was what I tried at first to reproduce the problem in a simple scenario.

For those who don’t know, ld resolves symbol references in a very specific order:

1. A compilation unit needs a particular symbol.
2. Libraries specified after that compilation unit provide this symbol.
3. Linker does its thing.

This is why specifying libraries in the right order matters, because if you provide the library ‘before’ you make the ‘request’, the ld will not go back and get it for you (but I think MSVC will, so 1 point to Microsoft I guess?).

I’ll leave this excellent writeup on the topic here, as I think it explains the topic far better than I ever could (he even talks about the circular dependencies!)

A little more in-depth: what does the linker actually pull in?

What I tried to do to reproduce the hidden symbol issue at first was to create the following scenario:

1. Compile two separate source files separately: we’ll call them static.cpp and static_exclude.cpp for reasons that will become apparent. They will share the same header, static.h, and we will place them all into the same static library.

// static.h
int addstatic(int a, int b);
int notinshared(int a, int b);
float exclude(float a, float b);

// static.cpp
#include "static.h"

int addstatic(int a, int b){
  return a+b;
}

int notinshared(int a, int b){
  return a-b;
}

// static_exclude.cpp
#include "static.h"

float exclude(float a, float b){
  return a/b;
}

Honestly, I don’t know how to write raw Makefiles (I’m more of a CMake dude), so I had ChatGPT spit one out for me:

STATIC_LIB = libstatic.a

# Compile static.cpp to a static library (.a)
$(STATIC_LIB): $(STATIC_SRC)
	$(CXX) $(CXXFLAGS) -c static.cpp -o static.o
	$(CXX) $(CXXFLAGS) -c static_exclude.cpp -o static_exclude.o
	ar rcs $@ static.o static_exclude.o

So we made our libstatic.a.

Fun fact: doing this taught me that Makefiles must use tabs in the recipe, not spaces!

2. Now we make a shared library from shared.cpp. This will use some functions from the static library we just made.

// shared.h
int sharedaddscale(int a, int b, int c);

// shared.cpp
#include "shared.h"
#include "static.h"

int sharedaddscale(int a, int b, int c){
  return addstatic(a,b)*c;
}

And another excerpt from ChatGPT:

SHARED_OBJ = shared.o
SHARED_LIB = libshared.so

# Compile shared.cpp to an object file
$(SHARED_OBJ): $(SHARED_SRC)
	$(CXX) $(CXXFLAGS) -c $< -o $@

# Link shared.o and static library to create the shared object (.so)
$(SHARED_LIB): $(SHARED_OBJ) $(STATIC_LIB)
	$(CXX) -shared -o $@ $(SHARED_OBJ) -L. -lstatic

This makes libshared.so.

3. Finally, we create our main executable file e.cpp.

#include "static.h"
#include "shared.h"
#include <iostream>

int main(){

  printf("%d\n", sharedaddscale(2,3,4));
  printf("%d\n", notinshared(5,3));
  printf("%d\n", exclude(5.0f,3.0f));

  return 0;
}

And we link our shared library to it:

E_SRC = e.cpp
E_OBJ = e.o
EXEC = final_executable

# Compile e.cpp to an object file
$(E_OBJ): $(E_SRC)
	$(CXX) $(CXXFLAGS) -c $< -o $@

# Link e.o with the shared library to create the final executable
$(EXEC): $(E_OBJ) $(SHARED_LIB)
	$(CXX) -o $@ $(E_OBJ) -L. -lshared

The question for you now is: does this build? If not, which function calls will cause errors?

You can ponder the code and build paths above before continuing down..

…

…

…

…

…

…

…

…

…

…

…

…

…

Alright that’s enough blank space. If you said only exclude() will cause issues, then you would be correct, so congratulations! You’re a qualified GCC nerd.

Why does exclude() cause an undefined reference?

When the linker finds a symbol reference, it searches the later inputs to try to find it. Here, we only linked the shared library, so the next question must be: why is exclude() not inside the shared library? Didn’t we link the static library when we created it?

The answer is that the linker resolves and includes only symbol definitions that it requires. The shared library never needed exclude(), so it never got included in the final output.

Okay, but the shared library also didn’t need notinshared(), so why is it there?

The above answer isn’t technically complete. What the linker does exactly is to find a symbol reference it requries, and then include the entire object file. It doesn’t matter whether it is a simple .o, or an archive (static library .a) of object files.

In the above example, we did need addstatic() inside shared.cpp. But addstatic() shares the same compilation unit (and hence the same object file) as notinshared() - both come from static.cpp which became static.o - so both of them end up inside the final shared library!

Static libraries are basically just containers of object files to the linker.

This is a pretty good post about the above linkage behaviour.

Fine, isn’t the solution easy? Just link the static library?

Yes, that is indeed the solution here. This way the linker sees the necessary symbol, which is present in libstatic.a. But technically, that’s not the only way to fix the problem..

Including everything from the archive

We’ve already seen that the linker has the freedom to pick and choose the objects it wants to include. But if you’re building a shared library, you probably just want to dump everything in; that way, the end-user can just use your library, instead of having to link in the original static library as well.

Well, we can do that too, with -Wl,--whole-archive:

# Link shared.o and static library to create the shared object (.so)
$(SHARED_LIB): $(SHARED_OBJ) $(STATIC_LIB)
	$(CXX) -shared -o $@ $(SHARED_OBJ) -L. -Wl,--whole-archive -lstatic

This inserts all object file components from the static library directly into the new shared library, so we can find the symbols again!

Finally we get back to the title question..

Of course, the above isn’t the full story. Most libraries nowadays limit the visibility of their functions. For those that want the full story, just read this.

You might be thinking: okay, I don’t really care about ‘protecting’ any part of my code, so should I ever use this if I’m writing a .so? The answer is still yes, because it is likely to generate more optimal and smaller code.

We could have built the shared library without linking the static one

Before we go into the details, let’s return to the example above, and assume we linked libstatic.a in the link step for the final executable, in order to fix the issue. You should have noticed that we are now linking libstatic.a at both the DSO creation of libshared.so as well as the executable itself, and this is not necessary; DSOs are perfectly happy with ‘postponing’ looking for a symbol until a later time, so in this particular scenario we can do the following:

# Build .so, leaving out all libstatic symbols for now
$(SHARED_LIB): $(SHARED_OBJ) $(STATIC_LIB)
	$(CXX) -shared -o $@ $(SHARED_OBJ)

# ...

# Build final executable, linking in libstatic symbols to solve both the DSO's
# used symbol (sharedaddscale) and the ones the symbols from object files
# that were left out (excluded)
$(EXEC): $(E_OBJ) $(SHARED_LIB)
	$(CXX) -o $@ $(E_OBJ) -L. -lshared -lstatic

This would have solved all our undefined reference problems, and compiles correctly.

Now we introduce hidden visibility..

Continuing from above, I’m now going to turn on -fvisibility=hidden for both static.cpp and static_exclude.cpp:

# Compile static.cpp to a static library (.a)
$(STATIC_LIB): $(STATIC_SRC)  
	$(CXX) $(CXXFLAGS) -fvisibility=hidden -c static.cpp -o static.o
	$(CXX) $(CXXFLAGS) -fvisibility=hidden -c static_exclude.cpp -o static_exclude.o
	ar rcs $@ static.o static_exclude.o

# Build .so, leaving out all libstatic symbols for now
$(SHARED_LIB): $(SHARED_OBJ) $(STATIC_LIB)
	$(CXX) -shared -o $@ $(SHARED_OBJ)

# Link e.o with the shared library to create the final executable
# Also link static library to solve undefined reference
$(EXEC): $(E_OBJ) $(SHARED_LIB)
	$(CXX) -o $@ $(E_OBJ) -L. -lshared -lstatic

I did not change the internal code whatsoever, just this Makefile. Building again now results in the following error (yes, this is a screenshot from an iPhone, shoutout to the makers of iSH):

Recap of what we did here and the possible fixes

The order of problems and solutions we just presented to reach this step is as follows:

1. First build the static library as per normal, then create the shared library using it with a simple -lstatic, and finally create the executable linking only the shared library. This causes an undefined reference, since our executable uses a function in an object that was dropped by the linker when creating the shared library.
2. We fixed this by linking our static library in the final executable as well. This fixes the undefined reference as we provide the object file containing the function we used.
3. Since we are linking our static library in 2 steps - the shared library and the executable - we can remove the static library link for the shared library and the executable will still compile correctly, so we do this.
4. We then turned on -fvisibility=hidden for all our initial static library source files. This causes the hidden symbol in DSO error, since ALL the functions we are using are now hidden by default, and hence cannot be used by our executable.

Now the fix for this is actually quite simple; we simply need to link the static library during the shared library creation again!

Doing this places our hidden static library symbols into the DSO, which can then be used by the shared library’s exported symbol. Dumping the symbols using nm libshared.so I see:

T _Z14sharedaddscaleiii
t _Z15notusedbysharedii
t _Z9addstaticii

where the T means that it is included and t means it is hidden (if it’s your first time, just look up nm or objdump output). Hence, the shared library now clearly contains the necessary hidden symbol for addstatic, and this is referenced by sharedaddscale in the final executable without having to look outside the current DSO; this is exactly what hidden visibility is meant to do.

Some concluding remarks

These recent discoveries have made me think about the importance of understanding not just the shiny algorithms (read: leet code) and computer science things, but also learning about the tools available to us as developers.

It’s like a carpenter learning about the optimal angle to hammer a nail but not knowing how to use an electric drill (I’m not a carpenter but I build IKEA things so hopefully that was reasonable).

Writing a printf-based C++ logger class was more of a journey than I originally thought..

I recently had to work with a codebase where the build process was so convoluted it couldn’t be run and debugged from within a Visual Studio instance, despite being a Visual Studio solution.

This was primarily because it had a bunch of Java components, which were mainly used for the UI, and that prevented it from being run as a standard C++ application within the debugger (or maybe you could? I couldn’t find a way..)

Regardless, the normal way - the way that everyone else working on it would do it - to debug was through logging. I had to make do, so I hopped on the faulty bandwagon.

printf enjoyers unite

Alright alright, it’s not C++, but I honestly think C-like printf with its format specifier syntax is the cleanest, most concise way to log different POD-type values.

printf(“a: %.6f\nb: %d\nc: %.2g\n”, a, b, c);

The equivalent of the above in C++ with stream-like things:

std::cout << “a: “ << std::fixed << std::setprecision(6) << a << std::endl << “b: “ << b << std::endl << “c: “ << std::scientific << std::setprecision(2) << c << std::endl;

Holy shit, what a monster. Even if you remove every namespace call to std:: and replace the endl calls with the newline characters, you’d still be sitting with a disgustingly long line of code.

It gets slightly better with C++20 and std::fmt:

std::cout << std::format(“a: {:%.6f}\nb: {:%d}\nc: {:%.2g}\n”, a, b, c);

But still, why write an extra function call to std::format and extra braces when I didn’t have to previously? No one is going to change my mind on this. Yes, I know it provides a lot more functionality, but I don’t really care for all those bells and whistles.

printf, but with RAII

Alright let’s just assume we want to have printf style logs, but still have access to the C-like defines in __FILE__ and __LINE__. That let us print things like

/some/path/badfile.cpp:237

by simply using some macro magic like

#define log(…) my_logger_func(__FILE__, __LINE__, __VA_ARGS__)

// on line 237 of badfile.cpp
log(“some msg”);

Libraries like this amazing one do exactly that.

But I wanted to encapsulate some other things, like the automatic closing of a log file upon destruction.

Here’s the first problem if you make a class and provide a method to log, instead of a macro; the preprocessor magic no longer works, because the substitution for __FILE__ and __LINE__ will now occur directly where it was written (inside your class method) instead of where you called the method.

// logger.h
struct Logger
{
…
// At line 25 for example
void log(…){
  printf(“%s %d: some log message.\n”, __FILE__, __LINE__);
}
…
}

// main.cpp
…
logger.log(…);
…
// This will always print 
// logger.h 25: some log message, regardless of where it’s called

This is obviously not what we want. The only way around this would have been to provide __FILE__ and __LINE__ on every log method call, which would obviously be a far worse experience.

// no one is going to do this
lgr.log(__FILE__, __LINE__, “what i actually want to log”);

C++20 and std::source_location

I said ‘would have been’ because in C++20 we now have a useful alternative with std::source_location. Now we can do this:

struct Logger{
  void log(std::source_location l = std::source_location::current()){
    printf(“%s:%d”, l.file_name(), l.line());
  }
}

The default argument gets substituted at the point the method is called, and we get back our true filename and line numbers!

But there’s now another problem: dealing with multiple, variable arguments for the substitutions.

In the world of C, we had __VA_ARGS__, or the corresponding variadic function helpers (see this). In C++, we have similar functionality with variadic templates (see this), also known as parameter packs.

Now this is great - we can do a variable number of arbitrarily typed substitutions, printf style - but there’s a catch.

Both of these requirements need to be at the end of the function signature. We need the default argument to be at the end, because otherwise it won’t compile.

struct Logger
{
    // this doesn’t compile
    template <typename… Args>
    void log(std::source_location l = std::source_location::current(), Args… args)
    {
      …
    }
}

But we also need the parameter pack to be at the end, otherwise it won’t be able to automatically predict the template types for us:

struct Logger
{
    // this may compile, but may give unexpected results
    // because the final argument in the function call 
    // will be directed to the std::source_location variable
    // in most cases it will fail since the supplied 
    // argument cannot be converted to a std::source_location
    template <typename… Args>
    void log(const char* s, Args… args, std::source_location l = std::source_location::current())
    {
      …
    }
}

// main.cpp
lgr.log(“my custom message %d %d”, lognumber, mylogvariable);
// the compiler will (attempt to) assign mylogvariable
// to replace the default std::source_location argument

There’s a very good discussion of this exact problem - which I referenced while coming up with my own flavour of a solution - here. For example, one way to ensure that the parameter unpacking template works as intended is to specify the exact types:

lgr.log<int, int>(“my custom message %d %d”, lognumber, mylogvariable);
// this will print what you would expect

But this is a lot more verbose than I would like; it should be clear that with 10 variable substitutions you’d need to specify 10 types, so it can quickly get out of hand.

The most concise compromise I could tolerate

My number one goal was to ensure as few characters typed, when compared to the original printf form.

I decided to use the ‘constructor template’ to achieve this. The final code looks something like this:

struct Writer
{
  std::source_location m_l;

  Writer(std::source_location l) : m_l(l)
  {
    …
  }

  template <typename… Args>
  log(const char* s, Args… args)
  {
    …
  }
  
}

struct Logger
{
  …
  Writer operator()(std::source_location l=std::source_location::current())
  {
    return Writer(l);
  }
  …
}

// main.cpp

// writer class constructed
//   ^   writer class method invoked
//   |   ^
lgr().log(“my message %d %d”, 123, 456);

The idea is this:

1. Use operator() to instantiate a writer class, which is constructed with the correct value of std::source_location.
2. The new writer class then contains the necessary logging methods like log(), with the parameter packs we want. The file and line numbers are substituted directly from the class member variable.

This has the benefit of separating the ‘file holding’ class from the ‘log writing’ class, and allows us to fix the fight between the parameter pack and default argument order.

Is this the most efficient? Probably not - I think the compiler will elide the copy during the construction of the writer, but you would still construct on every line you log.

Is it the safest? Also probably not - in the actual code I put some safeguards like having a private constructor, but I’m sure there will be some ways that something can go wrong (and that I’ll update the code with as I find out).

But I got what I wanted: a short logging method call that is barely longer than a printf, and works exactly the same way.

The final code is here if you’d like to see or use it!

IPP in particular has some very niche ways of packing R2C DFT output, but otherwise there’s a few pointers here to keep in mind for how they are implemented in most libraries.

It’s pretty well known that the output of a Fourier transform of real inputs has symmetric properties. This is due to the fact that real waves consist of two conjugate pairs of complex exponentials.

What may be a bit less obvious (at least to me, when examining some programming libraries) is exactly how many useful output elements there are, and under which scenarios. In particular, I looked at IPP (which has some special packed structure), and cuFFT/NumPy/SciPy (which follow the FFTW structure I think).

The 1D Definition

We should start here before going on to 2D things. It’s easily wiki-able, but here’s our 1D DFT:

where $x_n$ consists of real elements in a length $N$ array.

Conjugate pairs

The first obvious redundancy happens when you recognise

So roughly half the output is redundant. But for programming purposes, let’s be a bit more specific.

Odd $N$

The 0-th element is self-conjugate, so it must be completely real. The remaining elements can be paired up, so in general there will be $\frac{N-1}{2}$ conjugate pairs of complex values that are useful.

In total this makes for $2\frac{N-1}{2} + 1 = N$ useful real values.

Even $N$

The 0-th element is self-conjugate and is completely real again. However there is another element at $X_{N/2} = {X_{N-N/2}}^*$ that by definition must also be self-conjugate, and hence completely real. The rest of the elements then comprise the $\frac{N-2}{2}$ conjugate pairs of useful complex values.

This again makes for a total of $\frac{N-2}{2} + 2 = N$ useful real values.

In all cases, the number of useful, real values is equal to $N$. This shouldn’t be too surprising since the dimensionality cannot change!

Library output formats

FFTW-like complex output (NumPy/SciPy/cuFFT/IPP CCS)

Most libraries return a fully complex-valued, truncated output, with length $N/2 +1$. This handles both odd and even valued $N$ correctly (assuming you do integer division for $N/2$), with the last value having a non-zero imaginary component only for odd $N$.

This is equivalent to converting the input to complex first - NumPy/SciPy will do this for you - running a standard FFT, then dropping the second half(ish).

# x has length 5
In [8]: sp.fft.fft(x)
Out[8]: array([-2.3526919 -0.j        ,  1.37635917-0.27259233j,  0.5953175 -0.55606247j,  0.5953175 +0.55606247j,  1.37635917+0.27259233j])

# So this stops at 5//2 + 1 = 3, and last element has a non-zero imaginary component
In [9]: sp.fft.rfft(x)
Out[9]: array([-2.3526919 +0.j        ,  1.37635917-0.27259233j,  0.5953175 -0.55606247j])

# .....

# Now x has length 6
In [11]: sp.fft.fft(x)
Out[11]: array([-0.391538  -0.j        , -2.01281022-0.2942553j ,  2.76346742+2.30952267j,  1.36395615-0.j        ,  2.76346742-2.30952267j, -2.01281022+0.2942553j ])

# And this stops at 6//2 + 1 = 4, with the last element having zero for its imaginary component
In [12]: sp.fft.rfft(x)
Out[12]: array([-0.391538  +0.j        , -2.01281022-0.2942553j ,  2.76346742+2.30952267j,  1.36395615+0.j        ])

So far this shouldn’t be hard to accept; you’re just taking all the positive frequencies from a standard Fourier transform (with some handling for the edge bin).

Packed output (IPP Pack)

IPP has packed formats documented for its signal processing library part. I think the documentation there describes it sufficiently well, but in case you’d like to see a concrete example, here’s the extension using the above python array:

# This is the output of the rfft for the length 6 array
In [14]: y
Out[14]: array([-0.391538  +0.j        , -2.01281022-0.2942553j ,  2.76346742+2.30952267j,  1.36395615+0.j        ])

# And this is how IPP would effectively pack the output
In [15]: np.hstack((y[0].real, y[1:-1].view(np.float64), y[-1].real))
Out[15]: array([-0.391538  , -2.01281022, -0.2942553 ,  2.76346742,  2.30952267,  1.36395615])

Since there are always $N$ real values in the output, we can drop the 0-valued imaginary components (the 0-index, and the N/2-index for the even $N$ case) and simply squeeze everything together. This would make the output a bit shorter.

Notably, I haven’t seen any other library opt to this, since it would require operating on the output in a non-uniform way; you would have to remember that the first (and possibly the last) output has 1 real element, while everything in between has 2 real elements (1 complex element). That means you can’t traverse it with the same pointer.

Understandably, this is probably why it isn’t seen elsewhere. IPP has a variation of choices in what output formats you can select - but not always, since in its image processing section you are forced to use this output format, and then convert it to another format if you wish after.

2D transforms and their R2C output dimensions

At first glance, you might have thought that since 1D transforms can be defined by half the output, 2D transforms could be defined by $1/4$ of them right?

Okay if not, then you’re smarter than I was. After I had realised the dimensionality argument for the 1D case, the 2D case became equally clear.

In an $M \times N$ real matrix, the output of the transform must contain $MN$ real, useful values.

It shouldn’t be hard to see that this means that you can still only drop about half of the full complex 2D DFT output.

But how do libraries choose what to keep? It turns out everyone seems to have chosen to agree on this part.

Slice the fastest changing dimension

The title says it all. In a typical array defined by, for example

std::complex<float> out[a][b];

the b index is the one that gets truncated. This also follows for 3D transforms, though I’m not going to bother with that here.

Again, there’s no mathematical reason behind this; it’s merely a convention that seems to be held by every library: NumPy/SciPy, cuFFT, etc.

To see why you could slice it the other way, let’s consider an example.

# We now have a random 6x6 input array
In [25]: sp.fft.fft2(x)
Out[25]:
array([[ 11.4684-0.0000e+00j,   7.1864+9.1749e+00j,   4.1346-3.6415e+00j,   0.9137-0.0000e+00j,   4.1346+3.6415e+00j,   7.1864-9.1749e+00j],
       [  5.6947+1.9658e-01j,  -0.5213+2.2622e+00j, -10.9693-4.7503e+00j,  -3.6554+2.2839e-01j,  -1.3505+4.1515e+00j,   3.4155+3.9835e+00j],
       [  8.1551+2.6689e+00j,  -5.7527-4.1722e+00j,   0.6473-1.1867e+00j,  -4.3752+5.1961e+00j,  -3.7517+1.5745e+00j,  -2.1656+9.6490e+00j],
       [ -1.089 -2.9802e-08j,   1.31  -1.0136e+01j,  -4.7408-7.9127e-01j,   8.455 +3.7253e-08j,  -4.7408+7.9127e-01j,   1.31  +1.0136e+01j],
       [  8.1551-2.6689e+00j,  -2.1656-9.6490e+00j,  -3.7517-1.5745e+00j,  -4.3752-5.1961e+00j,   0.6473+1.1867e+00j,  -5.7527+4.1722e+00j],
       [  5.6947-1.9658e-01j,   3.4155-3.9835e+00j,  -1.3505-4.1515e+00j,  -3.6554-2.2839e-01j, -10.9693+4.7503e+00j,  -0.5213-2.2622e+00j]], dtype=complex64)

# As you can see the output of this is simply indexing the output above like [:,:4]
# i.e. taking all rows, but cutting at the equivalent 1D required column
# Note that numpy arrays are row-major by default, so this has truncated the fastest changing dimension
In [26]: sp.fft.rfft2(x)
Out[26]:
array([[ 11.4684+0.0000e+00j,   7.1864+9.1749e+00j,   4.1346-3.6415e+00j,   0.9137+0.0000e+00j],
       [  5.6947+1.9658e-01j,  -0.5213+2.2622e+00j, -10.9693-4.7503e+00j,  -3.6554+2.2839e-01j],
       [  8.1551+2.6689e+00j,  -5.7527-4.1722e+00j,   0.6473-1.1867e+00j,  -4.3752+5.1961e+00j],
       [ -1.089 +2.9802e-08j,   1.31  -1.0136e+01j,  -4.7408-7.9127e-01j,   8.455 -3.7253e-08j],
       [  8.1551-2.6689e+00j,  -2.1656-9.6490e+00j,  -3.7517-1.5745e+00j,  -4.3752-5.1961e+00j],
       [  5.6947-1.9658e-01j,   3.4155-3.9835e+00j,  -1.3505-4.1515e+00j,  -3.6554-2.2839e-01j]], dtype=complex64)

If you look closely at the original, full output of the 2D FFT, you can separate it into a few block matrix sections that contain the conjugate pairs:

|A|B|B|B|B|B|
|C|D|D|D|D|D|
|C|D|D|D|D|D|
|C|D|D|D|D|D|
|C|D|D|D|D|D|
|C|D|D|D|D|D|

In each block, the conjugate pairs are nicely positioned on what I’d like to call opposite ends of a line passing through the block origin. Here are some conjugate pairs you can verify, together with the example output above:

# Block D
|A|B|B|B|B|B|
|C|x| | | | |
|C| | | | | |
|C| | | | | |
|C| | | | | |
|C| | | | |x|

# Block D
|A|B|B|B|B|B|
|C| |x| | | |
|C| | | | | |
|C| | | | | |
|C| | | | | |
|C| | | |x| |

# Block D
|A|B|B|B|B|B|
|C| | | | | |
|C| | | |x| |
|C| | | | | |
|C| |x| | | |
|C| | | | | |

# Block D, self-conjugate i.e. 0 imaginary component
# Note that numerical accuracy sometimes leaves it as a small number instead..
|A|B|B|B|B|B|
|C| | | | | |
|C| | | | | |
|C| | |x| | |
|C| | | | | |
|C| | | | | |

# Block C
|A|B|B|B|B|B|
| |D|D|D|D|D|
|x|D|D|D|D|D|
| |D|D|D|D|D|
|x|D|D|D|D|D|
| |D|D|D|D|D|

# Block C self-conjugate
|A|B|B|B|B|B|
| |D|D|D|D|D|
| |D|D|D|D|D|
|x|D|D|D|D|D|
| |D|D|D|D|D|
| |D|D|D|D|D|

Of course, this is just graphically representing the conjugate pair relationship in 2D:

The idea is you’d need to keep at least one of each conjugate pair to have non-redundant output; and importantly, since the conjugate pairs are in block matrices, you can target to keep half of each block. The way all the libraries do it (probably because they do the transforms down the rows first, then down the columns) is to keep half of block $B$ and all of block $C$, chopping block $D$ vertically in the process.

|A|B|B|B|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|

However you could easily just chop block $D$ horizontally instead, and still keep an equivalent set of non-redundant output:

|A|B|B|B|B|B|
|C|D|D|D|D|D|
|C|D|D|D|D|D|
|C|D|D|D|D|D|
|-|-|-|-|-|-|
|-|-|-|-|-|-|

And finally, the IPP 2D packed format..

The RCPack2D format is found only in the image processing section.

Now this is really something. In theory, this is just extending the 1D packed format idea to 2D; everytime you meet an imaginary component of 0, you skip it (hence the Packed), but there’s a little more nuance to it.

The first thing to note is that the previous method can still contain redundant values after slicing. In our $6 \times 6$ example above, the final truncated output still has the following redundant pairs:

|A|B|B|B|-|-|
|x|D|D|D|-|-|
| |D|D|D|-|-|
| |D|D|D|-|-|
| |D|D|D|-|-|
|x|D|D|D|-|-|

|A|B|B|B|-|-|
| |D|D|D|-|-|
|x|D|D|D|-|-|
| |D|D|D|-|-|
|x|D|D|D|-|-|
| |D|D|D|-|-|

|A|B|B|B|-|-|
|C| | |x|-|-|
|C| | | |-|-|
|C| | | |-|-|
|C| | | |-|-|
|C| | |x|-|-|

|A|B|B|B|-|-|
|C| | | |-|-|
|C| | |x|-|-|
|C| | | |-|-|
|C| | |x|-|-|
|C| | | |-|-|

I didn’t include the self-conjugate centre indices in blocks $B,C,D$ as they should be treated the same as the 1D case.

So there can be a shorter first and last column. Hopefully, it shouldn’t be too hard to see that for an odd number of columns, the last column will not have this issue.

Okay, so let’s get rid of the extra redundant indices,

|A|B|B|B|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|
|C|D|D|D|-|-|
|-|D|D|-|-|-|
|-|D|D|-|-|-|

and then let’s label the fully real \(R\) and fully complex (*) indices:

|R|*|*|R|-|-|
|*|*|*|*|-|-|
|*|*|*|*|-|-|
|R|*|*|R|-|-|
|-|*|*|-|-|-|
|-|*|*|-|-|-|

Here’s where IPP’s packed format comes in:

1. The first (and in this case also the last) column is flattened to real values into a column. This becomes a length-6 column - 2 from reals and $4=2 \times 2$ from complex values - and is written into the first (and last) column of a $6 \times 6$ matrix.
2. The columns in between, which are flattened to real values into their respective rows. This will become - in this case - 6 rows and 4 columns of real values.

Hence, the final output becomes a $6 \times 6$ real-valued matrix, with no redundant values whatsoever.

Sounds like a lot of work for not a lot of gain?

I think so too. By the logic above, you’d save roughly $2M/2 = M$ values worth of memory for a general $M \times N$ matrix. Even worse, if someone had to operate on this other than IPP, you’d have to unfold this anyway, or have to remember to deal with these first/last column shenanigans.

Yes, it’s yet another blogpost about CRTP and how it’d be useful..

The 1st issue: method chaining

Have you seen a billion other blogposts about CRTP? Yes, so have I. But maybe there’s a reason for all of them; it wasn’t really apparent when reading them previously why it would be useful and/or why I would ever need it.

But recently, while writing some simple templated code for ufl, I had the bright idea of trying to make method chaining possible for the class. That’s when my templated class and its derived friend implementation started to fall apart. I fixed this by using CRTP for the first time.

I think this post is probably the best explanation for how CRTP solves this problem, so I’m not going to repeat it here, but rather leave a link (for myself). Bonus points for not having ads all over the place on that page.

The 2nd issue: static polymorphism

I remember stumbling on a video that removed virtual functions with templates a few years back. At the time, I didn’t really grasp the concept very well (is this what they call experience?), but it’s starting to dawn on me I think.

While writing some code for uhdeb, I tried to wrap 2 similar but related streamer object classes in a container. I wanted a parent class that would hold a different type of streamer object - a tx_streamer or a rx_streamer - and to perform this initialization in the constructor (using another common object type that was passed in).

My first thought was to define a parent templated class that contained a T m_stream. But now I had 2 choices:

1. I write a custom constructor for each of the derived classes. This would call the appropriate initialization for each type of m_stream. But I wanted the constructor to also do similar things for both derived classes - allocate some memory, start a thread - and so I would have had to copy all those calls, violating DRY.
2. Do some virtual calls.

The code (roughly speaking) looked like this:

template <typename T>
class Parent
{
    ...

    T m_stream;
};

class DerivedRX : Parent<RXStreamer>
{
    DerivedRX(...) : Parent(...)
    {
        // create the RXStreamer..
        ...

        // the rest depend on the streamer, so I couldn't throw it into Parent's constructor
        allocate();
        start_thread();
    }
};

class DerivedTX : Parent<TXStreamer>
{
    DerivedTX(...) : Parent(...)
    {
        // create the TXStreamer..
        ...

        // the rest depend on the streamer, so I couldn't throw it into Parent's constructor
        allocate();
        start_thread();
    }
};

Could I have refactored the code to not have this problem? Probably. But at the time I was adamant on finding a way of getting this to work.

The way I did it was to push all the ‘standard’ constructor logic to the parent class. The parent class would call the appropriate derived class’s create_stream method through CRTP:

template <typename T, typename U>
class Parent
{
    Parent()
    {
        static_cast<U*>(this)->create_stream();

        // the rest of the ctor..

    }


    T m_stream;
};

class DerivedRX : Parent<RXStreamer, DerivedRX>
{
    DerivedRX(...) : Parent(...)
    {
        // nothing else needs to be done..
    }

    void create_stream()
    {
        // custom RX code..
    }
};

class DerivedTX : Parent<TXStreamer, DerivedTX>
{
    DerivedTX(...) : Parent(...)
    {
        // nothing else needs to be done
    }

    void create_stream()
    {
        // custom TX code..
    }
};

This avoids any use of virtual methods - no vtables here hoho - and the correct derived method is called for each derived class!

Oh, I also had to add a friend class declaration within the derived class definitions in order for it to work.

Now, if you look at this and are thinking: couldn’t he have just moved the custom streamer types back into the derived class definitions? Then there would be no need for any of these shenanigans, and there would just be simple parent and derived classes. And you would be right.

But like I said, I was exploring, and it was interesting.

Going to leave some more references here for myself:

1. https://www.fluentcpp.com/2017/05/16/what-the-crtp-brings-to-code/
2. https://eli.thegreenplace.net/2013/12/05/the-cost-of-dynamic-virtual-calls-vs-static-crtp-dispatch-in-c

I’m sure some day soon I’ll find a stronger use-case for this. I never really liked doing virtual methods after all.

Carrier offset and baud rate estimation can be done blindly using cyclostationary (cyclic moment) methods, but why?

Introduction

In ReImage, there are options to blindly estimate a signal’s baud rate or residual carrier offset. These require some working knowledge, and only work on some types of modulations like PSK. But why do they work, and how do we explain the peaks we see in the resulting spectra?

First, some terminology. Here I’ll often mention the exponents in a CM XY form: this refers to applying an exponent onto a signal with a total of $X$, where $Y$ of that is the conjugate. These refer to the cyclic moments.

For example, CM 20 refers to $x^2$ whereas CM 21 refers to $x x^*$.

To see a proper treatment, refer to the man himself. Otherwise, onwards we go..

Carrier offset estimation

Let’s consider a generic QPSK signal given by:

where the symbols are given by

The baud period here is specified by $T$ and the carrier offset to be estimated is given by $f_c$.

The correct thing to do here is to find some values of exponents where the result contains additive tones. These will show up in a specturm of FFT plot as peaks that can be easily measured.

We claim now the required operation is CM 40 i.e.

Let’s see what this does to the signal. First, observe that the exponent applied directly to the message symbols always results in 1:

What happens when we first try to square the terms in the square brackets?

There are 2 terms at play here: intra-term products and inter-term products.

Let’s first consider the inter-term products:

It shouldn’t be too difficult to see that this should tend to 0, since for a random variable $m_k$, the expectation value of $m_k m_{k'}$ should approach 0. Moreover, most reasonable time-limited pulse shapes $h(t)$ only have non-negligible support within 1 symbol i.e. within $T$ of each other. Hence, the contribution of non-zero products is also limited.

Hence we turn our attention to the intra-term products:

We square again to get our exponent of 4, obtaining

where we have substituted ${m_k}^4 = 1$.

What can we say about this final expression? Let’s bring the summation back into the picture:

We start the analysis by considering what would happen if we used a $\delta$ function as a pulse:

Well the Fourier transform of this Dirac comb is simply another Dirac comb (see this):

Let’s put this together with the other parts of $s(t)$ to get

where we have left out any inter-term products and also the Fourier transform of ${m_k}^4$.

What does this suggest to us? The spectrum consists of components centred around $f = 4f_c$, with steps of $\pm1/T$, otherwise known as the baud rate.

But here we assumed the infinitesimal pulse shape $\delta$; what if we revert to a more practical pulse shape like an RRC? Heuristically, we need only consider the properties of Fourier transforms: the more support it has in time, the less support its transform has in frequency. Hence, we expect to have the Dirac comb in frequency space be multiplied (since we convolved with $h(t)$) with a pulse of some bandwidth.

That is, the amplitude of the Dirac comb in frequency space has an envelope defined by the Fourier transform of $h^4 (t) = H(f)$. Together with the other $\delta$-function, this shifts the entire enveloped Dirac comb to centre around $4f_c$.

Hence, we expect to see a large peak in the spectrum at $4f_c$, with smaller peaks uniformly at $1/T$ spacing.

My first steps into migrating to neovim, some clangd problems and my solutions..

Kickstart.nvim brought me here

I started using vim motions in VSCode a month or two ago, and decided to actually try to see if neovim would work for me. Honestly, I probably wouldn’t have gone down this path if kickstart.nvim didn’t exist. But it does, and I tried it, and it looks like I’m here to stay.

clangd as my first LSP

In kickstart.nvim’s template, clangd is listed as an example LSP, so I decided to uncomment that line and try it out. I opened my ffs repository at the time, and was immediately greeted with a flood of warnings on my code. This brings us to the first problem.

Header Resolution with clangd

My code was immediately flooded with warnings, to the point that clangd itself said it would stop reporting errors. A lot of them looked like this:

It seemed like most of them were related to header resolution; I had seen a similar type of error/warning in VSCode before, which always seemed to magically disappear after running cmake the first time.

Solution? Generate a compile_commands.json (see this). In particular, I needed to make sure this was near the root of the directory. In my original setup, I had separate CMakeLists.txt files in my examples/ and tests/ subdirectories, and would manually enter each subdirectory when I needed to do stuff there. However, this meant that when opening my include/ subdirectory’s files, I would be greeted with similar clangd errors, as it wouldn’t see the compile_commands.json generated from running cmake for the prior folders.

So I needed to write an outermost CMakeLists.txt, to ensure that the compile_commands.json appeared in the build/ subdirectory at the top. Enabling this explicitly is trivial by just placing this line in that top-level CMakeLists.txt:

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

Compile commands don’t work with MSVC

The second problem is related to the first. As it turns out, setting the above does absolutely nothing when using MSVC, which I was using since I was in Windows. I didn’t want to use MinGW-gcc here, so I thought I’d try the other generator in the link: Ninja.

After installing and calling cmake .. -G Ninja, the compile_commands.json was generated and everything worked as they should.

Other Ninja Benefits

1. I always didn’t like how MSVC had to be different by pushing the build type configuration to after the cmake call. Using ninja lets me have a more symmetric experience while retaining the use of MSVC, as now I can always specify -DCMAKE_BUILD_TYPE at the cmake step regardless of whether I’m on unix or Windows.
2. I get to make things by ninja and make clean by ninja clean. This is pretty satisfying.
3. Having compile_commands.json means I can see the literal command for every executable, rather than having to dig inside the .vcxproj to double-check every option that was enabled; previously I would just wait til compile time to see the command line arguments the project file would use.

Clang isn’t much better than MSVC for complex number multiplication, while Eigen is equivalent to GCC but uses slightly different instructions.

Complex Number Multiplication Is Probably Not Common Enough

I think I might have bashed MSVC too much on the last post; trying to vectorise our simple 4-element complex number multiplication using clang with AVX instructions delivers similarly poor results:

#include <complex>

void cmul(
    const std::complex<float>* __restrict__ x,
    const std::complex<float>* __restrict__ y,
    std::complex<float>* __restrict__ z
){
    for (int i = 0; i < 4; ++i)
        z[i] = x[i] * y[i];
}

void mul(
    const float* __restrict__ x,
    const float* __restrict__ y,
    float* __restrict__ z
){
    for (int i = 0; i < 8; ++i)
        z[i] = x[i] * y[i];
}

cmul(std::complex<float> const*, std::complex<float> const*, std::complex<float>*):          # @cmul(std::complex<float> const*, std::complex<float> const*, std::complex<float>*)
        vmovsd  xmm0, qword ptr [rdi]           # xmm0 = mem[0],zero
        vmovsd  xmm1, qword ptr [rsi]           # xmm1 = mem[0],zero
        vbroadcastss    xmm2, xmm0
        vmovshdup       xmm0, xmm0              # xmm0 = xmm0[1,1,3,3]
        vshufps xmm3, xmm1, xmm1, 225           # xmm3 = xmm1[1,0,2,3]
        vmulps  xmm0, xmm3, xmm0
        vfmaddsub231ps  xmm0, xmm1, xmm2        # xmm0 = (xmm1 * xmm2) +/- xmm0
        vmovlps qword ptr [rdx], xmm0
        vmovsd  xmm0, qword ptr [rdi + 8]       # xmm0 = mem[0],zero
        vmovsd  xmm1, qword ptr [rsi + 8]       # xmm1 = mem[0],zero
        vbroadcastss    xmm2, xmm0
        vmovshdup       xmm0, xmm0              # xmm0 = xmm0[1,1,3,3]
        vshufps xmm3, xmm1, xmm1, 225           # xmm3 = xmm1[1,0,2,3]
        vmulps  xmm0, xmm3, xmm0
        vfmaddsub231ps  xmm0, xmm1, xmm2        # xmm0 = (xmm1 * xmm2) +/- xmm0
        vmovlps qword ptr [rdx + 8], xmm0
        vmovsd  xmm0, qword ptr [rdi + 16]      # xmm0 = mem[0],zero
        vmovsd  xmm1, qword ptr [rsi + 16]      # xmm1 = mem[0],zero
        vbroadcastss    xmm2, xmm0
        vmovshdup       xmm0, xmm0              # xmm0 = xmm0[1,1,3,3]
        vshufps xmm3, xmm1, xmm1, 225           # xmm3 = xmm1[1,0,2,3]
        vmulps  xmm0, xmm3, xmm0
        vfmaddsub231ps  xmm0, xmm1, xmm2        # xmm0 = (xmm1 * xmm2) +/- xmm0
        vmovlps qword ptr [rdx + 16], xmm0
        vmovsd  xmm0, qword ptr [rdi + 24]      # xmm0 = mem[0],zero
        vmovsd  xmm1, qword ptr [rsi + 24]      # xmm1 = mem[0],zero
        vbroadcastss    xmm2, xmm0
        vmovshdup       xmm0, xmm0              # xmm0 = xmm0[1,1,3,3]
        vshufps xmm3, xmm1, xmm1, 225           # xmm3 = xmm1[1,0,2,3]
        vmulps  xmm0, xmm3, xmm0
        vfmaddsub231ps  xmm0, xmm1, xmm2        # xmm0 = (xmm1 * xmm2) +/- xmm0
        vmovlps qword ptr [rdx + 24], xmm0
        ret
mul(float const*, float const*, float*):                         # @mul(float const*, float const*, float*)
        vmovups ymm0, ymmword ptr [rsi]
        vmulps  ymm0, ymm0, ymmword ptr [rdi]
        vmovups ymmword ptr [rdx], ymm0
        vzeroupper
        ret

Compiled with clang 18.1.0, with -O3 -ffast-math -march=x86-64-v3, this still refuses to vectorise the cmul function correctly, only using the xmm registers. I included the normal real-valued float multiplication to check that clang is indeed able to vectorise that. Note that you still need the __restrict__ keywords for the vectorisation to work. Using -ffast-math doesn’t seem to do anything for us in the real float vectorisation, but it does make the complex-valued vectorisation less verbose.

There’s probably not enough complex number math code out there in the wild, and so only the most used compilers - like GCC - actually have decent vectorisation when it comes to this.

How does Eigen do it?

Since we’re on the topic, I decided to try and see what Eigen’s implementation looks like. After all - and I just discovered yet another amazing feature here - godbolt.org actually allows you to include popular libraries, right there in the online interface!

So we have the following, compiled with GCC:

#include <Eigen/Dense>
#include <complex>


void eigCmul4(
    const Eigen::Array4cf& x,
    const Eigen::Array4cf& y,
    Eigen::Array4cf &z
){
    z = x * y;
}

void cmul4(
    const std::complex<float> * __restrict__ x,
    const std::complex<float> * __restrict__ y,
    std::complex<float> * __restrict__ z
){
    for (int i = 0; i < 4; ++i)
        z[i] = x[i] * y[i];
}

We use Array here instead of Vector or Matrix since we want the element-wise products.

The assembly looks like this:

eigCmul4(Eigen::Array<std::complex<float>, 4, 1, 0, 4, 1> const&, Eigen::Array<std::complex<float>, 4, 1, 0, 4, 1> const&, Eigen::Array<std::complex<float>, 4, 1, 0, 4, 1>&):
        vmovaps ymm1, YMMWORD PTR [rdi]
        vmovaps ymm0, YMMWORD PTR [rsi]
        vmovsldup       ymm3, ymm1
        vmovshdup       ymm1, ymm1
        vpermilps       ymm2, ymm0, 177
        vmulps  ymm1, ymm2, ymm1
        vmulps  ymm0, ymm0, ymm3
        vaddsubps       ymm0, ymm0, ymm1
        vmovaps YMMWORD PTR [rdx], ymm0
        vzeroupper
        ret

cmul4(std::complex<float> const*, std::complex<float> const*, std::complex<float>*):
        vmovups ymm0, YMMWORD PTR [rdi]
        vmovups ymm3, YMMWORD PTR [rsi]
        vpermilps       ymm1, ymm0, 160
        vpermilps       ymm2, ymm3, 177
        vpermilps       ymm0, ymm0, 245
        vmulps  ymm0, ymm2, ymm0
        vfmaddsub231ps  ymm0, ymm1, ymm3
        vmovups YMMWORD PTR [rdx], ymm0
        vzeroupper
        ret

Aside from the aligned vs non-aligned mov instructions, the main difference is that Eigen uses vmovsldup and vmovshdup instead of vpermilps for two of the instructions. However, it turns out this choice is insignificant, as the same number of cycles are used for permute and shuffle:

Timeline view:
                    012
Index     0123456789   

[0,0]     DeER .    . .   vmovsldup     ymm3, ymm1
[0,1]     DeER .    . .   vmovshdup     ymm1, ymm1
[0,2]     D=eER.    . .   vpermilps     ymm2, ymm0, 177
[0,3]     D==eeeeER . .   vmulps        ymm1, ymm2, ymm1
[0,4]     D=eeeeE-R . .   vmulps        ymm0, ymm0, ymm3
[0,5]     D======eeeeER   vaddsubps     ymm0, ymm0, ymm1

Timeline view:
                    01
Index     0123456789  

[0,0]     DeER .    ..   vpermilps      ymm1, ymm0, 160
[0,1]     DeER .    ..   vpermilps      ymm2, ymm3, 177
[0,2]     D=eER.    ..   vpermilps      ymm0, ymm0, 245
[0,3]     D==eeeER  ..   vmulps ymm0, ymm2, ymm0
[0,4]     D=====eeeeER   vfmaddsub231ps ymm0, ymm1, ymm3

FMA instructions and using -march=x86-64-v3

If you noticed, the GCC code this time is slightly different from the last post. That’s because I discovered that it’s a lot easier to specify the entire instruction set using -march=x86-64-v3, which includes all of the following, taken from the wikipedia page: AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE.

Honestly, don’t know what half of them are right now, but hey, FMA and AVX are both in there and are relevant to what I do.

Surprisingly though, it seems like there’s very little real difference in terms of number of cycles (only 1 less, since we have to wait for the previous mul instructions to finish), so I guess the benefit for this case is primarily in higher precision using FMA.

MSVC has extremely lackluster auto-vectorisation, so I handrolled intrinsic calls by backtranslating GCC’s output.

The Motivation

I recently decided I wanted to spend some time understanding intrinsics and SIMD at a deeper level.

In developing code for my project ffs, I wanted to make sure that the code was running with at least AVX instructions (because that’s my target architecture, and honestly very few computers don’t have AVX these days..).

This led me down a path of discovery; first I discovered the amazing-ness that is godbolt.org, then I joined their discord, where I then asked for some help with understanding basic .asm compiler output.

Ultimately though, I needed to know how to write SIMD-tuned functions for complex numbers i.e. std::complex<T>. These were the types that were going into ffs.

Old but Gold Stackoverflows

Some google searching led me to one of the best references for complex-number SIMD multiplication in an answer by Peter Cordes on StackOverflow.

Although it’s almost 8 years old at this point, his recommendation is still valid: use -ffast-math and label arrays with __restrict__, and GCC will do a pretty damned good job vectorising with -mavx on.

So with that I placed essentially the same code and compiler arguments - -O3 -ffast-math -mavx - into godbolt again and re-tried it with newer compilers.

I tried first with a simple loop over 4 std::complex<float> values, enough to fill a 256-bit AVX YMM register:

for (int i = 0; i < 4; ++i)
{
    z[i] = x[i] * y[i];
}

And indeed, x86-64 GCC 13.2 outputs the following:

vmovups ymm2, YMMWORD PTR [rdi]
vmovups ymm0, YMMWORD PTR [rsi]
vpermilps       ymm1, ymm2, 160
vpermilps       ymm2, ymm2, 245
vmulps  ymm1, ymm1, ymm0
vpermilps       ymm0, ymm0, 177
vmulps  ymm0, ymm0, ymm2
vaddsubps       ymm1, ymm1, ymm0
vmovups YMMWORD PTR [rdx], ymm1
vzeroupper
ret

MSVC.. doesn’t auto-vectorise

Using MSVC, the equivalent compiler options would be to use /O2 /fp:fast /arch:AVX. You also have to change __restrict__ to __restrict. This, however, emits the following:

    sub     rsp, 24
    vmovss  xmm5, DWORD PTR [rcx]
    vmulss  xmm1, xmm5, DWORD PTR [rdx]
    vmulss  xmm2, xmm5, DWORD PTR [rdx+4]
    vmovss  xmm5, DWORD PTR [rcx+8]
    vmovaps XMMWORD PTR [rsp], xmm6
    vmovss  xmm6, DWORD PTR [rcx+4]
    vmulss  xmm0, xmm6, DWORD PTR [rdx+4]
    vsubss  xmm3, xmm1, xmm0
    vmulss  xmm1, xmm6, DWORD PTR [rdx]
    vmovss  xmm6, DWORD PTR [rcx+12]
    vaddss  xmm0, xmm2, xmm1
    vmulss  xmm1, xmm5, DWORD PTR [rdx+8]
    vmulss  xmm2, xmm5, DWORD PTR [rdx+12]
    vmovss  xmm5, DWORD PTR [rcx+16]
    vmovss  DWORD PTR [r8+4], xmm0
    vmulss  xmm0, xmm6, DWORD PTR [rdx+12]
    vmovss  DWORD PTR [r8], xmm3
    vsubss  xmm3, xmm1, xmm0
    vmulss  xmm1, xmm6, DWORD PTR [rdx+8]
    vmovss  xmm6, DWORD PTR [rcx+20]
    vaddss  xmm0, xmm2, xmm1
    vmulss  xmm1, xmm5, DWORD PTR [rdx+16]
    vmulss  xmm2, xmm5, DWORD PTR [rdx+20]
    vmovss  xmm5, DWORD PTR [rcx+24]
    vmovss  DWORD PTR [r8+12], xmm0
    vmulss  xmm0, xmm6, DWORD PTR [rdx+20]
    vmovss  DWORD PTR [r8+8], xmm3
    vsubss  xmm3, xmm1, xmm0
    vmulss  xmm1, xmm6, DWORD PTR [rdx+16]
    vmovss  xmm6, DWORD PTR [rcx+28]
    vaddss  xmm0, xmm2, xmm1
    vmulss  xmm1, xmm5, DWORD PTR [rdx+24]
    vmulss  xmm2, xmm5, DWORD PTR [rdx+28]
    vmovss  DWORD PTR [r8+20], xmm0
    vmulss  xmm0, xmm6, DWORD PTR [rdx+28]
    vmovss  DWORD PTR [r8+16], xmm3
    vsubss  xmm3, xmm1, xmm0
    vmulss  xmm1, xmm6, DWORD PTR [rdx+24]
    vmovaps xmm6, XMMWORD PTR [rsp]
    vaddss  xmm0, xmm2, xmm1
    vmovss  DWORD PTR [r8+28], xmm0
    vmovss  DWORD PTR [r8+24], xmm3
    add     rsp, 24
    ret     0

Obviously, not a single YMM register used, and no packed instructions either. You can enable the following MSVC compiler flag, /Qvec-report: 2, to ask it why it didn’t vectorise the loop:

<source>(15) : info C5002: loop not vectorized due to reason '1200'

Reason 1200 is apparently ‘Loop contains loop-carried data dependencies that prevent vectorization. Different iterations of the loop interfere with each other such that vectorizing the loop would produce wrong answers, and the auto-vectorizer can’t prove to itself that there aren’t such data dependencies.’, which obviously is complete garbage.

I posted all these to the godbolt discord to ask the grandmasters of assembly over there if there was any way around MSVC’s issues:

If you’d like to play around with the MSVC implementation, here’s the link on godbolt.

Since GCC works, just copy them?

I needed the above to work with MSVC and Windows, so the easiest thing I could think of was to just reverse-translate GCC’s output into the appropriate intrinsic calls; I just went to Intel’s catalogue and searched them one by one. The model answers are literally given, so why not right?

Word for word translation of the GCC instructions:

__m256 ymm2 = _mm256_loadu_ps(reinterpret_cast<const float*>(x));
__m256 ymm0 = _mm256_loadu_ps(reinterpret_cast<const float*>(y));

__m256 ymm1 = _mm256_permute_ps(ymm2, 160);
ymm2 = _mm256_permute_ps(ymm2, 245);

ymm1 = _mm256_mul_ps(ymm1, ymm0);

ymm0 = _mm256_permute_ps(ymm0, 177);

ymm0 = _mm256_mul_ps(ymm0, ymm2);

ymm1 = _mm256_addsub_ps(ymm1, ymm0);

_mm256_storeu_ps(reinterpret_cast<float*>(z), ymm1);

And their assembly output:

vmovups ymm1, YMMWORD PTR [rdi]
vmovups ymm0, YMMWORD PTR [rsi]
vpermilps       ymm3, ymm1, 160
vpermilps       ymm2, ymm0, 177
vpermilps       ymm1, ymm1, 245
vmulps  ymm0, ymm0, ymm3
vmulps  ymm1, ymm1, ymm2
vaddsubps       ymm0, ymm0, ymm1
vmovups YMMWORD PTR [rdx], ymm0
vzeroupper
ret

If you look closely, the assembly is not 100% identical, but it is functionally identical (just track the registers and you’ll see it).

Performance-wise, there is also no difference, as confirmed by the man himself:

Links to my godbolt are here. Note that if you change it back to MSVC, the original movups instructions will disappear (this seems to be a thing with MSVC assuming/optimizing the arguments away directly into registers).

Some Concluding Remarks

I didn’t go into the workings of how the complex number multiplies work, because that’s pretty much inside the stackoverflow link.

Also, some might ask why I didn’t use a library like Eigen. Yes, I could have, but that would require including a giant library just for one function which is ffs. Also, it was a pretty good learning opportunity.

Intel Performance Primitives is not guaranteed to work on non-Intel chips, but there are some ways around it..

Over the last year or so I’ve written a C++ wrapper of header-only templates around a library known as Intel Performance Primitives (IPP). I often use this for its signal processing library, which is - at least by my measurements - one of the fastest, if not the fastest one around. It also has the benefit of having almost everything I need in one place: FFT/DFTs, math array processing, low-pass filtering etc. You can see my templates at the ipp_ext repository.

Intel themselves make no promises about IPP’s performance on non-Intel chipsets; in fact, they recently pulled support for MacOS, since Apple has gone with Apple Silicon now.

This is trouble for people like me who develop for Intel chipsets, but on non-Intel chipsets (like my home computer and my Macbook).

Dynamic Linking is the Problem

On Apple Silicon, any linkage with IPP will refuse to build when dynamically linked. Only static linking works; that is, you link with the .a libraries directly. This still throws a whole shitload of warnings, but at least it builds. I did this when writing my Catch2 tests within ipp_ext.

On AMD however, the problem is more nefarious. I recently spent the better part of a week trying to debug an unknown issue with some code that uses IPP; it would randomly crash when referencing a DFT function, and explicitly checking every reference and pointer would prove that everything was okay. I opened it in Visual Studio debugger to really make sure everything was okay i.e. not that my copy/assignment methods were failing etc.

It seemed likely to me that the problem was the function itself was problematic. IPP dispatches the correct function that corresponds to your chip’s instruction set at runtime, so maybe on non-Intel chips this fails for the dynamically linked libraries?

So I just relinked with the static libraries (my AMD is on Windows) which are ippcoremt.lib, ippsmt.lib and ippvmmt.lib. The crashes stopped.

Why? I have no idea, and IPP is closed-source so I will never know.

Implications

I hope Intel doesn’t find this and then decide ‘oh shit he found a loophole’. You have to realise that we use this because we want to deploy on Intel eventually.

For the rest of us using IPP, remember that static linking is your friend if all other errors have been checked and you’re developing on a non-Intel chip. Do it even if it compiles properly, because the runtime crash is almost impossible to debug!

For the nay-sayers

I suspect someone who reads this is going to think: nah bro you definitely wrote some memory leak in, or something along those lines.

Please, if you manage to not crash consistently when linking with IPP dynamically, do let me know how. The setup is here at my non-master pybindCZTdebug branch of pydsproutines. I link with my ipp_ext repository there so maybe the error is inside ipp_ext? But I’ve never found the error (and why would the error only surface when dynamically linking..).

Build the solution and link with dynamic libraries - ippcore.lib, ipps.lib - and issue a fix for me if you can find it. I tested this on my AMD Ryzen 5600X (which crashed almost every time, but not every time mind you) and another 2 Intel-based chips (which never crashed with the dynamic linking).

Testing out this template..

Are we live?

Seems like we are. Playing around with next-js for like the 2nd time in my life here..