Recall that macros are basically programs that transform your program’s code, rather than runtime values. In doing so, they may introduce new variable bindings. If we’re not careful, these new bindings can end up capturing variables in your own code. That is, the new binding might shadow a variable you as the programmer have already created. All of a sudden, the variable you thought you were referring to is no longer the same thing, and because all of this code is hidden in a macro expansion, it will be very hard to figure out what’s going on.

Consider the following Scheme macro for or.

1
2
3
4
5

(define-syntax or
  (syntax-rules ()
    ((_ e1 e2)
     (let ((t e1))
       (if t t e2)))))

Like a good macro, this uses a temporary variable, t, to avoid calculating e1 twice and potentially duplicating any side effects in that expression. Unfortunately, if we’re not careful, this binding can capture an existing binding of t. Consider the following program.

1
2

(let ((t 5))
  (or #f t))

If you run this in the Scheme REPL, you should get 5. Let’s see what happens if we blindly expand the or macro without regard for hygiene. We would end up with the following program.

1
2
3

(let ((t 5))
  (let ((t #f))
    (if t t t)))

This program evaluates to #f, which is the exact opposite of what we were supposed to get! Expanding our macro has shadowed the binding of t to 5 with a binding of t to #f.

One way to work around this, which was a common trick for LISP programmers of yore is to choose variable names that a programmer is unlikely to guess. We could rewrite our or macro like this:

1
2
3
4
5
6
7

(define-syntax or
  (syntax-rules ()
    ((_ e1 e2)
     (let ((this-is-my-super-secret-name-which-you-will-never-guess e1))
       (if this-is-my-super-secret-name-which-you-will-never-guess
           this-is-my-super-secret-name-which-you-will-never-guess
           e2)))))

This works okay, except it’s a lot more typing. Eventually, some overly clever programmer is going to name their own variable this-is-my-super-secret-name-which-you-will-never-guess, and then their program will break in really unexpected ways. It’d really be great if our macro expander could take care of these issues on its own. That way, the macro writers could type less and use names they like, and macro users don’t have to worry about their variables being captured unexpectedly.

We could modify the macro expander to automatically rename any variables bound by a macro expansion. In this case, our simple test program would expand as follows, using our first definition of or.

1
2
3

(let ((t 5))
  (let ((t.1 #f))
    (if t.1 t.1 t)))

This program evaluates as expected, so things are looking good. But, what if someone writes this program?

1
2

(let ((if (lambda (a b c) b)))
  (or #f 5))

We’ll use our variable-renaming expander and see that we end up with the following program:

1
2
3

(let ((if (lambda (a b c) b)))
  (let ((t.1 #f))
    (if t.1 t.1 5)))

This program once again evaluates to #f instead of 5 like we’d like it too. This illustrates the second, and more subtle, class of hygiene error. The problem is that the programmer’s definition of if has captured the if used by the expansion of or. Now, many languages treat keywords like if specially and don’t let you name your variables after them. Enforcing a rule like that in Scheme would solve this particular case, but at the cost of a lot of the power that Scheme programmers love. The proper solution is to find some way of tracking what if was bound to when the or macro was defined and using that version in the expansion of or.

Properly maintaining hygiene in macro systems turns out to be really tricky. The reward is worth it, however, as programmers can then reason much more easily about the behavior of macros and start to rely on them in large projects. Macros are a powerful feature of programming languages, and many newer language have some form of macro system. Sadly, these are often not hygienic, or they so “Oh, we’ll add hygiene later.” Given how important hygiene is, and how tricky it is to get right, language designers should really implemented hygiene in their macro systems from the very beginning.

Kernel Inlining

Let’s consider an example program written in Harlan:

1
2
3
4

(let ((temp* (kernel ((x x*) (y y*))
               (+ (* x x) (* y y)))))
  (kernel ((t temp*))
    (sqrt t)))

I believe this is the first time I’ve included Harlan code snippets in my blog, so some introduction is in order. Harlan is a high level language for data parallel computing that I’ve been working on for a while now as part of my Ph.D. research. You can read our initial paper about Harlan here, although it will be immediately apparent from the drastically different syntax that the language has evolved significantly in that time.

The primary construct for parallelism in Harlan is the kernel form. Syntactically, this looks like Scheme’s let, in that it takes a list of variable names and expressions to bind the names to, and then a body describing what computation to do on those variables. The difference is that each of the bound expressions must be a vector, and the result of a kernel is another vector. Furthermore, each input vector must be of the same length, and the result of the kernel will have the same length as its inputs. Kernels operate on parallel, with the body being applied to each corresponding set of elements in the inputs. You can think of kernels as a parallel map or zipWith.

In the example above, if we imagine x* and y* as containing the X and Y coordinates of a set of 2D vectors, then the code snippet calculates the magnitude of each of these vectors. As written, however, we have a couple of obvious inefficiencies. This example allocates space for the result of the first kernel, temp*, which could be significant if x* and y* are very long. Secondly, both kernels are tiny—that is, they have low arithmetic intensity. Arithmetic intensity is a measure of how many arithmetic operations we have for each memory reference. If it is low, we won’t be using the full computational power of the GPU.

Fortunately, we can solve both of these problems with the first form of fusion, which I feel is better called kernel inlining. With kernel inlining, we would transform our program into the following:

1
2
3

(kernel ((x x*) (y y*))
  (let ((t (+ (* x x) (* y y))))
    (sqrt t)))

In case it’s not clear what happened, we took the body of the first kernel and inserted it into the second kernel. In doing this, we also got rid of the parameters to the second kernel and replaced them with the parameters from the first kernel. Now there is only one kernel, the temp* vector is completely gone, and the arithmetic intensity of the newly fused kernel is higher than we had before.

This transformation is usually a win when it’s possible. We may not want to do it if, for example, the temp* array is also referenced somewhere else. Also, while kernels with higher arithmetic intensity tend to work better because there is more computation for the GPU to use to hide memory access latency, it can also increase the number of registers needed by each thread and thereby limit how much parallelism is possible.

2D Kernel Fusion

For the next type of fusion, let’s consider the following code that would appear in a hypothetical program to render the Mandelbrot set:

1
2
3

(kernel ((i (iota height)))
  (kernel ((j (iota width)))
    (compute-mandelbrot i j)))

Harlan allows nested kernels like this, but Harlan compiles to OpenCL, which does not support nested kernels. Harlan generally deals with this by sequentializing all but one of the kernels. In this case, we could either transform the out or inner kernel into a loop. However, since the kernels form a grid with no dependencies between them, the compiler can fuse these into a single, two dimensional kernel, which OpenCL does support. The Harlan compiler is able to do this for simple cases like this one.

Kernel Concatenation

Let’s look at one more form of fusion, although the Harlan compiler does not currently do this form. Consider a program like this:

1
2
3
4
5

(let ((a* (kernel ((i inputs))
            (do-something i)))
      (b* (kernel ((i inputs))
            (do-something-else i))))
  (write-outputs a* b*))

Here we have two kernels that execute in sequence with no dependencies between them. Furthermore, they have the same set of inputs. If we pretend for a moment that Harlan has a multiple return value form, like Scheme’s values, we could combine these two kernels as follows:

1
2
3
4
5

(let-values (((a* b*) (kernel ((i inputs))
                        (let ((a (do-something i))
                              (b (do-something-else j)))
                          (values a b)))))
  (write-outputs a* b*))

In keeping with the general theme of fusion transformations, we now have only one kernel that performs more work instead of two smaller kernels.

Conclusion

We’ve looked at three different program transformations that could be called fusion. A key point is that they are transformations, not necessarily optimizations. In my experience, these sorts of transformations almost always improve performance, but that need not always be the case. By combining small pieces of code into larger chunks, we reduce the overhead of launching new kernels and also give the GPU more code to use for overlapping memory accesses with computation. On the other hand, larger code could lead to less efficient instruction cache usage, and also lead to more memory accesses if each thread’s working set can no longer fit in registers. It’s good to be able to do these transformations, but it’s important to benchmark and see what works best for each particular program.

Lately I’ve seen a lot of GitHub repositories that include an image like this in their Readme:

It turns out this is from a free service called Travis CI, which makes it easy to integrate continuous integration with GitHub projects. Basically, this means you can define a set of tests that run every time you push a commit to GitHub, and you’ll be notified if something breaks.

Scheme was not one of the languages with first class support, but fortunately we can make it work. Here’s how.

Travis CI gives a lot of flexibility with the scripts you can run on their servers. Thus, we will pretend that we are a C project, and that our build process installs Petite Chez Scheme. Once that’s done, we run a small wrapper script which launches our Scheme test suite.

First, we need a script to install Petite. You can find our script here, but I’ve included the contents below.

1
2
3
4
5
6
7
8
9
10
11

#!/bin/bash

wget http://www.scheme.com/download/pcsv8.4-a6le.tar.gz

tar xzf pcsv8.4-a6le.tar.gz

cd csv8.4/custom

./configure
make
sudo make install

Notice that we can use sudo within our script. This is because Travis CI gives you your very own virtual machine and allows passwordless sudo. Once your tests are done, the VM is destroyed and your next build will happen on an entirely new instance.

Next, lets assume our test suite is called run-tests.scm. We can then create a simple wrapper script to execute these. Be sure to mark it as executable before you check it in.

1
2
3

#!/bin/bash

/usr/bin/petite --libdirs lib --script run-tests.scm

Some of the parameters will be specific to your project. For example, the Elegant Weapons code lives in the lib subdirectory, so we are sure to add this to the compiler’s load path.

Finally, with all the necessary pieces, we can create the .travis.yml file that will actually enable our project with Travis CI:

1
2
3
4
5
6
7

language: c

compiler:
 - gcc

install: ./travis/install-petite.sh
script: ./run-tests

Once you add this file to your repository, push it to GitHub, create your Travis CI account and enable Travis CI for your repository, you should have your tests automatically running.

Obviously, these instructions have been unique to Petite Chez Scheme. It should not take too much trouble to adapt these to your Scheme implementation of choice. In the future, I hope to learn how to automatically run the tests on a variety of Scheme implementations to help keep us honest as far as portability is concerned.

I know where I am.

I was amazed at how many people felt the need to put “IPDPS 2013” on every single slide in their presentation, as if I might forget what conference I’m attending. I suspect this is the result of some Beamer template being helpful, but really it’s unnecessary. Some helpful presenters even thought it’d be useful to remind me on every slide that I’m in Boston. Even if I was somehow unaware of my location, the fact the we are in Boston likely does nothing to help me understand the information you are trying to convey to me. In the same way, if I wanted to know the date, I’d look at the cell phone in my pocket, rather than at your slides.

That said, often slides are posted on web sites afterwards for archival purposes, and in this case the conference name and date actually are somewhat useful. Putting this information on the title slide is sufficient. It doesn’t need to go on every slide.

I have probably forgotten what your talk is about.

One piece of information that I realized would actually be useful to have on every slide is the title of the talk. I was surprised at how many times we were on about slide 3, chest deep in technical details and I realized I had completely forgot what the point of it all was. The fact is, most of us in the audience are not giving you our complete attention. Make it easy for us to try to tune back in by keeping some kind of a reminder of the big picture on every slide.

If I wanted to read, I’d read your paper.

Put as little text as possible on your slides. If you can, come up with a clear graphic or tasteful animation that makes your main point clear. The sooner I can understand your main point, the easier it will be for me to put the rest of your slides in context and avoid the existential crisis mentioned in the previous point.

I don’t want to think about your graphs either.

Make your graphs easy to understand. Often they will have some of the smallest text in your presentation, which means I won’t be able to read it. That’s fine, since graphs are about presenting information visually. Generally, the only thing I’m looking for in your graphs is how much better your system is than the ones you compare against. Make this easy for me. Clearly indicate whether up is better or down is better. Even better, be consistent and set up all your graphs so that up is always better or down is always better. Make it really clear which bars are lines are your system. Make them a brighter color, bold your system’s name in the legend, or put an asterisk next to it. This is particularly important if you are comparing a bunch of systems whose names are all acronyms.

You don’t need to include every graph from your paper. Focus on the key ideas. What are the major results that will make me want to read the rest of your paper? I can get all the nitty gritty details once I decide to read your paper.

For each graph you do include, spend some time telling me what I am supposed to see in it. Remember the saying, a picture is worth a thousand words. You can’t adequately explain your picture with a dozen words. You need to say more than “here is our memory usage.” Why is this result significant? Does it show your system if far faster than anything else? Does it show you have excellent scaling characteristics? Does the chart show you are only marginally worse than other systems, but yours provides other important benefits to make it worth the sacrifice? You should be trying to convince me to adopt your ideas in my own work. How does your graph help convince me of that?

Map

Map takes as input a vector and an operation, and returns a new vector that is the result of applying the operation on each element of the vector. For example, we can map add1 over a vector to increment each element by 1:

1
2
3

map(add1, [1 2 3 4])

=> [2 3 4 5]

Related to this is zip-with, which takes some number of equal-length vectors and applies an operator to corresponding elements in each vector. For example, we can add two vectors using the + operation:

1
2
3

zip-with(+, [1 2 3 4], [0 1 0 1])

=> [1 3 3 5]

Map and zip-with can work on segmented vectors as well, and the result will have the same shape as the original:

1
2
3

map(add1, [[1 2 3] [4 5]])

=> [[2 3 4] [5 6]]

However, if we try to apply zip-with on vectors with differing shapes, we will end up with an error.

Reduce

Reduce takes a vector of elements and combines them into a single element using a given operation. The operator \(\oplus\) must be an associative binary operator. It helps if the operator has a 0 element, that is, one for which \(a \oplus 0 = a\) and \(0 \oplus a = a \). We can do even better if we know the operator is commutative.

Below is an example of how to add all elements in a vector:

1
2
3

reduce(+, [1 1 1 1 1 1 1 1])

=> 8

We can similarly define a segmented variant of reduce, which results in a vector containing the reduction of each segment. For example:

1
2
3

reduce(+, [[1 1 1 1] [1 1 1 1]]

=> [4 4]

The two operators we’ve seen before are very powerful. We can define many linear algebra operations in terms of them, and they all form the basis for MapReduce. Still, there are cases where more operators are extremely helpful.

Scan

Scan is like reduce, except that it returns a vector of all of the intermediate results. Here’s an example:

1
2
3

scan(+, [1 1 1 1 1 1 1 1])

=> [0 1 2 3 4 5 6 7]

Each entry in the resulting vector is the sum of all of the elements to the left of the corresponding element in the source vector.

Scan is often handy as an intermediate step in a larger computation. For example, we may have a vector of vectors where the lengths of the inner vectors vary wildly. In this case, scan can be used to help balance the work between some number of worker threads.

Permute

Permute is used to rearrange elements in a vector. It takes a vector, of course, and also a function that maps indices to indices. The element at the input index will be stored in the output index. Here’s a basic example that reverses a vector:

1
2
3

permute([1 2 3 4], i -> 4 - i)

=> [4 3 2 1]

There are a couple of caveats. Suppose our mapping function were i -> 0. If we apply this to the vector [1 2 3 4], all of the elements would be mapped to the first position and there would be nothing for the later positions. To solve the first problem, we can provide a combining function that is used when multiple elements map to the same location. For example, we may want the maximum value of all elements that map to a given location. For the second problem, we can provide a default element that is used when none of the inputs are mapped to a certain output location. Of course, now our permutation function is more complicated to use and also more complicated to implement.

Back-permute

In many ways, back-permute is a simpler version of permute. Back-permute also takes an input vector and an index mapping function, but instead of mapping input indices to output indices, it maps output indices to input indices. Because of this, we don’t have the problem of multiple values mapping to the same location, as the mapping function can only produce one value. This also means we can avoid the combining function. That said, because this function is simpler, it is also less powerful than permute.

Filter

All of our operators so far except for reduce have essentially mapped some number of inputs to the same number of outputs. Sometimes we need to remove elements that we are working with. An example is when writing Quicksort. We’d need to split a vector into one vector containing those elements greater than the pivot and another containing those elements less than the pivot. Using filter, we could do this as follows:

1
2
3
4
5
6
7
8
9
10
11

let pivot = 5;
let data = [5 2 8 6 4];

filter(x -> x > pivot, data)

=> [8 6]


filter(x -> x <= pivot, data)

=> [5 2 4]

What’s primitive?

Many of these operators can be written in terms of other ones. For example, you can get reduce from scan by reading the last element of the result and adding the last element of the input. This version will probably not be efficient, as it may allocate far more storage than necessary. In designing a data parallel system, one important decision is which operators will be considered primitive, and which will be built in terms of others. Ideally, these decisions should consider the characteristics of the underlying hardware. In Guy Bleloch’s book, Vector Models for Data-Parallel Computing, he argues in favor of a set of primitives that can execute in about the same amount of time it would take to read the input vector from memory. Similarly, when implementing a data parallel system for GPUs, it’s important to consider what operators are easy to implement. Map and back-permute are pretty straightforward, but others are more difficult. Ideally, upon choosing a good set of primitives, the compiler and runtime can optimize the other operators into efficient code.

Systems that support data parallelism typically provide a handful of data structures that can be manipulated with a set of parallel operators. These data structures include vectors, segmented vectors, and arrays. In this post, we’ll take a look at these different structures and in a later post we’ll discuss some of the parallel operators that manipulate them.

Data parallel systems typically provide some combination of vectors, segmented vectors and arrays. In the abstract, we’ll consider a vector to be an ordered, finite sequence of elements of the same types. Most often, these are scalar values, but some systems allow vectors of vectors. A segmented vector is a vector that has been divided into segments (surprise!). Below is an example of a vector (X) and a segmented vector (Xs).

1
2

X  = [1 2 3 4 5 6 7 8]
Xs = [[1 2 3] [4 5] [6 7 8]]

The first vector just includes the numbers one through eight, while the second is made up of three segments. Notice how the segments don’t have to be the same length. You might also notice that the segmented vector looks a lot like a vector of vectors of numbers. Often, languages that support nested vectors implement them as segmented vectors. Representing nested vectors as segmented vectors is essentially the idea behind flattening, which is used heavily in the implementations of NESL and Data Parallel Haskell, for example.

Hardware rarely supports segmented vectors directly, so they are instead represented as a pair of a data vector and a segment descriptor. There are several possibilities for segment descriptors, and different representations have different performance characteristics depending on the hardware. One representation for the segment descriptor is the flag vector, which we see below.

1
2

Xdata = [1 2 3 4 5 6 7 8]
Xseg  = [1 0 0 1 0 1 0 0]

This example is the same as Xs from above. The segment descriptor, Xseg, includes a 1 in each element that starts a new segment and a 0 otherwise. This representation can simplify the implementation of some operations, such as scan, but this representation also cannot describe 0-length segments.

Another option is to store the length of each segment in the segment descriptor:

1
2

Xdata = [1 2 3 4 5 6 7 8]
Xseg  = [3 2 3]

This representation makes it easy to tell how many segments there are, and how long each one is, but it makes accessing individual elements more difficult because it’s not obvious where a segment starts. Instead, we could have the segment descriptor store the indexes of the beginning of each segment:

1
2

Xdata = [1 2 3 4 5 6 7 8]
Xseg  = [0 3 5]

As one final option (I’m sure there are many more), we’ll consider a case where each element in the segment descriptor indicates the segment that the data lies in:

1
2

Xdata = [1 2 3 4 5 6 7 8]
Xseg  = [0 0 0 1 1 2 2 2]

Each of these representations has different tradeoffs. Some take more space, others enable more efficient implementation of certain operations. Which operations are important will help inform the best choice of representation. Fortunately, many of these can converted to another form with a constant number of operations. You might notice some similarities between these representations and various sparse matrix formats.

Finally, we consider the array type. This is similar to a segmented vector in which each segment has the same size. Below is an example.

1
2
3

[1 2 3]
A = [4 5 6]
[7 8 9]

We can certainly represent this exactly as we would a segmented vector, but we can do many operations more efficiently if we instead store the array as a data vector and a list of dimensions. For example:

1
2

Adata = [1 2 3 4 5 6 7 8 9]
Adim  = [3 3]

These structures provide a solid basis for data parallel computing. Though there are many possible representations, one commonality is that the actual data elements are stored in a contiguous block of memory. This feature means that many data parallel operators, such as those we’ll discuss in an upcoming post, can be implemented in a way that maps efficiently onto data parallel processors like GPUs.

On a recent paper submission, I included a graph comparing Harlan to CUBLAS on a trivial vector addition benchmark. Below is a recreation of that graph.

The green dots represent the execution time for the Harlan program, while the blue dots indicate the same for the CUBLAS version. This graph clearly shows that the Harlan version takes about twice as long as the CUBLAS version. On the one hand, being slower than a highly tuned linear algebra library by only a factor of two isn’t bad. On the other hand, this graph doesn’t cast Harlan in a great light.

Let’s see what happens if we put the Y axis on a logarithmic scale.

This looks much better! Here it looks like we only have an additive factor overhead, and there’s really not that much room between the Harlan and CUBLAS points. We also get the nice logarithmic shape, which makes it look like the time doesn’t get that much worse as the vectors get larger.

As a third option, we can use a logarithmic scale on the X axis as well.

Now we see two lines, which suggests the execution time for each program increases linearly in the size of the vector, which is true. But now, the lines are essentially parallel, again with only an additive factor between the two. As we move to the right, the lines seem to converge, suggesting that for extremely large vectors, Harlan might even outperform CUBLAS. Not bad!

The point of all this is that I have presented the exact same data in all three cases, yet the choice of scales leads us visually to very different conclusions. In this case, I believe the linear-linear plot is the most honest. For other data sets, one of the other plots may be more appropriate. As an author, it’s important to choose visual presentations of data that faithfully highlight the key features of the data. Likewise, as a reader, it’s important to pay attention to the scales and understand what the shape of a graph means in light of those scales.

Do start with, let’s look at an example macro. Imagine we wanted to take the or2 macro from last time, but we want to extend it to support any number of arguments. Certainly, writing (or2 a (or2 b c)) is a bit of a pain. It’d be much better to simply write (or a b c). We can do this by using an ellipses in our patterns, as seen in this macro, called my-or to avoid clashing with the built in or:

1
2
3
4
5
6

(define-syntax my-or
  (syntax-rules ()
    ((_ e) e)
    ((_ e e* ...)
     (let ((t e))
       (if t t (my-or e* ...))))))

This macro uses two patterns. The first matches my-or with exactly one argument. The second matches my-or with one argument and any number of remaining arguments. The second pattern expands into a recursive call to my-or, but it decreases the number of arguments each time until we finally hit the base case. We might trace the expansion like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

(my-or a b c)

=> 

(let ((t.1 a))
  (if t.1 t.1 (my-or b c)))

=>

(let ((t.1 a))
  (if t.1 t.1
      (let ((t.2 b))
        (if t.2 t.2 (my-or c)))))

=>

(let ((t.1 a))
  (if t.1 t.1
      (let ((t.2 b))
        (if t.2 t.2 c))))

Unfortunately, pattern matching and template instantiation become much trickier in the presence of ellipses. As we will see, however, when all is said and done we’ll have added just one more cond clause to both the matcher and the instantiation function.

The basic idea is that we want to keep everything more or less the same, but when we encounter a ..., we want to recursively match or instantiate a pattern as many times as we can and then pick up where we left off. In my mind, the most complicated part of this becomes representing the environment between the matcher and instantiation code. Last time, we represented the environment as an association list. It’s attractive to try to simply pair the pattern variable name with a list of things it matched instead of just a single value, but this doesn’t seem to preserve quite enough information. Suppose we matched (my-or (+ 1 2) a b c) against (_ e e* ...), yielding the bindings ((e . (+ 1 2)) (e* . (a b c))). Then let’s see what happens if for some reason we used this to instantiate the template '((e e*) ...). We might use a rule like “when instantiating a ... template, look up every variable in the environment and zip them together.” This rule would lead to the following instantiation:

1

'((+ a) (1 b) (2 c))

This rule wouldn’t be too hard to implement, but it’s also probably not what we want. In this case, we’ve broken apart the (+ 1 2) expression into its individual components, when instead we’d probably like to keep this together. The semantics we’d like instead lead to the following result:

1

'(((+ 1 2) a) ((+ 1 2) b) ((+ 1 2) c))

Admittedly, this is a somewhat contrived example, but as we write more macros it will become apparent that this is usually the behavior we want.

In order to preserve the information we need, we’ll represent the environment more like a tree, which includes a ... node that describes pattern variables bound under a .... Thus, if we matched (my-or (+ 1 2) a b c) against (_ e e* ...), we’d end up with the environment ((e . (+ 1 2)) (... ((e* . a)) ((e* . b)) ((e* .c)))). Each ... node in the environment includes a list of bindings matched under a ... pattern. The list of association lists approach allows us to keep track of which variables were matched together.

We will see that building this environment in the matcher is fairly straightforward, but more care is needed with template instantiation. We will define a flatten operation on environments, which takes in one environment, strips off a level of ellipses, and returns a list of environments. We then recursively apply the template instantiation function to each of these new environments. To look at our running example, ((e . (+ 1 2)) (... ((e* . a)) ((e* . b)) ((e* .c)))) would convert to this:

1
2
3

(((e . (+ 1 2)) (e* . a))
 ((e . (+ 1 2)) (e* . b))
 ((e . (+ 1 2)) (e* . c)))

It’s easy to see now how instantiating '((e e*) ...) just requires mapping the instantiate function with (e e*) over this list of environments.

We’re not quite done yet though. What happens if we had the pattern (_ (a ...) (b ...)), and matched this against ((1 2 3) (x y))? We’d end up with this environment:

1
2

((... ((a . 1)) ((a . 2)) ((a . 3)))
 (... ((b . x)) ((b . y))))

If we try to flatten this, we’ll see that we won’t have enough bs to correspond with each a. To avoid this problem, we will make the flatten procedure take a template as input, and only consider ... nodes that contain pattern variables referenced by the template. This way, we only expand the environment for either a or b, but not both.

So with all of this out of the way, we can look at an exension of match that handles ellipsis patterns:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

(define (match* p e sk fk)
  (cond
    ((and (pair? p) (pair? (cdr p)) (eq? '... (cadr p)))
     (let loop ((e e)
                (b '()))
       (if (null? e)
           (match* (cddr p) e
                   (lambda (b^) (sk `((... . ,b) . ,b^)))
                   fk)
           (match* (car p) (car e)
                   (lambda (b^)
                     (loop (cdr e) (snoc b b^)))
                   (lambda ()
                     (match* (cddr p) e
                             (lambda (b^) (sk `((... . ,b) . ,b^)))
                             fk))))))
    ((and (pair? p) (pair? e))
     (match* (car p) (car e)
             (lambda (b)
               (match* (cdr p) (cdr e)
                       (lambda (b^) (sk (append b b^)))
                       fk))
             fk))
    ((eq? p '_)
     (sk '()))
    ((symbol? p)
     (sk (list (cons p e))))
    ((and (null? p) (null? e))
     (sk '()))
    (else (fk))))

This is almost identical to match from before, but we’ve added lines 3—16. These lines first check if the pattern is a ... pattern, and if so, we go into a loop where we try to match the pattern as many times as possible. As we do this, we build up a list of environments from each successful match. Once we get a failure, instead of calling the failure continuation we were given, we just continue on matching the rest of the pattern. We can try matching this against a let pattern as follows:

1
2

> (match* '(_ ((x e) ...) b) '(let ((x 5) (y 6)) (+ x y)) (lambda (b) b) (lambda () #f))
((... ((x . x) (e . 5)) ((x . y) (e . 6))) (b + x y))

Based on our previous discussion of environments, this is what we wanted.

Now let’s look at how we combine a pattern with a template. We’ll start with a helper, called extract-..., which pulls out the relevant ellipsis bindings from the environment:

1
2
3
4
5
6
7
8
9
10
11

(define (extract-... p bindings)
  (if (null? bindings)
      '()
      (let ((rest (extract-... p (cdr bindings)))
            (b (car bindings)))
        (if (and (eq? (car b) '...) (not (null? (cdr b))))
            (let ((names (map car (cadr b))))
              (if (ormap (lambda (x) (mem* x p)) names)
                  (cons (cdr b) rest)
                  rest))
            rest))))

This function works by walking over the set of bindings. When we encounter a ... node, we check if any of the variables bound in that sub-environment are referenced in the pattern (this is what the (ormap (lambda (x) (mem* x p)) names) clause does). If any variables are relevant, we include this sub-environment on the list that is returned.

Doing this relies on another helper, mem*, which determines whether a variable is relevant to a pattern. As its name suggest, this was originally a blind tree walk that checked if the symbol occurs anywhere in the pattern. Sadly, it’s not that simple. Suppose we had the environment ((... ((a . 1)) ((a . 2)) ((a . 3))) (... ((b . x)) ((b . y)))) and we were instantiating the pattern ((a b ...) ...). Since there are two levels of ..., we only want to consider a on the first level, and then when we get to the second ..., we need to consider only b. This means we need to cut off search when we see a sub-pattern with an ellipsis. That’s what lines 3 and 4 do in this code:

1
2
3
4
5
6
7

(define (mem* x ls)
  (cond
    ((and (pair? ls) (pair? (cdr ls)) (eq? (cadr ls) '...))
     #f)
    ((pair? ls)
     (or (mem* x (car ls)) (mem* x (cdr ls))))
    (else (eq? x ls))))

With our helper functions out of the way, we can look at the extended version of instantiate:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(define (instantiate* p bindings)
  (cond
    ((and (pair? p) (pair? (cdr p)) (eq? '... (cadr p)))
     (let ((bindings... (extract-... (car p) bindings)))
       (if (null? bindings...)
           (instantiate* (cddr p) bindings)
           (append
            (apply map (cons (lambda b*
                               (instantiate* (car p)
                                             (append (apply append b*)
                                                     bindings)))
                             bindings...))
            (instantiate* (cddr p) bindings)))))
    ((pair? p)
     (cons (instantiate* (car p) bindings)
           (instantiate* (cdr p) bindings)))
    ((assq p bindings) => cdr)
    (else p)))

One again, this includes the same code as before, extended with lines 3—13 to handle ellipses. This clause works by finding the relevant sub-environments. If it finds any, we append each of the sub-environments to the full environment and use each of these new environments to instantiate the sub-template.

We can pass this the results of the our match* call above, and see that it is able to reconstruct a let expression:

1
2
3

> (instantiate* '(let ((x e) ...) b)
                '((... ((x . x) (e . 5)) ((x . y) (e . 6))) (b + x y)))
(let ([x 5] [y 6]) (+ x y))

Of course, we can also use this to convert let expressions to function applications:

1
2
3

> (instantiate* '((lambda (x ...) b) e ...)
                '((... ((x . x) (e . 5)) ((x . y) (e . 6))) (b + x y)))
((lambda (x y) (+ x y)) 5 6)

And, we can handle some harder cases too:

1
2
3
4

> (instantiate* '((a b ...) ...)
                 '((... ((a . 1)) ((a . 2)) ((a . 3)))
                   (... ((b . x)) ((b . y)))))
((1 x y) (2 x y) (3 x y))

And here’s my personal favorite:

1
2

> (instantiate* '((a a ...) ...) '((... ((a . 1)) ((a . 2)) ((a . 3)))))
((1 1 2 3) (2 1 2 3) (3 1 2 3))

1
2
3
4
5

(define-syntax or2
  (syntax-rules ()
    ((_ e1 e2)
     (let ((t e1))
       (if t t e2)))))

We can see how this works at the REPL by using expand:

1
2

> (expand '(or2 1 2))
(let ([t.8 1]) (if t.8 t.8 2))

This matches the expression (or2 1 2) against the pattern (_ e1 e2). The underscore means we don’t care what’s in that position (it’s always or2, since this is part of the or2 macro definition). The names e1 and e2 allow us to refer to what occurs in this pattern in the template. When instantiating the template, the macro system replaces all references to e1 with the chunk of syntax from that position (1 in this case), and likewise with e2. The result is the expression, (let ([t.8 1]) (if t.8 t.8 2)). Here we let-bind the first expression because we do not want to accidentally evaluate any side effects it might contain twice. Don’t worry too much about why the variable name is t.8 instead of just t. This has to do with something called hygiene, which I will hopefully say more on in a later post.

Now let’s take a look at how we would write a function to match patterns like this. We’ll start with a procedure called match that takes four arguments: the pattern to match against, the expression to match, a success continuation, and a failure continuation. The success continuation is called if the expression matches the pattern, and the failure continuation is called otherwise. When I first started writing this, it seemed like it would be cleaner to use success and failure continuations, although I’m not entirely sure that’s true anymore. At the moment, we will support three things for patterns:

1. Pairs – A pair pattern matches an expression if and only if the expression is a pair, and the car of the pattern matches the car of the expression and the cdr of the pattern matches the cdr of the expression.
2. Symbols – A symbol pattern binds a pattern variable. This matches any expression, and binds that expression to the variable.
3. Null – A null pattern matches an expression if and only if the expression is null.

Scheme’s pattern matching supports several other features, like auxiliary keywords, which we’re ignoring for the now for the sake of simplicity. Using our three kinds of patterns so far, we can translate this pretty directly into a Scheme program:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

(define (match p e sk fk)
  (cond
    ((and (pair? p) (pair? e))
     (match (car p) (car e)
            (lambda (b)
              (match (cdr p) (cdr e)
                     (lambda (b^) (sk (append b b^)))
                     fk))
            fk))
    ((and (symbol? p))
     (sk (list (cons p e))))
    ((and (null? p) (null? e))
     (sk '()))
    (else (fk))))

Notice that we pass a list of bindings to the success continuation. These will be used by the template instantiation code. Let’s try an example:

1
2

> (match '(_ e1 e2) '(or2 1 2) (lambda (b) b) (lambda () #f))
((_ . or2) (e1 . 1) (e2 . 2))

For the most part, this does what we want. The matcher says the expression matches the pattern, and tells us that e1 was bound to 1 and e2 was bound to 2. However, we also bound _ to or2. In reality, we want _ to mean ignore, which we can do by adding one line before the symbol? line in our cond clause:

1
2

((eq? p '_)
 (sk '()))

Now we get only the bindings we care about.

The next part is to use the resulting bindings to instantiate the template. We’ll do this with a function called instantiate, which takes a template (called p), and a list of bindings. This is a straightforward tree walk that tries to replace symbols with expressions when they are bound:

1
2
3
4
5
6
7

(define (instantiate p bindings)
  (cond
    ((pair? p)
     (cons (instantiate (car p) bindings)
           (instantiate (cdr p) bindings)))
    ((assq p bindings) => cdr)
    (else p)))

If we did this right, we should be able to instantiate the template from the or2 macro above.

1
2
3
4
5
6

> (instantiate
      '(let ((t e1))
         (if t t e2))
      '((e1 . 1) (e2 . 2)))

(let ([t 1]) (if t t 2))

It looks like we got what we wanted, although without hygiene.

At this point we have a pattern matcher and template instantiation function that is suitable for making a very simple macro system. One very nice feature that we are missing at the moment is ellipsis patterns, which allow us to match repeated patterns. For example, we could use this to write a custom let macro like this:

1
2
3
4

(define-syntax my-let
  (syntax-rules ()
    ((_ ((x e) ...) b)
     ((lambda (x ...) b) e ...))))

In a post at some point in the hopefully near future, I will show how to handle patterns with ... in them.

First of all, I meant to operate on both arrays, A and B, but through some sloppy coding I ended up only using A. Incidentally, I did some back-of-the-envelope calculations to figure out the memory bandwidth I was getting, and I was surprised to see that I was getting close to twice the theoretical peak for the cards I was working with. It looks like it’s because I was only reading half the data I thought I was. Here are the corrected figures (the experiment is the same other than the small corrections to my code):

We’re slower across the board, but the overall shape of the data is about the same. Interestingly, the fastest kernels are not much slower than the fastest kernels from before.

Next, reddit user ser999 pointed out that we could forego the branch entirely by doing some clever arithmetic. Instead of doing

1
2
3
4

if (i % 2 == 0)
    get(C, N, i, j) = get(A, N, i, j) + get(B, N, i, j);
else
    get(C, N, i, j) = get(A, N, i, j) - get(B, N, i, j);

we could instead do this:

1

get(C, N, i, j) = get(A, N, i, j) + get(B, N, i, j)*(1 - ((i&1)<<1));

This isn’t exactly as clear as the code we had before, but perhaps a sufficiently smart compiler could perform this optimization so the programmer could still write the nicer code. To be fair, my thread divergence optimization wasn’t great for code readability either. The important this is, how does this “no branching” version perform? The table below shows the performance along with the “unweaved” version from before for comparison.

For row-wise access, the “unweave” variant always wins. For column-wise access, the “nobranch” version wins on the C1060, while the GTX 460 and the ATI card do better with the “unweave” variant. In both the ATI and GTX 460 column-wise case, however, the two perform basically the same.

So why is this? Branches are often pretty expensive, especially when they create thread divergence. However, by removing the branch we always have to do a multiplication, and we also have to convert an integer value into a floating point value. Multiplication in particular is a fairly expensive operation. In the case, the branch isn’t so bad.

As before, the code from this post is available at https://github.com/eholk/bench-thread-diverge.

I was recently given a test question that presented a simple CUDA kernel and asked us to rewrite it to increase parallelism and to minimize thread divergence. Translated to OpenCL, the kernel looks like this:

1
2
3
4
5
6
7
8
9
10
11
12

#define get(A, N, i, j) ((A)[((i) * (N)) + (j)])

__kernel void MyAdd(const double __global *A, const double __global *B, double __global *C, unsigned int N) {
    unsigned int i = get_global_id(0);

    for(unsigned int j = 0; j < N; j++) {
        if (i % 2 == 0)
            get(C, N, i, j) = get(A, N, i, j) + get(A, N, i, j);
        else
            get(C, N, i, j) = get(A, N, i, j) - get(A, N, i, j);
    }
}

Let’s see how we can transform this program, and more importantly, let’s see how different variations perform.

First, let’s try to increase parallelism. This kernel is operating over a two dimensional array. Each thread processes one row, does a for loop to process each column in that row. The even number threads add the appropriate rows from A and A, while the odd number threads subtract. There are no dependences between any iteration of these loops, so it is straightforward to parallelize them by using a two dimensional kernel:

EDIT: I had intended this code to operate on elements on both A and B, but due to what was probably me being sloppy with copy and past, the code only uses values from A. Thanks to reader Gasche for pointing this out!

1
2
3
4
5
6
7
8
9

__kernel void MyAdd_2D(const double __global *A, const double __global *B, double __global *C, unsigned int N) {
    unsigned int i = get_global_id(0);
    unsigned int j = get_global_id(1);

    if (i % 2 == 0)
        get(C, N, i, j) = get(A, N, i, j) + get(A, N, i, j);
    else
        get(C, N, i, j) = get(A, N, i, j) - get(A, N, i, j);
}

Obviously, you’d need to adjust the host code to issue a 2D kernel instead. Now, each thread processes exactly one element of each array, which is the exact kind of parallelism that GPUs are very well suited for. I was able to test this on three different GPUs, and for the most part this yielded a substantial performance improvement. The table below shows the time to execute the kernel for a 1024×1024 matrix.

The two consumer grade GPUs give between about a 4x and 10x improvement. Oddly, the professional grade Tesla C1060 is nearly 4x slower. This is our first example of another common frustration in optimizing GPU codes: architectures vary wildly and behave very differently.

Now let’s see if we can do something about the thread divergence. Right now every other thread runs a different program. GPUs work best when threads can exectue in lock step, so right now the situation is pretty bad. What if instead we had the first \(\frac{N}{2}\) threads handle the even rows and the rest handle the odd rows? If we did this, our code would look like so:

1
2
3
4
5
6
7
8
9
10
11
12
13

__kernel void MyAdd_2D_unweave(const double __global *A, const double __global *B, double __global *C, unsigned int N) {
    unsigned int i = get_global_id(0);
    unsigned int j = get_global_id(1);

    if(i < N / 2) {
        unsigned int i = 2 * i;
        get(C, N, i, j) = get(A, N, i, j) + get(A, N, i, j);
    }
    else {
        unsigned int i = 2 * (i - N / 2) + 1;
        get(C, N, i, j) = get(A, N, i, j) - get(A, N, i, j);
    }
}

It’s mostly the same, except now we have some extra index arithmetic to map each thread onto each row in a different order. The table below shows the performance for this version.

All GPUs speed up on this version, but the two NVIDIA processors have especially high performance increases. This fits well with the way that NVIDIA GPUs divide a thread block into warps of 32 threads, which each execute in lock step. The fact that we didn’t see as extreme of a performance improvement on the ATI GPU suggests that they may not execute threads in warps the same way that NVIDIA GPUs do.

Overall, these two transformations greatly improved performance. Even though we took a hit initially on the high end C1060, after reducing branch divergence we have about a 4x improvement over our original code. The GTX 460 showed a more than 20x improvement! One surprising thing is that the fairly inexpensive GTX 460 ended up being about four times faster than the C1060. This is a testament to how fast GPus are improving. I suspect the main issue here is memory bandwidth. This benchmark does very little computation, so we should once again be memory limited. The GTX 460 has GDDR5 memory, while the C1060 only has GDDR3. According to Wikipedia’s handy list of device bandwidths, GDDR5 should be about 2.4x faster than GDDR3, which would account for much of the performance difference.

Column-wise access

At first, I didn’t expect the thread divergence transformation to improve the performance significantly. The reason is that GPUs read memory blocks at a time (similar to how CPUs read in entire cache lines at once). Before, each block was only read by one warp, whereas afterwards two warps had to read each block. In the absence of a cache that is big enough to hold the whole matrices, this means we’ll need to read twice as much data.

To test this hypothesis, I wrote versions that alternated on the j dimension instead of the i dimension. To save space here, I’ve put all three of the column-wise kernels in a gist. The performance for all three variants is given below.

My hypothesis holds for the C1060, as it actually got slower between the regular 2D kernel and the 2D kernel that has been reordered to reduce branch divergence. In all cases, the sequential code performs about the same as it did in the row-wise version. In the column-wise kernels, the GTX 460 and the Radeon HD 6750M both improve with less thread divergence, but the GTX 460 does not improve to nearly the same extent as it did in the row-wise version.

Summary

We’ve looked at several ways of doing very similar computations, and seen that their performance depends greatly on the memory access pattern and amount of thread divergence. Here’s all the data we looked at in one convenient table:

One thing I still have not figured out is why the C1060 takes such a performance hit when going to the 2D kernel, but the GeForce and the Radeon do not show this same trend. In the GeForce case, I suspect it might have something to do with the new thread scheduler in the Fermi series. I am not as familiar with the Radeon architecture, but it seems to have interesting performance characteristics and would be interesting to study further.

The code for this post is available at https://github.com/eholk/bench-thread-diverge. It’s written in Rust, using the rust-opencl bindings.

UPDATE: I added a follow up post taking into account several suggestions from readers.

Here’s the video. Be sure to check his post for commentary.

∞ Permalink

I was one of the test subjects, and Mike was kind enough to let me have a video of the eye tracker data superimposed on my screen. I’ve shared a small section for you to watch.

One of the things that stood out to me in watching the video was how much my mind seems to work like a computer. First I read over the whole program, and then I start interpreting it. The program in question consists of two calls to a function called between, followed by a call to common. For the first call to between, I spend a lot of time moving my eyes between the call site and the function definition. For the second call, however, I only glance up at the function definition once.

In programming language terms, I seem to be doing some kind of just-in-time compilation. The first time through, I read and interpret every instruction. Afterwards, it seems like I remember what this function does and am able to determine its output much quicker. Interpreting the first call takes about 24 seconds, while I blow through the second one in about 10 seconds.

Another observation is that naming things accurately seems to help. I was able to work through the call to common very quickly. While reading this program, I remember thinking “this should return the elements that are in both arrays.” I read over the program to verify that it does what its name suggests, and then I can do the equivalent operation in my head rather than by interpreting the code.

I’m excited to see what else Mike’s research uncovers. One aspect he’s interested in is how the approach of inexperienced programmers differs from that of experienced programmers. For example, there seems to be some evidence that following variable naming conventions helps experienced programmers understand the code much quicker, while breaking these conventions leads to a severe penalty. On the other hand, inexperienced programmers seem to take about as long regardless of how the variables are named.

If you happen to live in Bloomington, consider volunteering for Mike’s experiment. It’s a lot of fun, and you get $10 for participating. He’ll be collecting data all through next semester, and the more people he gets, the better. If you want to participate, send him and e-mail at mihansen@indiana.edu and he can schedule a time for you.

Theoretically, GPUs should still do quite well. According to this post, an NVIDIA GTX480 should be able to reach 177.4 GBps of global memory bandwidth. On the other hand, a CPU with PC17000 memory can only theoretically hit 17 GBps of main memory bandwidth. This means a GPU ought to be able to outperform the CPU by about a factor of 10. Unfortunately, unless you are computing on data that you have generated on the GPU or is part of a larger computation, you must consider the time to copy the data from the CPU memory to the GPU. This goes over the PCI Express bus, currently tops out at about 8 GBps.

This means that when you also include the time for data transfer, besides just the computation time, it should be almost impossible for a GPU implementation of dot product to beat a CPU implementation. To test this, I wrote several different variants of dot product and compared their speed. The results are below.

The code for these tests is available on my GitHub. I chose vectors of 33 million single precision floats so that the total amount of data for the two vectors would be about 256 MB. This is large enough to take a measurable amount of time, while being small enough to comfortably fit in most GPU memories. As usual, these tests were conducted on a Core i7 2600K with PC17000 memory and an NVIDIA GTX460 GPU.

The Simple version is the most obvious way to write a dot product:

1
2
3
4
5
6
7
8

float simple_dot(int N, float *A, float *B) {
    float dot = 0;
    for(int i = 0; i < N; ++i) {
        dot += A[i] * B[i];
    }

    return dot;
}

My hypothesis was that it’d actually be hard to do much better than this, given how fast CPUs are relative to main memory. I was surprised to find that you could actually do significantly better making use of the CPU’s SIMD instructions.

The bars are not actually organized in the order I tried them. The next test I did was the SSE version. This was mostly handled by the compiler. I just had to add __attribute__ ((vector_size (sizeof(float) * 4))) to the types I was working on, and add a few casts here and there. I checked the generated assembly code to be sure it was generating SSE instructions.

Generating AVX was a little bit trickier. For the most part, I just needed to change the vector size from 4 to 8. However, I was using a slightly old version of GCC that I needed to update. Also, I needed to add the -march=corei7-avx and -mtune=corei7-avx compiler flags. The AVX version did perform slightly better, but we only gained about a millisecond. This suggests that we’re pretty close to maxing out the memory bandwidth. Dividing 256 MB by 16.2 ms gives us 15.43 GBps, which is approaching the 17 GBps theoretical peak.

At this point, I had exhausted most of what I could think of to improve the performance. I decided to compare with the performance of ATLAS. ATLAS is a collection of linear algebra routines. The package includes several variants of each algorithm, and at install time the build system measures the performance of each variant on the machine you are running. Based on this, it decides the best variant to use. The ATLAS version performed quite a bit better than the simple version, but not quite as well as the SSE or AVX versions.

I pulled out gdb and tracked down disassembled the core computational loop for the ATLAS version of dot product. If you’re interested, the assembly is here. I noticed two things. First, it was using the prefetchnta to prefetch values into cache before they were need. Second, and more important, the loop had been unrolled four times. I had tried a similar thing using the -funroll-loops compiler flag, but this made only a marginal difference. This version of unrolling looks like this:

1
2
3
4

dot += A[i] * B[i];
dot += A[i + 1] * B[i + 1];
dot += A[i + 2] * B[i + 2];
dot += A[i + 3] * B[i + 3];

The problem is, this code must still run basically sequentially because each statement depends on the previous statement. What ATLAS did, and what I copied in the Unrolled version, is this:

1
2
3
4

dot1 += A[i] * B[i];
dot2 += A[i + 1] * B[i + 1];
dot3 += A[i + 2] * B[i + 2];
dot4 += A[i + 3] * B[i + 3];

This meant there was no longer any dependency between statements, and the CPU’s out of order execution engine could run all four of these as soon as the data was available. In a sense, the CPU is vectorizing loops on the fly.

Based on these observations, I tried to see if we could do better using this unrolling technique and prefetching on the AVX version. This got about another 0.1 milliseconds, but clearly we are at the point of diminishing returns.

Finally, I decided to see what a GPU implementation would be like. I used NVIDIA’s CUBLAS library, which should be a very highly tuned implementation for NVIDIA GPUs. Using the library was pretty easy. I just had to remember to allocate GPU buffers and copy data into them. As I predicted, however, the CUBLAS version was far slower than even the simple version.

This exercise proved interesting for me. It confirmed one of my hypotheses, that it should be very hard if not impossible to make a GPU implementation of dot product outperform the CPU when the data starts in the CPU memory. I was surprised, however, to find that you can do better than the simple version on the CPU. It was also interesting to see how minor changes in the C++ source code can cause the compiler to generate very different assembly language.

1
2
3
4

#[kernel]
fn add_float(x: &float, y: &float, z: &mut float) {
    *z = *x + *y;
}

There are two main parts to this project. The first is compiling Rust code into something suitable for running on the GPU. We do this using the PTX backend that is part of LLVM. The second part is loading and executing the kernel. For this, we use OpenCL and its clCreateProgramWithBinary API. In this post, I’ll focus on the issues encountered with generating PTX code.

The bulk of the work to generate PTX code was already done by the NVPTX backend that was recently contributed to LLVM by NVIDIA. We started out with a very manual process. First we used the --emit-llvm flag for rustc to save the generated LLVM bitcode. From there, we attempt to compile as PTX using llc:

1

llc -march=nvptx -mcpu=sm_13 trivial-kernel.ll -o trivial-kernel.ptx

I wasn’t terribly surprised to see this fail with one of LLVM’s typically opaque error messages. You can see it here if you wish. Basically, Rust was generating code that the NVPTX backend didn’t know how to handle. This makes sense; I expect NVIDIA primarily tests the backend on code generated by CUDA, which looks different from code that Rust generates. The next step was to pare down the generated LLVM to something a little more manageable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64"
target triple = "x86_64-apple-darwin"

%"~enum intrinsic::TyDesc[#0]" = type { [16 x i8] }
%tydesc = type { i64, i64, void (i1*, i1*, %tydesc**, i8*)*, void (i1*, i1*, %tydesc**, i8*)*, void (i1*, i1*, %tydesc**, i8*)*, void (i1*, i1*, %tydesc**, i8*)*, i8*, i8* }

define void @_ZN9add_float16_cb9e1b436595b333_00E(i1*, { i64, %tydesc*, i8*, i8*, i8 } addrspace(1)*, double*, double*, double*) uwtable {
static_allocas:
  %5 = alloca double*
  %6 = alloca double*
  %7 = alloca double*
  br label %8

return:                                           ; preds = %8
  ret void

; <label>:8                                       ; preds = %static_allocas
  store double* %2, double** %5
  store double* %3, double** %6
  store double* %4, double** %7
  call void asm "# *z = *x + *y; (trivial-kernel.rs:3:4: 3:16)", ""()
  %9 = load double** %5
  %10 = load double** %6
  %11 = load double* %9
  %12 = load double* %10
  %13 = fadd double %11, %12
  %14 = load double** %7
  store double %13, double* %14
  br label %return
}

Of course, LLVM still fails:

1

Assertion failed: (!isLiteral() && "Literal structs never have names"), function getName, file /usr/local/src/llvm/lib/VMCore/Type.cpp, line 605.

It seems that NVPTX was having trouble with the anonymous struct in the function arguments ({ i64, %tydesc*, i8*, i8*, i8 }). To test this theory, I replaced that type with an i8 *. The argument was ignored anyway, so this shouldn’t cause problems. With this change, we ended up with a PTX file.

At point, we could either hack the Rust compiler to avoid generating code that the NVPTX backend couldn’t handle, or we could improve the NVPTX backend. I opted for the latter, and ended up submitting my first ever patch to LLVM.

After another minor fix or two, it became clear that we were going to have to modify the way Rust generates code as well. For example, the PTX code I linked to above does not include a .entry line, which is required to indicate where a kernel function begins. One option would be to add a new PTX target for Rust, and basically set it up as a cross compiler. This isn’t quite what we want. We don’t want to run all of Rust on the GPU, just a few portions of a program. Other than the code generator, we want to PTX code to agree with the architectural details of the host system. Instead, I added a -Zptx flag to rustc and started making minor changes to the translation pass. Functions that have the #[kernel] attribute get compiled to use the ptx_kernel calling convention, which tells NVPTX to add the .entry line. According to Patrick, we should probably use a new ABI setting instead, as arbitrary attributes aren’t part of the function’s type.

At any rate, we could now pretty reliably go from Rust to PTX without any manual intervention. The next challenge was to execute the kernel. When we first tried to load the PTX file, OpenCL complained about an “invalid binary.” We had previously been able to load a PTX file generated with OpenCL and extracted using clGetProgramInfo, so we decided to compare the Rust-generated code with the OpenCL-generated code. It turns out that the parameters to the kernel were not being annotated with an address space. We manually added .global to the parameters in the Rust-generated code, and we were able to load and execute the kernel. Furthermore, we could manually annotate the LLVM code with addrspace(1) to get the same behavior.

For some types, Rust would have the addrspace(1) annotation, but for others it wouldn’t. It turns out Rust was already using address spaces for something related to garbage collection. Unfortunately, Rust and NVPTX disagree on what these address spaces mean. To work around this, I had Rust generate different address spaces when the -Zptx flag is given. At the moment these changes only take effect for & pointers. Others, such as @ pointers will be more difficult to get working.

The final missing piece on the code generation side of things is to have threads be able to do different things. This means providing equivalents of the blockIdx, blockDim and threadIdx variables. These show up in LLVM as intrinsic functions, so all we need to do is expose those as new Rust intrinsics. We expect to have this part working soon.

Our work here shows it’s possible to compile Rust to run on the GPU. We support an extremely limited subset of Rust at the moment. Most of the remaining challenges have to do with the way data is arranged in memory and how Rust provides safety at runtime. Rust uses a lot of pointer structures, and moving these between host and device memory can be difficult. Perhaps the best thing to do for now is simply to be careful about what data types we use in GPU code. Even if we use relatively flat types, however, we will still need to handle a few more things. For example, Rust does array bounds checks at runtime. If we want to allow arbitrary array indexing safely in GPU code, we’ll need a way to do bounds checks and report failures from kernel code. There are clearly a lot of design issues left, but the initial results for compiling Rust to run on the GPU seem very promising.

If you want to try it out, here are links to the code.

https://github.com/eholk/rust/tree/nvptx
https://github.com/eholk/llvm/tree/nvptx-rust

1

(choose-arg 1 (factorial 5) (+ 1 2) (fibonacci 40))

Here, choose-arg takes a number and some arguments. The whole expression should evaluate to the argument selected by the number argument. The example above should return 3. As a procedure, choose-arg might be written as follows:

1
2
3
4
5

(define choose-arg
  (lambda (i . args)
    (if (zero? i)
        (car args)
        (apply choose-arg (- i 1) (cdr args)))))

Using this definition (and appropriate definitions for factorial and fibonacci), we see that our starting example returns 3 as expected. Unfortunately, it takes a lot longer than we’d like:

1
2
3
4
5

(time (choose-arg 1 ...))
    no collections
    11167 ms elapsed cpu time
    11168 ms elapsed real time
    144 bytes allocated

That’s over 11 seconds to decide the answer is 3. What’s going on?

This takes so long because Scheme is an eager language, meaning it must fully evaluate all procedure arguments before applying a procedure. My version of Fibonacci takes exponential time, so (fibonacci 40) is somewhat nontrivial. What we’d really like is to only evaluate the argument we actually selected with choose-arg.

One way to do this is by using thunks. We just wrap all of the arguments in lambdas with no arguments, and wrap an extra set of parentheses around the call to choose-arg to evaluate the thunk we selected:

1

((choose-arg 1 (lambda () (factorial 5)) (lambda () (+ 1 2)) (lambda () (fibonacci 40))))

This performs much better:

1
2
3
4
5

(time ((choose-arg 1 ...)))
    no collections
    0 ms elapsed cpu time
    0 ms elapsed real time
    192 bytes allocated

Unfortunately, the program is now much harder to read. What we’d really like is to write our original program, but have Scheme execute it as if we had written something like this:

1
2
3
4
5
6

(if (= 1 0)
    (factorial 5)
    (if (= 1 1)
        (+ 1 2)
        (if (= 1 2)
            (fibonacci 40))))

This is exactly what macros let us do. Macros let us define syntax transformers, which take one part of your program as input and rewrite it into a different program. As a macro, we could write choose-arg like this:

1
2
3
4
5
6
7
8

(define-syntax choose-arg
  (syntax-rules ()
    ((_ i e)
     (if (= i 0) e))
    ((_ i e e* ...)
     (if (= i 0)
         e
         (choose-arg (sub1 i) e* ...)))))

While a full explanation of syntax-rules will have to wait for another post, let’s take a minute to make sure we have some idea what’s going on. We define macros with syntax-rules by giving a set of input and output patterns. Program fragments that match an input pattern are replaced with the output pattern. Our choose-arg macro has two sets of patterns. The first one matches things like (choose-arg 5 (+ 1 2)) and replaces it with (if (= 5 0) (+ 1 2)). Notice that the i and e in the output pattern are replaced with the code fragment from the input pattern. The second pattern is similar, except that the e* ... portion can match any number of instances. The second one would take (choose-arg 1 'a 'b) and transform it into (if (= 1 0) 'a (choose-arg 1 'b)). Macros keep expanding, so the next time around, the application of choose-arg would match the first pattern.

This version has the performance we would like, while also letting us write the code as we would like. Still, performance is not the most compelling reason to use macros. For any given problem, some languages are better at expressing the solution than others. Rather than having to choose the best language among many imperfect choices, macros let the programmer write their code in a language they have created for the program at hand. Additionally, macros make things easier for the language implementor because they can focus on high quality implementations for a small set of core forms and express the other forms as macros that expand to these core forms. For example, just lambda and if is enough to implement many more forms like let, let*, or, and, cond, etc.

$$ T_\mathit{total} = T_L + T_B $$

where \(T_L\) is the time due to latency and \(T_B\) is the time due to bandwidth. Typically, the \(T_L\) term is a constant. For example, when talking about two computers on the Internet, the latency term might be something like 35ms. When talking about the latency between main memory and the CPU, this term is on the order of hundreds of nanoseconds.

The \(T_B\) term normally depends on the size of the data being transferred. So, if the size of the data is \(S\) and the bandwidth is \(B\), we’d have,

$$ T_B = \frac{S}{B} $$

Sometimes there is a minimum amount of data that you can transfer. For example, many hard drives have a 512 byte sector size. These hard drives transfer data in units of 512 bytes, so even if you only need 4 bytes off of the disk you will still have to spend as much time as you would to copy 512 bytes.

My research group had a hypothesis that there is a similar minimum unit of data transfer for GPUs. Furthermore, we suspected this was a fairly large amount. This would mean for GPU programs we’d want to try to combine transfers to pay the latency overhead as little as possible. It would mean that in some cases we could get away with transfering more data than necessary in order to minimize the number of transfer operations.

In order to test this hypothesis, I wrote a simple program that copies data between the CPU and GPU in varying sizes. We expected to see a line that was basically flat up to a certain threshold size and then see the transfer time increase linearly. Here’s an example of what we saw in practice.

This is on a Core i7-2600K with an NVIDIA GTX460 GPU.

We see the general shape we expected to see. Up until about 8K, all transfers take around 6 or 7 microseconds. Afterwards, the transfer time increases linearly.

Though we saw what we expected, in some ways many of the expected implications do not hold. We expected the threshold to be in the range of several megabytes. Instead, the threshold was at 8K. It seems unlikely that your code will benefit from running on the GPU if you only have 8K of data. The second conclusion we expected to make was that it was okay to over-approximate the data to transfer. This is also invalid, because the size of data your program will typically be working at is so far above the threshold that you actually want to minimize the amount of data transfered to minimize the contribution of the bandwidth term to the total transfer time.

Another important lesson from this is that it’s important to test your intuition before basing design decisions on it. This test was pretty easy to write, and yet the decisions we would have made based on our assumptions might have had expensive and long-lasting consequences.

This should let me do some nice new things as well. For example, I could use D3 to generate nice inline graphs, or I could use MathJax to render nice inline math when needed.

Anyway, here goes my experiment with Octopress.