Gallium3D LLVM

I've seen the future. The blurry outlines sketched by such briliant audio-visual feasts as Terminator came to fruition as in the future the world is ruled by self-aware software.
That's the bad news. The good news is that we haven't noticed.
We're too busy due to the fact that we're all playing the visually stunning "The Good Guy Kills the Bad Guys 2" on Playstation 22. It really captured the essence of the first. The theaters are ruled by "Animals - Did they exist?" documentary with some stunning CG of a horse, although I could have sworn that horses had 4, not 3 legs but then again I'm no nature expert.
What everyone was wrong about though is which program became self-aware and exerted its iron-fist like dominance upon the unsuspecting humans and the last few, very suspicious cockroaches. It wasn't a military mistake. It was a 3D framework that evolved.

But lets start from the beginning. First there was Mesa, the Open Source implementation of the OpenGL specification. Then there came Gallium3D, a new architecture for building 3D graphics drivers. Gallium3D modeled what modern graphics hardware was doing. Which meant that the framework was fully programmable and was actually code-generating its own pipeline. Every operation in Gallium3D was a combination of a vertex and a fragment shader. Internally Gallium3D was using a language called TGSI - a graphics dedicated intermediate representation.

Gallium3D was generating vertex and fragment shaders at run-time to describe what it was about to do. After that system was working, some engineers decided that it would make sense to teach Gallium3D to self-optimize the vertex/fragment programs that it, itself, was creating. LLVM was used for that purpose. It was used because it was an incredible compiler framework with a wonderful community. The decision proved to be the right one as Gallium3D with LLVM proved to be a match made in heaven. It was pure love. I'm not talking about the "roll over onto your stomach, take a deep breath, relax and lets experiment" kind of love, just pure and beautiful love.

So lets take a simple example to see what was happening. Lets deal with triangles, because they're magical.
Now to produce this example Gallium3D was creating two small programs. One that was run for every vertex in the triangle and calculated its position - it was really just multiplying the vertex by the current modelview matrix - that was the vertex shader. The other program was run on every fragment in this figure to produce the resulting pixels - that was the fragment shader. To execute these two programs they were being compiled into LLVM IR, LLVM optimization passes were run on them and LLVM code generators were used to produce executable code. People working on Gallium3D quickly noticed that, even though, their code wasn't optimized at all and it was doing terribly expensive conversions all the time, it was up to 10x faster with LLVM on some demos. They knew it was good.

So Gallium3D was in essence, at run-time, creating and optimizing itself. Which lead many Free Software enthusiast to create, wear and rarely even wash shirts with a slogan "We might not have a billion dollar budget but our graphics framework is smarter than all the people in your company together".

Then in the year 2113 Gallium3D got bored with just creating graphics and took control of the entire world. Which realistically speaking wasn't hard to do because we were willingly immersing ourselves in the worlds it was creating for us anyway.

But that's still many, many years away from our boring present. So for now, while you wait for sex robots, dinners in a tube and world without insects (or for that matter absolutely any animals at all) you can just go, get and play with Gallium3D where LLVM is used. At the moment only in the software cases, but the fifth of November is going to mark the first day in which work on code-generating directly for GPU's using LLVM is going to start.

Remember, remember the fifth of November... (oh, come on that's one heck of an ending)

Gallium3D, Shaders and LLVM

Today we're going to talk about shaders. Well, I'll talk, or to be more specific write, or to be blunt I'll pretend like I'm actually capable of putting my thoughts into readable excerpts that other human beings (hopefully you) and some of my imaginary friends (they're not all winners) can understand.

The question I've been asked a few times during the last week was "who are you and what are you doing in the bushes outside my house", which isn't related to computer graphics at all and what I do in my spare time is none of your business so I won't be talking about that. Now the other question that I've heard a few times during the last week was "will Gallium3D use LLVM?", the short answer is "yes, it will".

First of all a little about graphics hardware. A common thing to do in modern graphics hardware is to have very wide registers and allow stuffing arbitrary vectors inside those registers. For example one register might very well store 8 2 component vectors. Or 16 components of 16 different vectors with other components being stored in subsequent registers. To support writing to those wide registers, usually there's another register, often a stack of them, which is used as a write mask for all operations. Cool, eh? So now when your language supports, god forbid, branches or loops and you want to code generate something for graphics hardware, you're left with two options. Option one is to give up and go ride donkeys in a circus and option two which is to do something crazy to make it work. To be honest I've never even seen a real life donkey. I've seen a cow but we just didn't hit it off. So I knew that option one is just not right for me.

So one of the big worries that we had was whether we'll be able to code generate from LLVM for graphics hardware. After some discussions about pattern matching in code generators and opcode lowering it finally looks like the answer is "yes, we will be able to generate something usable". So the way it will work in Gallium3D is largely similar to the I wanted to do it in the LLVM GLSL code that Roberto and I have been working on for Mesa a few months back. The difference is that the IR in Gallium3D is completely language agnostic.

You can run OpenGL examples already, granted that some of them will not produce correct results,but if it all would just work then I'd have nothing to blog about. I'll start integrating LLVM parts within the next two weeks which is when the performance should get a major boost and flowers should bloom everywhere. You might think that the latter is not, technically, related to our work on Gallium3D and the fact that Autumn is here makes that last statement even more dubious, but you're wrong. Who would you rather trust, you or me? I bet you thought "me" and so I rest my case.


And all of that is brought to you without any sheep sacrifice and hardly any virgin sacrifice ("hardly any" because I, as a representative virgin, am making a small sacrifice, but from what I understand it doesn't count as a full fledged "virgin sacrifice").
How do you like them apples? (or oranges... or strawberries... I like raspberries... They're all good is I guess my point).

Mesa and LLVM

I've been toying for a while with the idea of rewriting programmable pipeline in Mesa. The most obvious reason is the fact that fragment shaders are absolutely crucial when it comes to modern graphics. We use them extensively in Qt and I'm using it all over the place in my special effects library. As those small (mostly GLSL based) programs become more and more complicated a need for an extensive compiler framework with especially good optimization support becomes apparent. We already have such a framework in LLVM.

I managed to convince the person I consider an absolutely genius when it comes to compiler technology, Roberto Raggi to donate some of his time and together with me rewrite the programmable pipeline in Mesa. The day after I told Roberto that we need to lay Mesa on top of LLVM I got an email from him with GLSL parser he wrote (and holly crap, it's so good...). After picking up what was left of my chin from the floor I removed the current GLSL implementation from Mesa, integrated the code Roberto sent me, did some ninja magic (which is part of my job description) and pushed the newly created git repository to freedesktop.org.

So between layers of pure and utter black magic (of course not "voodoo", voodoo and graphics just don't mix) what does this mean, you ask (at least for the purpose of this post). As I pointed out in the email to the Mesa list last week:

• it means that Mesa gets an insanely comprehensive shading framework
  
• it means that we get insane optimization passes for free (strong emphasis on "insane". They're so cool I drool about being able to execute shading languages with this framework and I drool very rarely nowadays... And largely in a fairly controllable fashion.)
• it means we get well documented and understood IR,
• it means we get maintenance of parts of the code for free, (the parts especially difficult for graphics people)
• it means that there's less code in Mesa,
• it means that we can basically for free add execution of C/C++, soon Python, Java and likely other languages, code on GPU's because frontend's for those are already available/"in work" for LLVM. (and even though I'm not a big fan of Python the idea of executing it on GPU is giving me goose-bumps the way only some of Japanese horror movies can)

I think it has all the potential to be by far the best shading framework in any of the OpenGL implementations out there. Now having said that there's a lot of tricky parts that we haven't even begin solving. Most of them are due to the fact that a lot of modern graphics hardware is, well, to put it literary "freaking wacky" (it's a technical term). We'll need to add pretty extensive lowering pass and most likely some kind of transformation pass that does something sensible with branch instructions for hardware that doesn't have support for them. We'll cross that bridge once we get to it. Plus we'll need to port drivers... But for now we'll bask in the sheer greatness of this project.

Ah, the git repository is at http://gitweb.freedesktop.org/?p=users/zack/mesa.git;a=shortlog;h=llvm , Roberto and I have tons of unpushed changes though . Of course this is an ongoing research project that both Roberto and I work on in our very limited spare time (in fact Roberto seems to now have almost what you'd call a "life". Apparently those take time. Personally I still enjoy sleepless nights and diet by starvation patched by highly suspicious activities in between. Which by the way does wonders to my figure and if this is not going to work I'll try my luck as a male super-model) so we can only hope that it will all end up as smoothly as we think it should. And in KDE 4 most graphics code will be able to utilize eye-popping effects with virtually no CPU price.