Constant state objects

I know you wither away, like a leaf in the darkness of the chilly, Autumn night, without my posts. I've been depriving you for too long of the sunshine that is me. Writing an interesting post explaining the new way one is doing register allocation or how the code generation has changed to benefit the drivers is a little difficult and I didn't feel like it would interest anyone. Especially that I know that if I write a blog about a little demo app I wrote in half an hour to showcase some technology I get 50 comments but when I blog about the process of creating that new technology I get two comments with one of them being "What?". On top of that I was lacking those new nude photos (it's art you pig!) of myself that I've promised you and which you yearn for so much.

Lately I've been working on the Gallium3D i965 driver. I had to read up some documentation because, as it turned out, being a "graphics ninja" did not give me intrinsic knowledge of all graphics hardware ever built. I know! I was as shocked as you are right now.

It started with the fact that I wanted to experiment with the layout of vectors in shaders for the code generation using LLVM facilities.

To experiment with different layouts I've decided to write i965 driver for Gallium3D and experiment with LLVM code generation for i965 in that driver. Keith Whitwell and I have been hacking on it and it's going pretty well. It's amazing how much code we've removed from the old i965 driver while porting it to Gallium3D. It was a rather nice feeling to see so much of the complexity of the driver disappear.

The great thing about writing a Gallium3D driver is that a lot of the complexity of the high level API goes away, as it's being moved to "state tracker". State tracker is responsible for all API specific state handling and tricky conversions. The driver never sees the state tracker, it implements a very thin interface, which corresponds rather closely to the way modern hardware works.

One of my favorite changes is the new way we handle state changes. It used to be that the driver had to check whether any of its state changed and if it did upload it before drawing anything. It turned out to be a rather serious bottleneck and it made reading our driver a little painful.

In Gallium3D we went away from that. Now we use similar semantics to what Direct3D10 and OpenGL3 (will) use. Which is that states are largely immutable objects. Their usage follows the

• create from a template
• bind the state to make it active for subsequent rendering calls
• delete when not needed anymore

pattern. This way the driver can do all of its conversion on creation (which ideally happens only once) and then on bind calls (which we have multiple off) it can just reference the id of this state to have it used instead of having to do a full conversion from the Mesa state and upload of the converted state.
When I wrote the constant state objects code I ported our simple i915 driver and, even though on i915 we don't have hardware state caching just doing the state conversions in create calls improved the performance in simple examples by about 15fps. For more complicated examples where the state changes are more frequent it will be a lot more. Not even mentioning drivers which will do full state caching in the hardware where this is going to fly like Superman with diarrhea.

Less complexity in the driver and faster code, is what I think love is all about. Granted that my idea of love might be a smidge skewed on account of me being crazy and all but no one can argue with the "simpler/faster" being 'awesome'.

And to really top this graphics talk off, a picture of me naked:

In retrospect not my best day. The lighting was all wrong...

Gallium3D LLVM

I've seen the future. The blurry outlines sketched by such briliant audio-visual feasts as Terminator came to fruition as in the future the world is ruled by self-aware software.
That's the bad news. The good news is that we haven't noticed.
We're too busy due to the fact that we're all playing the visually stunning "The Good Guy Kills the Bad Guys 2" on Playstation 22. It really captured the essence of the first. The theaters are ruled by "Animals - Did they exist?" documentary with some stunning CG of a horse, although I could have sworn that horses had 4, not 3 legs but then again I'm no nature expert.
What everyone was wrong about though is which program became self-aware and exerted its iron-fist like dominance upon the unsuspecting humans and the last few, very suspicious cockroaches. It wasn't a military mistake. It was a 3D framework that evolved.

But lets start from the beginning. First there was Mesa, the Open Source implementation of the OpenGL specification. Then there came Gallium3D, a new architecture for building 3D graphics drivers. Gallium3D modeled what modern graphics hardware was doing. Which meant that the framework was fully programmable and was actually code-generating its own pipeline. Every operation in Gallium3D was a combination of a vertex and a fragment shader. Internally Gallium3D was using a language called TGSI - a graphics dedicated intermediate representation.

Gallium3D was generating vertex and fragment shaders at run-time to describe what it was about to do. After that system was working, some engineers decided that it would make sense to teach Gallium3D to self-optimize the vertex/fragment programs that it, itself, was creating. LLVM was used for that purpose. It was used because it was an incredible compiler framework with a wonderful community. The decision proved to be the right one as Gallium3D with LLVM proved to be a match made in heaven. It was pure love. I'm not talking about the "roll over onto your stomach, take a deep breath, relax and lets experiment" kind of love, just pure and beautiful love.

So lets take a simple example to see what was happening. Lets deal with triangles, because they're magical.
Now to produce this example Gallium3D was creating two small programs. One that was run for every vertex in the triangle and calculated its position - it was really just multiplying the vertex by the current modelview matrix - that was the vertex shader. The other program was run on every fragment in this figure to produce the resulting pixels - that was the fragment shader. To execute these two programs they were being compiled into LLVM IR, LLVM optimization passes were run on them and LLVM code generators were used to produce executable code. People working on Gallium3D quickly noticed that, even though, their code wasn't optimized at all and it was doing terribly expensive conversions all the time, it was up to 10x faster with LLVM on some demos. They knew it was good.

So Gallium3D was in essence, at run-time, creating and optimizing itself. Which lead many Free Software enthusiast to create, wear and rarely even wash shirts with a slogan "We might not have a billion dollar budget but our graphics framework is smarter than all the people in your company together".

Then in the year 2113 Gallium3D got bored with just creating graphics and took control of the entire world. Which realistically speaking wasn't hard to do because we were willingly immersing ourselves in the worlds it was creating for us anyway.

But that's still many, many years away from our boring present. So for now, while you wait for sex robots, dinners in a tube and world without insects (or for that matter absolutely any animals at all) you can just go, get and play with Gallium3D where LLVM is used. At the moment only in the software cases, but the fifth of November is going to mark the first day in which work on code-generating directly for GPU's using LLVM is going to start.

Remember, remember the fifth of November... (oh, come on that's one heck of an ending)

KHTML future

I've read Harri's blog about WebKit and I figured it makes sense for someone to respond. First of all I liked the blog, It was full of drama, action, despair, marketing and bad and good characters. Which is really what I'm looking for when reading fiction.

Especially the part that mentioned QtWebKit as an irrelevant fork of KHTML sources. That was awesome. It's the kind of imagination we need more of in the blogosphere. For the purposes of the point Harri was trying to make, which I think was "no matter what's the reality, our ego is bigger than yours", it was a well suited argument.

Describing the WebKit project as a fork of KHTML sources is like calling GCC a fork of EGCS, or to use a more popular analogy it's like calling chicken a fork of an egg. If you want to talk about forks then technically nowadays KHTML is a fork of WebKit. Not a terribly good one at that. It's real easy to back that statement up by comparing the number of submits to KHTML to the number of submits to WebKit. In fact that comparison is just embarrassing for KHTML.

I also found it funny that people like Lars Knoll, Simon Hausmann, George Staikos or myself are not part of the KHTML team. "We are the 'KHTML team' (except KHTML's author and ex-main developer Lars who's one of the biggest supporters of WebKit now and other people who used to work on KHTML but now work on WebKit as well... but they were all ugly... honestly!)" you can go make shirts with that.
We're working on WebKit now hence we're not KHTML team members. Any KDE developer who works on WebKit (hey, Niko, Rob, Adam, Enrico...) is automatically dissociated from the KHTML team.

The fact is that there is more KDE developers contributing to WebKit than there is KDE developers contributing to KHTML.

So since there's more of us, I think technically that means that we are the official KDE web engine team. KHTML team, we would love to work with you, the fork, but you're kind of a pain in the butt to deal with.

Which is ok, because like I mentioned a number of times KDE community lives of the "who does the work decides" dogma. And ultimately the Apple guys, the Trolltech guys, people from George's company who work on this stuff full-time and tons of Free Software contributors working on WebKit do much, much more work than people do on KHTML.

On a more serious note, let me explain a very important point: bug for bug compatibility with the latest Safari would be worth much, much more to KDE than any patches that are in KHTML and haven't been yet merged to WebKit could ever be worth.
Web works on the principle of percentages - web-designers test their sites with engines that have X% of market reach. Konqueror with stock KHTML isn't even on their radar. WebKit is. Having web designers cater towards their engine is worth more than gold to KDE users.

And if you care more about some personal grudges than the good of KDE, that's also OK, because we, the official KDE web rendering team will do what's right for KDE and use WebKit.

Gallium3D, Shaders and LLVM

Today we're going to talk about shaders. Well, I'll talk, or to be more specific write, or to be blunt I'll pretend like I'm actually capable of putting my thoughts into readable excerpts that other human beings (hopefully you) and some of my imaginary friends (they're not all winners) can understand.

The question I've been asked a few times during the last week was "who are you and what are you doing in the bushes outside my house", which isn't related to computer graphics at all and what I do in my spare time is none of your business so I won't be talking about that. Now the other question that I've heard a few times during the last week was "will Gallium3D use LLVM?", the short answer is "yes, it will".

First of all a little about graphics hardware. A common thing to do in modern graphics hardware is to have very wide registers and allow stuffing arbitrary vectors inside those registers. For example one register might very well store 8 2 component vectors. Or 16 components of 16 different vectors with other components being stored in subsequent registers. To support writing to those wide registers, usually there's another register, often a stack of them, which is used as a write mask for all operations. Cool, eh? So now when your language supports, god forbid, branches or loops and you want to code generate something for graphics hardware, you're left with two options. Option one is to give up and go ride donkeys in a circus and option two which is to do something crazy to make it work. To be honest I've never even seen a real life donkey. I've seen a cow but we just didn't hit it off. So I knew that option one is just not right for me.

So one of the big worries that we had was whether we'll be able to code generate from LLVM for graphics hardware. After some discussions about pattern matching in code generators and opcode lowering it finally looks like the answer is "yes, we will be able to generate something usable". So the way it will work in Gallium3D is largely similar to the I wanted to do it in the LLVM GLSL code that Roberto and I have been working on for Mesa a few months back. The difference is that the IR in Gallium3D is completely language agnostic.

You can run OpenGL examples already, granted that some of them will not produce correct results,but if it all would just work then I'd have nothing to blog about. I'll start integrating LLVM parts within the next two weeks which is when the performance should get a major boost and flowers should bloom everywhere. You might think that the latter is not, technically, related to our work on Gallium3D and the fact that Autumn is here makes that last statement even more dubious, but you're wrong. Who would you rather trust, you or me? I bet you thought "me" and so I rest my case.


And all of that is brought to you without any sheep sacrifice and hardly any virgin sacrifice ("hardly any" because I, as a representative virgin, am making a small sacrifice, but from what I understand it doesn't count as a full fledged "virgin sacrifice").
How do you like them apples? (or oranges... or strawberries... I like raspberries... They're all good is I guess my point).

Gallium3D

Critics are raving: "Gallium 3D is the best thing that ever happened to Free Software graphics", "It's breathtaking!", "Never before has nudity been so tasteful!"... Alright, maybe not the last one. Actually none of them, since it's a brand new project. In fact that's the point of this entry. To introduce you two.

You, a brilliant (as derived from the fact that you're reading this blog) Free Software enthusiast or simply my very own stalker (both options very satisfying to me personally). And Gallium3D, the foundation of Free Software graphics for years to come.

Gallium3D is a redesign of Mesa's device driver model. It's a new approach to the problem of accelerating graphics. Given tremendous investment that free desktops make in OpenGL nowadays I'm very excited to be working on it.

At Tungsten Graphics we've decided that we need a device driver model that would:

• make drivers smaller and simpler
• model modern graphics hardware
• support multiple graphics API's

The basic model, as presented by Keith Whitwell on XDS2007, looks as follows:


You can follow the development of Gallium as it happens in Mesas gallium-0.1 branch.

Also you can read a detailed explanation of what it is on our wiki .

Now why should you be excited (besides the fact that, like I already pointed out, there's no developer nudity in it and that being excited about the stuff I'm excited about is in general a good idea).

• Faster graphics
• Better and more stable drivers
• OpenGL 3
• Ability to properly accelerate other graphics APIs through the same framework. Did someone say OpenVG?

This is a huge step on our road to tame the "accelerated graphics" demon in Free Software. We've been talking about it for a long time and now and we're finally doing it. There's something zen like about working on free software graphics for years and finally seeing all the pieces falling into place.

Git cheat sheet

Due to the fact that I've been moving I forgot to point out that about three weeks ago I created a small Git cheet sheet. Quoting my email to the Git mailing list: I took a short break from being insanely handsome (which takes a lot of my time - gorgeous doesn't just happen) and based on similar work for Mercurial created a little SVG cheat sheet for Git. I'm not sure if it's going to be useful for anyone else (the target audience was composed of engineers who agreed to move to and work from Norway so you know right of the bat that historically they already made some bad decisions), but the times when I do art are so rare that I feel the need to share.

The thing that I took from the Mercurial sheet, besides the idea, is the flow-chart (people dig icecream and flow-charts, the first one is really hard to get into a SVG rendering so I went with the second) so the license is the same as of the Mercurial sheet which was Creative Commons. There's likely a few errors in it and if you have any suggestions or if you sport latex pants and a fancy green hairdo that goes with those pants (which equals the fact that you're an artist) and would like to pimp the sheet out, it would be my pleasure to help you.




The SVG is at:
http://byte.kde.org/~zrusin/git/git-cheat-sheet.svg
Sample png's are here:
http://byte.kde.org/~zrusin/git/git-cheat-sheet-medium.png
http://byte.kde.org/~zrusin/git/git-cheat-sheet-large.png

I also got up to speed on all the latest announcements. I thought that the Novell's Spotlight collaboration announcement was disappointing. I'm referring to the "Microsoft will provide Novell the specifications for Silverlight".

Richard Leakey once said "We are human because our ancestors learned to share their food and their skills in an honored network of obligation". I love that quote because it so beautifuly describes what we, so heavly, rely on in the Open Source community. For a company to take from the great ocean of free knowledge, led by an open standard of SVG and end up with a closed specification is just disgusting. Seeing an Open Source company strike a deal to cooperate on that closed technology is just sad to me. I understand why they did it but understanding something doesn't make it morally right.

I really hope, pointlessly as it might be, that the work on the Silverlight specification and the specification itself will be open. You obviously thought that SVG isn't good enough for your purposes and you built on top the experiences and ideas taken from SVG. Let us improve SVG based on your experiences and ideas. Once we've done that, you'll be able to repeat that process again. That's the way it works and that's the way our society has always worked.

Despite what you might think, you don't own ideas, they belong to us all.

Quasar

Does working on the 3D stack make me look fat? Give it to me straight. I promised myself I won't cry no matter what the answer is. As I mentioned in my last blog I'll be getting awfully kinky with our 3D stack and people wondered whether my work on it affects Quasar. Well, just as me moving didn't affect my habits with regards to taking showers (which I do frequently, i.e. every christmas or so), smoking (never ever), being awake (at least once per day) it also didn't affect my spare time habits (except the "starving" thing which was one of my favorite past-times in Norway). Since Quasar was always my spare time project, my move affected it as much as me buying new sneakers affects temperature in Japan (which it doesn't, assuming of course that the folks in Japan won't find the vision of me in the new sneakers so insanely hot to increase the temperature of the whole country).

Aaaanyway, so what is Quasar? Besides being a word that starts with a Q, has some other letters in the middle and sounds funky. It's a dynamic rendering framework, which in turn begs the question "what the hell is that?" ("the hell" being an optional part of this sentence).

Let me lead you through the evolution of Quasar and hopefully at end of this trip we'll be on the same page with regards to what we really need in terms of a graphical framework on the desktop.

Quasar started as a pretty basic image manipulation framework. I wanted to make sure that people can chain filters e.g. read-image->scale image->blur image->render image. It was a linear pipeline. The thing that made it important to me was that it was focused on hardware acceleration. This is a very important point which I want to underline.

I'm not interested in software based effects. This is not to say that the software versions of all of them shouldn't work, I just refuse to bother. It's 2007, even phones come out with some kind of gpu in them, worrying about software based graphical effects does to your time what listening to financial advises from blogs of Open Source engineers does to your money - wastes it (which reminds me, I also have some financial tips for you. Well, tip, just one. It's not the quantity but quality as they say. Invest money in winning lottery tickets. Note that I said "winning", buying any other kind would be silly. You're welcome). With the work we're doing right now at Tungsten Graphics our graphics stack will be extremely good at it.

When I was a kid we didn't have fancy graphics. We had crackheads, alcoholics, pimps, thugs and plenty of other shady characters none of which, I'm fairly certain, did computer graphics (childhood memories warm your heart, don't they?). If my years of working on a vector graphics framework taught me anything (besides how to write a vector graphics framework) it's that the most fundamental part of a high level graphics framework isn't the framework at all. It's the way in which you let people design their applications, their interfaces. What use are blurred perspective transformations if two people know how to use them? What's the point of 20 composition modes if people can't be bothered to understand the basic 2.

So I started seeing effects as a step, a very crucial one, on the way to very appealing interfaces, but not my ultimate goal. In a desktop world we're dealing with graphics on a fairly abstract level, because we're doing graphics for non-graphics people. Good effects framework is not one that has every possible effect ever invented but one that makes it possible to quickly add those effects to ones application/interface and end up with something usable and pretty.

Finally inherent complexity of graphics bothered me. Maybe not even the complexity itself (especially since it's where my job security stems from), but the fact that it takes a lot of knowledge to understand and know the fast paths. The knowledge necessary to extract that logic can be easily abstracted in a graph like structure. That's what I started doing in Quasar.

Quasar is a dynamic graph. One that can rearrange itself to produce the output in the fastest possible way.
Those are the three basic tenants of Quasar, to rehash, they are:
- hardware accelerated,
- easy to incorporate and produce good looking results,
- smart enough to produce the results in the quickest possible way, no matter how little knowledge its users posses.

The way it works is that one creates the rendering in a graphical builder (which at the moment looks very much like Quartz Composer), loads it up in application and, well, that's it.

Of course, the graph can be created or modified at run-time just well. It follows the "tell me what you'd like and I'll give you that in the best way possible" principle.
Having said that Quasar is still far from complete but I just posted where to get it from on the relevant KDE list (if you don't know which one, trust me you want to wait a little bit longer before trying it out). Oh, and of course Quasar will support full effects pipelines on top of movies (integrated with Phonon).

In London

Today was my last at Trolltech. Some of you already knew that, the others are likely getting mouth to mouth reciprocation right now while a heavy set man is charging defibrillator paddles next to them (breath, breath!).

Technically Qt is years ahead of all of its competition and with Trolltech's amazing engineering team it will continue doing great. I felt that other parts of the Open Source graphics stack could use my work a little more at the moment.

We (Open Source community) move at an astounding pace but the lower parts of our graphics framework are lagging a little bit behind. I wanted to go back to working on those parts of the graphics stack and it made sense for me to join a company that would directly profit from my work on Mesa, DRI, drivers and X.

This Monday will mark my first day at Tungsten Graphics. The engineering team at TG includes the likes of Brian Paul, Keith Whitwell, Michel Dänzer, Alan Hourihane and others, which reads almost like "Who's who in Open Source graphics". Clearly the only thing that was missing there was someone who looks really good naked. That's me. I also know a thing or two about this whole "graphics" thing which tends to help.

I'll be living in London for the next ~2 months or so. In England, while mainly working on Mesa, I'll be filling the void left by the departure of Victoria and David Beckham.

Small steps

I was on vacations last week but I'm being all jealous of my luggage. It got a free trip around the world. During my last 8 flights my luggage has been lost 5 times. Is that a record? Confetti anyone? It's a celebration. If you're going to meet me during any of the upcoming conferences I'll be the outgoing and highly sarcastic naked guy with a sign on my chest saying "for my face look this way" and an arrow pointing up.

I neglected to mention that, as Simon said, QtWebKit is working on Windows. Simon did an amazing job of porting all the quirks of the build system but "amazing" is the default state for all of his code so it's not a surprise at all. While he was doing that I've sat down and ported XML tokenizer to QXmlStream from LibXML. If you never wrote a web rendering tokenizer (and unless you're crazy, the chances of that are pretty high, and if you did you're crazy and won't remember doing it anyway) you know that "fragile" is a term that nicely describes it. After it was ported Lars and I sat down to fix the regressions and they didn't even know what hit them (ha! ninja reference).

In other news I've merged in FreeType2 rasterization algorithm patches in Qt. Our raster engine, uses the beauty that is FreeType's rasterizer, with a few patches on top. Because they break BC in FreeType's public interfaces we can't merge them back at the moment. In any case the patches improve rendering speed in general antialiased paths of the raster engine (meaning on Windows, Qtopia Core and in general whenever rendering to a QImage) by about 10% which is gangsta awesome ("gangsta awesome" is a very high level of awesomeness, at least judging from MTV).

I've also optimized the path clipping code. Andreas uses the path clipping code in GraphicsView for collision detection, so when I say "path clipping code" you should read "path clipping and GraphicsView collision detection". A lot of the time in that algorithm has been spent on vertex allocation for tested paths. I've used a few tricks to speed it up by about 15%. The code for that algorithm is the number two reason why baby seals die (the first is still undisputed). It's not even the algorithm itself but the inherent complexity of the problem. I'm a big fan of computational geometry in computer graphics because it makes grown man cry, except me and I like feeling like the lean, mean, killing machine that I am. My favorite part of the path clipping problem is that there are two ways of solving the precision problems and neither of them really works. The trick is that paths operate in double coordinate system, efficient snap-rounding implementations that I've seen operate in fixed-point coordinate system which falls apart in this case because of absolutely random distribution of vertices across the full double spectrum. Tessellation and clipping itself can be done in a screen coordinate system, which makes it possible to consistently represent your coordinates with fixed-point representation. That doesn't work for paths because, e.g. boolean operations on paths need to be done in native path coordinates not screen coordinates. So the algorithm forces an absolutely crazy mix of dynamic fixed-point size, reduced-predicates, magic and good-will to work. Aren't you happy that I'm doing it for you? You better be.

Yours(1) Latino(2) Lover(3)

1) Not really "yours", more "community". I love "you" but "you" need to realize that I need to be seeing other people.
2) Not really "Latino". Unless of course my Spanish or Brazilian friends would like to name me an "honorary Latino" or "Latino by association". I'd be definitely down with that. The only food I can make that is eatable and doesn't force the fire department to evacuate the building before are nachos. I'm a definition of grace in the kitchen. "Whatever you have in the kitchen I will make it burn" is my motto. Plus I'm sporting quite an attitude to boot. "Make Zack a Latino" campaign. We can make it work!
3) Not really "lover". More "no feelings haver". Though technically I've worked on software for so long that hate is, next to sarcasm, my primary export.

Web on canvas and Dashboard widgets

There are days when I do something quite interesting and in my mind I can almost see myself on a stage in tight, tight spandex pants, long hair, perm, cowboy boots yelling angrily "are you ready to roock?!". People cheering, babies laughing, women throwing their bra's on the stage. It's poetic. Then I remember that I'm a computer scientist and I snap right out of it. I go back to the life filled with math equations on napkins, sleepless nights in front of buzzing computers, stacks of books in corners and no spandex pants (although I can deal with the last one just fine). The fact that I hate rock lessens the blow, but it doesn't make it any less disappointing. So in those moments of sadness I blog, yearning attention and approval, so readily available on the internet. Cough, cough...

I was wondering how hard would it be to create a QGraphicsItem that uses QtWebKit to render pages on a canvas. The idea being that combining full blown canvas like QGraphicsView framework with web rendering engine would give us quite a killer combination. So I've sat down today and done it. At first I had to redo some of the rendering code in QtWebKit and once I was finished I had a QWebGraphicsItem that beautifully renders pages. It being a QGraphicsItem all the effects available to graphics items in Qt are available for free to it. So you can animate, scale, rotate, perspective-transform and do a whole bunch of neat effects on it for free. Once I've done that I figured that it's obvious that this is the best way of getting Apple's Dashboard widgets to work. So I've done that too. I quickly hacked up a class that reads-in Apple Dashboard widget bundles and can render them on a QWebGraphicsItem. The compatibility is not 1:1 quite yet, because some of the Dashboard widgets use JavaScript objects that I haven't implemented yet, like AddressBook object. To be honest I'm not 100% sure whether I want to implement them, I think we can get those things done a lot nicer, it's just a question of whether 1:1 compatibility with Apple Dashboard is worth the extra effort needed to make all those JavaScript objects work on KDE.
First a screenshot of one Apple Dashboard widget rendered and on top a scaled to half its size KDE homepage:


Now a Dashboard widget with a perspective-transformed dot.kde.org page. Since this is QGraphicsView I can interact with the item while it's transformed so I've selected some text on it.

Crackalicious. (no drugs were used while hacking on this, but I did touch myself a little after getting it to work). Furthermore (yes, there's more... what can I say, I'm a giver...) in QtWebKit we have this neat interface that allows you to inject QObject's into the framework as JavaScript objects at run-time, so adding new JavaScript objects is trivial and getting Opera widgets to work would be very, very simple. No spandex pants included though.

Scripter

Before boarding a flight I'm eagerly awaiting the security presentation that is about to ensue. I figure that these people spent their valuable time learning how to point to the ground, back, forward and to the side and someone needs to appreciate that effort. During my last flight I even went ahead and tried to inspect my life jacket. "Tried" because as it turned out my seat was missing it. As the security announcement advised, I "calmly" informed the fellow sitting next to me that "in the unlikely event of a water landing" he's screwed because as a stronger man I'm taking his life jacket. This also prompted me to think about tools that make our life easier (just to clarify, when I say "our" I mean "my"). I've spent a little time yesterday creating a tool that will make my (when I say "my" I mean "our") life a lot easier. So today I wanted to tell him (when I say "him" I mean "you") about us (when I say "us" I mean "it").

I've spent a lot of time writing simple C++ applications to test out some kind of rendering algorithm. Internally we had a tool that automated a lot of it. The tool uses a very simplistic, reg-exp based language to specify commands. I wanted something more powerful. This is how scripter came to be. Scripter is a very simple application that uses QtScript's bindings to Arthur to do its rendering. It allows for rapid prototyping of algorithms and most importantly for me, quick testing of Qt's rendering framework. At first it was a whole IDE with its own code editor, very quickly though I decided to remove the editor and just make it a content widget that monitors the file it was opened with for changes. The reason for that is that I wanted to keep working in my own editor and just have a dynamic, visual preview of everything I was doing. So with Scripter one can be writing rendering code while the visuals effects of the editing are immediately visible. Here are two screenshots of examples included with it:



But the whole beauty of this application is ability to create animations, while seeing the changes done in real-time. I recorded two demos:

• The first one shows me just playing around with an example. In this case it's a freedesktop.org clock.
• The second shows me writing a simple animation from scratch. This one has an added benefit of seeing me hack in real-time, no copy&pasting, a little bit of chaos (next time I should probably figure out what kind of animation I want to do before start recording, but oh, well, live and learn). Fun. This one is 12mb though.

Scripter requires Qt 4.4. If you don't have Qt 4.4's snapshot it won't work. Get it from the SVN at labs.trolltech.com with :

 svn co svn://labs.trolltech.com/svn/graphics/scripter

Revelations

Like many other geeky people I've spent the last week in Scotland. The insane abundance of overgrown egos caused wrinkles at the very fabric of weather conditions in Glasgow and it rained the whole week. It happened to be my second birthday in a row that I've spent on a KDE hackfest. If things continue like this Aaron will have to adopt me, because I refuse to marry him and we're spending way too many "family holidays" together to be "just friends".
I've been always highly supportive of the "Earth is the center of the universe" theory. I considered this to be a side-effect of the fact that I'm the very center of the creation. By association this small piece of rock has got to be in the smack-middle of the universe.
That is until I find a way of developing a dust-powered interstellar spacecraft (dust being the only material possession that I have at my disposal in numbers even remotely close enough to those necessary for intergalactic travel) and I leave this place in pursuit of happiness/aliens. Unlike all other people I don't really want to meet aliens or exchange knowledge/ideas with them, I just want to punch a foreign life form in whatever is their equivalent of a face. Stay with me here, because there's an elaborate reason for that: I'm spending a lot of my time talking to people and for the purposes of a story "I've seen aliens, met them and we chatted a bit about the meaning of life, technology, universe and then I got anal probed" makes a boring story. One that is so prevalent in Alabama. Now if I've met someone who has seen an alien, met an alien and punched it in the face - that person has got my full attention. A way better story.
But for now I'm stuck here, so I used this week to look over some of the graphics code that was written for KDE4. It looks pretty good. Clearly a lot of it has been influenced by heavy drugs and some people have got to be shooting it straight down their aortas for them to have that severe effects. I'm trying to figure out whether to write a FAQ or some scripts to clean some of the mess up. I've also done a little fun hacking. I added sample 3D plasmoids looking like this:


They work very nicely and taking that code allows you to write 3D plasmoids. The examples animate (pretty nicely I might add, but everything I do is at least "pretty nice" so that's a redundant statement). The code is in KDE SVN playground/base/plasma/applets/samplegl* module.

Since for some cards we still don't have terribly good 3D drivers I created a sample plasmoid that shows how to rotate it and have something shown on the back. The code is in the playground/base/plasma/applets/rotator. A movie with the rotating widget is here. And statically it looks like this:



With much love
Yours Truly: militant interstellar dominator (by day), graphics voodoo practitioner (by night)

Mirroring widgets

"The life of man is divided between waking, dreaming and dreamless sleep." or so it's written in "The Upanishads"... I wouldn't know because the cartoon version still hasn't been released and I refuse to spend even a second of my life pumping a stream of information into my brain that hasn't been properly sprinkled with commercials and product placement. Which reminds me: use Qt...

I can almost see you sitting at your desk with the same expression the great Plato had when he said: "What?" (not one of his greatest quotes, but I'm sure he said it at one point or the other). In my last blog I described how the engineering department at Trolltech spent the last few months fixing bugs in, what could be described as, a constant "waking" state. By natural progression the next Qt release is putting us in the "dreaming" state.

It's been a while since the last time that I've posted an example on how to do something funky so today I'll partially make up for it. I get a lot of questions asking me how to do something windowing system specific. People at the office can approximate the exact time of delivery of each one of the emails relating to that topic as the seismographic vibrations, originating in the vicinity of the area where the table meets with my head (in a repeated and aggressive fashion), cause ripples to appear in coffee mugs around the office. Hopefully today's example will satisfy the most vicious desires for X11 wackiness.

6'2" (height by association - as measured by the height of the author), weighting at about 139 lines of code (while wearing the license header), the undisputed (mainly because the only one) champion (questionably) of... well, nothing: QX11Mirror. QX11Mirror is a class that can monitor and return the contents of any X11 window in real time. So you could start your favorite media player, pass its window id to QX11Mirror and then render the contents half-the-size with perspective transformation. It would look like this:
One thing that you can't see is that the contents of the movie is updated while the movie is playing and the perspective transformation is animating. All of which is done in real-time. It's really cool.

The original reason for writing this, seemingly, silly example was not "making something pretty". Even though the inability to make applications look gorgeous is the number one cause of hair loss (as shown in a study by doctor "me". Note: "me" is not really a doctor. In fact "me" doesn't even fulfill grammatical requirements of the previous sentences.), which is number one reason for you not looking gorgeous and I'm a big proponent of having KDE developers look beautiful. "KDE - we got hair... In all the right places...". (I know... I'm as shocked as you are that I'm not being paid a zillion dollars to do marketing.) No, the original reason for all of this was to make web plugins behave a lot nicer. Currently the problem is that they don't compose correctly within the web rendering tree. So what I wanted to do is correctly fetch the offscreen contents of those windows, render them in correct stacking order and propagate the events back to them. This goes along the "make stuff work" ideology which I consider myself to be a big fan off. Oh, and here's Flash plugin rendering inside of a Qt applications (as always in my example, things are animating inside):
Oh, and the code is available at Graphics Dojo.

Keyhole

Today's blog might sound a little muffled, that's because it's coming straight from the heart. Which is in direct opposition to all of the "What's Open Source community missing" blogs/articles, which are coming straight from a vastly less prominent body part. To not dwell to much on human anatomy, I'm going to move on to the main topic today which is peace, love and Qt 4.3. The first two are overrated and got their fair share of treatment in all kinds of literary works, therefore the only reasonable conclusion is that my perky-self focuses on Qt today.

A number of people surely have already pointed out in their blogs that Qt 4.3.0 has been released. The first thing you'll notice about this release is the version number. Rightfully so because 4.3.0 is the highest Qt version we've ever released. 4.2.0 was already taken and we felt very strongly about reusing a ".0". Although "13.13.0" was available we really came together as a team/body/unit/crew (pick one) to release 4.3.0.

Now, I'm not going to be doing marketing for 4.3.0 (mainly because others are being paid better to do that) or listing the "like totally awesome new features", what I wanted to do is present the perspective of people who actually spent days and nights working to make this piece of software the best they could. Whether from the loins of those geeks came something exceptional is a judgement call that I leave to you. This is your keyhole into our world.

The main focus of this release was for us the general increase in quality of Qt. In the darkness of our meeting rooms we were moving tons of paper, while from time to time lonely tears danced on our cheeks (due the fact that the light smoke coming from our pens and pencils was irritating our eyes). Immersed in this mysterious darkness, we sat and watched. The cracking of the projector, seemed to be the only noise that dared to challenge the insanable silence. We knew that we wanted to make people smile a little more with this new release. Very early in the release process we also agreed that shipping drugs with Qt was not an option. We turned for help to the happiness champions - Care Bears and Popples. We watched and analyzed. In the very end we decided that while we agree that Gi-Joe's are like totally cooler (the engineering department is unfortunately male dominated) we know what we have to do. We came to the conclusion that people seem to be a lot happier if things work the way they planned and there's no unpleasant surprises. That's what we focused on. Fixing bugs, making sure that things work the way they are expected to, all in all improving the quality of Qt. If you have ever spent extended periods of time just staring at the code and rerunning tests in order to fix bugs, you know it's a mundane process that takes a lot of concentration. This is what we've been doing for the last few months. Fixing bugs and running around screaming (a lot of screaming, not a whole lot of running) if someone broke one of the tests. Have you ever seen engineers play ping-pong after fixing bugs for weeks? Oh, it's quite a sight. A lot of raw energy (very raw, one could say that energy hardly touched by any kind of skill). Flying balls, paddles and often engineers were a common sight late in the evening (low flying engineers bring bad luck - especially if the flight schedule predicts a landing at "you").

No one is more critical of us than we are but we work together to improve all the things we don't like. While I was thinking about this today I couldn't stop thinking about Pythagoreans who despite achieving many great things, were often described as a group who cherished authority beyond anything else. That approach is completely different from any discussion we have here. Every argument is judged solely based on its soundness no matter who's making it. I think that what I'm trying to say in this, severely not hysterical, paragraph is that one thing you can be sure off is that every decision we have reached and every change we have made in Qt was not due to any kind of hidden agenda, religious beliefs or beauty of people pushing for it. It was done because 20+ engineers decided that the arguments for it are stronger than against it. I realize that due to the fact that we have those discussions at the office and in person, a lot of the transparency of them remains hidden behinds clouds for people outside Trolltech. We're working to improve the flow of the information from the heart of Qt's home to the outside and while we do that please remain assured that behind the clouds there's a world adhering to the strictest physical and logical rules and not an ocean of strings with muppets drowning underneath.

Having said that, we're very happy with Qt 4.3.0 so hopefully you'll enjoy it too.

If you want pure Qt release goodness with pictures of the chosen few who get to stare at Qt code until their eyes bleed make sure you read Girish's blog.

Browser plugins

One of the perks of being a software engineer is that during the day you get to use such phrases as "oh, what the hell", "that's pure crack" and a large number of "expletive-deleted" without it seeming weird in any way. I can only feel sorry for doctors who rarely get the opportunity to scream "oh, what the hell" while looking at new patients. It's quite therapeutic.

I've spent a little time this week implementing support for plugins in WebKit Qt and the general situation of cross-browser plugins is the reason for the above, spot-on, observations. My blog is, of course, famous for hard-hitting political satire but not today. Today I'll talk technology. No politics, hardly any environmentalism, virtually no vegan propaganda - just straight to the point geek talk (well, "write" but "talk" sounds a lot better).

Given how important browsers are in our day-to-day computer usage it would seem that cross-browser plugins should be a well understood and in fact solved problem. Nope, that's definitely not the case here.

So while tapping your fingers on the messy desk, you roll your eyes and ask "so zack, what is wrong with browser plugins?".
The role of zack - young, careless and wild wacko is played by critically acclaimed me. The role of "you" is played by "you" and "you" really need to step up because you're not convincing anyone right now.

Currently cross-browser plugins use a very sexy (assuming of course that you're heavily into s&m) mix of Xt and another toolkit of their choice. The usage of Xt is just an utter disaster. In fact mixing event loops in any application is a cause of many subtle problems. George had the best idea when he implemented netscape plugins in an out of process application for Konqueror. It's really the only thing that makes sense.

In the Konqueror when a page has a plugin, an external application, the plugin viewer, is started. That process in turn loads and initializes the actual plugin and via DCOP informs the browser what is its window id. Once the hosting browser knows the window id of the plugin, it can XEmbed it. The neat side-effect of this approach is that if a plugin misbehaves one can kill it without any damages to the current browser session (besides the fact that the plugin doesn't render anymore).

The new plugin standard tries to use XEmbed all way. Unfortunately it assumes in-process communication between the browser and the plugin which again is a bad idea. It's a bad idea because it removes this neat concept of an "event loop" from the equation and assumes that whatever event-loop the plugin uses is the same as the one hosting browser is running. Which in turn doesn't do wonders for the whole "cross-browser" aspect of plugins (as long as the definition of "browsers" includes at least two entries and at least one of them is not Firefox).

So lets say you'd like go nuts (like young people tend to do) and display a slider in your plugin which, of course, good, god-fearing people never do. But lets push forward with this highly dubious example and say that maybe you wouldn't want to go all out and display, maybe not right away a slider but some kind of an animating widget. Well, you're pretty screwed because you need the event-loop native to the toolkit in which the widget is implemented to receive the timing events.. Just ask SwfDec guys. They wrote this awesome piece of software, then wrote a browser plugin using, what claims to be, a cross-browser standard just to find out that in that particular standard "cross-browser" meant that it works in Firefox and Firefox. So SwfDec guys found out that Konqueror is not part of that set (unfortunately they also found out a few other things).

So how could we fix it, without having a large groups of grownups sharing some quiet time in a corner, while the sound of weeping fills the room. Hosting plugins in an external process has clear advantages. To the list that I mentioned above we can add another point - by running a plugin in an external process plugins could easily specify what kind of an event loop they need. If your plugin needs Glib's event loop, GTK+ container would be started, if your plugin uses Qt, Qt container would be started (which might or might not use Glib's event loop in Qt 4) if your plugin needs Xt then you should be punched in the face and it all would just work. Containers would announce their window id's through DBUS so that hosting applications could XEmbed them and the world would be a better place (because clearly "browser plugin woes" are occupying one of the top spots on the list of "what's wrong with the world", right between "dust" and "rich people").

At the moment I'm trying to see how feasible it is to lay npruntime implementation, which serves as a bridge between the plugin and the hosting browser on top of DBUS. If that works then we can move to running all plugins as external processes. Which means:

• plugins taking a lot of CPU can be stopped/killed without affecting the browser
• when a plugin crashes the browser keeps running without any problems
• external event loops don't pollute the browser
• a lot of code for plugin viewers could be shared (besides networking interface) between browsers on systems running X11. Which means that for a change plugin working in one browser would run in all of the others using those containers.

This is basically what we're doing in WebKit Qt. I just really want a simply working plugin infrastructure for browsers. I'm not a man who holds grudges, but last year I didn't get what I wanted, even though I've been exceptionally good, so this better work (last year I wanted to win the lottery. Sure, some might point out that I made it exceedingly difficult by not playing (did you see the odds?) but lets not get bogged down by minor details).

Oh, and a mandatory screenshot of the Diamond plugin in QtWebKit.

Mesa and LLVM

I've been toying for a while with the idea of rewriting programmable pipeline in Mesa. The most obvious reason is the fact that fragment shaders are absolutely crucial when it comes to modern graphics. We use them extensively in Qt and I'm using it all over the place in my special effects library. As those small (mostly GLSL based) programs become more and more complicated a need for an extensive compiler framework with especially good optimization support becomes apparent. We already have such a framework in LLVM.

I managed to convince the person I consider an absolutely genius when it comes to compiler technology, Roberto Raggi to donate some of his time and together with me rewrite the programmable pipeline in Mesa. The day after I told Roberto that we need to lay Mesa on top of LLVM I got an email from him with GLSL parser he wrote (and holly crap, it's so good...). After picking up what was left of my chin from the floor I removed the current GLSL implementation from Mesa, integrated the code Roberto sent me, did some ninja magic (which is part of my job description) and pushed the newly created git repository to freedesktop.org.

So between layers of pure and utter black magic (of course not "voodoo", voodoo and graphics just don't mix) what does this mean, you ask (at least for the purpose of this post). As I pointed out in the email to the Mesa list last week:

• it means that Mesa gets an insanely comprehensive shading framework
  
• it means that we get insane optimization passes for free (strong emphasis on "insane". They're so cool I drool about being able to execute shading languages with this framework and I drool very rarely nowadays... And largely in a fairly controllable fashion.)
• it means we get well documented and understood IR,
• it means we get maintenance of parts of the code for free, (the parts especially difficult for graphics people)
• it means that there's less code in Mesa,
• it means that we can basically for free add execution of C/C++, soon Python, Java and likely other languages, code on GPU's because frontend's for those are already available/"in work" for LLVM. (and even though I'm not a big fan of Python the idea of executing it on GPU is giving me goose-bumps the way only some of Japanese horror movies can)

I think it has all the potential to be by far the best shading framework in any of the OpenGL implementations out there. Now having said that there's a lot of tricky parts that we haven't even begin solving. Most of them are due to the fact that a lot of modern graphics hardware is, well, to put it literary "freaking wacky" (it's a technical term). We'll need to add pretty extensive lowering pass and most likely some kind of transformation pass that does something sensible with branch instructions for hardware that doesn't have support for them. We'll cross that bridge once we get to it. Plus we'll need to port drivers... But for now we'll bask in the sheer greatness of this project.

Ah, the git repository is at http://gitweb.freedesktop.org/?p=users/zack/mesa.git;a=shortlog;h=llvm , Roberto and I have tons of unpushed changes though . Of course this is an ongoing research project that both Roberto and I work on in our very limited spare time (in fact Roberto seems to now have almost what you'd call a "life". Apparently those take time. Personally I still enjoy sleepless nights and diet by starvation patched by highly suspicious activities in between. Which by the way does wonders to my figure and if this is not going to work I'll try my luck as a male super-model) so we can only hope that it will all end up as smoothly as we think it should. And in KDE 4 most graphics code will be able to utilize eye-popping effects with virtually no CPU price.

Folding

One of the more frequent questions that I see is how to do interesting transition effects for applications. So here's how, first of all prototype your algorithm. Here we're going to use a page folding to visualize transitions as turning of pages. So that it looks like this:

Then we figure out how to adopt it to application. The trick is how to get contents of an application without extra repaints. In Qt the contents of top-level's is buffered and there is a private but exported way of fetching that contents (not fully platform independent at the moment). The example #2 shows how to grab the contents of the window without extra repaints and copies and use it to implement the effect, so it looks like this:

The forms were taken from Kate config dialogs so they're real config pages (hence also why they're, well, not too pretty). The effect is fully animated so to get a better idea of how it looks you'll have to look at the ogg file provided here. It's also worth noting that this effect is very efficient and works smoothly with virtually 0 cpu usage.

As mentioned in the last blog, to see more examples and code for this one visit Graphics Dojo. The above mentioned examples are at the examples 2 page.

Graphics Dojo

I have been traveling and working a lot lately. I was attending Bossa Conference, which was the best organized Open Source conference I had the pleasure of attending. I got to hang out with people I normally spend very little time with like Rasterman. Raster and I had some time to sit down and talk about graphics, Qt, Evas, chickens (don't ask), him being a crab killer (again, don't ask) and just have a lot of fun. Chris explained a few things about LLVM to me. It's always fun to hang out with Marcelo as well. All around it was a great time.

I went to spend a little time with my "family" in England last week which was also just amazing.

The most important news though is that thanks to amazing work that Marius did on labs.trolltech.com I've got a Qt focused graphics corner. Being a Graphics Ninja, I've decided to call it the Graphics Dojo. The first action point was the release of some of the examples that I wrote in my spare time. You can take a look at an overview here or go directly to page 1 or page 2. There's a few ones I've never shown before, like Mario Klingemann influenced Ring of Fire.



I hope that the "Graphics Dojo" and the articles which I'll be posting on it will help everyone write better looking interfaces.

Reflections

In the spirit of my never ending "pimp your Qt application" series comes another example. This time, "how to make iTunes-like album selector". It's funny how much attention this widget got. I looked at it yesterday on one of the Macs at the office and just implemented it. I probably should make it a view for a list model but for now it's just a simple widget. It runs at perfectly smooth 60 frames per second while utilizing ~7% cpu on my 3GHz Pentium4 with NVIDIA GeForce 6600, so by no means a monster of a machine. Oh, and of course this is all done with pure Qt. Reflections are actually a vector effect, so they would work equally well for any vector based graphics (by using the same code as this example, you can reflect your svg's or even whole widgets with no problem). Mandatory screenshot:
And as with every animated example a movie (again, framerate of the screencapture does not come close to the real world performance) is available here . Finally the code is available here.

Gradient bounds

The most unfriendly thing about rendering gradients with Qt has been the fact that you had to specify gradients in coordinates of the shape they were to be rendered on. It wasn't ideal especially for all applications which included any kind of animations or were rendering large number of items because it meant that the gradient had to be individually created for each and every one of items/frames. I've fixed that two days ago by adding a coordinate-mode property to QGradient, which now accepts ObjectBoundingMode. Object bounding mode, just like in SVG, means that the gradient coordinates are percentages of bounding box of the shape that the gradient is about to fill. So all the coordinates are between 0 and 1 and Qt automatically adjusts the bounds for gradient when it's being rendered. This makes it possible to easily use QGradient's with QPalette. An example where I'm drawing a bunch of rectangles and animate them along while the gradient is set only once with (0,0, 1, 1) coordinate box (meaning starting at the topleft and ending at the bottom right corner of each rectangle).
And since it's an animation (including a widget show/hide effect) here's a movie showing it in action.

Browsers, performance and interventions

This week I beat WebKit Qt's rendering into shape. I rewrote the theming and canvas code.
It works so nicely that I'm actually pretty happy with it. The rewrite helped me fix issues with statically positioned elements (which just didn't work on a scrollview/canvas combination I did before). Due to which my blog finally looks and behaves correctly as seen on the screenshot above. That plus scrolling is about 10x faster and you never see gray areas on scrolling as you did before.
This week George Staikos has been here in Oslo to work with us on the networking code. George and I have been friends for five or six years and he has this nasty habit that forces me to start an intervention and try to get him off it. What I'm referring to is the fact that George has been Canadian for, well his entire life, and it is my professional opinion (but I'm not a doctor) that he needs to move on. I'm not sure in which culture it is customary to bring Snapple ice-tea to your friends when you come to see them but we need to change him to that because he definitely didn't bring me any this time and that's just rude.
Going back to work, I'm sure Lars will blog about the networking magic they did so I'll keep my mouth shut. Once the code they were working on will be ready (which should happen within next week or two) and we'll start using in WebKit, we'll make the transition and start using WebKit Qt based browser on a daily basis. So we're close which is pretty exciting.

We have also started optimization run for Qt 4.3. My grand plan is to make sure Qt 4.3 is 2x faster in general rendering code than Qt 4.2 was. I have a list of algorithms to shave and rewrite over the next few weeks which will be rather challenging.
By the way of performance some people noticed that Qt OpenGL in recent snapshots with Antialiasing turned on is actually slower than it was in Qt 4.2. It's because of an experiment that we're trying on. We're now using GLSL to antialias primitives. Of course it's a lot slower than the old code but produces very high quality results. I don't really like it though because the performance hit is just too big.

Core dump

Today I'm joining the rapid fire blogging squad.

• Roberto is moving back to Italy and last Wednesday was his last day at the office. Roberto is one of my best friends at Trolltech so I was incredibly bummed out by his departure. The office won't be the same place without him. Plus who will help me hack on a prototype GLSL jit engine that I wanted to do some research on? (rhetorical question, I'd only trust him with that anyway)
  Roberto is also, by far, one of the best hackers who ever worked on KDE so I'm very excited to say that Roberto and Trolltech came to an agreement by which Trolltech will sponsor Roberto for two days of the week to work on KDE.
  


• This has been a bug fixing week so I didn't have any time to work on WebKit. Last week I did manage to finish off integrating SVG into WebKit Qt
  I also played a little bit with combining native Qt application rendering from within WebKit. One can embed a whole native Qt window inside an html frame and then control the Qt application from javascript (signals, slots and properties of the window and its children are dynamically exposed). Here's a native Qt application (my transform example) rendered inside the advertisment frame on cnn.com (sorry for the depressing image).
  
  


• For the last two days I've been thinking about adding basic high dynamic range color support to Qt. QColor is already suitable for it and extending it wouldn't be a big issue. QColor's internal representation is a union of unsigned shorts. Basic HDR support requires 64bit, where every channel is a so-called, "half". Half values have 1-bit sign bit, 5 exponent bits, and 10 mantissa bits. It's a format used, among others, in OpenEXR and Cg. In fact OpenEXR project provides a C++ half implementation under a BSD-like license. Along a CIELAB colorspace support that I wanted to add to Qt for a while now, support for HDR would open a slew of new possibilities. On a silly, desktop level - imagine a desktop background the is light proportionally to the sun-light that you'd see for your currently configured timezone. On a more basic level HDR is pretty much essential for any kind of image processing/editing nowadays. I'll most likely write another blog dedicated to color-theory and things I'd like to fix/implemented in Qt that are related to it. Google returns hundred thousands of HDR images so if you'd like to see what people do with it, you won't have to look for long. One of my favorite examples is New York at night by Paulo Barcellos Jr that can be seen on fickr here.


• During lunch at Trolltech there's only one kind of yellow cheese available. Climbing to the highest-levels of good-faith and stupidity I assumed the cheese would absolutely have to be vegetarian (many cheeses is still made with natural rennet, which is taken out of lining of young cattle's stomachs). Yesterday one of the people pointed out that it most likely isn't. Today morning I called Tine, which is the biggest dairy producer in Norway (and one whose cheese Trolltech buys) to ask them about the situation. The result was, that of course, the cheese is not vegetarian. If you're a vegetarian, do not eat cheese or any products containing it while in Norway. For me the bottom line is that for the last few months I've been having non-vegetarian cheese for lunch putting me right now in levels of pissed off/disappointed that I've never been at (you know that silly questions about graphics you wanted to ask me? you might want to wait with that a few days).

Generating SVG's with Qt

I've been working on a lot of things at once over the last week but I just remembered that I never mentioned something that I should have. In the beginning of the Qt 4.3 development phase I added a class called QSvgGenerator to QtSvg. QSvgGenerator is a paint device, meaning you can open QPainter on it and yes, that's right, you'll get an SVG XML stream back that you can save to a file or manipulate.

So now you can redirect your widget/window and have a snapshot of your application dumped to SVG format... Come on that's pretty cool. The generator still needs a lot of work, especially when it comes to complex text. But here's a weird, example. I made Qt render a simple animating bezier curve, then loaded html with Qt documentation and rendered that with opacity of .75 (which is why it looks very blend) and finally redirected to a painter opened on a QSvgGenerator. As a result I get a SVG (viewed, of course with, svgviewer) looking like this:

And the full output is here. So yeah, the concept of being able to load simple html and then save it as SVG is pretty neat.

Also boolean ops on paths are now in Qt. You'll find them in QPainterPath's. Since I already had the code and Andreas asked me for it, I added QPainterPath::intersects(const QPainterPath &other) and QPainterPath::contains(const QPainterPath &other) that can be used to check whether two paths intersect or whether one path fully contains the other. I still want to do some heavy optimisations on it, but I'd like to implement "Snap Rounding of Bezier Curves" instead of trying to find the right shortcuts to use reduced-predicates for the intersection finding algorithm.

The magic of animation

Creating a decent animation, albeit a lot of fun, is quite complex. Besides being able to create the content (and therefore exhibiting basic artistic skills) one has to go through the mundane process of trying to figure out what exactly should be on every frame.

SVG, as a format, is absolutely atrocious at expressing animations. KDE 4 is moving more towards dynamic metaphors, where animations will be used to convey a lot more information. Now the question is how do we let people who inherently have no artistic skills create animations that look and behave correctly. While "look correctly" is pretty simply defined as: it's visually appealing. Then "behave correctly" has a more complex definition that will depend on many factors. We'll define a "well behaved animation" as one which simply doesn't irritate or inhibit the work flow of the users.

While the latter is mostly a challenge for usability engineers I'd like to help people create animations - quickly, with minimal amount of artistic skills and no mundane tasks. So ideally I'd like to be able to say "this is what I'm starting with" and "this is what I'd like to end up with" and see computer animate this process. The technique is known under "shape interpolation", "shape blending", "image morphing" and many other names. There was a lot of research done on this topic but so far no one used this technique to do real-time animations on desktop.

The most popular research papers related to this problem include "As-Rigid-As-Possible Shape Interpolation", "A physically based approach to 2-D shape blending" and finally "As-Rigid-As-Possible Shape Manipulation". If you haven't seen it, you definitely want to look at the last one, there's a short movie on that site showcasing some of the things one can do utilising their algorithm. Some of the examples from the above research follows:

I think that shape interpolation together with as-rigid-as-possible shape manipulation done on QPainterPath's, or whole SVG's in fact, could potentially be the answer to our animation woes. The pain, once again, is that people always work on polygons in those papers so I'd need to spend a bit of time to figure out how to mix it in with paths and inject curves into the algorithm.

Oh, and yesterday was the "World Hug Day" so if you haven't hugged anyone yet, go right ahead.

More on WebKit Qt

Two days ago Lars and I sat down and did the initial implementation of classes which will represent the public API for the Qt/KDE version of WebKit. In KHTML almost everything has been handled by KHTMLPart. We'd like to avoid having one huge object that does everything this time so we went with similar abstraction to the one present in WebCore - meaning Page/Frame distinction. The top level classes are now: QWebPage which represents (surprise - a web page) and QWebFrame which is (another shocking discovery) a frame (remember that a page can have multiple frames).

It's pretty interesting what one can do with it. Me being me (aka. weird) had to try to do something visually funky so I made our test browser have a transparent overlay progress loader that is hidden when nothing is being loaded and made the pages render in grayscale until they are fully loaded, so a loading page looks like:And before someone complains - no this won't be a default and won't even be included in WebKit by default, it's just to give people a rough idea what one can do with this framework (and yes, the filter that I did for it messes up fonts but I just didn't even bother to make it flawless) Or more precisely I wanted to make sure one can do anything. Here's another example with webpage rotated a little bit (this time without grayscale filter on it) :
So, yeah, I think you'll be able to do a lot of new and pretty exciting things with WebKit once KDE 4 hits the streets.

OpenVG

OpenVG is a royalty-free, cross-platform API that provides a low-level hardware acceleration interface for vector graphics libraries such as Flash and SVG. Or at least that's how http://www.khronos.org/openvg defines it.

The problem is that there is no Open Source implementation. And the implementation that claims to be the reference implementation is closed. It'ss a rather silly situation, especially given how important vector graphics is becoming on our desktops/mobile devices nowadays.

After this introduction I'm sure you already know that I went ahead and started an Open Source implementation of OpenVG. I did it on top of QtOpenGL. The main reason for it is that QtOpenGL, quality and speed wise (in Qt 4.3) beats everything out there - every vector graphics framework.

So, there's a git repository at:
http://gitweb.freedesktop.org/?p=users/zack/openvg.git;a=summary
there's an example included, that animates and looks like this:Gears demos are mandatory for any kinds of hardware accelerated API it seems ;) Oh, and due of all the extra features that went in Qt 4.3, Qt snapshots are currently required to compile it.

I'm not sure where this is going to go. It will largely depend on whether there's real interest in it. In general I think it'd be interesting to see KDE have a library that would allow running of OpenVG based content right on KDE. It could be potentially an incredibly powerful feature of KDE as a platform, to support the hardware accelerated vector graphics api.

More boolean ops

I've spent most of the day just playing around with different degenerate cases of boolean operations. I've recorded a short movie showing some live path merging - the one case that should be interesting to application developer is combining two simple rounded rectangles to produce a frame with, what one can easily imagine to be, a highlighted selection on the left. Note that in this case two simple rectangles are enough to produce animation of the selection rectangle on the left going up and down between the "imagined" items :)

And also I forgot a very important thing, which is that Gunnar and Eskil hooked me up with a very cool QtJambi mug. They're doing an amazing job (working on QtJambi, not giving out cups). And here's the mug (and the last I checked, the thing holding it, was me).

Set operations on paths

I'm virtually never excited about the things I have implemented. Partially because I've been working on computer graphics for quite a while and a lot of what I do comes down to doing things that I've done before for different software, or in a different scenario. Writing code while knowing exactly what problems you'll end up facing and how to solve each one of them is very mundane and it makes it hard to get excited about it once you're done. So whenever I have a little bit of time I try to sit down and implement something that hasn't been done before.

So, here's problem #1: one of my embarrassments in the Qt X11 engine is that fact that we don't do anti-aliased clipping there. So if you'll clip out a rounded rectangle in your viewport/widget it will leave out nasty jaggies on the sides. We do it correctly on Windows but not on X.

Main reason for why we haven't done it, is that Qt has this nice concept of combining clip-regions which requires basically a full fledged implementation of boolean set operations on polygons to work correctly.

And here's problem #2: in Qt we have this class called QPainterPath which is a resolution independent, geometrical representation of some rendering. We got reports from people saying it would be really great if people could use boolean set operations with QPainterPath's. And I agree. Set operations on QPainterPath's would be an incredibly powerful feature, especially for animations.

Requirements for a state of the art algorithm here would be: fast, works with all kinds of degeneracies and operates directly on paths without intermediate conversion to polygons. The issue is that none of those things is usually associated with set operations on polygons.

In 1998 a paper entitled "Efficient clipping of arbitrary polygons" was published by Günther Greiner and Kai Hormann. It's a great paper, unfortunately the algorithm doesn't handle degenerate cases at all. This November Dae Hyun Kim and Myoung-Jun Kim published a paper entitled "An Extension of Polygon Clipping To Resolve Degenerate Cases" which addressed the shortcoming of the algorithm described by Greiner. The paper was a little short on details though so I've emailed Dae Hyun Kim who was kind enough to send me the full, original version of their papers. It's a tremendous paper and I'm very grateful to Dae Hyun Kim for sending me it.

Equipped with the paper I've sat down to modify the algorithm to handle paths, which was the last of my requirements for it. The research done in both papers was dedicated to polygons, while I wanted to apply it to paths, where curves are first class citizens. The main difference between polygons and paths. for the purposes of our new algorithm. is that with polygons any two segments can intersect in at most one point, with paths two arbitrary segments can intersect each other in at most 9 points. Furthermore with paths one segment can intersect itself. That plus a few smaller issues are causes of serious woes when working with paths in computational geometry - reason why basically no one does it.

Qt does now though. And yeah, I'm very excited about it. The new algorithm deserves a research paper of its own and if I'll have some spare time I'll definitely write a little bit about it.

I really needed this. I needed to do something that was very unique from a purely mathematical/computer science perspective rather than "oh, hey i fixed this bug/implemented this feature" type of stuff to keep me motivated. A screenshot showing a path (in dark gray fill and black outline) created from an intersection (boolean "And" operation) of two paths (the dashed blue and red objects) is shown below.

The number of effects and things that can be done with boolean set operations on paths is tremendous (not even mentioning the clipping :) ). You can be sure I'll write at least a few demos to show how to do very neat effects by differently combining few very simple paths.

Blend modes and dash offsets

Draft of SVG 1.2 Full adds composition modes to SVG. Besides the typical Porter&Duff operators some additional blend modes are defined in it. Those modes are also very commonly seen in many Adobe Photoshop tutorials and people frequently asked for them in Qt. They include "screen", "multiply", "overlay", "plus", "difference", "hard and soft light" and a few less prominent ones.

We decided to implement them for 4.3. Unfortunately Gunnar said that he simply won't have time to implement them and since I consider them to be very important I sat down after work today and implemted those blend modes in Qt. So if you checkout tonight's snapshot of Qt you'll nice that QPainter has new CompositionMode's. If you looked at some of the Photoshop tutorials on the net on how to do lightning effects, now you'll be able to do them all in Qt which is rather nice.

Once I did that I decided that I'll go ahead and implement the other feature that was missing in Qt and is required by SVG which is: dash offsets for stroke. API wise to QPen and QPainterPathStroker I just added setDashOffset(qreal offset) methods which respectively set the dash offset for the rendering operations in Qt.

To play around with blend modes I took two input images - CD cover image from a band called H2O and a flare generated with Gimp, they looked as follows:

And I wrote a simple application that blends those in real time, and the results of two of the modes look as follows:

One can do a lot of really interesting things with the new blend modes.
Oh, and since I'm weird like that I just went ahead and added support for composition modes (and dash offsets of course) to QtSvg and from what I see QtSvg is currently the only open source SVG renderer supporting the extended blend modes which is a nice bonus :)

Rotations

I didn't have time or motivation lately to talk about the things I am doing but today I wanted to mention something because I think it will make your life a little more exciting (that's assuming you like adding eye-candy to your applications).

Andreas and I were talking yesterday and to my great surprise he mentioned that not everyone can do 3D math in their heads. Furthermore he insisted that some people might have a problem with using QTransform::quadToQuad method which I added as a convenience method to the QTransform class. He went as far as to say that some people might not know how to transform one quad into another in 3D space. Of course in no way do I believe him, but after a short discussions we decided that adding QTransform::rotate(qreal angle, Qt::Axis axis) method to QTransform which lets one rotate an object about any arbitrary axis (X, Y and Z) would help some people.
I'm swamped with work so Samuel took this task of me and implemented that method today.

Anyway, if you quickly need to add some nice effects to your application without feeling like doing any work just use QTransform::rotate and blow people's mind ;) A short Ogg movie showing qtransform2 (you'll of course need the latest Qt snapshot to compile it) in action is here and it looks like this: