Intermission: Why I Needed To Fork QProcess For K3b

Yes, I know. This is supposed to be the Nepomuk blog. And yes, I have a blog over at k3b.org. But this one is so much nicer. Also: the blog’s name is “trueg”, not “nepomuk”. And anyway: I write what I damn well please! ;) … OK, enough mindless exaggeration, I doubt that anyone really has a problem with me blogging about K3b a bit. So here goes.

K3b for KDE 4. Where is it? Well, it is on its way, as it has been for the last few years. It started with Laurent porting the core of the build system and running all those nice KDE3->KDE4 scripts. He then invited me to Paris for a prelonged K3b-porting-session. We worked good but K3b was far from fully ported. Then time passed again and I only worked on Nepomuk. Then there was the week without Internet. Again some K3b porting. But still no end in sight. Finally Mandriva really wanted K3b done for 2009 Spring and put two developers on it (part time and under my supervision). This was a nice but also wise move. Everybody who ever worked on K3b or sent patches knows how protective I am of its code. So by letting two developers work on K3b Mandriva kind of forced me to work on it, too. Not the bad kind of being forced, not at all. I am finally enjoing hacking on K3b again.

Right, nice story, but what does that actually mean for the KDE 4 port? When will it be done? Well, if all goes well K3b 2 will be released with Mandriva 2009 Spring. But that is not certain. K3b is big and the port from KDE3 to KDE4 is not trivial at all. This brings us to the actual topic of this blog entry: QProcess and why I had to fork it.

A little background: In KDE 3 KProcess was non-cross-platform and used pipes to communicate. These pipes were even accessible through the API. This allowed me to implement K3bProcess which featured raw modes, i.e. blocking reading and writing. This was used all over libk3b to pipe data from one process to the other or between processes and some image creating thread. The advantage is obvious: the highest data-throughput, no handing of signals and unnecessary async behavior. Life was good.

Then came KDE 4 and with it KProcess became a convinience wrapper around the cross-platform QProcess. Sounds like a good concept and actually is one, too. However, the internals of QProcess are hidden and its API is way too high level for K3b’s needs. Why? Well, In K3b I used to pump data between processes and threads in a separate thread. QProcess has some methods for usage without an event loop which I thought would solve my problems. I thought I could just block using QProcess::waitForReadyRead and QProcess::waitForBytesWritten in my pumping thread. However, this is not possible. Simply because the processes are started in the GUI thread and thus, they have an event loop. Then QProcess::waitForReadyRead clashes badly with the internal QSocketNotifiers. OK, so that was not an option.

My next step was to change the multi-threaded design of K3b to an async one. I did not really like the idea since that would mean that data pumping (and thus the stream of data to cdrecord and others) would have to share event cycles with GUI updates and user interaction. Anyway I went for it, ported it to use signals and slots, only act on QProcess::readyRead and QProcess::bytesWritten. Looked OK. Well, it was not. QProcess is fully buffered, double-buffered even since QIODevice already has its own buffer. This makes sense for its async design (although I fail to understand the need for two buffers). But here it really messed things up. The read buffer has no maximum and can grow to an arbitrary size. That is exactly what happended. Mkisofs was happily creating a 4 GB ISO image while cdrecord was still initializing things. In the end, K3b used 4 GB of memory. Unacceptable. And yet another useless coding round.

So finally I realized that I really needed a fork of QProcess (and with it KProcess since that is what K3bProcess is based upon). And that is what I did. K3b now has classes with weird names like K3bQProcess and K3bKProcess. I did not change much in QProcess though. After a detour implementing an Unbuffered mode (I actually posted the crude patch as a Qt task) I ended up introducing flags RawStdin and RawStdout which do exactly what I want: reading and writing becomes blocking and full sync. Now K3b can pump its data as before and it works swimmingly. All jobs use QIODevice for communication, all of those devices need to be unbuffered and sync.

This whole development took forever. But in the end I think it is clean and fast again. And once QtSoftware will include the unbuffered mode in Qt, all communication becomes cross-platform, too (I did only patch the unix portion).

Well, this is it for now. Just a little telling of my adventures with QProcess and forking Qt classes.

Sadly I just read that Qt will not implement a blocking API. But the unbuffered mode does only make sense for me if it is blocking. Otherwise I am in multi-threading hell again, with QSocketNotifiers and waiting methods killing each other! Thus, I will have to maintain the fork :( Too bad, it could have been so easy…