Intermission: Why I Needed To Fork QProcess For K3b


Yes, I know. This is supposed to be the Nepomuk blog. And yes, I have a blog over at k3b.org. But this one is so much nicer. Also: the blog’s name is “trueg”, not “nepomuk”. And anyway: I write what I damn well please! ;) … OK, enough mindless exaggeration, I doubt that anyone really has a problem with me blogging about K3b a bit. So here goes.

K3b for KDE 4. Where is it? Well, it is on its way, as it has been for the last few years. It started with Laurent porting the core of the build system and running all those nice KDE3->KDE4 scripts. He then invited me to Paris for a prelonged K3b-porting-session. We worked good but K3b was far from fully ported. Then time passed again and I only worked on Nepomuk. Then there was the week without Internet. Again some K3b porting. But still no end in sight. Finally Mandriva really wanted K3b done for 2009 Spring and put two developers on it (part time and under my supervision). This was a nice but also wise move. Everybody who ever worked on K3b or sent patches knows how protective I am of its code. So by letting two developers work on K3b Mandriva kind of forced me to work on it, too. Not the bad kind of being forced, not at all. I am finally enjoing hacking on K3b again.

Right, nice story, but what does that actually mean for the KDE 4 port? When will it be done? Well, if all goes well K3b 2 will be released with Mandriva 2009 Spring. But that is not certain. K3b is big and the port from KDE3 to KDE4 is not trivial at all. This brings us to the actual topic of this blog entry: QProcess and why I had to fork it.

A little background: In KDE 3 KProcess was non-cross-platform and used pipes to communicate. These pipes were even accessible through the API. This allowed me to implement K3bProcess which featured raw modes, i.e. blocking reading and writing. This was used all over libk3b to pipe data from one process to the other or between processes and some image creating thread. The advantage is obvious: the highest data-throughput, no handing of signals and unnecessary async behavior. Life was good.

Then came KDE 4 and with it KProcess became a convinience wrapper around the cross-platform QProcess. Sounds like a good concept and actually is one, too. However, the internals of QProcess are hidden and its API is way too high level for K3b’s needs. Why? Well, In K3b I used to pump data between processes and threads in a separate thread. QProcess has some methods for usage without an event loop which I thought would solve my problems. I thought I could just block using QProcess::waitForReadyRead and QProcess::waitForBytesWritten in my pumping thread. However, this is not possible. Simply because the processes are started in the GUI thread and thus, they have an event loop. Then QProcess::waitForReadyRead clashes badly with the internal QSocketNotifiers. OK, so that was not an option.

My next step was to change the multi-threaded design of K3b to an async one. I did not really like the idea since that would mean that data pumping (and thus the stream of data to cdrecord and others) would have to share event cycles with GUI updates and user interaction. Anyway I went for it, ported it to use signals and slots, only act on QProcess::readyRead and QProcess::bytesWritten. Looked OK. Well, it was not. QProcess is fully buffered, double-buffered even since QIODevice already has its own buffer. This makes sense for its async design (although I fail to understand the need for two buffers). But here it really messed things up. The read buffer has no maximum and can grow to an arbitrary size. That is exactly what happended. Mkisofs was happily creating a 4 GB ISO image while cdrecord was still initializing things. In the end, K3b used 4 GB of memory. Unacceptable. And yet another useless coding round.

So finally I realized that I really needed a fork of QProcess (and with it KProcess since that is what K3bProcess is based upon). And that is what I did. K3b now has classes with weird names like K3bQProcess and K3bKProcess. I did not change much in QProcess though. After a detour implementing an Unbuffered mode (I actually posted the crude patch as a Qt task) I ended up introducing flags RawStdin and RawStdout which do exactly what I want: reading and writing becomes blocking and full sync. Now K3b can pump its data as before and it works swimmingly. All jobs use QIODevice for communication, all of those devices need to be unbuffered and sync.

This whole development took forever. But in the end I think it is clean and fast again. And once QtSoftware will include the unbuffered mode in Qt, all communication becomes cross-platform, too (I did only patch the unix portion).

Well, this is it for now. Just a little telling of my adventures with QProcess and forking Qt classes.

Sadly I just read that Qt will not implement a blocking API. But the unbuffered mode does only make sense for me if it is blocking. Otherwise I am in multi-threading hell again, with QSocketNotifiers and waiting methods killing each other! Thus, I will have to maintain the fork :( Too bad, it could have been so easy…

38 thoughts on “Intermission: Why I Needed To Fork QProcess For K3b

  1. Great work! Was there any reason given why this feature was rejected by Qt? I looked at the task page and there was none. Did they not like it for some principled reason, or was it just that they did not deem it important enough to support?

  2. Thank you for the wonderfull K3b, Sebastian. It is simply the nicest application for burning CDs I’ve ever used. And thanks to Mandriva for putting people and resources behind the port to KDE 4.

  3. K3b has spoiled me to such an extent that I can’t bring myself to use anything else for CD/DVD authoring. It just works great every time. I’ve been looking forward to a KDE 4 port for sooo long. Thanks for putting time and effort into such a great app.

  4. I’m trying to understand the problems you ran into; but there is an inconsistency in your blog; you talk about threads and QProcesses as if they are releated.
    But a QProcess creates a new task, not a thread.
    If you indeed use threads and not tasks then you should be able to do the ‘pumping’ in old fashioned shared memory (avoids loads of copying too) and QMutex or its more convenient QWaitCondition are there to help out.

    Anyway, if your solution works for you, go for it :) Forking for your own usage is ok, thats why its open source.

    • Well, I use threads and processes. apps like mkisofs and cdrecord are run through QProcess. audio file conversion and track generation as well as iso image extraction are done in separate threads. Now the data needs to be pumped from a thread to a process or from one process to the other. I do that in yet another thread. Why?
      Well, I cannot use QProcess::setStandardOutputProcess since I need the checksum of the data, too. I don’t want to do async stuff in the GUi thread to keep the data stream as stable as possible (+ the 4gb buffer problem I explained).
      I hope this clears is up a bit.
      If Qt 4.6 does contain an unbuffered but non-blocking mode I could do the async GUI thread thing I tried before. But that is undesirable and would entail more changes to K3b’s job classes (cannot use pumping threads anymore, all needs to be able to wait until more data can be read/written).

  5. This is great news! As others have said, K3B is simply the best burning application I’ve ever used!

    I’m sure it would win quite a few windows people over if it is made cross platform.

  6. Keep up the good work. K3b is one of, if not the best open source apps I’ve used. Actually it is one of the best apps full stop.

    Will it be renamed K4b for KDE4?

  7. Hopefully I’ll be able to burn dvd’s for the Wii console with it, because all other linux cd burners seem to stupid while trying to be smart and phail burning correctly an encrypted iso image most of the times.
    Not even wine + ImgBurn cut it for some reason…
    Someone said the only way is to use emulation + windows…. lame..

    • burning DVD for the wii always worked for me. I never had to setup speed or thing like that you can read all other the places.
      And even if I’m using kde4 k3b(using kdelibs3) work great @home.

      • Funny how all other people are saying the opposite, including myself. It works right with some of the dvd’s only some of the times.

        So while i managed to burn certain 2 or 3… i ruined about 15-20.

    • because K3b links to QtCore and kdecore. The symbols would then clash. Also I reuse all enums from the original QProcess and KProcess to make future conversion back to the originals easier.

    • No, there is no need to boo them at all. Qt’s design aims to be async. There is no room for a blocking method. And in a way K3b is a special case. I respect their decision although I would have liked another one – some advanced QProcess mode.

      • Well, I think who are they, to take that choice away? If you develop a library/API, always leave options. You never know who will need them. And *then* give good defaults (which would be the async mode).

    • I suspect the Qt guys will figure this out eventually. It’s not a good story to sell a development framework as, ‘great for many types of applications, unless you need to do certain types of I/O’. Because developers never know where their apps are going to go in the future.

      I mean, sure, you can write your own I/O classes and port them to every OS you want to target, but that’s a bit of a drag once you’ve decided to use a cross-platform framework in the first place.

      Perhaps somebody will figure out a design that meets all needs.

  8. SebT: Since it is a fork of QProcess it is not possible to add it to KProcess unless you want to convert KProcess into a fork of QProcess. I don’t think that is desirable for the kdelibs.

  9. How about calling them QSerialProcess and KSerialProcess (in case that’s not already used elsewhere)? This way you don’t have nonsensical sounding names and you could possibly move both combined as KSerialProcess into KDE should Qt Software not end up with a better solution by Qt 4.6.

  10. Hello!
    I found a nasty bug in current K3B version, but unfortunately I didn’t find any bug tracking tool for K3B (tried K3B site and bugs.kde.org). Anyway the bug occurs when I try to record mixed bin/cue image. The data is fine, but audio tracks are completely messed up. All I get is awful noise. I tried the same image in trial version of Nero Linux and everything worked fine, so I suppose it’s K3B fault.
    Thanks for great app and keep up the good work!

  11. Very sad that the Qt guys don’t want to give developers the possibility to write a CD/DVD/whatelse burning application easily. I mean, if the concept they have doesn’t cut it for some applications, they should offer some other method for these applications.

  12. Hi Sebastian,
    I’ve always wanted to leave you some appreciation feedback, but for some reason had never found your web page/blog. Anyway like so many others of said K3B is truly a killer app for open source, in fact it has surpassed Nero since Nero became bloatware. On windows people have nothing that is free that comes even close to K3B (ImgBurn is close but not as flexible etc ec), and poor gnome users get stuck with a product about half as useful as K3B by default…
    Anyway I was worried for a bit that it might have been abandoned by I’m soooo glad your working hard on it. Please take these comments as encouragement to keep developing, I know it can often be a thankless task but so many of us would be screwed without K3B!

    One thing I’ve always wanted to ask was….a few years ago when I first used k3b I was recording conference talks on my pc and then burning them to CD available for sale at the end of the session. So I had 3 burners in my pc, and all I did was open K3B 3 times and choose a different burner each time and I had all 3 going at once. Somewhere down the line K3B stopped me from loading more than one program at a time, so I got around it by running it from the terminal as another user =) Anyway that would be the only thing nero CAN do the K3B can’t…anyway no huge biggy but would be really nice if it was a feature you could select ‘burn to multiple drives’.

    Thanks again, you and Paul from Ardour are my favourite developers in the whole wide world

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s