Nepomuk Gives Back Your CPU Cycles…

…at least partially. With the introduction of the Data Management Service we got a very powerful way to put your data into Nepomuk: storeResources (thank you Vishesh). I will not go into the details here but the important fact is that the file indexer, the Akonadi indexer, the TV namer, the movie integration, and maybe some more rely on this method.

Thus, it is obvious that improving its performance means improving the performance of the overall system in general.

Now like I said it is a very powerful method. Sadly this results in very complex code that is not easy to wrap your head around. So optimizing it is not trivial. I tried anyway and tackled one of the main parts of it: the resource identification. After playing around a little, trying different ways to break up the main query I got to a point where I am satisfied. And here is why:

Total time spent in the resource identification code for 196692 indexed Akonadi items.

Average time spent in the resource identification code for 196692 indexed Akonadi items.

Finally some nice bar graphs! So what does this tell us? Well, the most important bit is that with my patch we save roughly 3 hours when indexing the 196692 Akonadi items I used for testing. But maybe more importantly: if the identification is faster the whole indexing is smoother and eats less CPU since it is throttled and, thus, has shorter phases of high CPU usage.

I took the Akonadi indexing as an example since the difference here is impressive. The file indexing is also slightly improved but by far not as significant as the Akonadi indexing. I suppose this is due to the very simply identification that is required during file indexing (some very basic nco:Contact merging is all that is required most of the time).

So, this is what you will get with KDE 4.8.2. One of several going away presents…

27 thoughts on “Nepomuk Gives Back Your CPU Cycles…

  1. That is AMAZING! Is there any chance we can see some more of that magic? I’m really worried that we might not now that you are leaving, specially because the code is complicated as you mentioned, and i don’t know if there would be that many other people out there that can optimize it as good as you.
    Anyway, keep up the amazing work. All i can say, i’m sure i would not be the only one to participate again in a fundraising so that you can continue such amazing work (even if its not as a fulltime job)

  2. Less then 20 milliseconds is not bad assuming and average system. What’s next, will you work with the query API issues?

  3. If you want the whole semantic bloat to work for users this is way to go! Not features but optimizations, performance, stability! This is what users love the most. KDE have a lot of great ideas but things like nepomuk/akonadi/kdepim/gui consistency still needs a lot of work and I’m afraid nepomuk without your amazing work will end up bad ;(

    • Fear not; the DMS is something made also by Vishesh Handa, and I won’t worry if Nepomuk lies in his hands now.

      I see this more as the transition from KDE to Qt some of the core KDE3 developers had to make, than as a full departure. So, don’t be so sad, Sebastian… We support you!

  4. Hi Sebastian,

    it turns out, some amazing projects mainly rely on a couple of individuals. I figured that out with Gimp for instance, Dolphin etc. Nepomuk is so important for KDE, but it took *so many* releases to make it really usable for daily usage, I’m quite saddened by your leaving — at the same time, I understand it cannot be a volunteer lifetime commitment :-)

    Just a suggestion : what about doing what Aurélien Gâteau did : he reduced his working time (1 day off per week) and asked users to compensate the salary loss through donations. Sure, he didn’t get that much. But maybe it could work in your case, considering the importance of Nepomuk & co ?

    There are a few OSS pieces of software that are funded by the community, such as Ardour. It wouldn’t be possible in your case, but maybe, 1 day a week would ?

    Anyway, you are entitled to your choice, and it’s very important that projets don’t just rely on a couple of persons. It’s too frequently the case (and is logical : once you get to “master” a code base, you stick to it and are much more efficient).

    Cheers !

  5. This could be better projected as resource reduction by ~ 80% or 5 times performance enhancement. As a user and wellwisher of the project (kde in kubuntu) I would suggest getting publicity for genuine efforts is desirable. The not so pleasnat aspect of advertisement is runninga flashy or catchy line more and more. Instant gratification aside, it may help you to expand adoption as well as contributor base. Both these aspects are needed for good health and longevity of the projects.

  6. Departure? What departure? :)

    On the contrary. I assume that a) Nepomuk will be maintained by Vishesh Handa, and that means Nepomuk will be in good hands, and b) you will work in OpenLink, you’ll learn more about Virtuoso, and you’ll bugfix Virtuoso and optimize it, specially where Nepomuk uses it. So, I don’t see this as a “departure” ;)

    Well, I promised myself to not open again the git tree and clone the kde-runtime repo. I’m breaking this promise, right now.

  7. I echo the comments from others. This is awesome, just like all the work you have put in place for Nepomuk for so long… I also feel it’s a bit of a shame that you have to part ways (somewhat) with your baby now that it’s growing mature, but I guess that’s life.

    All the best in your new endeavor, Seb.

  8. BTW, Sebastian, let me elaborate… I’m an user of both the Facebook Resource and the Google Contacts resource. I have more than 1,000 contacts. If I got you well, then this optimization would mean for me even better results. Am I correct?

  9. Pingback: KDE: Nepomuk più performante entro la release 4.8.2 | oneOpenSource

  10. Pingback: Akonadi, Nepomuk, and A Lot Of CPU | Trueg's Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s