…at least partially. With the introduction of the Data Management Service we got a very powerful way to put your data into Nepomuk: storeResources (thank you Vishesh). I will not go into the details here but the important fact is that the file indexer, the Akonadi indexer, the TV namer, the movie integration, and maybe some more rely on this method.
Thus, it is obvious that improving its performance means improving the performance of the overall system in general.
Now like I said it is a very powerful method. Sadly this results in very complex code that is not easy to wrap your head around. So optimizing it is not trivial. I tried anyway and tackled one of the main parts of it: the resource identification. After playing around a little, trying different ways to break up the main query I got to a point where I am satisfied. And here is why:
Total time spent in the resource identification code for 196692 indexed Akonadi items.
Average time spent in the resource identification code for 196692 indexed Akonadi items.
Finally some nice bar graphs! So what does this tell us? Well, the most important bit is that with my patch we save roughly 3 hours when indexing the 196692 Akonadi items I used for testing. But maybe more importantly: if the identification is faster the whole indexing is smoother and eats less CPU since it is throttled and, thus, has shorter phases of high CPU usage.
I took the Akonadi indexing as an example since the difference here is impressive. The file indexing is also slightly improved but by far not as significant as the Akonadi indexing. I suppose this is due to the very simply identification that is required during file indexing (some very basic nco:Contact merging is all that is required most of the time).
So, this is what you will get with KDE 4.8.2. One of several going away presents…
Trueg's Blog 
That is AMAZING! Is there any chance we can see some more of that magic? I’m really worried that we might not now that you are leaving, specially because the code is complicated as you mentioned, and i don’t know if there would be that many other people out there that can optimize it as good as you.
Anyway, keep up the amazing work. All i can say, i’m sure i would not be the only one to participate again in a fundraising so that you can continue such amazing work (even if its not as a fulltime job)
Less then 20 milliseconds is not bad assuming and average system. What’s next, will you work with the query API issues?
Yes! I drafted the changes I have to make in order to include your ideas into the query API – let’s hope it works.
The first graph shows “seconds” at left. Or it is “milli-seconds” ?
No, that is correct, I could have used minutes…. but the point is the difference…
If you want the whole semantic bloat to work for users this is way to go! Not features but optimizations, performance, stability! This is what users love the most. KDE have a lot of great ideas but things like nepomuk/akonadi/kdepim/gui consistency still needs a lot of work and I’m afraid nepomuk without your amazing work will end up bad ;(
Fear not; the DMS is something made also by Vishesh Handa, and I won’t worry if Nepomuk lies in his hands now.
I see this more as the transition from KDE to Qt some of the core KDE3 developers had to make, than as a full departure. So, don’t be so sad, Sebastian… We support you!
Hi Sebastian,
it turns out, some amazing projects mainly rely on a couple of individuals. I figured that out with Gimp for instance, Dolphin etc. Nepomuk is so important for KDE, but it took *so many* releases to make it really usable for daily usage, I’m quite saddened by your leaving — at the same time, I understand it cannot be a volunteer lifetime commitment :-)
Just a suggestion : what about doing what Aurélien Gâteau did : he reduced his working time (1 day off per week) and asked users to compensate the salary loss through donations. Sure, he didn’t get that much. But maybe it could work in your case, considering the importance of Nepomuk & co ?
There are a few OSS pieces of software that are funded by the community, such as Ardour. It wouldn’t be possible in your case, but maybe, 1 day a week would ?
Anyway, you are entitled to your choice, and it’s very important that projets don’t just rely on a couple of persons. It’s too frequently the case (and is logical : once you get to “master” a code base, you stick to it and are much more efficient).
Cheers !
Thanks for all your work!
I would participate again in a fundraising too :)
This could be better projected as resource reduction by ~ 80% or 5 times performance enhancement. As a user and wellwisher of the project (kde in kubuntu) I would suggest getting publicity for genuine efforts is desirable. The not so pleasnat aspect of advertisement is runninga flashy or catchy line more and more. Instant gratification aside, it may help you to expand adoption as well as contributor base. Both these aspects are needed for good health and longevity of the projects.
What a lovely departure gift it is. Thanks and good luck at your new gig :)
As evidenced by the comments, the KDE community recognizes and loves its contributors.
Thank you for your fabulous work, Sebastian.
Departure? What departure? :)
On the contrary. I assume that a) Nepomuk will be maintained by Vishesh Handa, and that means Nepomuk will be in good hands, and b) you will work in OpenLink, you’ll learn more about Virtuoso, and you’ll bugfix Virtuoso and optimize it, specially where Nepomuk uses it. So, I don’t see this as a “departure” ;)
Well, I promised myself to not open again the git tree and clone the kde-runtime repo. I’m breaking this promise, right now.
Thank you Sebastian for your ongoing hard work!
Good job guy!
Thank you for your amazing work!
I echo the comments from others. This is awesome, just like all the work you have put in place for Nepomuk for so long… I also feel it’s a bit of a shame that you have to part ways (somewhat) with your baby now that it’s growing mature, but I guess that’s life.
All the best in your new endeavor, Seb.
Tried your patch, PIM indexing (with idle detection disabled) seems to be disk bound on my machine instead of CPU bound without it. More scientific testing to follow.
Where are those patches? I’d love to test them :)
Thanks in advance!
I attached the patch to https://bugs.kde.org/show_bug.cgi?id=289932. I will push it as soon as I have some feedback.
As you wish, master ;-)
With this patch, my computer has finally come a a rest, before, it was always stuck in a “select disting ?r count(?r) …” query. isql-vt status() showed these queries running for hours, sometimes even several of them.
So, for me, this fix is working great, although I wonder if it fixes the root cause or just makes it just very unlikely to happen (some kind of a race condition maybe …)
Anyway, thanks for the fix and Nepomuk in general, and good luck at openlink.
BTW: openSUSE KDE 4.8.1 packages with patch applied are available from my home repository: http://download.opensuse.org/repositories/home:/StefanBruens:/branches:/KDE:/Release:/48/
thank you Sebastian. :)
BTW, Sebastian, let me elaborate… I’m an user of both the Facebook Resource and the Google Contacts resource. I have more than 1,000 contacts. If I got you well, then this optimization would mean for me even better results. Am I correct?
Well, yes. Akonadi indexing should improve all around.
Pingback: KDE: Nepomuk più performante entro la release 4.8.2 | oneOpenSource
Pingback: Akonadi, Nepomuk, and A Lot Of CPU | Trueg's Blog
Thank you so much. Great work!
cheer! many thanks!