One Bug has been driving people crazy. This is more than understandable seeing that the bug was an endless high CPU usage by Virtuoso, the database used in Nepomuk. Kolab Systems, the Free Software groupware company behind Kolab, a driving force behind Akonadi, sponsored me to look into that issue.
Finding the issue turned out to be a bit harder than I thought, coming up with a fix even more so. In the process I ended up improving the Akonadi Nepomuk Email indexer/feeder in several places. This, however useful and worthwhile, turned out to be unrelated to the high CPU usage. Virtuoso was not to blame either. In the end the real issue was solved by a little SPARQL query optimization.
Application developers against Akonadi and Nepomuk might want to keep that in mind: The way you build your queries will have dramatic impact on the performance of the whole system. So this is also where opimizations are likely to have a lot of impact in case people want to help improve things further. Discussing query design with the Nepomuk team or on the Virtuoso mailing list can go a long way here.
So thanks to the support from Kolab Systems, Virtuoso is no longer chewing so much CPU, and Akonadi Email indexing will work a lot smoother with KDE 4.8.2.
Trueg's Blog 
Analog to other dbs is there the possibility in virtuoso to log slow running queries or log queries with execution time > 10 ms or something like this?
I know I can see the currently running queries, but to really identify problematic queries some logging would be better.
That would be really useful i think, specially if the way queries are written can influence the runtime so much. That way it would be easier to detect those offending queries.
What you can do is to create a ~/.odbc.ini file with the following section:
This will log all queries into the given logfile. However, it does not include the execution times and the log will grow very big. But it is a start.
Maybe I can come up with a debugging mode which does exactly what you are asking for.
Thanks! Maybe someday this will actually be usable ;) this kind of posts just make me sure that this day is approaching faster then ever before
Yeah maybe some day KMail, Akonadi and such will be usable :) We look forward for it…
Am glad to see that this is speeding up; after recent troubles w/KMail I resorted to cleaning up and then re-importing all of my mail back into KMail. It took FOUR DAYS for Akonadi/Nepomuk to calm down and finish importing and indexing everything; my Core2 Quad here indexed them at the blistering rate of *one* message per second. :o
Anything that can make Akonadi (and thus KMail) more performant is a good thing IMHO; after the switch to akonadi as the back-end I just about gave up on KMail after having used it for many years (so many, that I can’t even remember what I used before it) (so it wouldn’t surprise me to see if it was more than 10yrs).
Have to say, I’m *really* happy to see performance improvements here.
After major problems w/KMail after the switch to akonadi, I switched off of kmail and tried other packages. Unfortunately, found that they either couldn’t import maildir formatted messages (Thunderbird), or found that although they started fine that they eventually just crapped out and stopped working (Evolution). I was _shocked_ to see how poor the state of affairs is w.r.t. e-mail apps these days; I’ve used kmail for so long (probably 10+ yrs) that I’d just taken for granted what it could do and just expected that other apps at least had the _basic_ functionality that it provided. Needless to say, I was wrong.
But, after seeing that nothing else out there could handle my mail (~193k messages, 4.7GB, across 2500 folders), I came back to KMail. Well…. my Core2 Quad imported them at the blistering rate of…. *ONE* message per second. Yes, one. And when finished, they were all marked as “unread”. So, had to mark them all read, which then went at the rate of…. *one* message per second.
So…. five days after I started re-importing back into kmail (4.8.1), its finally finished.
So yeah… anything that can improve the performance here is appreciated.
Thank you very much for this!
Performance is actually my only real gripe with akonadi, so I cherish any improvement there.
Oh, CPU usage is improved…, really?
I suggest you do your research better then. In KDE 4.9.2., Akonadi is now able to set a dual core processor to 100% for both cores and is trying to consume 5 GB memory in total.
I thought Unity was bad but KDE is 3 times worse.
I’ve installed it and removed it within 5 minutes. Absolute garbage.
Try claws-mail. It does not have maildir but it has mh, which is as good. There are tools to convert maildir to mh for you. Even in 4.9.5 Kmail/Akonadi is unusable. This centralized crap is too complex in may opinion to be stabilized. One remark by Sebastian that the performance of Nepomuk depends on the quality of the queries written by application creators made me certain that I wil not return to kdepim until akonadi will be completely removed. I don’t think that big is beautiful and definitely not efficient. I use claws mail and “notmuch with emacs”. Especially the latest is blazing fast. I am not looking back, but I am sorry, Kmail used to be a great tool. Not anymore!
“So thanks to the support from Kolab Systems, Virtuoso is no longer chewing so much CPU, and Akonadi Email indexing will work a lot smoother with KDE 4.8.2.”
It’s KDE 4.9.2 now and nothing has changed.
I’m using KDE 4.10.3 via Ubuntu/Mint Olivia and I have turned off all indexing features on my laptop to prevent it slurping the battery. Despite this, Akonadi insists on indexing my 6G of IMAP folders *every* time I start kmail. Needless to say this destroys my battery and my wireless data plan, so I can’t use KMail on my laptop when I’m mobile. Not that KMail is the best or most stable mail client I could be using to begin with, but I got comfortable with it using KDE 3.5.x and now that KMail has started to approach the features/stability of KDE 3.5’s version it would be nice to use it again, if only I could make it behave.
Any ideas how I can shut off the Akonadi indexing?
The real solution, for me, is to set the directory for the Local Folders agent to /dev/null
Shell script to run at kde start, containing:
cpulimit -b -e virtuoso-t -l 10
– it throttles virtuoso-t nicely, as a workaround