… but it would not work. Not on my main system, not even in the VirtualBox which I only set up for that purpose. That is frustrating. There is a set of crash reports in bugs.kde.org for which I attached patches. But I cannot test them. Nepomuk simply does not want to crash for me!
Thus, I need your help. Anyone who manages to crash Nepomuk and is able to test a patch please have a look at those patches, bug 257176 even suggests that it might be prudent to apply more than one at the same time.
Anyway, I am now down to 111 open crash reports for which I did not request additional info yet (this is either: “does it still crash in 4.7.1” or “please test this patch”). That means a total of 140 crash reports have been closed since 17.09.2011. Most of them are in fact duplicates but that is tedious work also. ;)
Trueg's Blog 
Sorry,
I can’t help you because Nepomuk simply works fine with me.
…I though it was important to say that, too. :)
What happened to the valgrind guide? As far as I understood Michael, he wrote something and sent it to you for review. Looking at most crashes, they need a valgrind log attached. I was hoping to add a link to this guide to each memory corruption crash we got (that’s why I did not close any of them).
I will have a look for it again. But actually so far I do not need a valgrind check. I need patch testing. :)
Have you actually tried valgrind? I.e. is it not reporting any issues with nepomuk? If so, good work. If not, you should do that asap. Valgrind is good at finding bugs that might not always lead to crashes, or manifest in “heisenbugs”.
If you want to boast about stability, being able to say: no warnings/errors reported by memcheck/helgrind is a really good argument.
I will second this. Valgrind is amazing. If you have not used it on Nepomuk yet, you will almost surely discover quite a number of bugs—almost any sophisticated piece of software that has not yet been checked by Valgrind has them. Their effect might not always result in an obvious, immediate problem on any given system, but on someone else’s system they may have a more dramatic effect.
Of course, it only reports errors for the code paths that are tested.
Thanks for the awesome crash testing work because I remember when Nepomuk was first introduced it would crash every five seconds.
I have nepoumkservicestub crashing everytime I attempt to start strigi, but it is neither of the bugs listed in your query.
Hi.
Please dont forget that polishing isnt only about fixing crashes. Its also about polishing existing features.
For example, this is what i get when i search about a contact in my krunner, which is basically useless:
Oh, i forgot to say:
Keep rocking!
Actually, last week for the first time I used krunner to search in my emails using nepomuk. And it worked perfectly fine. And i thought ‘shape of things to come’.
First the crashes then the rest. :)
Hm, I think on my machine there were two cases when Nepomuk crashed.
When shutting down the machine (not reproducibly, but frequently) and stupid Dr. Konqi of course delayed/prevented shut down -.-
When having a file that rapidly changed, i. e. downloading a Linux ISO or copying many files.
I will have to test if it still crashes then. :)
Copying many small files from some place to another or downloading a big file did crash nepomuk for me on kde 4.7.1 (on Arch Linux), too. I also encountered that Nepomuk crashes sometimes when I shut down my machine. Unfortunatly, I have no access to a kde machine atm.
I’ve seen the shutdown crash on Arch/4.7.1. University is madness right now, but I’ll try to follow your work and test if needed. :)
working perfectly fine here too, sorry… ;)
The collective blogs of you, Martin and Aaron have made me feel guilty enough to start an attempt at setting up a KDE development environment, even though my main languague is really Fortran ;) So, although I have no clue what I am going to do exactly (bug triaging, testing, fixing my own annoyances are likely candidates), or even have to see how much time I can actually spend on it, I will try and apply your patches as soon as I manage to crash Nepomuk. Not that my operational box (F15 with KDE 4.6) ever manages to crash Nepomuk right now….
thanks. :)
I haven’t relevant nepomuk crashes in my computers for a long time but I take care to index only files that previously I clean and normalize and don’t use Akonadi so I think I’m not very useful.
It works fine for me, no crashes on my system for a while. And I am even using good ol’ 4.6, so I guess the packages are quite outdated. Maybe you could try to contact the people who experience crashes ,and try something like a remote bug testing session.
Your work is terrific, keep going on.
What a surprise. No bug, no matter how small, is stupid enough to raise its head in front of an angry developer, waiting eagerly to smash it with a large mallet in his hands.
Something I noticed, while not strictly being a bug, is the bad behavior of the digikamnepomukservice. It’s normally not running until I start digiKam (with Nepomuk integration activated), then it will go on like forever, even long after digiKam has exited. Along with nepomukstorage, both have now clocked more than 4000 (!) minutes of cpu time in about a week and are still going at at. They don’t create that much of a cpu load and memory consumption is within limits, but I get a lot of hw waitstates due to disk activity. (virtuoso-t has also accumulated a nice amount too, but not _that_ much). Also, digiKam gets really slow when tagging a large number of photos while Nepomuk is active. But it works fine and the tags are available to Dolphin & Co.
One crash happened to me yesterday with 4.7.1 build. I am pretty sure this was a ‘most unfortunate event’: I deleted a large folder (kdesrc, as it happens…) and Nepomuk crashed. I can not be certain, but I think Nepomuk was indexing that folder when it was deleted and that is why it crashed. I’m not sure how I could set up this scenario in a reproducible fashion to prove it though…
Nonetheless good information that may allow me to reproduce.
thank you for such a great effort! Really appreciate the work you’re doing on Nepomuk. I would make a donation if I weren’t such a poor student.
A problem that really annoys me is that sometimes virtuoso_t/nepomukindexer CPU usage goes 100% without any particular reason after some time, which makes me have to kill the process in order to be able to keep using the system. And I don’t know how to track that.
which one is it? both of them?
Usually it’s virtuoso-t, but the filewatch service and the backupsync service follow suit (I’ve seen this same behavior every now adn then). They need to be killed with SIGKILL or they willl not quit.
I don’t know if it’s related but I had problems with CPU usage and crashes while Nepomuk tried to analyze a file which I was downloading using Chrome.
I guess I should add the .crdownload extension to Nepomuk ignore list.
This is something many users reported now and something I need to test and fix.
Nepomuk can’t detect if a file is growing constantly. I already blacklisted .crdownload files, but, if I try to download something with Opera, Virtuoso will reindex constantly the file.
This is a side effect of well done file notifications, so, we must enable workarounds for those apps that make ever-growing files.
Hi there,
i had never a real crash with nepomuk but most of my files are saved on external hard drives.
I use tagging a lot and most of the time the tags stay on the files but every now and then i loose a huge chunk of my tag-file relations. That happens four times now, what is not acceptable anymore. I don’t blame anyone, i even don’t know myself whats happening but i cannot use nepomuk until it is 100% (99%) reliable.
Beside that, the dolphin search (the most important part for me) never worked for me. The last iteration in the GUI is much better but the results are always incomplete. I cannot even tell which files are missing. The results seems totally random to me but always incomplete.
Last but not least the nepomuk backup system seems to be not ready for public. The manual backup seems to work ok but when you try to reimport a backup later nepomuk crashes a lot. Last time i tried i was not able to reimport an old backup.
The automatic (once per day) backup is also pretty useless to me cause it only saves a fixed number of backups. How is this usefull to anyone? I would suggest to set a fixed number (for example 20) and when 20 is reached delete the oldest one after tell the user about it.
I hope this post is useful in any way.
Keep up your hard work, i’m pretty sure your work is the foundation of the future, semantic desktop.
— Andy
Do you have multicore processor? Maybe some of crashes are related to thread synchronization?In that case not all problems could be replicated (or difficult to replicate) on single core processor.
However I don’t remember any Nepomuk crash last time. However I don’t use it very often because of performance issues.
I have 4 cores here…. I even simulated 2 in VirtualBox for exactly that reason. Maybe I need to do the opposite. :P
Doing full builds on a large java codebase while using Intellij IDEA is a sure way to do it for me (Chakra 9.2011), the build does a LOT of delete-create-write files, and Intellij does a lot of updates in its indexes to match a few copied resources. I tried to exclude a few folders to see if the crash would go away, but it turned into a time-sink and I just disabled the whole indexing service.
I can always let it crash if i do a compile job in my home folder that is also being indexed. Nepomuk, strigi or whatever indexes it keeps crashing.
Wow, that are a lot of fixed bugs. Cool!
Did you apply at the company that develops Virtuoso?
Since you are the reason Virtuoso is installed on every KDE Plasma Desktop, they could use it as promotion.
Personally I have still the major problem with soprano/nepomuk warnings spamming my ~/.xsession-errors and hence leading eventually to a disk-full error and probably data loss:
milian@thor:~/.kde/env$ du -hs ~/.xsession-errors 15G /home/milian/.xsession-errors milian@thor:~/.kde/env$ head -n 10000 ~/.xsession-errors | sort | uniq -c | sort -nr | head 5580 "/home/development/projects/compiled/kde4/bin/akonadi_nepomuk_email_feeder(1668)" Soprano: "Unsupported operation (2)": "Invalid model" 3220 "/home/development/projects/compiled/kde4/bin/akonadi_nepomuk_email_feeder(1668)" Soprano: "org.freedesktop.DBus.Error.UnknownObject - No such object path '/org/soprano/Server'" 43 "/home/development/projects/compiled/kde4/bin/akonadi_nepomuk_email_feeder(1668)" Soprano: "Invalid iterator." 32 X Error: BadDrawable (invalid Pixmap or Window parameter) 9 27 *: nothing appropriate. 24 "/home/development/projects/compiled/kde4/bin/kmail(1851)" Soprano: "Unsupported operation (2)": "Invalid model" 23 Database "akonadi" opened using driver "QMYSQL" 18 X Error: BadWindow (invalid Window parameter) 3 18 "/home/development/projects/compiled/kde4/bin/kmail(1851)" Soprano: "org.freedesktop.DBus.Error.UnknownObject - No such object path '/org/soprano/Server'" 16 X Error: BadValue (integer parameter out of range for operation) 2The file grows in the order of megabytes per second. You said there is a KDE bug for it – I cannot find it? Could you tell me which one it is, so I can follow it?
Thanks.
This is it: https://bugs.kde.org/show_bug.cgi?id=269442. The problem is two-fold:
1. For some reason Nepomuk is not running anymore and the index feeder still tries to do something (normally it should instead show the nice “Nepomuk has been disabled” dialog)
2. Soprano is Qt-only, thus does not use kDebug/kdebugdialog. Not sure how to solve that yet.
Today I logged in and Nepomukstubservice went though the roof. Killing Akonadi settled down the disk-activity and CPU-usage. No idea what happened but will erase all config and datafiles and find out if that solves the problem.
The problem Emil Sedgh describes is present on my 4.7.1 too. Sometimes Krunner only returns ‘Recources’, sometimes it returns those cryptic names (and for some reason it doesn’t find filenames).
I can’t remember the last time Nepomuk crashed so I am pretty sure the last 2 days it did not*
Keep up the good work! Will donate some more next month.
*alcohol is bad ;-)
I know this is a out-of-track comment, but I really think the indexing should not happen in files that were modified less than 2 seconds ago (and probably directories that modifications – adding/removals in the same period). That would solve a ton of problems for people that use indexing in dev boxes…
That is a very good point. It would avoid indexing of all those temp files.
If you also rule out files that keep modifying themselves with this 2 seconds rule, then you’ll also solve the issues with browser downloads and processes that keep making changes to files.
very good idea, make it 5 seconds just to be sure the file is ‘stable’. This will solve a lot of problems and users don’t need a faster response from nepomuk/strigi imo (it almost takes 5 seconds for typing your search query)
Btw: I’m also one of those people that can’t wait for nepomuk to become useful. My first usage will be for searching scientific paper pdf’s, which never worked as good as recoll.
I’m not so sure. The indexer needs to handle the case that the file has been changed or removed while indexing, otherwise the problem is still there, just harder to reproduce.
That said: If a file is getting updated a lot, then re-indexing it all the time is not a good plan. But how long to wait between indexing? In an ideal world it would increment, so initially it doesn’t wait. When a change happens while indexing, set a flag saying wait 5 seconds (in the Nepomuk database?). If it changes during the next indexing, double wait time. Keep doubling the ‘wait time’ every time the files changes while it is being indexed. If indexing completes with no changes to the file, re-set the wait time to 0.
Or something along those lines… I might need to think a little more about it…
Well, we already have a queue of file modifications to act upon. So whenever a file changes it is queued to be re-indexed. Now if we were to wait for 5 seconds before we do the re-indexing, and then if during these 5 seconds the file changes again we would simply restart the 5 second count, then the problem should be solved.
really!? hummm … so strange!, in real world this crazy thing is crashing all the time! well, you can try this:
install mm3-webassistant from http://www.proxy-offline-browser.com, use it as proxy-cache for kde and after some days of online browsing, including online mp3 players like we7 and youtube, you will get a really huge bunch of cached files from small gif ads to youtube videos and so on and then point nepomuk infrastructure to index all this madness ( in very short time will be many files in mm3 offline cache!)… will crash! or if will not crash you will get a lot of strigi indexing errors and nepomuk chokes …
It would be great if you could try to find any crashes from running the following command with the master version of strigi:
strigiutils/xmlindexer -j 1 /home 2> /dev/null|../strigiutils/bin/xmlindexer/validateStdInXML.py
I have to say that the whole multi-process architecture of KDE, multiplied by the usage of tons of wrappers, all of them with their own API, and by the un-debuggable nature of Qt signals-and-slots / meta-object system makes KDE app debugging and bug fixing a royal pain. I once tried to trace down a Dolphin bug regarding previews. I found like 5 tiers of abstraction, lots of processes, insanely complex APIs and swore not to try to fix KDE bugs any more, at least in Dolphin or the KIO subsystem.
This is very sad. I’d like it to change one day, but this day is very very far away, somewhere in the wonderland.
Actually, for this not to look like a simple rant, I’d add the following:
KDE Core Libs and Frameworks *do* need good up-to-date documentation, which would cover such aspects of the named Libs and Frameworks as the API (with comments and probably examples), the process structure with MSDs (arguably the most important part, the graphics!), the design rationale for certan complex pieces of Frameworks. This would lower the barrier for bug-fixing as currently only the main application / library dev knows why the heck his app is as complex as a space shuttle when all it does is simply list files that are on the hard disk.
As far as Nepomuk goes: the only reason we have several processes is stability – if one service crashes it does not bring down the rest of the framework.
Erhm, Nepomuk does not index symlinks? As soon as I configure it to index the ‘real’ files it indexes thousands of them. Using my symlinked /home does result in 8 indexed files.
It does index them. But searching for stuff in a symlink pointing to a folder is something I never took into account.
Should I file a bug, a wish or nothing at all?
Hm, maybe a wish…
Doesn’t crash for me either (Chakra, latest, 64bit, Intel C2D) but this bug is very troubling and persistent, grinding system to halt in single-core computers:
https://bugs.kde.org/show_bug.cgi?id=231936
Now see this from bug you posted:
nepomukstrigiservice(15493)/nepomuk (strigi service)
Nepomuk::Indexer::slotIndexedFile: Indexing of “/home/dave2/2005.pdf” finished
with exit code 0
nepomukstrigiservice(15493)/nepomuk (strigi service)
Nepomuk::IndexScheduler::slotIndexingDone: Nepomuk::Indexer(0x22f9a90)
local socket error: QLocalSocket::PeerClosedError
I’m wondering…
nepomuk crash… did not look carefully at which component.
Few minutes before the crash I did pkill plasma-desktop; sleep 3; plasma-desktop
…..
plasma-desktop(9772)/kdecore (KConfigSkeleton) KCoreConfigSkeleton::writeConfig:
void DBusMenuImporterPrivate::slotItemsPropertiesUpdated(const DBusMenuItemList&, const DBusMenuItemKeysList&): No action for id 3
Application asked to unregister timer 0x700003d which is not registered in this thread. Fix application.