Nepomuk Tasks: KActivityManager Crash

After a little silence during which I was occupied with Eastern and OpenLink related work I bring you news about the second Nepomuk task: the KActivityManager crash.

Ivan Cukic already “fixed” the bug by simply not using Nepomuk but an SQLite backend (at least that is how I understood it, correct me if I am wrong). However, I wanted to fix the root of the original problem.

Soprano provides the communication channel between Nepomuk and its clients. It is based on a very simple custom protocol going through a local socket. So far QLocalSocket, ie. Qt’s implementation was used. The problem with QLocalSocket is that it is a QObject. Thus, it cannot live in two threads at the same time. The hacky solution was to maintain one socket per thread. Sadly that resulted in complicated maintenance code which was impossible to get right. Hence crashes like #269573 or #283451 (basically any crash involving The Soprano::ClientConnection) were never fixed.

A few days ago I finally gave up and decided to get rid of QLocalSocket and replace it with my own implementation. The only problem is that in order to keep Windows compatibility I had to keep the old implementation around by adding quite a lot of #ifdefs.

And now I could use some testers for a Soprano client library that does only create a single connection to the server instead of one per thread. I already pushed the new code into Soprano’s git master. So all you need to do is run KDE on top of that.

Oh, and while at it I finally fixed the problem with re-connecting of clients. So now a restart of Nepomuk will no longer leave the clients with dangling connections, unable to perform queries. That fix, however, is in kdelibs.

Well, the day was long, I am tired, and this blog post feels a little boring. So before in addition to that it gets too long I will stop.

Nepomuk Tasks: Let The Virtuoso Inferencing Begin

Only four days ago I started the experiment to fund specific Nepomuk tasks through donations. Like with last year’s fundraiser I was uncertain if it was a good idea. That, however, changed when only a few hours later two tasks had already reached their donation goal. Again it became obvious that the work done here is appreciated and that the “open” in Open-Source is understood for what it actually is.

So despite my wife not being overly happy about it I used the weekend to work on one of the tasks: Virtuoso inferencing.

Inference?

As a quick reminder: the inferencer automatically infers information from the data in the database. While Virtuoso can handle pretty much any inference rule you throw at it we stick to the basics for now: if resource R1 is of type B and B derives from A then R1 is also of type A. And: if R1 has property P1 with value “foobar” and P1 is derived from P2 then R1 also has property P2 with value “foobar“.

Crappy Inference

This is already very useful and even mandatory in many cases. Until now we used what we called “crappy inferencing 1 & 2”. The Crappy inferencer 1 was based on work done in the original Nepomuk project and it simply inserted triples for all sub-class and sub-property relations. That way we could simulate real inference by querying for something like

select * where {
  ?r ?p "foobar" . 
  ?p rdfs:subPropertyOf rdfs:label .
}

and catch all sub-properties of rdfs:label like nao:prefLabel or nie:title. While this works it means bad performance, additional storage and additional maintenance.

The Crappy Inferencer 2 was even worse. It inserted rdf:type triples for all super-classes. This means that it would look at every added and removed triple to check if it was a rdf:type triple. If so it would add or remove the appropriate rdf:type triples for the super-types. That way we could do fast type queries without relying on the crappy inferencer 1 which relies on the rdfs:subClassOf method. But this meant even more maintenance and even more storage space wasted.

Introducing: Virtuoso Inference

So now we simply rely on Virtuoso to do all that and it does such a wonderful job. Thanks to Virtuoso graph groups we can keep our clean ontology separation (each ontology has its own graph) and still stick to a very simple extension of the queries:

DEFINE input:inference <nepomuk:/ontographgroup>
select * where {
  ?r rdfs:label "foobar" .
}

Brilliant. Of course there are still situations in which you do not want to use the inferencer. Imagine for example the listing of resource properties in the UI. This is what it would look like with inference:

We do not want that. Inference is intended for machine, not for the human, at least not like this. So since back in the day I did not think of adding query flags to Soprano I simply introduced a new virtual query language: SparqlNoInference.

Resource Visibility

While at it I also improved the resource visibility support by simplifying it. We do not need any additional processing anymore. This again means less work on startup and with every triple manipulation command. Again we save space and increase performance. But this also means that resource visibility filtering will not work as before anymore. Nepoogle for example will need adjustment to the new way of filtering. Instead of

?r nao:userVisible 1 .

we now need

FILTER EXISTS { ?r a [ nao:userVisible "true"^^xsd:boolean ] }

Testing

The implementation is done. All that rests are the tests. I am already running all the patches but I still need to adjust some unit tests and maybe write new ones.

You can also test it. The code changes are, as always, spread over Soprano, kdelibs and kde-runtime. Both kdelibs and kde-runtime now contain a branch “nepomuk/virtuosoInference”. For Soprano you need git master.

Look for regressions of any kind so we can merge this as soon as possible. The goal is KDE 4.9.

Nepomuk Tasks – Sponsor a Bug or Feature

Thanks to a very successful fundraiser in 2011 I was able to continue working on Nepomuk and searching for new enterprise sponsoring. Sadly that search was not fruitful and in 2012 Nepomuk has become a hobby. Several people proposed to start another fundraiser or try to raise money on a monthly basis. I, however, will try to get sponsoring for specific bugs or features. Depending on their size the sponsoring goal will differ. This would allow me to keep working on Nepomuk as more than a hobby.

The Nepomuk Tasks page lists the current tasks that can be sponsored. Of course you can propose new tasks but I will try to keep the list of current tasks small. Donate to the tasks you would like to see finished, ignore the ones you do not deem important. I will simply remove tasks if there is no activity within a certain period of time. So please have a look at

The Nepomuk Tasks Page

Nepomuk Fundraiser – Badamm (Or Some Other Really Clever and Funny Title I Cannot Think of at the Moment)

It happened. Alf Rustad donated the missing 356€ which broken the magical barrier of 9000€ in the Nepomuk Fundraiser I started nearly three months ago.

While the actual goal – securing long-term funding for Nepomuk – has not been reached yet this is a great opportunity to thank Alexander, Alvar, Andreas, Andre, Andrew, Angelo, Anton, Antonio-J, Ardy123, arkub, Baltasar, Bernd, Bernhard, Calogero, Carl, Ceferino, Christopher, Christoph, Claude, Cristiano, Daniel, David, dunkelschorsch, Eduard, Efthymia, Elias, the two Enriques, Fabio, Felix, Florian, Francisco, Friedhelm, Fux, Gael, Giacomo, Giorgio, Guillaume, Günter, Hans, Han, Hartmut, Hector, Hendy, Huftis, Jaroslav, Jérôme, Jesus, Josep, Jos, Jramskov, Juan, Juanjo, Junichi, Kai, Kenneth, Kevin, Kilian, Kulomi, Leopoldo, Linopolus, Luca, Luis, Luiz, Maik, Manoel, Manuel, Marco, Marc, the three Markusses and Martins, Maxime, Mguel, the two Michaels, Mikael, Mike, Morgan, Nicolas, Olaf, Olivier, Orestes, the two Pauls, Paulo, the two Peters, Philipp, Pierre-Hugues, Régis, Robert and Robert, Rodrigo, samtuke, the Sebastians, Simone, Sören, Stefano, Steffen, Stian, tanghus, Thiago, Thomas, Thomas, and Thomas, Tiago, Timothy, Tommi, Tuukka, Ulrich, Wakeley, Xavier, Yaroslav, and all the anonymous doners for their support. You have given me time to keep looking.

A special thanks goes to Carl Symons for his great dot article, his many tips and continuous encouragement.

Thank you also to Peter, George, Ivan, Vishesh, Christian, Andrew, Martin, and Laura for their great developer comments on Nepomuk.

And last but not least thanks for all the positive feedback on my blog articles, the translations into strange and exotic languages such as spanish :P and all the encouraging words which showed how many actually get what the semantic desktop is all about and want Nepomuk to go on and change the way we work with information today.

Nepomuk – Fundraiser Update

It was only two days ago that I started my fundraiser for my work on Nepomuk and already I got a lot of donations and very nice and encouraging comments. Thank you for that.

It was pointed out that it might be good to describe in more detail what work you are actually funding. That is exactly what I will do now. I will update nepomuk.kde.org which is hopelessly outdated, try to establish a better way of publishing my progress in Nepomuk, and draft a vision of what Nepomuk sets out to do (well, at least part of it – the semantic desktop simply has too many use cases, small and big, to list them all).

In the meantime I created a Nepomuk pledge on pledgie.com. If nothing else it allows to nicely publish the names of all you generous people and track the development of the donations.

Again: thanks a lot for your donations so far. It is very much appreciated.