At this week‘s OpenSym Conference we will present our evaluation of property recommender systems for Wikidata and generally, collaborative knowledge bases. The (admittedly way too long) title of our paper is: “An Empirical Evaluation of Property Recommender Systems for Wikidata and Collaborative Knowledge Bases“.
Wikidata is a popular example for collaboratively filled and mainained knowledge bases. These mostly rely on a community of committed people who edit and add data. These users are often supported by recommender sytems during the process of entering and editing data. Wikidata also provides a so-called „property suggestor“ which basically recommends further properties to be added. In principle, data is entered in a triple-form: subject-property-object, where property-object pairs (so-called „statements“) are used to describe a subject. Wikidata supports its editors by providing recommendations for further suitable properties for a given subject.
In this work, we evaluate different recommendation algorithms serving this purpose. In principle, we compare an approach by Abedjan and Naumann, the current Wikidata recommender and the Snoopy-approach we developed a couple of years ago.
We identify three important influence factors regarding the quality of recommendations: (i) the use of classifying properties into the rule creation process as incorporated in the Wikidata approach (WD), (ii) ranking according to confidence values as performed by Abedjan and Naumann (AN) and (iii) incorporating contextual information into the ranking process as proposed by the Snoopy approach (SN_context). We find that the current implementation of the Wikidata Entity Suggester works better than the other presented approaches. In the course of our analyses, we identify two key aspects which are essential for the quality of recommendations: incorporating classifying properties and making use of contextual information for ranking the property recommendation candidates. Combining the current Wikidata Entity Suggester approach with Snoopy’s ranking strategy, which facilitates contextual information, significantly increases the performance of the current Wikidata recommender approach as can be seen in the followRecall@k evaluation (WD_context).
You can find our open source implementation of the underlying evaluation framework and the evaluated algorithms here: https://github.com/dbisibk/PropertyRecommenderEvaluator/
For more details on this, please check out or OpenSym paper:
[bibtex file=publications.bib key=opensym16,format=plain, sort=year order=desc]]