Evaluation Methodologies for Multilabel Classification Evaluation
This event took place on Friday 18 December 2009 at 11:30
Stefanie Nowak
Semantic indexing of multimedia content is a key research challenge in the multimedia community. Several benchmarking campaigns exist that assess the performance of these systems. My PhD thesis deals with approaches for the annotation of images with multiple visual concepts and evaluation methodologies for annotation performance assessment.After a short outline of the different parts of my thesis, I would like to illustrate three case studies that were performed based on the results of a recent benchmarking event in ImageCLEF in more detail. In ImageCLEF 2009, we conducted a task that aims at the detection of 53 visual concepts in consumer photos. These concepts are structured in an ontology which covers concepts concerning the scene description of photos, the representation of photo content and the photo quality. For performance assessment, a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and generates a score per photo. Starting from this benchmark, three case studies have been conducted related to evaluation methodologies. The first study deals with the ground truth assessment for benchmark datasets. We investigate how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and whether these annotations can be obtained with a crowdsourcing approach. A second case study examines the behaviour of different evaluation measures for multilabel evaluation and points out their strengths and weaknesses. Concept-based and example-based evaluation measures are compared based on the ranking of systems. In the third case study, the ontology-based evaluation measure is extended with semantic relatedness metrics. We apply several semantic relatedness measures based on web-search engines, WordNet and Wikipedia and evaluate the characteristics of the measures concerning stability and ranking.
This event took place on Friday 18 December 2009 at 11:30
Stefanie Nowak
Semantic indexing of multimedia content is a key research challenge in the multimedia community. Several benchmarking campaigns exist that assess the performance of these systems. My PhD thesis deals with approaches for the annotation of images with multiple visual concepts and evaluation methodologies for annotation performance assessment.After a short outline of the different parts of my thesis, I would like to illustrate three case studies that were performed based on the results of a recent benchmarking event in ImageCLEF in more detail. In ImageCLEF 2009, we conducted a task that aims at the detection of 53 visual concepts in consumer photos. These concepts are structured in an ontology which covers concepts concerning the scene description of photos, the representation of photo content and the photo quality. For performance assessment, a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and generates a score per photo. Starting from this benchmark, three case studies have been conducted related to evaluation methodologies. The first study deals with the ground truth assessment for benchmark datasets. We investigate how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and whether these annotations can be obtained with a crowdsourcing approach. A second case study examines the behaviour of different evaluation measures for multilabel evaluation and points out their strengths and weaknesses. Concept-based and example-based evaluation measures are compared based on the ranking of systems. In the third case study, the ontology-based evaluation measure is extended with semantic relatedness metrics. We apply several semantic relatedness measures based on web-search engines, WordNet and Wikipedia and evaluate the characteristics of the measures concerning stability and ranking.
Future Internet
KnowledgeManagementMultimedia &
Information SystemsNarrative
HypermediaNew Media SystemsSemantic Web &
Knowledge ServicesSocial Software
Semantic Web and Knowledge Services is...
Our research in the Semantic Web area looks at the potentials of fusing together advances in a range of disciplines, and applying them in a systemic way to simplify the development of intelligent, knowledge-based web services and to facilitate human access and use of knowledge available on the web. For instance, we are exploring ways in which tnatural language interfaces can be used to facilitate access to data distributed over different repositories. We are also developing infrastructures to support rapid development and deployment of semantic web services, which can be used to create web applications on-the-fly. We are also investigating ways in which semantic technology can support learning on the web, through a combination of knowledge representation support, pedagogical theories and intelligent content aggregation mechanisms. Finally, we are also investigating the Semantic Web itself as a domain of analysis and performing large scale empirical studies to uncover data about the concrete epistemologies which can be found on the Semantic Web. This exciting new area of research gives us concrete insights on the different conceptualizations that are present on the Semantic Web by giving us the possibility to discover which are the most common viewpoints, which viewpoints are mutually inconsistent, to what extent different models agree or disagree, etc...
Our aim is to be at the forefront of both theoretical and practical developments on the Semantic Web not only by developing theories and models, but also by building concrete applications, for a variety of domains and user communities, including KMi and the Open University itself.
List all Semantic Web and Knowledge Services Projects
Check out these Hot Semantic Web and Knowledge Services Technologies:
List all Semantic Web and Knowledge Services Technologies
