KMi Seminars
Evaluation Methodologies for Multilabel Classification Evaluation
This event took place on Friday 18 December 2009 at 11:30

Stefanie Nowak

Semantic indexing of multimedia content is a key research challenge in the multimedia community. Several benchmarking campaigns exist that assess the performance of these systems. My PhD thesis deals with approaches for the annotation of images with multiple visual concepts and evaluation methodologies for annotation performance assessment.After a short outline of the different parts of my thesis, I would like to illustrate three case studies that were performed based on the results of a recent benchmarking event in ImageCLEF in more detail. In ImageCLEF 2009, we conducted a task that aims at the detection of 53 visual concepts in consumer photos. These concepts are structured in an ontology which covers concepts concerning the scene description of photos, the representation of photo content and the photo quality. For performance assessment, a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and generates a score per photo. Starting from this benchmark, three case studies have been conducted related to evaluation methodologies. The first study deals with the ground truth assessment for benchmark datasets. We investigate how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and whether these annotations can be obtained with a crowdsourcing approach. A second case study examines the behaviour of different evaluation measures for multilabel evaluation and points out their strengths and weaknesses. Concept-based and example-based evaluation measures are compared based on the ranking of systems. In the third case study, the ontology-based evaluation measure is extended with semantic relatedness metrics. We apply several semantic relatedness measures based on web-search engines, WordNet and Wikipedia and evaluate the characteristics of the measures concerning stability and ranking.

 
KMi Seminars
KMi 2013 - A review of the year

Download the KMi 2013 Review of the year iBook to your iOS device or alternatively as a PDF.

Journal | 25 years of knowledge acquisition
 

Knowledge Management is...


Knowledge Management
Creating learning organisations hinges on managing knowledge at many levels. Knowledge can be provided by individuals or it can be created as a collective effort of a group working together towards a common goal, it can be situated as "war stories" or it can be generalised as guidelines, it can be described informally as comments in a natural language, pictures and technical drawings or it can be formalised as mathematical formulae and rules, it can be expressed explicitly or it can be tacit, embedded in the work product. The recipient of knowledge - the learner - can be an individual or a work group, professionals, university students, schoolchildren or informal communities of interest.
Our aim is to capture, analyse and organise knowledge, regardless of its origin and form and make it available to the learner when needed presented with the necessary context and in a form supporting the learning processes.