KMi Seminars
Evaluation Methodologies for Multilabel Classification Evaluation
This event took place on Friday 18 December 2009 at 11:30

Stefanie Nowak

Semantic indexing of multimedia content is a key research challenge in the multimedia community. Several benchmarking campaigns exist that assess the performance of these systems. My PhD thesis deals with approaches for the annotation of images with multiple visual concepts and evaluation methodologies for annotation performance assessment.After a short outline of the different parts of my thesis, I would like to illustrate three case studies that were performed based on the results of a recent benchmarking event in ImageCLEF in more detail. In ImageCLEF 2009, we conducted a task that aims at the detection of 53 visual concepts in consumer photos. These concepts are structured in an ontology which covers concepts concerning the scene description of photos, the representation of photo content and the photo quality. For performance assessment, a recently proposed ontology-based measure was utilized that takes the hierarchy and the relations of the ontology into account and generates a score per photo. Starting from this benchmark, three case studies have been conducted related to evaluation methodologies. The first study deals with the ground truth assessment for benchmark datasets. We investigate how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and whether these annotations can be obtained with a crowdsourcing approach. A second case study examines the behaviour of different evaluation measures for multilabel evaluation and points out their strengths and weaknesses. Concept-based and example-based evaluation measures are compared based on the ranking of systems. In the third case study, the ontology-based evaluation measure is extended with semantic relatedness metrics. We apply several semantic relatedness measures based on web-search engines, WordNet and Wikipedia and evaluate the characteristics of the measures concerning stability and ranking.

KMi Seminars
KMi 2013 - A review of the year

Download the KMi 2013 Review of the year iBook to your iOS device or alternatively as a PDF.

Journal | 25 years of knowledge acquisition

Future Internet is...

Future Internet
With over a billion users, today's Internet is arguably the most successful human artifact ever created. The Internet's physical infrastructure, software, and content now play an integral part of the lives of everyone on the planet, whether they interact with it directly or not. Now nearing its fifth decade, the Internet has shown remarkable resilience and flexibility in the face of ever increasing numbers of users, data volume, and changing usage patterns, but faces growing challenges in meetings the needs of our knowledge society. Globally, many major initiatives are underway to address the need for more scientific research, physical infrastructure investment, better education, and better utilisation of the Internet. Within Japan, USA and Europe major new initiatives have begun in the area.

To succeed the Future Internet will need to address a number of cross-cutting challenges including:

  • Scalability in the face of peer-to-peer traffic, decentralisation, and increased openness

  • Trust when government, medical, financial, personal data are increasingly trusted to the cloud, and middleware will increasingly use dynamic service selection

  • Interoperability of semantic data and metadata, and of services which will be dynamically orchestrated

  • Pervasive usability for users of mobile devices, different languages, cultures and physical abilities

  • Mobility for users who expect a seamless experience across spaces, devices, and velocities