Better Science Through Benchmarking: Lessons for Software Engineering
This event took place on Friday 04 June 2004 at 14:30
Susan Elliott Sim
Benchmarking has been used to compare the performance of a variety of technologies, including computer systems, information retrieval systems, and database management systems. In these and other research areas, benchmarking has caused the science to leap forward. Until now, research disciplines have enjoyed these benefits without a good understanding of how they were achieved. In this talk, I present a process model and a theory of benchmarking to account for these effects. These were developed by examining case histories of existing benchmarks and my own experience with community-wide tool evaluations in software reverse engineering. According to the theory, the tight relationship between a benchmark and the scientific paradigm of a discipline is responsible for the leap forward. A benchmark operationalizes a scientific paradigm; it takes an abstract concept and turns it into a guide for research. Application of this theory will be illustrated using an example from reverse engineering: the C++ Extractor Test Suite (CppETS), a benchmark for comparing fact extractors for the C++ programming language. This talk will conclude with a discussion of how insights from studying benchmarking can improve the science in software engineering and collaboration in scientific communities more broadly.
(No replay available due to a shortage of technical staff to record event on the day)
This event took place on Friday 04 June 2004 at 14:30
Benchmarking has been used to compare the performance of a variety of technologies, including computer systems, information retrieval systems, and database management systems. In these and other research areas, benchmarking has caused the science to leap forward. Until now, research disciplines have enjoyed these benefits without a good understanding of how they were achieved. In this talk, I present a process model and a theory of benchmarking to account for these effects. These were developed by examining case histories of existing benchmarks and my own experience with community-wide tool evaluations in software reverse engineering. According to the theory, the tight relationship between a benchmark and the scientific paradigm of a discipline is responsible for the leap forward. A benchmark operationalizes a scientific paradigm; it takes an abstract concept and turns it into a guide for research. Application of this theory will be illustrated using an example from reverse engineering: the C++ Extractor Test Suite (CppETS), a benchmark for comparing fact extractors for the C++ programming language. This talk will conclude with a discussion of how insights from studying benchmarking can improve the science in software engineering and collaboration in scientific communities more broadly.
(No replay available due to a shortage of technical staff to record event on the day)
Future Internet
KnowledgeManagementMultimedia &
Information SystemsNarrative
HypermediaNew Media SystemsSemantic Web &
Knowledge ServicesSocial Software
Multimedia and Information Systems is...
We focus on content-based information retrieval over a wide range of data spanning form unstructured text and unlabelled images over spoken documents and music to videos. This encompasses the modelling of human perception of relevance and similarity, the learning from user actions and the up-to-date presentation of information. Currently we are building a research version of an integrated multimedia information retrieval system MIR to be used as a research prototype. We aim for a system that understands the user's information need and successfully links it to the appropriate information sources, be it a report or a TV news clip. This work is guided by the vision that an automated knowledge extraction system ultimately empowers people making efficient use of information sources without the burden of filing data into specialised databases.
Visit the MMIS website
List all Multimedia and Information Systems Projects
Check out these Hot Multimedia and Information Systems Technologies:
List all Multimedia and Information Systems Technologies
