|
|
|
Poster Submissions
| |
| |
Knowledge-Based Approach to Modality Choice
Yulia Bachvarova, Betsy van Dijk, Anton Nijholt
Human Media Interaction Group, University of Twente The Netherlands |
|
Choosing the most appropriate modality or combination of modalities that best conveys the intended message is central to any multimedia or multimodal application. We address modality choice by proposing a unified knowledge-based approach where we describe the knowledge about modalities in modality ontology and formulate rules for modality choice.
The modality ontology describes the nature of each modality, that is, those features pertinent to a particular modality regardless of the content it holds. The ontology models the knowledge about modalities at three different levels: (i) features that robustly distinguish a particular modality from the others (ii) features that describe how the modality is perceived and processed by the human perceptual-sensory system (iii)the structural dependencies that exist between the composite modalities and that have direct bearing on the way multimodal messages construct and convey meaning.
We formulate rules for modality allocation and modality combination - the two main processes defining modality choice. Modality allocation rules establish the mapping between the types of information that has to be presented and the modalities that are best at representing these types of information. Modality combination rules determine the optimal modality combinations. The rules use the knowledge modeled by the ontology and formalize certain theories from psychology, cognitive sciences and multimodal semiotics.
|
|
|
