Description logics and rules for multimodal situational awareness in healthcare

Abstract

We present a framework for semantic situation understand- ing and interpretation of multimodal data using Description Logics (DL) and rules. More precisely, we use DL models to formally describe contex- tualised dependencies among verbal and non-verbal descriptors in multi- modal natural language interfaces, while context aggregation, fusion and interpretation is supported by SPARQL rules. Both background knowl- edge and multimodal data, e.g. language analysis results, facial expres- sions and gestures recognized from multimedia streams, are captured in terms of OWL 2 ontology axioms, the de facto standard formalism of DL models on the Web, fostering reusability, adaptability and interoperabil- ity of the framework. The framework has been applied in the eminent eld of healthcare, providing the models for the semantic enrichment and fusion of verbal and non-verbal descriptors in dialogue-based systems.