Free Choice Massively Multimodal Interaction

Cyber-physical environments (CPE) are highly sophisticated systems of systems of systems. For the user to retain control and overview, it is essential to maximize usability in these environments. One major aspect of this is to allow the users to interact as naturally as possible with their surroundings, at any time employing the modality they choose. With more systems than ever being combined into a single CPE, the available modes are growing, as well as the amount of combination and switching that must be supported. Massively multimodal interaction uses whatever modality is available, whenever the user selects it.

Free choice modalities enable a seamless transition between input and output devices. From the user’s point of view, it does not matter which modality she uses, as long as there is some input device within the CPE able to capture the interaction. Having different devices for both near and far field interaction installed in the environment would provide ideal conditions for this task. In a similar way, the means of output should not be determined by technical considerations, but by usability considerations, for example the paradigm of symmetric multimodality.

In MADMACS, we will extend the concepts of fusion and fission built into our dialogue platform to support an even larger set of modalities and allow for rapid switches between them. Furthermore, we allow dialogue applications to be written in a modality-agnostic way.