Integrated Multimodal Interaction Using Texture Representations

Auston Sterling and Ming Lin


Multimodal Interaction: A user interacts with a virtual scene through sight, sound, and touch.

Documents
Integrated Multimodal Interaction Using Normal Maps
Graphics Interface 2015 [bibtex]
In this paper, we explore texture mapping as a unified representation for enabling realistic multimodal interaction with finely-detailed surfaces. We first present a novel approach to modifying collision handling between textured rigid-body objects; we then show how normal maps can be adopted as a unified representation to synthesize complex sound effects from long-lasting collisions and perform rendering of haptic textures. The resulting multimodal display system allows a user to see, hear, and feel complex interactions with textured surfaces. By using normal maps as a unified representation for seamlessly integrated multimodal interaction instead of complex triangular meshes otherwise required, this work is able to achieve up to 25 times performance speedup and reduce up to six orders of magnitude in memory storage. We further validate the results through a user study which demonstrates that the subjects are able to correctly identify the material texture of a surface through interaction with its normal map.
Integrated Multimodal Interaction Using Texture Representations
Computers & Graphics [bibtex]
http://dx.doi.org/10.1016/j.cag.2015.10.010
In this paper, we explore texture mapping as a unified representation for enabling realistic multimodal interaction with finely-detailed surfaces. We show how both normal maps and relief maps can be adopted as unified representations to handle collisions with textured rigid body objects, synthesize complex sound effects from long lasting collisions and perform rendering of haptic textures. The resulting multimodal display system allows a user to see, hear, and feel complex interactions with textured surfaces. By using texture representations for seamlessly integrated multimodal interaction instead of complex triangular meshes otherwise required, this work is able to achieve up to 25 times performance speedup and reduce up to six orders of magnitude in memory storage. We further validate the results through user studies to demonstrate the effectiveness of texture representations for integrated multimodal display.
Demonstrations

Download mp4 (112mb)
This demo shows multiple uses of both the normal and relief map representations and comparisons with previous work.

Download mp4 (39mb)
This demo shows multiple uses of the normal map representation and comparisons with previous work.
Acknowledgments
This research is supported in part by the National Science Foundation and the UNC Arts and Sciences Foundation.
Related Links
The GAMMA group at UNC
Sound research at GAMMA