Interaction in Smart Environments: Use of Ubiquitous User Interfaces

The widespread use of computing technology raises the need for interactive systems that adapt to user, device and environment. Multimodal user interfaces provide the means to support the user in various situations and to adapt the interaction to the user’s needs.

Computer technology is currently changing our lives and the way we handle technology. The computer has moved from a business machine dedicated to specific tasks in a well defined environment to a universal problem solver in all areas of live. Powerful mobile devices that are always online, and the ongoing paradigm shift towards ubiquitous computing concepts provide increasingly complex functionality and allow remote access to additional services and information. Wireless ad-hoc network technologies and the upcoming “Internet of Things” drive the trend to local networks and smart environments.



This poses challenges to applications and their user interfaces that now have to support various situations instead of the well known scenario of the user sitting in front of his desk. The widespread use of computers in all areas of life also continuously affects new groups of users. As their number grows, so does their diversity, with each user having different personal preferences, different experience levels and different capabilities. Smart environments confront user interfaces with a variety of available (mobile) interaction resources supporting diverse modalities, and heterogeneous users with different capabilities and preferences.

A user interface supporting smart environments requires a high degree of adaptability to innumerable contexts of use. Unfortunately, today’s user interfaces do not sufficiently support the creation of ubiquitous systems and smart environments and a significant improvement of the communication, interaction and adaptation capabilities is required. At the same time the user must be given the power of understanding and controlling her smart environment in a flexible and comprehensible way

There’s therefore need for Ubiquitous User Interfaces (UUIs) addressing the challenges of the ubiquitous computing paradigm within the following dimensions:

 multi-situation: support of multiple UI layouts for different usage contexts;

 multi-device: support for the usage of multiple devices simultaneously (or sequentially);

 multi-modal: support for multiple interaction modalities according to the needs of the interaction;

 multi-user: support to share applications, information and interaction devices between multiple users;

 multi-application: support to use multiple applications per user and device simultaneously and sequentially.

Based on these five features we can define Ubiquitous User Interfaces as interfaces that are shapeable, distributable, multimodal, shareable and mergeable. Ubiquitous User Interfaces thus support the utilization of multiple modalities, devices and interaction concepts to provide robust interaction for different purposes. Furthermore they facilitate flexible interaction, mobile and stationary, within changing contexts and situations. Developing UUIs now poses the challenge to express the increasingly complex interaction concepts and to handle the adaptive distributed multimodal interaction at runtime.

Smart environments raise the need of highly adaptive user interfaces, supporting multiple situations, devices, modalities, users and application. With current state of the art technologies, the complexity of such user interfaces is almost impossible to handle. Existing languages, tools and methodologies are not expressive enough and usually target specialized contexts instead of the broad adaptive applications. Further research is needed in engineering factors like expressive and standardized languages, runtime adaptation, self-aware and self-adaptive systems, multimodal input processing, the understanding of the meaning of interaction and the handling of the required world knowledge as well as human factors as the usage and combination of (multiple) modalities, natural and robust interaction or the collaborative usage of applications and resources.

From the perspective of mobile computing, future trends include the seamless integration of mobile and personal devices with their surrounding environment. Thus mobile devices and especially their applications are required to make use of varying available (wireless) connected resources and to expose information and services to their human users. Additionally, to more dynamic service combination and utilization, new interaction concepts are needed that on the one hand make these increasingly complex and distributed systems accessible and usable for human users and on the other hand better exploit the broad spectrum of human communication capabilities. While current interaction resources like keyboard, mouse and even touch screen are well adapted to the needs of interactive systems, they do not reflect human communication means like speech, sounds, gestures or facial and body expressions.

Moving the computer from a business machine dedicated to specific tasks in a well defined environment to a universal problem solver in all areas of live continues to pose great challenges to technicians and scientists. Making computers more powerful does not automatically makes them smarter. Making machines ubiquitously available does not automatically makes them usable and enjoyable. The main goal for future developments has to be to prevent systems from being a hassle to users and to build systems providing supportive and enjoyable interaction for users.