Andreas Zimmermann, Andreas Lorenz, and Marcus Specht, Fraunhofer Institut for Applied Information Technology, Germany

When art and technology meet, a huge information flow has to be managed. In the LISTEN project conducted by the Fraunhofer Institut in St. Augustin, we augment every day environments with audio information. In order to distribute and administer this information in an efficient way, we decided to employ an information brokering tool for the management of information items. Furthermore, the generation of user profiles and the personalized presentation of information are possible by this means. This contribution depicts an approach of transferring our information brokering experience to Web museum applications. We show how the LISTEN domain model can easily be extended by an overlay model and adapted to a Web museum environment.

Keywords: Audio-Augmented Environments, Information Brokering, Ontology Modeling, User Modeling, Personalization, Context-Awareness

Introduction

A variety of guidance systems or information systems have been developed in the last few years for the support of a museum visit or for preparation for such a visit. In most cases the authenticity and the possibility of contextualizing the information presentation to the current position or situation of a user were seen as central issues. Furthermore, classical museum audio guides were much more acceptable to museum visitors because of the easy handling and the quality of sound presentation. For the flexible use of information about art objects and museum exhibitions, we propose a centralized model around a domain ontology that is described in a software tool for information brokering. We think that the proposed model is a very flexible way to reuse existing information and support curators and exhibition experts to in the design of a variety of personalized museum experiences, ranging from an exhibition Web site to a interactive audio experience in the museum space to a personalized CD-ROM production for taking home.

Context as a mean for adaptation of information selection and presentation has been described in a variety of ways and approaches (Gross & Specht, 2001; Shilit, Adams & Want, 1994; Dey & Abowd, 1999). Nevertheless the underlying problem of identifying similarities and differences between various constellations of context parameters has not been discussed intensively in the literature. From our point of view identifying important context parameters to describe user interaction is an essential issue when designing context aware information services.

The identification and the description of the context parameters is a non-trivial task. A central issue is to find a solution for structuring an information domain appropriate not only from the point of view of an information engineer but also from the point of view of a user. Especially for designing personalized information services the structure and the intuitiveness of the information structuring is essential for the successful application of user modeling and personalization methods.

The LISTEN project conducted by the Fraunhofer Institut in St. Augustin deals with the audio augmentation of real and virtual environments. A first LISTEN prototype has been successfully installed at the Kunstmuseum in Bonn (UnnÔtzer, 2001). Visitors walk through the museum and experience personalized audio information about paintings by August Macke through wireless headphones. In such a scenario, the distribution and authoring of this valuable information on exhibits is a non-trivial task.

Audio-Augmenting Real and Virtual Environments

Combining high-definition spatial audio rendering technology with advanced user modeling methods creates audio-augmented environments. Visitors are immersed in a dynamic virtual auditory scene that consistently augments the real space they are exploring. The physical environment is augmented through a dynamic soundscape, which users experience over motion-tracked wireless headphones for 3D spatial reproduction of the virtual auditory scene.

A sophisticated auditory rendering process takes into account the current position and orientation of the visitor's head in order to seamlessly integrate the virtual scene with the real one. Speech, music and sound effects are dynamically arranged to form an individualized and situated soundscape offering exhibit-related information as well as creating context-specific atmospheres. Next to the automatic adaptation of sound scene rendering to the position and orientation of the user's head, the audio stream is controlled in two ways: events (mediated interaction) that are used to start and stop the playback of information items in form of audio recordings; and continuous control (immediate interaction) changing parameters in the audio-generation of the presentation (e.g. a sound that gets continuously louder as you approach a certain position within the space).

The dynamic composition of the soundscape is personalized through each visitor's spatial behavior, the history of the visit, and interests or preferences either expressed explicitly by the visitor or inferred from the visitor's behavior (Eckel, 2001). To present individualized media and create augmented environments, such systems have several models in common (Gossmann & Specht, 2001) that are described in more detail in this section: the World model, the augmentation layer, the domain model and the user model.

The World Model (Space Model, Location Model) describes the physical environment the user moves through while interacting with the system. In the LISTEN environment, the space model contains the geometric information of the exhibition space and its objects. The LISTEN world model is a detailed VR-based geometric model. It is created for the AVANGO application (Tramberend, 1999) and is described as a geometric scene graph. Therefore, a LISTEN environment can be tested and prototyped in a CAVE system (Eckel, 2001), or be explored in real space with virtual audio content displayed through a wireless motion-tracked headphone.

The Augmentation Layer on top of the World Model defines areas (Zones, Segments, Triggers) within the world model that contain active information or sound objects the users of the system interact with. The augmentation layer filters the position and motion of the user by dividing the dimensions the user moves through (location and orientation) into meaningful constraints and deriving continuous parameters from them. By defining zones and segments, the visitor's focus obtains a valuable meaning.

The Domain Model holds information about sound objects and other hypermedia objects connected to the physical space via the augmentation layer by using meta data. The domain model builds up a virtual acoustic space in which the location of virtual sound sources and spaces are defined. Stopping in front of an exhibit generates aural information about the art piece. Moving the head and body activates a further audio source, where music deepens the visitor's impressions, or the voice of a commentator talks about the artist or describes the period the painting originates from.

The User Model contains knowledge and profile information about the system's users. While the user moves in physical space, events are sent to the user model, and by these events the model is refined. The user model stores knowledge about the visitor - like preferences, interests in arts, and the history of the visit. In combination with the visitor's spatial position (delivered by the tracking system), the content of the user model strongly influences the presentation of information according to the current visitor's context.

Information Brokering

In this real world application of LISTEN, we decided to use an information brokering tool for authoring valuable information on exhibits and distributing requested information among the visitors in an efficient way. Information brokering is a value adding process of mediation between information demands and information offers. The added value emerges from the understanding of the domain complexity and the definition of a useful vocabulary. Additional structures and interpretation rules (implicit and explicit) in information exchange simplify the processing and comprehension of exchanged information. Figure 1 illustrates the interrelation between data, information, and knowledge.

Three roles participate in the information brokering process: the provider offers information, the consumer demands information, and the broker mediates between the other two. The quality of the information brokering process depends to a great extent on the knowledge available to the broker. Knowledge about available sources, the domain, consumers and other brokers is needed. Source knowledge is created in the domain representation or maintenance processes and describes the quality of sources and how they can be accessed. Domain Knowledge is about the contents of the brokering domain and should reflect the provider's understanding of the domain as well as the consumers' perception of the domain. Consumer Knowledge is created in the consumer-oriented process and describes the consumer and his concrete information need. Consumer knowledge has to map onto domain knowledge to fulfill the information need. To ensure an optimal service to consumers, they should be served by the best broker according to their information need. This assignment task depends on the availability of Expert Knowledge about different brokers.

In the LISTEN project, the brokering tool can be understood as a server mediating information items between the exhibition's curator and the LISTEN application. In the role of the information provider, the curator defines the domain model and authors the information items, as illustrated in the left column of Figure 2. From the point of view of the information broker, the information is consumed by the LISTEN application, although it finally passes the information to the visitor as the real consumer. In the role of the information consumer (Figure 2, right column), the LISTEN application defines an interest model regarding to the visitor's location, preferences, interests, and history of already visited objects.

To mediate between the provider and the consumer, a brokerage tool acts as a server to deliver filtered information on demand. Such a brokerage development and management environment (the Broker's Lounge; Jarke, Klemke & Nick, 2001) has been developed at the Fraunhofer Institut for Applied Information Technologies (FIT) in St. Augustin. With the aid of this application, a large variety of scenarios within the general framework of Figure 1 and Figure 2 can be quickly developed, and efficiently and flexibly executed.

Information brokering processes make use of and create a number of information items that describe single units of information (cf. Klemke & Koenemann, 1999). As Klemke (2002) has defined, each information item is an instantiation of a concept which describes the structure of the brokered items. In order to organize information items, categories describe fundamental principles or ideas. In many cases, these categories define hierarchical trees. In Figure 1 conceptualization and categorization is illustrated as a procedure for the concretion of raw data. The next section provides an outline of definition of the LISTEN project's information items and illustrates a hierarchical tree to categorize them.

Modeling Information Items

Bringing arts and technology together means the administration and distribution of a huge amount of information. In this paper we propose the use of a central management component for all information items. We present an approach of modeling information items in a museums environment by means of domain ontology in an information brokering tool. In the case of the LISTEN project, the set of information items is composed of sound items. We describe the application of the ontology methodology (concepts and categories) to generate a meaningful domain model and depict the association of information items to exhibition objects.

In recent projects developing guiding systems and electronic art guides, the structuring and internal representation of information appeared to be a central issue for delivering the right information at the right time to a specific user. Even more, for the personalization of presentations and the contextualized delivery of information pieces, the underlying structure of the represented information needed to be a core part of the work. Nevertheless there was a trade off between the efforts of authoring information items into a highly enriched information representation and the daily work of curators and information providers for museum environments.

To find the right balance between user efforts for authoring and creating museum information and meta data in previous projects, we often used an object-oriented approach, where the main entities were the art objects as such. This led us to typical presentation and structuring of information as found in art databases and museum information systems.

By detailed user needs analysis of museum visitors and by the analysis of human guides, we realized that the event character of museum visits in most cases is much more important than plain information about the art object as such. The presentation of data about artworks is just one style of presentation that can be appropriate but in practice is rarely used by human guides. Furthermore, through our work with museum curators and artists in the LISTEN workshops we elicited different strategies and methodologies to structure information items and to combine those information items in a highly flexible way for different users.

The main entities for generating presentations are therefore not the artworks as such, but the parts or chunks of a presentation that can come from a diversity of sources and use a variety of stylistic means to make the museum visit an interactive experience. Therefore, for creating interactive audio augmented LISTEN spaces, we decided to choose the sound item as the main entity. Sound items can be classified in a category system with several dimensions describing the sound items technically and stylistically. An overview of the dimensions can be seen in Figure 3. This figure illustrates the domain model inside the graphical interface of the information brokering tool, organized as a tree, and a list of information items of the LISTEN project.

The single sound items as such are independent episodes or chunks of presentation that can be combined flexibly because they contribute to a variety of art objects. The expressive power of such a structure can be easily explained with an example:

In an introduction to several artworks of August Macke, the different artworks are connected by personal episodes, events and experiences of August Macke, by reactions and interaction with his social environment (personal letters, media, press articles), or by the “Zeitgeist” and other factors of the environment of August Macke at that time. In most cases human guides extend the visual perception of such artworks with these “stories or impressions” of that time period to immerse the user in an authentic experience of the works and the life of the artist. By the structured description (meta tagging) of a huge collection of such “stories or impressions,” the LISTEN system allows a highly flexible way of “immersing the user” into the August Macke experience.

Therefore we designed the domain ontology (meta data) for the Macke-Exhibition to allow for the description of small information items on a variety of dimensions, allowing for the connection and the individualized sequencing and presentation of these items. Beside the simple classification of the information items in a tree structure, the information brokering tool allows for classification of sound items into multiple categories.

Especially speech sound items could be further classified into subcategories like Citation, Collage, Diary, Letter, Newspaper and others to describe their style of presentation. As described in the example of the Macke-Exhibition, the multidimensional classification of the sound items allows for a variety of sequences and presentation styles, even combining sound items on several channels; i.e. typically music, effects, and speech. An enormous advantage of the description in such away is that the curator of an interactive experience is not forced to design a complete sequence of information presentation but can combine resources in a collage style or define sequencing rules on the level of possible connection categories and not for single sound items.

In the LISTEN application, the visitor's position and orientation are observed by a tracking system and interpreted within the context of a virtual environment, which connects the real world objects with virtual objects. Since the museum's visitors are mobile in physical space, their spatial positions are translated into virtual positions in the electronic space relative to virtual objects. The virtual environment enables the definition of virtual sound sources and the segmentation of the physical space into virtual zones. Invisible to the user there exists a specific category in the broker's domain model for each exhibition object. Depending on the visitor's location, preferences and so forth, combinations of such categories are dynamically selected in order to pre-filter the information items. The next section illustrates how this solution can be adapted to an electronic museum.

Adaptation to a Web Museum

In order to transfer our information brokering solution to the domain of an electronic museum, we present a modus operandi for mapping the physical space into an electronic space. For this reason we follow an overlay model approach for context-aware information systems as introduced by Gross & Specht (2001). This methodology enables us to apply and reuse the algorithm for information filtering described in the previous section within another domain. Thus, only three concepts of the LISTEN application must be adapted or extended: tracking, modeling of the information items, and domain modeling.

As described in the last section, the domain model connects the objects to categories. If the user of the LISTEN system enters the zone connected with a certain domain object, the LISTEN application selects the category associated with this object, in order to filter information items according to the users preferences and position. In information brokering terminology, the application builds an information model and uses the broker as triggered by events. In an electronic museum application, the tracking system would deliver the current URI as position and the selected objects (e.g. text links, images, or clicks on maps) as a kind of the visitor's movement. Here, an event is fired when the user visits any Web source that is connected with a category of the broker's domain model.

The information architecture of the LISTEN system is limited to audio presentations. The information broker falls back on a set of audio pieces (e.g. spoken text, music, effects) and returns the best matching item according to the inquired interest model of the user. In the domain of an electronic museum, the curator may probably extend this set of information items by any kind of multimedia presentations, text fragments or links to Web pages. Therefore, the broker's domain model must be extended also to adequately reflect the changed information architecture. New categories have to be added and the meta data description has to be augmented, easily accomplished with the assistance of the information brokering tool. As a result, the system provides additional facility for specification of preferences to the users (e.g. users can turn off streamed video-presentations), and thereby the user model is extended at the same time.

In contrast to the fixed sound installation in the real world LISTEN environment, an electronic museum has to take into account the abilities and features of the visitor's particular environment; i.e. the Web browser. An initially executed piece of software can ask for the user's specific settings and automatically preset the information brokering tool. Based on known environmental properties, the information items are filtered in advance and presented in an appropriate format. The ontology model of the information brokering tool has to be adapted in a suitable way.

The mentioned extensions of the information brokering tool do not affect its capability of filtering information items that fit best the user's context. The overlay model enables the retention of the presented concepts no matter whether the user moves in physical or in electronic space. The next section shows how the presentation of information and exhibits in an electronic museum can further be personalized and adapted to the visitor's needs and behavior. By means of a user model, additional filters can be triggered or refined.

Personalization of a Web Museum

A Web museum environment offers the ability to adapt the order of the presented exhibits and the attached information to the visitor's profile and position within the exhibition. In order to provide a personalized adaptation of the environment according to the visitor's context (i.e. interests, preferences and “motion”), the system has to build up and maintain a user model.

By using sensor data, the user is tracked in the information space in the same way a location tracking system tracks users in physical space. This allows for a variety of new applications and is also an important data resource in user modeling and adaptive systems. Basically speaking, the growing number of data resources about a user allow more valid inferences and much more contextualized interactions between users and adaptive systems.

The enrichment of information items with significant meta information enables the personalization and customization of information offers. By requesting user preferences ,different user profiles can be built up to facilitate information filtering according to the user's needs. Besides information presentation, the system can provide recommendations to the visitor regarding context. These recommended exhibition objects may attract the visitor's attention by emitting an attractor cue (so-called “prompting” of the user). On the basis of meaningful user profiles, several adaptive strategies can be applied to guide the visitor to the Web museum on a specific (strictly predefined or context-aware adapted) tour.

In our approach, the personalization process is divided into four steps: information collection, modeling, controlling and rendering. Each step fulfils a certain role within the user modeling process. The next subsections describe these modules in more detail.

Information Collection

A network of sensors is placed in the environment and connected to variable parameters of the domain. These sensors are used for recognizing changes within the environment, and especially for the perception of the user's interaction with this environment. An observation module receives all incoming events sent by the Web-Server. These event descriptions are pushed into a database. Thus, an event history for every visitor is saved, and an implicit user profile is recorded.

Modeling

By the means of statistical models, the implicit user profile already allows the deduction of valuable information that can be used for standard adaptation activity (e.g. the more time the visitor spends with the art exhibit, the more s/he likes it). This deducted information builds a behavior model of the visitor. The behavior model can be treated like explicit representation of implicit user feedback and may be consulted to draw an assumption about the user's interest in a specific object. It is planned to gain more significant information relating to the behavior of the user by implementing different machine learning and data mining algorithms to extract semantically enriched information.

In our user modeling approach for a museum environment, we chose to employ our above mentioned adapted information-brokering tool for modeling user preferences. Parts of the domain model of the museum environment are mapped to an ontology model (as shown in Figure 3). The users specify their preferences by simply selecting topics they are interested in from the displayed categories. From these requests different visitor profiles can be built up and stored within the tool. Based on these user selections and on significant meta data describing information items, the personalization process performs a pre-selection and pre-filtering of information offers and customizes according to the users' needs (cf. Pazzani & Billsus, 1997).

The utilization of stereotypes is common in adaptive systems. With the aid of stereotypes, the personalization engine defines the observation type of a visitor, and thus, the system is able to adapt the scenery accordingly. Every stereotype causes a different presentation and different strategies (cf. next subsection). These stereotypes are also part of the ontology model of the mentioned information-brokering tool. At the moment, the visitor's classification into one stereotype is done manually by that visitor and cannot be changed automatically during runtime. Additionally, the personalization engine is not able to perform an automated clustering of user profiles and derive new stereotypes from this process.

Controlling

Meaningful user profiles accurately document visitor activity within the museum and can be exploited to adapt the environment in order to support the visitor, to provide personalized information or to invest the exhibition with an even more artistic touch. Therefore, a controlling component is necessary to decide what consequences must occur if certain conditions in the visitor's environment and in the individual user model configuration appear together. Based on these information sources ,the control layer assembles a sequence of commands in order to adjust certain variable properties of the environment. Thus, different sequences of commands lead to different kinds of information presentation. This adjustment of environmental parameters is used for realizing domain independent adaptive or strategic methods, and domain dependant expressive methods. Examples for domain independent adaptation methods on a strategic level are adaptive prompting and adaptive annotation for objects. Expressive methods that seem to be appropriate for the personalization of Web museums are the following:

Adaptation of the presentation

The basis for every kind of adaptation is the presentation of the exhibits with some associated information. Besides the decision about which item is to be shown, the ways presentation can be modified are manifold, and with combinations of these possibilities, a wide range of adaptability is already accomplished.

Adaptation to social context

If visitors are spatially and temporally similar (e.g. two visitors looking at the same exhibit), they may obtain similar information or may be brought together for a chat. Through building such clusters of people, for example, a subsequent discussion about seen objects is possible (cf. Zimmermann, Lorenz & Specht, 2002).

Adaptation to the level of “immersiveness”

Within a museum environment, interest in objects may be expressed by the time a visitor's focus lingers on these objects. The level of interest corresponds to the complexity, the amount, and the style of already received information about one object and is transferred to succeeding objects. If one of these objects complies with the visitor's interests, the presentation style directly steps into the right level of interest, and information items that are classified at the adequate information depth and style are displayed.

Adaptation to movement and perception styles

Several kinds of common behavior can be identified with people “moving” through the environment (e.g. clockwise in real museums). Attractor cues (e.g. sounds, blinking or marked links) emitted from different sources are used to draw the user's attention to certain objects in the environment. Thus, entire tours through the Web museum can be recommended. The selection and dynamic adaptation of tour recommendations can be adjusted to the visitor's stereotypical type of movement and preferred perception style.

Rendering

Rendering means handling the connection back to the domain. This engine translates the assembled sequence of domain-independent commands into domain-dependent commands. The implemented domain-dependent methods directly change variable parameters of the domain (i.e. content of a HTML page) according to the user's behavior. Thus, the decisions taken by the controlling component are to be mapped to real world actions.

Conclusion

Information brokers mediate between information demands and information offers. The information items to be brokered in the LISTEN system are sound pieces the users experience during their movements through everyday environments. The use of an information brokering tool facilitates the management and distribution of a huge amount of information within this domain. Through mapping domain properties to an ontology model, we benefit from a better understanding of the domain's complexity. In addition, the ontology model, in combination with the enrichment of information items with meta data descriptions, enables a personalized presentation of audio information.

In this contribution we presented a concept for reusing our experience, gained during the application of the information brokering tool within audio-augmented environments, in the context of a Web museum. We followed an overlay model approach that builds on and extends the LISTEN methodology in three ways: tracking, information item modeling ,and domain modeling. The information brokering tool supports the adaptation of these three concepts, so that an electronic museum may take advantage of the benefits.

References

Dey, A.K. & G.D.Abowd (2000). Towards a Better Understanding of Context and Context-Awareness. In the 2000 Conference on Human Factors in Computing Systems (CHI 2000): Workshop on The What, Who, Where, When, and How of Context-Awareness. Hague (Netherlands)

Eckel, G. (2001). Immersive Audio-Augmented Environments. In Proceedings of the 8th Biennial Symposium on Arts and Technology. Connecticut College, New London, CT.

Gossmann, J. & M. Specht (2001). Location Models for Augmented Environments. In the Workshop Proceedings of Location Modelling for Ubiquitous Computing, Ubicomp, 94-99.

Gross, T. & M. Specht (2001). Awareness in Context-Aware Information Systems. In Oberquelle H., R. Oppermann & J. Krause (Eds.) Mensch & Computer - 1. Fachübergreifende Konferenz. Bad Honnef (Germany). 173-182.

Jarke, M., R. Klemke, & A. Nick (2001). Broker's Lounge - an Environment for Multi-Dimensional User-Adaptive Knowledge Management. In HICSS-34: 34th Hawaii International Conference on System Sciences, Maui, Hawaii.

Klemke, R. (2002). Modelling Context in Information Brokering Processes. PhD Thesis, RWTH Aachen (Germany).

Klemke, R. & J. Koenemann. (1999). Supporting Information Brokers with an Organisational Memory. In 5. Deutsche Tagung Wissensbasierte Systeme - Bilanz und Perspektiven, Workshop Wissensmanagement und Organisational Memory (XPS-99). Würzburg (Germany).

Pazzani, M.J. & D. Billsus. (1997). Learning and Revising User Profiles: The Identification of Interesting Web Sites. Machine Learning 27. 313-331.

Shilit, B.N., N.I. Adams, & R. Want,. (1994). Context-Aware Computing Applications. In Proceedings of the Workshop on Mobile Computing Systems and Applications. IEEE Computer Society, Santa Cruz, CA. 85-90.

Tramberend, H. (1999). Avango: A Distributed Virtual Reality Framework. IEEE Virtual Reality Conference. Houston, Texas, USA.

Unnützer, P. (2001), LISTEN im Kunstmuseum Bonn, KUNSTFORUM International. Vol. 155. 469-470.

Zimmermann, A., Lorenz, A. & M. Specht (2002). Reasoning From Contexts. In Henze,N. (Ed.) Personalization for the Mobile World: Workshop Proceedings on Adaptivity and User Modeling in Interactive Systems (ABIS). Hannover (Germany). 114-120.