Experiencing Art on the Web with Virtual Companions

Ido A. Iurgel, ZGDV e. V. (Computer Graphics Center), Germany

http://www.zgdv.de/zgdv/departments/z5/index_html_en

Abstract

Experiencing artworks in the Web aggravates the problem of the absence of any historical, cultural and social context, because the Web is a veritable nowhere. This paper stresses the importance of a social and narrative access to art, and presents an interactive group of virtual characters as an alleviation of this problem. The benefits and overall concepts of this approach are examined. Particularly, the importance of guiding the virtual characters to establish close emotional relations with the user for achieving narrative immersion is stressed, and ways of accomplishing this task are sketched.

Keywords: Virtual Characters, Hybrid Discussion Group, Interactive Storytelling, Emotional Computing, Social Computing

Introduction

Viewing artworks on the Web is an experience absent of any historical, cultural, or social context. While this problem is well known to traditional museums, the Web, as a veritable nowhere, aggravates the situation. Consider someone looking at a Byzantine icon at a Web museum with a home monitor. A visitor — henceforth the "user"- not very well acquainted with the subject is likely to miss much of the meaning of the artwork, and the art experience will be shallow and uninteresting.

It is certainly helpful to offer some additional written explanation or even video material, but still, the experience remains intellectual and solitary, rather than emotional and cultural enrichment. Additionally, it is not easy to motivate the visitor to study the additional material, instead of clicking away after some superficial glance.

What can be done to provide a more involving and enriching art experience to the user? According to the European hermeneutical tradition, understanding of an art work is inherently a social, dialogic and emotional process (Cf. Gadamer, 1993; Heidegger, 1973). There is no absolute meaning, but only restricted perspectives that belong to a whole live world. In order to understand a Byzantine icon, a user must understand the artists who made it and the worship of the believers, their way of living, their feelings, and their convictions. This is not simply a matter of increasing a person's knowledge, but to fuse his own horizon with the horizons of the people to whom the icon had an essential meaning. This is not only an intellectual but also an emotional process of feeling empathy and identifying with particular individual motivations and biographic experiences.

At this point, narration enters the scene as a means of promoting identification, empathy and understanding. Through stories, it is possible to boost the fusion of horizons in art experience. Examples of this process in traditional media are abundant, e.g. historical novels or films.

A new approach is "interactive storytelling". Interactive storytelling is the discipline that deals with the run time creation and adaptation of stories, observing emergent constraints like global duration of the interaction or user choices and interests. It aims at combining the narrative immersion of traditional story telling with the advantages of user behaviour adaptation, employing, for example, educational strategies.

In this paper, I will present a narration driven, interactive group of virtual characters with whom the user can fraternize and discuss each other's art experience and perspectives. This group allows reconstituting the contingent meanings of art and regaining the missing emotional and existential perspectives on it by preserving the narrative horizons. At the same time, the group involves the user through interaction and adaptation.

Figure 1 — Two virtual characters speaking about a religious icon

This interactive group is being developed in the project Art-E-Fact (Cf. www.art-e-fact.org ). This EU-funded project, a collaboration of several institutions of four European countries, aims at developing generic tools for artists to create dialogic, narrative worlds inhabited by personality-rich virtual characters. The interaction with these worlds is enhanced by specialized utilities like exploration tools that allow the user to uncover different painting layers. Though the project does not aim specifically at Web based use, incorporating an internet version poses no technical problem. A VRML viewer, together with some small plug-in and a freely available speech synthesizer, are the only technical necessities. The initial demonstrator deals exactly with a museum presentation of the Byzantine icons mentioned above (Cf. www.monastery-artdiagnosis.gr for a link to one partner).

The next section gives an overview of previous work. Then, I will present the overall architecture of the system. After that, I will explain the benefits and some ideas behind the project. I will stress especially the importance of automating the process of creating social and emotional bonds with the virtual characters to foster the experience of narrative immersion.

Related work

The importance of narration for the structuring of experiences is also stressed in psychology, e.g. in the work of Shank and Abelson (1995) and Graesser and Ottati (1995).

André et al. (2000) developed a non-interactive presentation group without narrative structures. Their emphasis lies in natural language generation independent of personality traits of the virtual participants. Mateas and Stern (2000) are developing a very promising system for interactive storytelling. In contrast to our approach, they do not aim at education and presentation, which is probably the reason for the different architectonic and conceptual details, since there is no need to present information in their system, and therefore a less centrally guided approach is possible.

Rickel et al. (2002) are engaged in an ambitious project that comprises natural language interaction, emotional and social simulation, but not information presentation and narration. A. Paiva, I. Machado and others have developed various interactive applications with storytelling and/or tutorial components (cf. Machado et al., 2001, Paiva, 2001), though none of these systems uses interactive discussion for multifaceted edutainment.

Another project in which our own group is involved is "Geist" (cf. www.tourgeist.de), where interactive storytelling is used, but without any natural language interaction , and "Virtual Human", which aims at the creation of a narrative tutoring system with talking virtual characters (cf. www.virtual-human.org).

Architecture of the System

Figure 2 — The basic architecture

The basic architecture is depicted in figure 2. The information process is centralised and hierarchic, the flow of control starting at a story engine and passing through a scene engine to the dialogue manager or directly to the scene player. The player executes the scripts delivered either by the scene engine or by the dialogue manager and controls the partially autonomous virtual characters and objects. The current architecture separates strictly interactive and non-interactive episodes. In an interactive episode, the user can talk to the virtual characters using a keyboard, and the dialogue manager is responsible for the generation and distribution of adequate answers. In the case of a non-interactive episode, the scene engine controls the virtual characters directly. Non-interactive episodes are fixed sequences authored in advance, much like the dialogues in a traditional theatre play.

Story Engine

Our basic approach to the automated generation of stories is to use a central narration component, called "story engine", that combines and adapts prefabricated minimal story components during run time, according to rules e.g. of narration or pedagogy, and to the interaction results. Those rules are stored as story models. The minimal story components are called "scenes". For details of this process, compare Braun (2002), Spierling et al. (2002). The scene engine plays the scenes that were chosen and adapted by the story engine controlling the subordinate modules.

Dialogue Manager

The dialogue manager is granted control by the story engine if an interactive episode is part of a scene. It is centralized in order to allow using distributed reactions, where, for example, two virtual characters mention two different aspects of a problem as an answer to some question of the user. Its output is a script to the dialogue manager. The script may consist of a single utterance or of a series of commands to the subordinate modules. Currently, the dialogue manager relies on a modified version of the chatbot ALICE (Cf. www.alicebot.org) which makes use of a "natural languagee processing database" (NLP-DB). The use of ALICE is only a provisory solution until some more powerful system is developed.

Scene Player

The scene player executes scene scripts. The scripts control the appearance of the characters and their behaviour at varying levels of abstraction, and govern the objects in the scene; for example, the appearance of icons or the starting of films.

Object Manager

The object manager is controlled directly by the scene engine. It manages the display of the objects of the scene; for instance, the appearance and disappearance of the artworks under discussion.

Virtual Characters

The virtual characters are responsible for the generation of consistent behaviour according to their personality traits and situation appraisal. This behaviour includes turn-taking signals and conversational feedback as well as emotional reactions and goal management. The virtual characters are composed of a mind called agentive person, a simulation of body processes called bodily person, and a graphics output module called manifest person. The parameters of these personality aspects work together to create the final visible personality of the virtual character. The prescriptions may vary in their level of abstraction, ranging from detailed commands of movements to mere annotations in the sentences to say, informing the characters about the pragmatic meaning of the sentences. In the last case, every detail of the performance is in the charge of the virtual characters. Thus, their function is analogous to actors in a theatre play, interpreting with leeway the instructions of a director according to their characters and situation awareness. For more details, cf. Iurgel 2002.

One of the central research goals of the project consists in understanding how to combine story elements prepared in advance with less guided, highly interactive episodes. Both aspects are necessary in interactive storytelling, since only the use of prepared dialogues and instructions guarantee an interesting and consistent storyline, and only the introduction of free interaction enables user adaptation and involvement.

As in any story, the main responsibility for success and suspense lies in the hand of an author. Story creation without an author — "emergent narrative"— is not likely to provide the intended quality of emotional experience. For this reason, in the project "art-e-fact", artists are involved in every step of the development. Thus, suitable authoring tools for interactive storytelling are vital and one of the main challenges of the project (Cf. Scheider, 2003).Ways of Fostering Art Understading. Fostering art understanding through the use of a virtual conversation group.

In this section, I want to elaborate more on some aspects of using a virtual conversation group to promote the understanding of art works. In order to concretize the discussion, I introduce two scenarios:

1. Distant horizon scenario.

In this scenario, the virtual characters are contemporaries of the human participant and maintain themselves at a distant perspective from the art work, though it has an important meaning for their lives. For example, a scientist who analyses materials and colours, and an historian who knows about cultural context, might be present. This scenario is a good starting point for the participant, because it is easy to understand the virtual characters, and the interest of the characters in the icon can spread to the user.

2. Near horizon scenario.

Here, the virtual characters live in the past and share the religious fervor that once accompanied the icon. A monk and a mother of a sick child might be present. This constellation must be introduced more carefully, but promises a more intense experience, involving more central feelings like suffering, compassion, hope, and so on.

Figure 3 — The virtual characters as talking heads

Keeping these scenarios in mind, I will now mention some advantages of using a conversation group to present the art work on the Web:

1. Perspective. Every character can support a definite perspective, e.g. a historian and a scientist. This allows elucidating the theme from different, possibly incommensurable points of view. Personality trait differences have here the important function of establishing a link between opinion and emotion. For example, a reserved and sceptical character is suitable for the role of a scientist, but probably not for the horizons of a religious person.

2. Dialogic deployment. The use of a group of characters facilitates the dialogic deployment of thoughts, because insightful sequences of questions, contributions and answers can be prepared in advance. Even if the user is included in the deployment, the presence of a group is technically and conceptually advantageous, because one virtual character can provide the keyword needed by the other to continue the dialogic developments.

3. Narration. It would be very difficult to maintain a narrative structure using only a single virtual character because of the unpredictability of user behaviour. If, for example, the author prepares a narrative structure where the character necessarily has to speak about his early childhood experiences, but the user refuses to keep at it, the story line is likely to fail. This cannot happen if there is another character willing to listen in the place of the user.

4. Natural language understanding. Communicating with a single virtual character is difficult simply because it very often won't understand the user. The use of a group alleviates the problem, because it is quite natural that some contributions do not receive a proper answer in a discussion group. Shortcomings of natural language processing do not lead to a breakdown in the conversation.

5. Social processes. Social and group processes are among the strongest elicitors of emotional arousal. According to Kemper (1991), there are only two basic dimensions that govern the arousing of emotions in a social relationship; namely, "status"and "power". Those dimensions can be exploited to generate emotionally thrilling plots involving the virtual characters and possibly including the user. For example, the user might be exposed to a planned power change in the course of the narration, his influence as a group member increasing or decreasing depending on his behaviour and on the plot. This can be achieved by a sequence of scenes in which the characters initially disregard the suggestions of the user; in the course of the interaction, his contributions would be "recognized"by the virtual characters as important and increasingly mandatory.

The nowadays prevalent use of anthropomorphic agents on the Web considers only the usage of a single character. I argued that this is a disadvantageous restriction that impedes narration and weakens the interactive experience. "Interactivity"in interactive storytelling

A non-interactive example for the combination of story and presentation is "Sophie's World" (Gaarder, 1996), where the adventures of a girl serve as a background for explanations in the history of philosophy. Here, the figure of Sophie is used to promote identification, gradual presentation of knowledge, and emotional involvement in the theme.

Certainly, stories in interactive storytelling currently are at risk of not reaching the narrative quality level of non-interactive media where every detail can be controlled by the author or director. Because of this, understanding the role of interactivity in interactive storytelling remains an essential and still not satisfactorily accomplished task. What are the benefits of introducing interactivity into storytelling? Of course, the conflicting goals of the application provide good reasons, even if it should turn out that the intensity of narrative immersion is affected. Some examples are:

1. Navigation. In an edutainment application, it is important that the user can navigate; for example, in order to go back to themes already discussed.

2. Reviewing and testing. Interaction is necessary for feedback about some educational goal, e.g. the acquired knowledge could be tested.

3. Joy of expression.Expressing personal opinions and observing the virtual character's reactions is a joyful experience.

4. Exploration. The user can be engaged in explorative learning strategies, e.g. a virtual character can gradually guide the user to discover more details of an icon.

However, the starting point of this paper was the social and emotional nature of art understanding and the importance of narrative immersion in this context. The question at hand is whether it is possible to use interactivity to intensify the experience of narrative immersion. Traditionally, emotional involvement in a story in achieved by identification with a protagonist. The prospect of placing the user in the protagonist's position in interactive storytelling might be thrilling, but would be opposed to the current goal of rendering another person's emotional and cultural perspective understandable — aside from the technical constraints.

Therefore, I follow the indirect path of creating planned emotional proximity — henceforth "friendship"— between the user and a chosen protagonist virtual character. The reason for this is that friendship is known to boost identification and empathy, and thus, should increase narrative immersion.

An example is a virtual character that is emotionally touched by discovering, in the course of an adventure, the symbolic meaning of a religious icon. Because of the establishment of some sort of friendship with the character, his emotional reaction should also touch the user, promoting his understanding and appreciation of the icon and of its religious context.

Improving Art Understanding Through Friendship With Your Virtual Companion

Previously, I argued that it is possible to foster the narrative immersion through the establishment of some sort of friendship between the user and the virtual character, thus intensifying the identification with the character and his emotions. In pedagogy, those phenomena of learning facilitation under the influence of a liked person fall under the broad and well studied heading of "social learning" (Bandura, 1977, is a classical source). In other words, the idea presented here is that "social learning"is also useful when the main channel is a narration. The research issue under way, now, is to automate the creation of friendship between character and user. In the present context, the relevant psychological factors that determine friendship ("attraction") are the following (Cf. Fehr, 1996, for an overview of those factors):

1. Similarity. There is plenty of empirical evidence that similar character traits and opinions are favourable to the establishment of close emotional relationships.

2. Reciprocity.Basically, reciprocity means that we tend to like someone who like us. This dimension is particularly sensitive to temporal factors. For example, increasing sympathy over time is more favourable to the establishment of emotional proximity than decreasing it, even if the final sympathy level remains the same.

3. Self-Disclosure. Self-disclosure is an important indicator of friendship. A person who discloses his feelings to me probably likes me.

4. Maximized gains in social exchange. Praise is also very favourable for the establishment of friendship. For example, it is advantageous for me if another person praises my doings, thus raising my status and my self-esteem.

5. Common enemy. The existence of a common enemy is also favourable for the establishment of camaraderie as a variety of friendship.

Every one of those factors can be incorporated into a story-telling system. For example, the virtual character chosen for identification can be made to mimic similarity ("You do not like this icon? Nor do I!"); reciprocity ("That's fine that you are enjoying our conversation! I am also learning very much from your contributions!"); or self-disclosure ("I am full of anguish when I think that my answers might not match your questions.") This can already be achieved by integrating an appropriate plot with fairly simple extensions of ALICE, which allow the story engine to guide this chatbot. More powerful methods for interactive episodes, based on the state-update-theory of dialogue moves, are under investigation.

The details and distinctions are tricky and the object of current research. Therefore, the next section will elucidate more the process of automated creation of friendship in interactive storytelling by presenting an example. Example

In this example, it is assumed that an appropriate story is generated and envelops the scenes under discussion. An abstract initial scheme for the automated establishment of friendship is depicted in figure 4. It is assumed that "discovering similarities", "taking sides", "self-disclosure", and "conflict resolution"are scenes controlled by the story engine according to friendship establishment models.

Figure 4 — Possible steps of automated friendship forming

The creation of this model is a process that involves computer science, psychology, artistic intuition and narrative skills. The scene "taking sides"is included because it allows the virtual character to show how his emotional well-being depends on the behaviour of the user, thus expressing the close friendship he presumes to have with the user. At the same time, the scene "taking sides"creates a common enemy. Short interaction examples follow, in which the system tries to establish friendship between Virtual Character 1 and the user: