1. Motivation: Authenticity Whenever Possible
Conveying cultural heritage is undoubtedly one major and publicly the most recognized task of museums. This task is often achieved through exposition of cultural artifacts. Generally, museums try to exhibit originals – authentic artifacts of history. Explanatory media such as textual descriptions are arranged with the artifact in order to mediate the role, functionality and context of the artifact in a culture.
However, this strategy is not always possible. This becomes apparent when original artifacts do not exist any more (or never existed). But even if they do, a variety of reasons prevent objects from being exhibited or fully exposed: fragile rarities might not always be suitable for visitor hands-on experience. Objects may be sensitive to the environmental circumstances of the exhibition or be simply too fragile to be used on a regular basis.
Exhibitions may pose danger not only for the exhibited objects but also for visitors: for example, Marie Curie’s workspace that led her to insights about radioactivity is unlikely to be exposed in its original form.
Besides, even if it is feasible to expose an artifact, it may not be comprehensible just by letting visitors play with it. Sometimes hands-on experience does not provide pedagogical benefit. A device may require specific environmental conditions to work – for example, optical devices may need special lighting conditions, astronomical devices may require specific sky constellations, etc. An artifact may simply be too complex to be understood just by randomly turning a knob. Or its effect may be unnoticeable directly.
In all these cases, Virtual Reality (VR) and Augmented Reality (AR) techniques offer significant benefit. Robust and harmless stand-in objects can avoid danger to both artifacts and visitors – a virtual augmentation substitutes for breakable or dangerous elements. In addition, since virtual augmentations are not strictly bound to physical laws, they may expose or exaggerate aspects that could otherwise not be seen and understood.
This paper is structured as follows: after a brief introduction of VR and AR concepts and an overview of current technological development in these fields (section 2), we discuss the application of AR in museums in section 3. We present two examples of bringing AR to museums ,g different strategies: the Fraunhofer Spektralapparat is a “bottom-up” example that tries to leverage existing and robust technology to an AR level, whereas the Spinnstube® follows the opposite approach: we try to raise the robustness and simplicity of a fully featured AR system to a level that makes it applicable to the museum field. In the following section, results are discussed first. We conclude with proposing to merge the two different approaches.
2. Technology Development
Since the late 1980s, Virtual Reality has attempted to bring computer graphics to a level that allows immersion – virtual worlds should look like and behave similarly to the real world so that users experience being physically there. The main techniques to achieve this effect are 3D graphics and direct interaction. While VR remains a vivid research topic, it has not yet reached a broad mass market in spite of blockbuster movies like Avatar where no visitor interaction is possible. One reason is the high technical complexity of fully immersive environments and the resulting high effort and costs to implement.
Nevertheless, many insights from VR research have made their way to consumers in limited but robust and cost-effective forms. Current 3D computer games show interactivity and photo-realistic graphics, but sacrifice stereo graphics, user tracking and large-field of view displays. Despite recent approaches to bringing stereo vision and head tracking to consumer games via technology such as the NVIDIA 3D Vision kit (NVIDIA, 2010) and the NaturalPoint TrackIR head tracking system (NaturalPoint, 2010), these features have not yet diffused into the mainstream market. An even more limited incarnation is 3D cinema that lacks both interactivity and correct spatial representation – only center seats have the correct perspective representation.
These recent developments were made possible by former VR research. The split of high-end VR in research from limited incarnations for consumers has led to interrelations between these paths: availability of consumer electronics has significantly reduced prices for the required technology, also lowering costs for high-end installations. Consumer VR has brought the immersion experience closer to a broad audience. Users are now more familiar with artificial interactive 3D worlds, and show increasing acceptance of more advanced installations, even driving the demand to bring richer virtual experiences. Museums and science centers are good candidates for closing the gap between research labs and consumer electronics. By embracing technology that is mature enough to leave research labs, but has not yet entered private living rooms (due to cost, market development or other limitations such as space constraints), museums can attract visitors through novel and rich media experiences.
Augmented Reality is related to VR to a large extent. On one hand, AR extends VR concepts; on the other hand, it can be seen as a counter concept to VR. Instead of creating virtual worlds, AR just augments existing reality. From a technology perspective, AR inherits all its technical complexities from VR. In addition, virtuality has to be related to the physical world, adding the challenge of mapping both half-worlds to each other. As a consequence, AR applications have not yet spread to consumer markets in significant numbers. Recently, first attempts have emerged; i.e. in mobile phones, smartphones, and computer game markets.
A plethora of Mixed Reality (MR) and Augmented Reality approaches exists, all with a target of combining physical and virtual worlds – either the real world is supplemented with virtual aspects, or vice versa. Augmented Reality itself can be categorized by its degree of visual realism, types of objects that can be augmented, interaction possibilities, and the techniques used for combination of both half words.
The aforementioned migration of VR techniques to consumer markets has boosted technology development, which has in turn led to wide and cheaper availability of base technologies: visualizations that require dedicated 3D graphics workstations can now be achieved using consumer graphics cards in desktop PCs. The challenge has moved from building technologies to combining existing technology in a smart way.
However, some technological obstacles still exist. Two of them currently still hinder broad market acceptance of AR:
- Most stereoscopic vision techniques still require users to wear glasses in order to get a spatial impression. First glasses-free autostereoscopic products are available. Affordable and robust solutions will presumably become available within the next decade.
- Tracking of real world features requires advanced computer vision techniques that are not yet stable enough for a broad market. A current workaround is either to assist computer vision by machine-detectable markers or to supplement vision with non-optical (i.e. mechanical) tracking. We expect computer vision algorithms to mature in the coming years.
In spite of the aforementioned obstacles, Augmented Reality seems to be the next great thing for museums.
Thanks to recent advances in optical technologies and computer vision methods, new accurate 3D scanning devices have appeared on the market, enabling the creation of digital copies from real objects. These technologies have naturally caught the attention of museums, galleries, archives, and collections whose needs of archiving tools are crucial, in terms of both preservation and communication. In principle, i.e. ignoring costs, a whole museum collection can be digitized.
Unfortunately, 3D digitization devices still remain expensive. Lengthy post-processing times, in addition to advanced technical skills in users, are generally required before a faithful and exploitable digital copy can be obtained. Therefore, despite the interest, these technologies are not yet widespread in cultural fields, and museums still prefer to keep on working with standard digital photography for their archiving purposes.
3. AR in Museums
Museums nowadays are willing to participate in research projects or experiments, e.g. (AMIRE, 2004), that temporarily introduce augmented reality technology into their exhibitions and museum halls. In most of the cases however, the project’s results do not make it to the permanent exhibitions.
Fig 1: AR museum guide in the Guggenheim Museum, Bilbao [AMIRE]
Looking at them from a museum visitor’s perspective, bringing AR to the museums is a challenging endeavour. There is not much time to experience AR in a museum, typically 4–5 minutes at the most per exhibit. This means that there is very little time for teaching a visitor how to experience an AR-based exhibit. Most museum visitors do not like to read lengthy instructions. It is imperative, therefore, that the interface be easy and fun to learn. It has to be simple. There is no time to strap on cumbersome and fragile tracking devices or display systems that require complex adjustment. Finally, the AR experience has to be enjoyable by visitors of all ages and both genders.
One of the rare exceptions of AR in museums is the Virtual Showcase on photosynthesis in the Deutsche Museum, Bonn, on display between 2002 and 2009 (Bimber et al, 2001):
The Virtual Showcase is a new Virtual and Augmented Reality display device that has the same form factor as a real showcase traditionally used for museum exhibits. The Virtual Showcases consist of two main parts: the actual showcase covered with half-silvered mirrors and a graphics display underneath. This configuration allows the three-dimensional graphical augmentation of real scientific and cultural artifacts placed inside the Virtual Showcase. Another interesting aspect of the system is its support for four simultaneously tracked users looking at the Virtual Showcase from different sides. This feature allows the collaborative exploration of artifacts shown in the Virtual Showcase.
Fig 2: Virtual Showcase in the Deutsche Museum, Bonn
In the Louvre – DNP Museum Lab visitors are allowed to receive both audio and visual information intuitively as they make their way around a presentation:
The visitor is given a portable guidance device equipped with a camera which makes it possible to add images to the traditional audio guide system. Six specific parts of the presentation area are linked to the AR system; when the visitor films these places with the camera, a virtual character appears on the captured image, and gives details about the visit.
(Louvre, 2008),
Fig 3: “A Future for the Past” exhibition in the Allard Pierson Museum, Amsterdam
The Allard Pierson Museum, the archaeological museum of the University of Amsterdam, chose to use Augmented Reality during its latest exhibition, "A Future for the Past" (Allard Pierson Museum, 2010). Two Augmented Reality applications are presented on a movable screen: Aavirtual reconstruction of Satricum, and an annotated landscape on a 1855 photograph of Forum Romanum in Rome.
Fig 4: Augmented Dinosaur in the National Museum of Nature and Science, Tokyo (Kondo et al, 2007)
Museums need efficient, cost effective and simple methods of creating AR based exhibits and exhibitions based on their collection of 3D models. At the same time, an AR based exhibit must provide museum visitors with an intuitive human-computer interface based on well-known metaphors. Users should be able to interact with an AR based exhibit as easily and naturally as they can interact with museum objects in a real world. Everything that does not meet these criteria will not be understood and, therefore, will not be generally accepted in a museum context.
4. A Simple Approach: The Virtual Fraunhofer Spektralapparat at the Deutsche Museum Munich
The Deutsches Museum in Munich is the most important museum in Germany, with 1.5 million visitors per year. It possesses over 100 000 objects from the fields of science and technology. The large number of valuable original exhibits makes the Deutsches Museum one of the most important museums of science and technology anywhere in the world. The collections are not restricted to any specialized range of topics: they include objects from mining to atomic physics, from the Altamira cave to a magnified model of a human cell. They extend from the Stone Age to the present time. Collecting historically significant objects is still one of the Museum’s central tasks, and so the stock is constantly growing. About a quarter of the objects are on exhibition (DM, 2010).
One of the objects on display is the Fraunhofer Spektralapparat. The Fraunhofer Gesellschaft was named after Joseph von Fraunhofer, a German optician (1787-1826). He is known for the discovery of the dark absorption lines known as Fraunhofer lines in the sun's spectrum, and for making excellent optical glass and achromatic telescope objectives. In 1814, Fraunhofer invented the spectroscope, later on called the Fraunhofer Spektralapparat, and discovered 574 dark lines appearing in the solar spectrum. These lines are still called Fraunhofer lines in his honor.
We developed a procedure for the reconstruction of digital copies based on the data that museums are nowadays really skilled at acquiring; namely, photographs. It was applied for the first time in the Deutsche Museum Munich to create the Virtual Fraunhofer Spektralapparat (see figure 5). The principle is to create, from only this information, a realistic 3D model with texture information providing a good impression of the materials the object is made of.
Fig 5: The Virtual Fraunhofer Spektralapparat at the Deutsche Museum Munich
Here the authentic and valuable artifact (right) is exhibited side-by-side with its virtual counterpart on the left side. Visitors have the possibility to interact with the Virtual Fraunhofer Spektralapparat. They understand its function and the importance of the invention by Joseph von Fraunhofer much better compared to just looking at it in a vitrine. This is not an immersive stereoscopic 3D display; i.e. no tracking, no glasses, no separate interaction devices. Instead of technical superposition, augmentation takes place through side-by-side presentation of the original artefact next to its virtual counterpart. The focus is on robustness, simplicity, ease-of-use, and finally hands-on visitor experience, with superimposed explanations related to the visualization of light, eliminating original limitations of use; e.g. restrictions caused by the original device’s requirement of specific sun constellations to work properly.
5. An Advanced Approach: Spinnstube®
In contrast to approaches that sacrifice vision or interaction features to circumvent the inherent complexity of Augmented Reality, the Spinnstube® implements a full featured Augmented Reality workspace. At its core, a semi-transparent mirror merges the real world with its virtual augmentation. The augmentation itself is stereoscopic, resulting in three-dimensional images that merge with the physical world on the workspace of the user. This approach, called see-through AR, is distinct to video-based AR, in which the real world is captured by a camera, the worlds are combined digitally, and both are presented on a video display. Thanks to its technical simplicity, video-based AR is the currently predominant form; i.e. AR applications based on mobile devices are video-based.
See-through AR is more complex because the user’s eyes, the world and the mirror have to be tracked individually and merged to form a convincing superimposition of real and virtual world (figure 7). Its main advantage is the ability of the user to see the real world naturally and directly. This also affects interaction, where users can interact with real objects in a common way, without the indirection of a video screen.
The Spinnstube® was developed primarily for education in a school context. Within the EC research project ARiSE (2006 – 2008, see ARiSE, 2010), AR workspaces were built, and interaction concepts were developed and evaluated in three different educational settings.
Fig 6: The Spinnstube® used during the ARiSE research project
Figure 6 shows the setup of the original Spinnstube® device. For affordability, low cost professional or consumer electronics components were chosen. The complex construction is mainly required to incorporate the light path of a stereoscopic video projector, one of the few affordable stereo vision solutions available in 2006. Current constructions can be significantly simplified using novel display techniques (see section 6). Despite its construction, the device is mobile and independent of the workplace. Robustness requirements for everyday use were deliberately neglected for the prototype in favor of open access for experimentation and development.
Fig 7: The variety of light and camera tracking paths within the Spinnstube®
The AR setup was tested in multiple educational scenarios for different subjects. For example, a chemistry learning unit utilized colored rubber balls as surrogates for atoms. Learners could assign a chemical element to a specific ball color, in which case the balls were augmented with electrons floating around them. When learners brought two atoms close to each other, electrons interacted – this way, learners could build molecules with their own hands and see submicroscopic effects not visible in the physical world (figure 8).
Fig 8: Chemistry in Augmented Reality
6. Evaluation
The Spinnstube® and its accompanying interaction and pedagogical concepts were thoroughly evaluated during three summer schools (Pribeanu et al, 2008) with respect to usability, pedagogical and social interaction aspects, and technology acceptance, according to the Technology Acceptance Model by Davis et al (1989).
Pribeanu highlights a set of positive evaluation results that support AR as a useful aid for understanding abstract and invisible processes as well as for those having special educational needs. Positive results include:
- AR increases students’ motivation to learn
- AR increases vividness of complex or abstract subjects
- Haptical elements of AR support kinesthetic learning
- Users confirmed high perceived usefulness and perceived enjoyment
- Pedagogical evaluation certified that AR has the potential to be an effective pedagogical instrument
Qualitative observations from the three summer schools show high acceptance for AR. In several cases, learners did not want to leave the workspace after evaluation. Students of the third evaluation deliberately spent significant amounts of their spare time in the Spinnstube® because they liked to work in AR.
The main shortcomings identified by Pribeanu include difficulties of getting in and out of the device, inaccuracies concerning the overlay of virtual augmentations over physical objects and inconvenient stereo glasses. These ergonomic problems are mainly related to limitations of underlying technology and the prototypical configuration of the device. They will be addressed in an improved version of the Spinnstube.
Besides an overall positive result for AR, the evaluation showed potential for simplification of the device because anticipated requirements turned out to be irrelevant for the result. For example, the mobility requirement turned out to be less important than expected. In addition, the semi-transparent mirror was designed to be freely movable – in fact, few students used this feature. The identified ergonomic problems, the shift of requirements and new base technologies led to a complete redesign of the Spinnstube®, the Spinnstube VT.
Fig 9: Spinnstube VT design, two-seat configuration
The Spinnstube VT design takes advantage of new 3D-capable flat screen displays, eliminating the need for complex projection light paths and resulting in higher visual quality. Next to the downward-facing screens, only the semi-transparent mirror is exposed to users. Besides causing less trouble to enter and leave the workspace, the system benefits from increased robustness since active components are unreachable. Up to four devices can be grouped around a table to form a shared augmented workspace, enabling collaborative augmented learning scenarios.
The device can either stand around a table, sit on top of a table, or be mounted from the ceiling, in which case no support frame is required. The evolution of the Spinnstube illustrates the disappearance of technology during the maturing process of AR. A prototype of the Spinnstube VT is being built while we write this paper.
The virtual Fraunhofer Spektralapparat (section 4) was installed in November 2009 in a newly opened section of the Deutsches Museum, Munich, the ZNT (“Zentrum Neue Technologien”). No formal evaluation has been undertaken yet, but early informal observations and visitors’ feedback show good acceptance of the device. We intend to perform an evaluation when the new exhibition has settled to normal operation.
7. Conclusion: Current and Future Work
In this paper, we have discussed possible benefits of bringing Augmented Reality technology to museums. We have argued that AR is currently at the frontiers of diffusing from research into application. After examining a small selection of existing AR projects in museums, we have illustrated two different approaches of bringing AR into museums:
- bottom up (Spektralapparat): extending proven and robust installation technology towards augmentation
- top down (Spinnstube): maturing fully featured AR technology to a state that is applicable in everyday life
Following the two approaches mentioned, we have gained enough experience to bring us to say that a convergence of both approaches will happen in the near future so that museums can convey cultural heritage and its legacy with innovative AR-based solutions. We will continue to develop both directions until they meet. Spinnstube® VT will be our next proof of concept.
Acknowledgements
The ARiSE project (2006-2008) was co-funded by the European Commission within the 6th Framework Programme under the contract no. 027039. The Virtual Fraunhofer Spektralapparat was co-funded by the Fraunhofer Headquarters in Munich on occasion of the 60th anniversary of the Fraunhofer Gesellschaft. We would like to thank Andreas Geiger and Gerhard Hartl of the Deutsche Museum Munich and our colleagues Roland Kuck and Frédéric Larue for their co-operation while implementing the Virtual Fraunhofer Spektralapparat.
References
Allard Pierson Museum (2010). Web site of the Allard Pierson Museum, Amsterdam, NL. http://www.allardpiersonmuseum.nl/english/exhibitions/index.html (Consulted Jan 29, 2010).
AMIRE (2004). AMIRE – authoring mixed reality, http://www.amire.net (Consulted Jan 29, 2010).
ARiSE (2010). ARiSE project Web site. http://www.arise-project.org (Consulted Jan 29, 2010).
Bimber, O., B. Fröhlich, D. Schmalstieg, and L.M. Encarnação (2001). „The Virtual Showcase”. IEEE Computer Graphics and Applications, vol. 21, no. 6, pp. 48-55, Nov./Dec. 2001, doi:10.1109/38.963460
Davis, F. D., R.P. Bagozzi, & P.R Warshaw (1989). “User acceptance of computer technology: A comparison of two theoretical models”. Management Science, 35, 982-1003.
DM (2010). Deutsches Museum Web site. http://www.deutsches-museum.de/en/collections/ (Consulted Jan 31, 2010).
Kondo,T., et al, (2007). "Mixed Reality Technology at a Natural History Museum ». In J. Trant and D. Bearman (eds). Museums and the Web 2007: Proceedings. Toronto: Archives & Museum Informatics http://www.archimuse.com/mw2007/papers/kondo/kondo.html (Consulted Jan 31, 2010).
Louvre (2008). Louvre - DNP Museum Lab, http://www.louvre.fr/media/repository/ressources/sources/pdf /src_document_53888_v2_m56577569831210172.pdf (Consulted Jan 29, 2010).
NaturalPoint (2010). NaturalPointTrackIR home page http://www.naturalpoint.com/trackir/ (Consulted Jan 29, 2010).
NVIDIA (2010) NVIDIA 3D Vision Kit product page. http://www.nvidia.com/object/product_GeForce_3D_VisionKit_us.html (Consulted Jan 29, 2010).
Pribeanu, C., V. Lamanauskas and L. Pemberton (2008). Report on the methods and the usability including report on all summer schools. Public deliverable 5.1 of the ARiSE project, 2008. http://www.arise-project.org (Consulted Jan 28, 2010).
