• Register
• Key Dates
• Schedule
• Local Info

April 11-14, 2007
San Francisco, California

Mixed Reality Technology at a Natural History Museum

Introduction

There are actual skeletons of dinosaurs in a natural history museum. However, visitors do not really know what a dinosaur looked like when it was alive; for example, the contour of the body or its skin color. In this research, by using mixed reality (MR) technology with a handheld PC, they can view and reconstruct the flesh of the dinosaur on display in the gallery. In this paper we will describe this system, its contents and the authoring tool.

In the current museum display system, to superimpose an animation on to an exhibit a system using a half mirror, called “fanta-view” or “magic-vision", has been used in the gallery. However, it is not an interactive system. On the other hand, a three-dimensional computer graphics (3DCG) technology used with a personal computer has been gaining popularity in recent years. Because of this, the potential of using interactive 3DCG technology for museum display is currently increasing.

Mixed Reality (MR) is in between the virtual and the real environment (Milgram & Colquhoun, 1999). MR is similar to Virtual Reality (VR) technology except that it can merge interactive 3DCG with the real world. In other words, although the majority of VR contents are made from only computer graphics, MR contents are made from computer graphics in the real world. In this research, the real world is a museum gallery or an exhibit. To use both virtual and real is important in the educational field at a museum because the technology can overlap layers of information such as comments or interactive explanations.

Examples of MR applications for museums, include, for example, a user who wears a head-mounted display (HMD) with a camera can see virtual buildings reproduced at the actual archaeological site of Olympia (Stricker & Kettenbach, 2001) and can see an animation of people living at the archaeological areas of Pompei (Papagiannakis et al., 2005). The virtual showcase is a museum display showcase that combined stereo 3DCG with a real exhibit using half mirrors (Bimber et al., 2001). A handheld augmented reality museum guide is a mixed reality system using a PDA (Schmalstieg & Wagner, 2005).

The purpose of this study is to use a museum display to illustrate how practical and simple it is to use MR technology. The system used will be an MR display system, a content authoring system, and a Web-based 3DCG texture painting system.

A Mixed Reality System

Media Characteristics of Virtual Reality, Web3D and Mixed Reality

In order to apply 3DCG in a museum, there are various possible methods; for example, Virtual Reality, Web3D, and Mixed Reality. These consist of different systems and each system has specific media characteristics. Figures 1-3 present the contents of the same church in France. Figure 1 shows the result of using the immersive VR system called CAVE. The operator enters into a 3m cube, and by using stereoscopic vision, can have the experience of being in the actual church. Figure 2 shows the same building using Web3D, which displays 3DCG on an Internet browser. Although an immersive VR system is more realistic than Web3D, financially Web3D is more practical. Figure 3 shows the exterior of the same church displayed on top of a hand. In an MR environment, the displayed image can be shown on a person’s hand; therefore, the operator can experience an animated image, which is embedded on an actual hand in a normal environment. MR also can be used in the real church. For example, it can show murals and paintings with annotations etc. on the actual ceiling painting.

Fig 1: Immersive VR

Fig 2: Web3D

Fig 3: Mixed Reality

Elemental Technology of Mixed Reality

There are several MR display systems; for example, a half mirror system (Bimber et al., 2001), a video see-through HMD with built-in small video cameras (Uchiyama et al. 2002), and a PDA with a video camera (Schmalstieg & Wagner, 2005) etc. All of these systems are a combination of reality mixed with 3DCG technology.

Moreover, MR technology also has tracking technology in which an animated image can follow a person’s head or hand. In such a tracking method, there are a magnetic position sensors, gyroscopes, etc., to calculate position and inclination by recognizing the images of the two-dimensional markers in the real world – image recognition (Papagiannakis et al., 2005), etc. This two-dimensional marker tracking method was used in this study.

Development and Operating Environment

ARToolKit (Kato & Billinghurst, 1999) was used to develop the contents in this study. It is a software library for creating a mixed reality environment. It uses a two-dimensional marker for tracking. A key feature of using ARToolKit is that no special equipment is needed. Therefore this technology is well suited for educational purposes. The development was done in the following environment:

• OS: Windows 2000 Professional
• Compiler: Microsoft Visual C++6.0
• Graphics library: OpenGL+GLUT
• Multimedia API: Microsoft DirectX 8.0 SDK
• Mixed Reality library: ARToolKit 2.65

The application can be operated on an Intel Pentium III 600MHz or higher with Windows2000 / XP. A high-speed graphic accelerator card works well with OpenGL, and a Web camera or a video camera connection is necessary.

Museum Display System

In this system, using a handheld pc with a camera allows replication of the look of the dinosaur, overlaying the actual skeleton. Figure 4 is a picture from the National Science museum in Tokyo. Figure 5 shows a half-transparent 3DCG with the names of the body parts, all displayed on a handheld PC.

The purpose of this content is not only for children to have a better understanding of what dinosaurs looked like before, but also for children to learn to think scientifically about dinosaurs. Using academic theories, children can modify the skin color. For example, Figure 6 is an example of skin color. This system offers an interactive function which asks the children to choose a specific skin color for a specific purpose.

Fig 4: Configuration of museum display system

Fig 5: Display of handheld PC

Fig 6: Examples of skin color of the dinosaur

An Authoring Tool for Mixed Reality Contents

The MR content that was used in this research project was developed using ARToolKit. Knowledge of C++ and OpenGL is required for this development, and it is difficult for general users to develop their own contents. An authoring tool will eliminate the hassle of traditional C++ programming which needs a lot of time and experience. Then the general user will be able to spend more time developing content instead of programming.

The user-friendly authoring tool that was developed to produce MR content such as the museum display system permits content to be easily developed in a short period of time using a two-dimensional marker or markers.

To create the contents, it is necessary to prepare a 3D object or a 3D animation separately, with 3D modeling software. The fundamental function of the authoring tool makes this 3D object correspond to the selected marker. The number of markers, the position, and the size can be specified, and positioning of the 3D object and zooming are possible. Furthermore, it also allows annotations, sounds, and hyperlinks. Figure 7 shows an authoring process to set a marker and a 3D model.

Fig 7: Authoring tool for mixed reality contents

Authoring Tool Specifications

• Marker position: The number of markers, the position, and the size can be specified.
• 3D Object's file formats: This software supports MQO (Metasequoia) and VRML file formats.
• Annotation: VRML file
• Hyperlink: It can set hyperlinks from an annotation.
• Cut: It can set cuts in a scene.
• Preview
• Pattern file: Pattern files can be generated automatically.
• Marker: It can print real size markers.

Web-Based Painting Tool For Dinosaur's Skin

MR technology can be used not only for museum guides but also for hands-on experience. Currently, a Web-based painting tool for dinosaur's skin is now under development (Figure 8) using Flash, mySQL and PHP. A picture designed by children can be saved on the server. . Children can take pictures of animals, for instance, and use those pictures as the base of developing their own choice of skin textures to be used when viewing the 3D model of the dinosaur skeleton. Then, using MR technology at the museum (Figure 9), those children can see the dinosaur with their choice of skin over its skeleton The purpose of the application is not to create art, but to understand more about science by optimizing the dinosaur skin and by using some filters such as shade and a splash effect,. In the end, a picture will be taken of the student creators with their skin texture choices overlapping the dinosaur and themselves in the picture, and they will be able to receive printed copies of those pictures.

This paint tool under development has the following functions:

select a dinosaur, select the brush thickness, select a solid or blur style brush, transparent brush, loading a picture file, stamping, editing the stamp, undo/redo, save the design and upload to the server. The server has the highest level design ranking functions.

Fig 8: Web-based painting tool for dinosaur's skin

Fig 9: Execution screen using mixed reality

Conclusions And Future Work

In this research project, we have given an example of the use of MR technology at a natural history museum. First, we explained about the media characteristics of MR, VR and Web3D. Secondly, we presented the MR content of a dinosaur exhibit with its museum display. Thirdly, we introduced an authoring tool for MR contents. Finally, we reported on a Web-based painting tool which is under development for MR. In concluding, we propose possible and practical uses that may change current museum display systems by using MR technology. Our aim is to give children in a museum a tool to help them think more scientifically. MR content can change the skin image of a dinosaur, and the painting tool can recreate a virtual animal; MR technology can change the way children learn.

A future version of this museum display system that uses MR technology will be a marker-less system and will have a more intuitive interface. Specifically, we are planning to use a magnetic positioning sensor, a haptic device, and an acceleration sensor such the Nintendo Wii controller as an operating interface. Lastly, we will release shortly an English version of the authoring tool.

Acknowledgements

A part of this study was subsidized by The Ministry of Education, Culture, Sports, Science and Technology, and Japan Society for the Promotion of Science with Grant-in-Aid for Scientific Research (No.16650219 and No.17300284).

References

Bimber, O., B. Fröhlich, D. Schmalstieg, and L.M. Encarnação (2001). “The Virtual Showcase”. IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55.

Kato, H. and M. Billinghurst (1999). “Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System”. In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality ’99 (Iwar'99). San Francisco, 85-94.

Milgram, P., and H. Colquhoun (1999). “A taxonomy of real and virtual world display integration”. In Y. O. H. Tamura (Ed.), Mixed Reality: Merging Real and Virtual Worlds. pp 5-30. Tokyo: Ohmsha / Springer-Verlag.

Papagiannakis,G., H. Kim, and N. Magnenat-Thalmann (2005). “Believability and Presence in Mobile Mixed Reality Environments”. IEEE VR2005 Workshop on Virtuality Structures. February 2005.

Schmalstieg, D. and D. Wagner (2005). “A handheld augmented reality museum guide”. In Proceedings of IADIS International Conference on Mobile Learning 2005.

Uchiyama, S., K. Takemoto, K. Satoh, H. Yamamoto and H. Tamura (2002). “MR Platform: A basic body on which mixed reality applications are built”. In Proceedings of IEEE and ACM Int. Symp. on Mixed and Augmented Reality (ISMAR 2002), pp. 246-253

Cite as:

Kondo,T., et al., Mixed Reality Technology at a Natural History Museum, in J. Trant and D. Bearman (eds.). Museums and the Web 2007: Proceedings, Toronto: Archives & Museum Informatics, published March 1, 2007 Consulted http://www.archimuse.com/mw2007/papers/kondo/kondo.html

Editorial Note