Hands-On versus On-Line:  Evaluating MarsQuest On-line

James B. Harold, Space Science Institute, Randi Korn, Randi Korn & Associate, Paul B. Dusenbery, Space Science Institute, and Christopher Randall, TERC, USA

Abstract

In October, 2000, the Space Science Institute of Boulder, Colorado, launched MarsQuest: a 5000 square foot traveling exhibition on Mars and Mars science. Two years later NSF and NASA gave their support to an associated Web project called MarsQuest Online, a partnership between TERC, the Space Science Institute, and JPL. The goal of the MarsQuest Online project was to create a Web site that extended both the reach and scope of the traveling exhibition. The program was premised on the idea that the effort expended on the museum exhibition - the exhibition framework, interactive concepts, educational goals, etc. - could be leveraged into a highly interactive on-line experience.

MarsQuest Online has now been live for over a year, and has had well over a half million visits.  Site visitors have flown over Mars in 3D, driven rovers, chosen potential landing sites, and followed NASA's MER rovers as they explore the surface. However, quoting numbers of visitors begs the real question for any education project: what have visitors learned?  More broadly, is the project a success as an informal learning experience, as the physical exhibit was?  If so, how does one define and demonstrate on-line learning?

Answering these questions has always been a central part of the project.  In addition to a more traditional formative evaluation, which examined the usability of the pieces, MarsQuest Online includes a summative evaluation that contains two primary components.  First, the interactives on the site were programmed to record user activities into a database that can be mined to obtained detailed information about visitors' interactions.  Second, Randi Korn & Associates, were hired to conduct a series of standardized interviews.  These interviews, conducted with science center visitors, included three groups: those who visited the MarsQuest exhibit but not the Web site; those who visited the Web site but not the exhibit, and those who visited both.  In that way we hoped to determine how the exhibit and the Web site, together and separately, affect visitors' attitudes and knowledge about Mars and Mars exploration. 

This paper will discuss the MarsQuest Online project, the motivations behind this evaluation program, its challenges, and results.

Keywords: mars, science, exhibits, evaluation, on-line activities, public web site

Introduction

The MarsQuest Exhibition

Funded by the National Science Foundation and NASA, the MarsQuest traveling exhibition uses data from NASA missions to bring Mars to science centers around the country. This multi-media exhibition includes sixteen interactive components for hands-on learning experiences; five computer stations where visitors navigate through virtual presentations, manipulate computer models, or find information; a video monitor that plays a brief animation of the formation of our solar system; seven scale models depicting spacecraft and portions of Mars and Earth; and the Mitsubishi Digital Imagination Theater, which presents a video of high-definition images of Mars.

Thematically, the exhibition is structured around sites on the planet.  Early on in the exhibition's development process, we decided to treat Mars the way one would a National Park.  Rather than discussing prospects for colonization, for instance, the developers asked the question: if you were a tourist, where would you go?  As a result, the exhibition is organized around locations of compelling interest: Volcanoes, Canyons, the Poles, and Rover Landing Sites. The exhibition components in each site expand on the dual themes of Earth-Mars comparisons and Mars exploration.

Fig. 1: A section of the MarsQuest Exhibition, showing part of the Volcano area.

MarsQuest faired well in its summative evaluation, performed by Randi Korn & Associates (RK&A, 2002).  Visitors moved more slowly through MarsQuest than visitors in other exhibitions of a similar size.  In interviews, visitors indicated that they enjoyed the exhibition and were able to articulate at least part of its message.  They recalled specific facts about Mars, and the comparison of Mars characteristics and features with those on Earth helped many interviewees learn new facts about Mars.

MarsQuest's initial tour was three years, during which it visited nine museums and science centers across the country.  In the fall of 2003, the exhibition was pulled in for refurbishment. It began its second three-year tour in late January of 2004 at The New Detroit Science Center, complete with new interactives and a full-scale model of a MER rover.

MarsQuest On-line

The idea for the MarsQuest Online site developed independently on two fronts.  SSI was pursuing support for a Web site based on the exhibition, while TERC was pursuing support for an educational Web site on Mars. Eventually TERC contacted SSI, and a joint proposal to the Division of Elementary, Secondary and Informal Education of the National Science Foundation was submitted, with TERC as the project lead.  TERC had also developed a relationship with a group inside NASA's Jet Propulsion Laboratory (JPL), and they formed the third partner on the project.  The project Principal Investigator (PI) is Dan Barstow (TERC); co-Investigators (I's) are Paul Dusenbery (SSI), Paul Andres (JPL) and Eric DeJong (JPL).  Chris Randall is the project director at TERC, and James Harold is the project director at SSI.  The proposal was submitted to NSF's Informal Division in part because of the connection with the exhibition, but also because we were strongly advocating the Web as a potentially powerful informal learning environment. This is a view that has been gaining momentum as Web technologies allow more interactive and engaging experiences.  In fact, the Association of Science and Technology Centers (ASTC) gave its year 2000 award for innovation not to a physical exhibit, but to the Exploratorium's Web team (ASTC, 2000)

Fig 2: The MarsQuest Online home page (http://www.marsquestonline.org).

The MarsQuest Online site consists of three major components:

• Explore Mars is modeled after the MarsQuest exhibition.  Following the exhibition's thematic framework, it invites visitors to explore various areas of Mars. Each area includes a series of light, short duration interactives, comparable to what one might find in an exhibition.
• Mars Mysteries includes more complex interactives, requiring more time and thought to complete. For instance, a visitor might try to determine where to land a rover, requiring an analysis of Mars' surface features to find a location that is both relatively safe and scientifically interesting
• Rover Images is a real-time archive of the images received from NASA's Mars Exploration Rovers (MER).  This piece includes navigable, high resolution panoramas that allow visitors to explore the rover's findings in a visual context.  New images are ingested into the system automatically through a dedicated feed from the Jet Propulsion Laboratory, as part of their Museum Visualization Alliance program.

Other areas include resource materials for teachers and the public.  The site has been running since mid-2003, with mirroring support through the University Corporation for Atmospheric Research (UCAR) and the San Diego Supercomputer Center (SDSC).

In 2004, NSF granted the project supplemental funding.  The supplement will allow us to expand the 3D Flyover activities into a global set of guided tours of Mars, and to continue to support the Rover Images area in the face of the unprecedented success of those missions.  The nominal design lifetime of the MER rovers was three months: at the time of this writing, they are celebrating their one-year anniversary of Mars and still functioning well.  This is tremendously exciting and represents an enormous achievement, but it presents obvious maintenance challenges for us as well.

The Evaluation Program

When the MarsQuest Online project was first beginning, "success" for an educational Web site was typically cast in terms of hits, and evaluation seemed to be limited to usability analysis.  In contrast, our history with exhibit development predisposed us to think that other questions ought to be addressed: in particular, what and how are visitors learning? While a few Web projects were examining these issues (e.g. Allison-Bunnell et al., 2002, and more recently, Honeyman, 2004), they seemed few and far between, and we felt that MarsQuest Online presented an opportunity to further examine these questions.

Some guidance for our evaluation existed in the approaches used for evaluating museum exhibits. However, while parallels exist, there are also clear differences between physical exhibits and the Web: among other things, the actual audience for a Web site is largely unknown, and in some ways inaccessible. Nevertheless, it seemed to us that establishing a methodology for reviewing educational Web sites was critical if the Web is to be taken seriously as an informal learning environment.  As a result, a broad goal for the evaluation of this project was to contribute to the base of information about how educational Web sites can be evaluated, what success looks like, and how exhibits and Web sites compare in their abilities to meet their goals. 

The evaluation approach included one-on-one interviews by external evaluators and SSI staff, Web log analysis, and database driven collection of usage information in interactive pieces.  In addition, because the physical MarsQuest exhibition had also undergone extensive evaluation, we were able to use the summative results for that program to provide additional information on visitor perceptions of the exhibition.

The remainder of this paper will touch on some of the results we have obtained to date.  This paper is not intended to provide an exhaustive review of the results of the evaluation (which at the time of writing is still ongoing).  Instead, we are isolating a subset of results that we feel are of broad interest to those working at the interface of museum exhibits and the Web. 

Leveraging from the Exhibition: Front End Results

Methodology

A significant part of the case for developing the MarsQuest Online Web site was the claim that we could both extend the exhibit experience and leverage from the exhibition development program.  To do this, it was important to identify those exhibition experiences that were most effective, both in engaging the public and achieving our learning goals.  In addition, since we saw the Web site as a mechanism for expanding on the exhibition's content, we wanted to understand which exhibition components stimulated the greatest interest in seeking additional information.

In order to assess what aspects of the exhibition might be most productively borrowed, RK&A performed a series of front-end evaluation interviews for the project. The interviews were conducted with MarsQuest exhibition visitors at the Peggy Notebaert Nature Museum in Chicago in May, 2002.  The goals of the evaluation included:

• Determining which aspects of the exhibition sparked MarsQuest exhibition visitors' curiosity;
• Identifying exhibition content and experiences that should be complemented on the MarsQuest Web site;
• Determining exhibition visitors' expectations for what they would be able to see and do on a MarsQuest Web site;

The visitor population interviewed included nearly equal numbers of children (15 years and younger) and adults (20 years and older).  When children younger than 9 years participated, they were always with a companion in the target audience.  Almost twice as many interviewees were male as female, (n=14 and n=8, respectively).

Visitor Expectations Of The Web Site

As part of the survey, visitors to the MarsQuest exhibition were asked about their expectations for a MarsQuest Web site. Based on their exhibition experiences, interviewees - especially youngsters - expected the Web site to be fun and interactive, and adults expected it to be rich in interesting information.  As they talked about expectations for the Web site, interviewees sometimes added "interesting...like the exhibition" or "it will have cool stuff..." and pointed to an example in the exhibition that was "cool" to them.  Children and preteens often requested three-dimensional simulations of travel to and on Mars, and sometimes said they expected to be able to play entertaining, in-depth games to explore Mars.  A few youngsters also expected to do research for school projects on the Web site.  But all stressed that while they expect the Web site to be educational, they want it to be more fun than typical Web sites they use for school.

Adults' expectations for the Web site were that it would provide more in-depth information than the exhibition provided, and would enable them to follow up on topics from the exhibition: find out more about Mars' environment; and learn about scientists' latest Mars-related theories, research, and discoveries.  Many adult interviewees appreciated the exhibition's unique perspective, and expected the Web site to continue that level of quality while presenting more in-depth content.

These results supported many of the goals of the MarsQuest Online program: to borrow from and extend the content in the exhibition, while preserving the interactive nature of the exhibits.  Note, however, the high bar set by children: they expected the site to be interactive, "cool",  and include 3D simulations, in depth games, etc.  

What's In A Name?

One interesting result of these interviews was that visitors to MarsQuest were not aware of the exhibition's title.  This could perhaps be corrected through a more extensive marketing campaign, but since MarsQuest is a traveling exhibition, such marketing is largely out of the hands of the developers.  The practical consequence is that visitors do not have a priori knowledge of the URL of the exhibition's Web site.  The assumption had been that we could hope at least for a "Google" of "MarsQuest", but even that belief seems to be misplaced. This suggests that in the future, if we expect visitors to find the Web site, it should be directly marketed within the exhibition.

Thematic structure

Perhaps even more significant is that the interviews indicated that visitors were unfamiliar with the exhibition's thematic structure. MarsQuest is structured around various "locations of interest" on Mars: Volcanoes, Canyons, Poles, and the Pathfinder/Sojourner landing site (a floodplain).  The failure of visitors to perceive this structure is not surprising: exhibition visitors are usually more attuned to discrete exhibit experiences than to an exhibition's thematic organization. While the "national park" concept may have helped the developers produce a cohesive exhibition, visitors tend to remember particular interactive experiences, such as driving a rover, and pieces of interesting information, i.e. that Mars is cold and has volcanoes.

However, in the interviews, when a sample MarsQuest Online home page presented those themes as its organizational framework, it piqued the interest of many adult interviewees and several youngsters.  Those interviewees often associated the themes with discrete MarsQuest exhibits they enjoyed or exhibition information that intrigued them.  This suggests that replicating the exhibition's structure in the Web site is still beneficial.  While visitors may not directly connect the Web site's structure with the exhibition, they may be able to place their exhibition experiences within that framework after the fact.  And of course the framework remains a valuable tool for developers in creating a coherent story, just as it did for the exhibition.

Now that the site has been running for some time, it is clear that Web visitors both perceive and take advantage of this thematic structure.  Analysis of paths from the home page of the Mars Tour shows visitors preferentially going to the rover and volcano areas, behaviour which mimics the preferences mentioned in the interviews.  It may be that ultimately this thematic structure is more valuable to the Web site than it was for the original exhibition.

Data Collected Through the Site

A central premise of the MarsQuest Online project was that the site should be highly interactive: a direct result of our science center backgrounds.  Science centers place tremendous value on interactivity: it is widely believed to be an effective way of engaging visitors and achieving learning goals.  However, the resulting heavy use of Flash and Shockwave on the site meant that traditional log analysis techniques would provide only limited information on how visitors used the site.  On the other hand, the interactives themselves could be coded to collect usage data.  Ultimately we used a combination of approaches, programming many of the interactives to collect data, and using traditional log analysis to examine broader questions (e.g., overall visitorship, and paths through the site).

Visitorship and activity

Visitorship to the site is strongly driven by the news coverage of the Mars rovers.  When the rovers landed in early 2004, the site received over 17,000 visits in a single day.  Over the next month traffic decreased to a few thousand a day, and in the last six months it has been relatively stable at 500-600 visits per day. 

On average, visitors view 8-12 pages per visit, which in our experience is reasonably high. Between 20% and 25% of visitors run at least one of the interactive activities, but this activity is highly dependent on the source of the visitor.  For traffic following a link from the Web site of a science center hosting the MarsQuest exhibition, this number may reach 33%, while for traffic arriving from the Yahoo search engine, it can be as low as 16%. 

Time Spent in Activities

In a typical science center exhibition, visitors spend very little time interacting with exhibit components: a minute or two is considered a "win". These short dwell times are a strong driver of both the content and design of exhibit computer interactives: learning goals must be well conceived and pieces must be able to achieve those goals quickly.  The MarsQuest exhibition is no exception, and in Table 1 we show a sampling of results from the MarsQuest summative evaluation (RK&A, 2002), showing the median time spent at a few of the exhibition components. We include physical as well as computer based exhibits to illustrate that these issues are not specific to computer stations.

Table 1: Median dwell times at several MarsQuest exhibit components.

One question we wished to answer in evaluating MarsQuest Online was whether these dwell times would be mirrored in an on-line environment.  The point can be argued either way.  Web surfers probably have fewer time constraints than exhibit goers, and because they can return to the site for free as often as they like, there is no pressure to "move on" to the next part of the exhibition.  On the other hand, Web surfers may be even more easily distracted than exhibit goers: they have an entire world of possibilities at their fingertips.  They also lack the investment represented by paying an admission fee, which can encourage visitors to spend more time in an exhibition to "make the most" of their money.

Table 2: Median dwell times at several MarsQuest Online interactives

Table 2 shows the results of time tracking for several MarsQuest Online interactive activities. Time is tracked from leaving the introduction screen to closing the interactive.  We choose to begin timing after the introduction screen because it typically also serves as a "loader", delaying visitors until enough of the piece has been downloaded to begin.  This approach also eliminates people who do not run the piece at all (by choice, or for technical reasons).

The values in Table 2 are quite similar to the values obtained for the physical exhibits.  This is not a direct comparison, since the pieces are not identical.  But the results suggest that, for activities with comparable complexity, exhibit and Web visitors do not demonstrate substantially different dwell times.

Depth of Interaction With Piece

Dwell time is an easily measured quantity; however the more substantive measure of the success of an interactive is the degree to which its learning goals were met.  Answering this question presents us with one of the double-edged swords of Web sites. It is easy to amass large amounts of data on specific questions, and that data has the advantage of being in situ: it is an accurate accounting of how visitors actually used your site. However, at the same time you have no personal contact with your users.  You do not know the most basic information about your visitors (gender, age), never mind whether the Web experience has changed their opinion of Mars science.

However, it is possible to use indirect proxies for the success of these pieces.  The metrics vary by activity, but in general we're looking for evidence that users either

• explored the activity thoroughly (playing to completion, if relevant for the activity), or
• accomplished a specific task that indicates they met our learning goal. 

By these measures, the MarsQuest Online interactives have been quite successful.  Significant percentages of visitors either complete the piece or at least reach a point that suggests our learning goals have been met. We summarize the results for some of the interactives below.

Find the Features

Fig. 3: The "Find the Features" activity, showing a topographic Earth globe

This activity includes 3D topographic globes of Earth and Mars.  Users are challenged to find 5 parallel geographic features on each of the two planets (e.g., highest and lowest points). The globe images can be switched between a conventional "visible" image (as the planet would appear from space) and a topographic image (where terrain is colored based on altitude).  The 3D spheres can also be "stretched", exaggerating the vertical span of features.  Both of these features were included specifically to expose users to tools and visualization techniques commonly used by scientists.

Results

• Of those who tried at all, 25% stopped at the first feature, and 27% completed all 10 features.
• Half of the visitors examined Earth as well as Mars: this is key to the "comparative planetology" goals of the activity
• 45% used the vertical exaggeration tool
• 52% switched between visible and topographic views

Fly to Mars

This simple orbital simulation challenges visitors to launch a spacecraft from Earth to Mars.  Success for our purposes does not require the visitor to succeed in getting to Mars: only to recognize that timing (the "launch window") is an issue.  In addition, the more time visitors spend in the simulation, the more they are exposed to some basic solar system concepts: relative orbital periods of the planets, relative positions, etc. A "hint" button gives some instruction, with an added "Show me" option that tells users when to launch.

Results

• Over 60% of the users launched the spacecraft two or more times; half launched four or more times. 
• Almost 60% succeeded in reaching Mars.
• 60% of the visitors used the hint option, and 38% opted to have the program tell them when to launch (this isn't surprising: it's quite difficult to successfully reach Mars in this piece.)

Drive a Rover

This is a simple, "virtual" version of the programmable rover in the MarsQuest exhibition. Visitors upload a series of commands to a rover, which then executes them.  Visitors have 7 days to navigate to as many target rocks as possible before their rover "dies". The learning goal is simply to understand that rovers are not "joysticked" around the planet like a radio controlled car, but must be given pre-planned sequences of instructions that are then carried out autonomously.

Results

• 65% of users found one or more rocks, demonstrating that they understood the basics of rover navigation.
• 43% played to completion (when the rover expired).

Fly in 3D

Visitors choose one of several different Martian terrains to explore in 3D.  They can chose to view the terrain as an object, tilting it and changing the angle of the sun, or to fly over it in more of a game-like mode.  In the fly-over mode, there is a series of markers to locate, each of which presents some additional information about the area. In either mode the terrain can also be vertically exaggerated ("stretched") in order to enhance features.

Results

• Choice of terrain: In order of preference, visitors chose Olympus Mons, the largest volcano in the solar system (44%); a section of Valles Marineris, a canyon system as long as the U.S. (21%); the "Face on Mars", a small mesa which created a stir in the 1970s (16%); one of the MER landing sites, Gusev Crater (11%); and an ancient floodplain (8%).  These results are consistent with previous evaluation results that indicated a preference for Volcanoes and a lack of knowledge of specific Mars features. 
• Choice of mode ("tilt" or "fly"):  86% of visitors used the "fly" mode; 56% used the "spin" mode, and 41% used both.  This is more balanced than had originally been predicted: there had been a belief that the more "fun" flying mode would dominate.
• Marker searching: this was less successful, with 40% finding one marker, and only 10% finding all three.

Summative Evaluation Interviews

What is clearly missing from the data presented in the previous section is actual contact with the users.  The evaluation research design for this project presented a number of challenges. 

As noted earlier, the research question for the evaluation was: does a Web site extend the exhibition experience?  This question is reasonable to ask under one condition: if the Web site was specifically designed to extend the exhibit experience.  In the case of MarsQuest Online, extending the exhibition experience was a central goal of the project.  To answer the question with any degree of confidence, we had to design an approach that would allow us to examine Web experiences across three groups of users: those who visited the exhibition only, those who visited the Web site only, and those who visited both the exhibition and the Web site. 

Research and Instrument Design

We knew that we wanted to obtain large samples of each group to give us confidence in the results.  We also wanted to control who would participate in this study.  We know from conducting museum visitor research that more museum visitors have four or more years of college than the public at large.  The exhibition was designed for museum audiences, and the Web site was designed for life long learners interested in science and Mars exploration - likely a subset of museum visitors.  MarsQuest Online developers were clear in their desire that MarsQuest Online provide a learning experience for users.  We designed a standardized questionnaire that we would use as an interview with users and nonusers over the telephone.  This standardized questionnaire had multiple-choice questions and open-ended questions that sought to identify users' learning experiences. 

We wanted all three groups to have the same data collection experience - that is, we wanted everyone to be interviewed over the telephone.  We assumed that some of these individuals would visit the Web site and others would not - providing two of the three groups needed.  We intended to rely on other museums' volunteers to help us collect phone numbers of their visitors, but we learned doing so was too much of an imposition, and after a year of trying to persuade other museums to help answer this research question, we simple gave up.  Instead, we hired graduate students to collect people's phone numbers.  We invited people to participate in this study by approaching them at the exit of the exhibition and giving them a beautiful photograph of Mars with a sticker on which was printed the URL.  We asked people for their phone numbers and the best time to call them. 

We also needed Web only users.  To solve that problem, we had to let go of our original desire to control our respondents (early on, we decided science center visitors were a reasonable proxy) and decided we should capture Web users only via the Web site.  The survey was placed on-line, via a Flash-driven prompt inside the main navigation bar.  This made the survey immune to pop-up blockers, and insensitive to the part of the site being visited.  The prompt appears after the visitor has been on the site for one minute.  Once the survey link is followed, the prompt will not appear again on that computer.

To identify exactly what on-line survey respondents have done, the survey asks them whether they have visited the exhibition, and if so, their responses will be placed in the Web and exhibition user group with fellow telephone respondents.  Likewise, we also ask telephone interviewees whether they visited the Web site, and if so, whether they completed the on-line survey.  If they have, we thank them and move on to the next call. 

Outcomes

In museum exhibition evaluation, response rates of 80 percent and higher are expected.  When non-profits solicit for funds or when market researchers conduct a study with the general public, a 1% response rate is not uncommon, nor is it frowned upon.  Our Web survey currently has a response rate of 2.7 percent and a smaller percent choose to complete the survey after clicking on it.  The response rate for collecting phone numbers was 77%.  It is difficult to judge response rates because as of yet, there does not appear to be an industry standard.  Prior to conducting any evaluation, however, it is paramount to identify objectives of the Web site, as objectives are the gauge against which outcomes are measured.  In this way, a Web site, as a product, is not different from any other product that is evaluated.  Evaluation is the systematic collection of information about a program that examines the successes and shortcomings of the program against stated goals and objectives.  The MarsQuest Online project had a specific target audience and specific learning objectives which set the bar high for evaluation.  The team's attentiveness to identifying success, as well as their passion for their subject, helped make this evaluation very directed, but logistically, it was problematic because no one anticipated how difficult it would be to collect data.  

Complete results (including the telephone interviews) are still pending; however, some preliminary results from the on-line survey are available.  To date, 217 visitors (2.7%) have clicked into the survey, and 91 (1%) have completed the survey.  83% of the respondents are male.  26% report high school or below as the highest level of education completed; 26% report having a graduate or technical degree.   With respect to education outcomes, we find:

• 92% agreed that MarsQuest had shown them something they had not seen or did not know before;
• 53% agreed that their experience with the site changed how they think about Mars
• 49% agreed strongly (rating the statement a 7 on a 7-point scale) that they had greater knowledge of Mars and Mars exploration as a result of their experience with the site;
• 42% agreed strongly (rating the statement a 7 on a 7-point scale) that the experience made them more interested in learning about Mars and Mars exploration;

Conclusions

One of the primary goals of the MarsQuest Online project was to determine whether we could leverage from the original exhibition to extend its reach and scope.  Embedded within that are additional questions. Are effective exhibition approaches effective on-line as well?  Do we understand what we can accomplish in on-line learning? Can we successfully evaluate that learning? This project clearly is not going to answer all of those questions, but we believe it can contribute to a baseline of data that can help future projects.

Some aspects of our approach clearly worked: the story, themes, and framework of the exhibit resonate with Web visitors as well as with exhibition visitors.  The visitor preferences measured on the site align with those obtained in interviews at the exhibition. Even when visitors do not perceive the framework in the exhibition, it helps them organize their experiences when they visit the Web site.

The emphasis on interactive activities also appears to be successful.  High percentages of visitors complete the activities, and even in the absence of interviews, it is reasonable to conclude that they met the learning goals set out for the pieces.  Dwell time is comparable on the Web site and at the exhibits.  This doesn't speak to an upper limit for how long visitors can be engaged on the Web, but it does demonstrate that they will spend at least as much time with a piece on the Web as they would at an exhibit: enough to complete the piece and meet our goals.

Interviews remain problematic.  Reaching individuals who have used the site "naturally" (at home, unprompted by an evaluator) is challenging.  Our original plan was to assert that science center visitors constituted a reasonable proxy for our intended audience, then reach them directly.  An alternative, which we are now pursing as well, is on-line "pop up" surveys.  At 2-3%, our response rates have been typical for this type of survey (e.g., Goldman and Schaller, 2004).  These rates are considered extremely low by the standards of exhibit evaluation, though not by the standards of other professions (marketing, for instance).  It may be that ultimately our community will have to accept this response rate as the expected baseline for this type of evaluation.

The Future

The last year has been a remarkable one for Mars exploration.  The twin rovers, designed for three month missions, are still functioning well a year later.  The rovers' discoveries are even more impressive than their performance: we now know that Mars once harbored a salty, liquid sea at a time when life was first starting on Earth. Future Mars missions will follow, studying the planet from orbit at high resolution, digging below the surface to search for life and eventually returning samples of Martian rock to Earth. 

The power of these events to capture the public's interest is extraordinary.  NASA received billions of Web hits in the weeks surrounding the rover landings, and the numbers will only be higher for future missions.   The challenge for science educators is to use these opportunities to create lasting change in the public's understanding of Mars, Mars exploration, and planetary science in general.  Science centers and museums have been refining their education strategies for years.  Our results with the MarsQuest Online project support our belief that educational Web sites can connect to and learn from museum exhibitions to become powerful informal learning experiences in their own right. 

References

Allison- Bunnell, S., M. Borun, D. Schaller, and M. Chambers (2004). "How Do You Like To Learn? Comparing User Preferences and Visit Length of Educational Web Sites". In D. Bearman and J. Trant (eds.). Museums and the Web 2002: Proceedings. Toronto: Archives & Museum Informatics, 2002, consulted January 26, 2005.  http://www.archimuse.com/mw2002/papers/schaller/schaller.html

Association of Science-Technology Centers (2000). 2000 Award for Innovation Goes to Exploratorium Web Team. Consulted January 28, 2004.  http://www.astc.org/members/innovation.htm.

Goldman, K.H., and D. Schaller, 2004. "Exploring Motivational Factors and Visitor Satisfaction in Online Museum Visits". In D. Bearman and J. Trant (eds.). Museums and the Web 2004: Proceedings. Toronto: Archives & Museum Informatics, 2004, consulted January 26, 2005.  http://www.archimuse.com/mw2004/papers/haleyGoldman/haleyGoldman.html

Honeyman, Brenton (2004).  "Online vs. On-site: The ‘Burarra Gathering' Experience". ASTC Dimensions, September/October 2004.

Randi Korn and Associates, (2002). MarsQuest Summative Evaluation. Consulted January 28, 2004. http://www.informalscience.org/tools/summative.html

Randi Korn and Associates, (2002). Front-end Evaluation for MarsQuest Online. Randi Korn and Associates, unpublished.

Cite as:

Harold, J., et al., Hands-On versus On-Line:  Evaluating MarsQuest On-line, in J. Trant and D. Bearman (eds.). Museums and the Web 2005: Proceedings, Toronto: Archives & Museum Informatics, published March 31, 2005 at http://www.archimuse.com/mw2005/papers/harold/harold.html