What Clicked? A Report on Audience Research and Media Resources

Diane Herman, Cincinnatus Inc.; Kate Johnson and James Ockuly, Director of Interactive Media, The Minneapolis Institute of Arts, USA

http://www.artsmia.org/what-clicks

Abstract

The Minneapolis Institute of Arts has completed a major research and development project — called What Clicks? — assessing its audiences' awareness of, and satisfaction with, its interactive media/Web resources. Following up on last year's mid-project interim report (Ockuly 2003) this paper will draw some conclusions and discuss lessons learned, insights gained, and other pertinent aspects of the project. The Minneapolis Institute of Arts, an encyclopedic art museum, produces and maintains two Web sites (http://www.artsmia.org and http://www.artsconnected.org  — the latter in conjunction with Walker Art Center). It also provides its physical visitors with a host of interactive media programs in the museum itself - from an interactive museum directory to a large number of permanent collection-based programs located throughout the building. The research was designed to a) measure audience awareness of and satisfaction with these resources; b) respond to those findings with improvements; then c) re-measure to gauge the effect of the improvements. A further goal of this project was to share its logic model, methodology, instruments, and findings with the museum community.

Key words: What Clicks?, Audience satisfaction, improvements, logic model, museum evaluation

Introduction

The Minneapolis Institute of Arts' (MIA) What Clicks? project was funded by a National Leadership Grant from the Institute of Museum and Library Services (IMLS) for the purpose of evaluating and enhancing the audience effectiveness of the MIA's digital media resources. The full report is available at http://www.artsmia.org/what-clicks .

Using multiple research tools and measurements, the What Clicks? project team evaluated the key concepts of visitor awareness, use, and satisfaction as they apply to the museum's Directory, seventeen Interactive Learning Stations (ILSs) that are immediately adjacent to the galleries, and the MIA's Web site, artsmia.org. The project included four phases: baseline studies in 2002 to determine initial findings; a period of analysis and interpretation; followed by the enrichment and redesign phase; and concluding with follow-up studies in 2003 to determine the impact of the changes. Various marketing initiatives were undertaken throughout the project to increase awareness of all the technology-based resources offered by the Institute. The overall project goal was to benefit not only the MIA and its audience but also other museums through the publication of the What Clicks? process and findings.

Logic Model and Evaluation Plan

As specified in the grant, the design of the What Clicks? project used an outcomes based model for program evaluation: the logic model. This model clarified the team's thinking at the outset of the project. The following key questions helped identify those factors we wanted  and were able to measure: awareness use and satisfaction with three specific applications of technology.

• What are the desired outcomes for each?
• Who will benefit from each?
• What key activities will bring about the intended changes?
• What are the inherent values of the proposed actions?
• What are the - perhaps hidden - assumptions about the actions?
• What kind of lasting impact is expected?

Key What Clicks? Elements

The What Clicks? project included numerous quantitative and qualitative studies both before and after the enrichment and redesign phase: museum visitor surveys, surveys of in-museum technology users, a focus group study of small video monitors in the galleries, a usability lab for the museum's Directory, Web site visitor surveys, Web site usability labs, a Web site terminology survey, and a thorough analysis of user statistics for the in-museum technologies and the Web site. Following the initial baseline research, the project team concentrated on the following enrichment and redesign activities:

• Redesign of the Museum Directory
• Physical modifications to all seventeen ILSs to enhance visibility and usability
• Testing of small video screens with focused, object-specific content in proximity to works of art
• Numerous changes to the artsmia.org Web site to improve visitor usability
• Launch of a comprehensive advertising and promotion plan for the Web site and in-museum technologies

Museum Directory Findings

Figure 1: The redesigned Museum Directory is organized in four main sections and emphasizes information to meet the immediate needs of the visitor.

At the beginning of the project, the MIA's electronic Directory consisted of three touch-screens located in the inner lobby of the museum and, after its redesign, a keyboard and mouse for each screen.  The Directory's contents include information about special exhibitions, permanent collection galleries, lectures, films, Family Days, tours, membership and amenities such as restrooms, coat check, and cafes.

The What Clicks? pre and post surveys and a Usability Lab provided direction for a major redesign of the museum's Directory.  The What Clicks? project team increased visibility with an attract video, larger screens and keyboards; modified the program to include more timely information related to the current visit and added an Art Finder function; and made behind-the-scenes improvements to increase accuracy of the information and reduce staff time to gather and enter it.  As of this writing, additional Directories have just been installed on the 2^nd and 3^rd floors of the museum.  Key findings related to the Directory include the following:

Directory Awareness: Visitors to the museum are surprisingly unaware of the Museum Directory; however, the addition of an attract video and keyboards dramatically increased Directory awareness from 35% in 2002 to 44% in 2003.  Visitors generally became aware of the Directory because they saw it on a visit, suggesting that if a Directory is to benefit visitors, it has to be prominent and inviting.

Directory Use: About 1 in 5 visitors in both 2002 and 2003 said they used the Directory on any previous museum visit, and the primary reason for use was a need for information about a specific exhibition or event, or a general overview of galleries and exhibitions.  Repeat use as measured in 2002 was low (41% had used it previously), and the primary reason for non-use in both years was that visitors did not see a need or already 'knew their way around.'  The addition of an Art Finder function was a very positive step because 24% of Directory users in 2003 were motivated to use the Directory 'to find a specific work of art.'  Duration of use increased to 39% of users reporting 5 minutes or more vs. only 15% in 2002; it is likely that more pertinent and timely Directory content in 2003 held their interest for longer periods.

Directory Satisfaction: When asked how to improve the Directory, users in 2002 said they wanted information to help them find works of art, more information about current happenings, and additional Directory locations throughout the museum.  What Clicks? improvements to the Directory resulted in a dramatic increase in overall satisfaction from 59% of users in 2002 to 77% of users in 2003 reporting they were 'extremely or very' satisfied.  Similarly, more users in 2003 felt the information was 'extremely or very' clear (86% vs. 73% in 2002), and more users in 2003 said they found what they were looking for (92% vs. 82% in 2002.) In addition, high percentages (87% and 88%) in both years found the Directory 'extremely or very' easy to use.

Additional Directory Findings: During the Enrichment and Redesign phase, museum staff learned that they could create a program that drew from existing databases, thereby eliminating the need for redundant and inconsistent updating.  An on-site Usability Lab, which tested the design and function of the new Directory program, yielded insights for improvement and confirmed that the redesign was on track.

Interactive Learning Station Findings

Figure 2: The Arts of Asia Interactive Learning Station - one of 17 such collection-based resources - has been modified to make the screen visible through the doorway.

The MIA installed its first Interactive Learning Stations (ILSs) in 1990 and now maintains a total of seventeen (17) ILSs immediately adjacent to many of the galleries.  Each ILS concentrates on a specific area of the museum's permanent collection and is designed to encourage visitors to connect what they learn from the programs with the actual works of art.

The What Clicks? project team used information from pre and post visitor surveys, focus groups and ILS user statistics to redesign and enhance the ILSs.  Because early research revealed low ILS awareness coupled with high user satisfaction, the team focused primarily on physical changes to the ILSs to increase visibility and promote use. The team's ability to detect statistically measurable differences from one year to the next was limited somewhat by relatively low sample sizes and the team's change in focus from only two ILSs in 2002 to a redesign effort that affected all 17 ILSs. In 2003, the team also tested visitor reaction to small video screens with focused content located in direct proximity to specific works of art.

Learning Station Awareness: In the 2002 survey, half the users said the ILSs were 'not very visible' or 'easy to miss.'  The team's efforts to increase ILS visibility led to a significant increase in visitor awareness from 43% to 53% in 2003.  An evaluation of user statistics for all the ILSs further demonstrated that the more visible the individual ILS, the greater the visitor use.  Many ILS users in 2002 (43%) were not aware that other ILSs existed in the museum, and efforts to 'cross sell' with rack cards placed within every one of the seventeen ILSs did not achieve measurable improvement at the time of the follow-up surveys.  As with the Directory, the primary source of awareness for ILSs (75% of visitors) is direct observation in the museum.

Learning Station Use: About a third of museum visitors in both years used the ILSs on a previous visit.  Among ILS users who were surveyed, the majority in both years reported using the ILS for 5 to 14 minutes.  Repeat ILS usage appeared to be relatively low in both years, with more than half identifying themselves as first time users.  The main reason ILS users were motivated to use them was a desire to 'learn more about works of art.'  The primary stated reason for not using ILS's among non-users was they 'would rather look at art.'

Learning Station Satisfaction: Due to high ILS satisfaction, the What Clicks? team focused primarily on increasing visibility and usability rather than making changes to content.  Not surprisingly, there were no 2003 increases in the already high satisfaction measures for overall satisfaction, ease of use, clarity of information and amount of information learned while using the ILS.  Most important, very high percentages of ILS users (76%) in both said the experience 'enhanced their understanding and appreciation of the art.'

Additional Learning Station Findings: An important finding related to the ILSs in both years was that an overwhelming majority of users (78% in 2002 and 85% in 2003) said they prefer the stations to be 'located in the gallery close to the art.'  In addition, a strong majority of ILS users said they preferred the information to be 'tied closely to specific works of art in the gallery.'  The percentage of users who preferred only general/background information about art in the gallery declined from 23% in 2002 to 10% in 2003, a statistically significant drop.

Small LCD Video Displays in the Galleries: As a result of these 2002 findings related to location and content of the ILS's, the What Clicks? team added a focus group study  during the Enrichment and Redesign phase to test the use of small video screens, with object-specific content, in direct proximity to works of art.  Three groups of diverse respondents were exposed to six different installations of the video monitors and interviewed by a moderator at several points during the 2 1/2 hour visit. 

Focus Group Conclusions

The following are among the key conclusions from the focus group study: 

The presence of small video displays in the galleries enhanced the visitors' understanding and appreciation of the works of art.  All types of visitors, including those who might be considered more 'art savvy,' felt they benefited from the videos.  Importantly, the video displays did not detract from the visitors' experience of the works of art; indeed, only a small number of visitors even noticed the video screens upon entering their first gallery.  (Initially, visitors did not know that the videos were the focus of the study.)  Most respondents felt that the soft, culturally appropriate music accompanying some of the videos added significantly to their experience.  In some cases, the video displays filled what visitors described as a gap between the more general information provided by museum signs and the labels for specific works of art, creating a mid-level contextual piece that resulted in deeper understanding of the works of art in the room.

Web Site Findings

Figure 3: The Minneapolis Institute of Arts' Web site home page.

At the beginning of the study, the museum's Web site, artsmia.org, provided on-line activities, an event calendar, educational resources, and up-to-the-minute information and previews of current and upcoming exhibitions.  The What Clicks? team evaluated the artsmia.org site with pre- and pos-t on-line surveys of Web site visitors, pre- and post- Usability Lab studies, an analysis of the museum's Web site statistics, and a survey related to Web site terminology conducted among visitors to the museum.

Based on the 2002 research which showed relatively high Web site visitor satisfaction and very high repeat visits, the project team's enrichment and redesign efforts focused on two primary areas:  (1) attracting new visitors to the Web site and (2) making site improvements based on the Usability Lab and Web Terminology research.

Web Site Awareness: The What Clicks? Pre- and post- museum visitor surveys revealed that more than half museum visitors are aware that the museum has a Web site (54% in 2002 and 59% in 2003). Web site awareness in both years was higher among museum members than non-members. The Web site survey revealed that most visitors find the Web site by using a search engine. At the time of this writing, there were no measurable increases in awareness.

Web Site Use: Overall, visits to the site based on the MIA's Web statistics showed that visits continued to grow at an historic increase of 50% per year during the What Clicks? study.  It is unknown whether this growth would have occurred in the absence of the What Clicks? improvements to the site and increased marketing. MIA staff knew from the Web statistics that visitors to the site tended to stay on the site for a relatively long period of time (10 to 12 minutes) and this was confirmed by the Web survey in which 54% said they visited for at least 10 minutes.

A major finding in the on-line surveys is that a very high percentage of visitors are repeat visitors (78% in both 2002 and 2003.)  The ever-changing, content rich site appears to provide ample reason to come back.  The survey research afforded the MIA its first opportunity to learn who, specifically, visits the Web site.  Among other demographic findings, the team learned that Web site visitors, in contrast to museum visitors, are younger, live further from the museum, and are more likely to be employed.

Another important finding confirmed the important relationship between visits to the Web site and the museum itself.  In 2003, more Web site visitors had visited the museum (78% vs. 70% in 2002). In addition, more Web site visitors in 2003 said they searched for information to plan a museum visit (52% vs. 38% in 2002).  A Technology Awareness survey of museum visitors revealed that 37% of museum visitors said they had visited the Web site in the past year. One-fourth of Web site visitors are museum members, while one-third of museum visitors are museum members.

Web Site Satisfaction: The 2002 Usability Lab provided excellent guidance for improving the overall navigability and usability of the site.  Examples of changes made during the redesign period included  making it easier to return to the Home page; clarifying calendar functions; adding gallery locations to specific works of art; improving search functions; and improving the image zoom interface (for examining thousands of permanent collection objects).  These and other changes were retested in the 2003 Usability Lab and in a Terminology Survey among museum visitors.

The on-line Web surveys indicated that overall satisfaction among Web site visitors was high in both years, with a majority reporting they were 'extremely or very' satisfied.  (The increase to 65% in 2003 from 62% in 2002 did not meet tests of statistical significance.)  There was an increase in the percentage of visitors who said they found the information on the site 'extremely' current (29% vs. 24% in 2002). Reassuring to the team was the fact that both the 2002 and 2003 surveys revealed that most (80%) of the Web site visitors found what they were looking for.  (An additional 10% said they were not looking for anything in particular.)

A new survey question in 2003 helped the museum prioritize new ways in which Web site visitors would like to interact with the site.  44% would like to buy tickets to exhibitions and events; 31% want to buy museum shop merchandise; 30% want to sign up for e-mail news and reminders; and 25% want to register for classes. Note: on-line shopping and membership purchase were made available on the site just after the follow-up research was conducted.  The relative success of on-line membership purchases and renewals has been a surprise.

General Web Site Findings: A key finding of both the Usability Lab and the Web Site Terminology Survey was that terminology familiar to museum staff may not be clear and meaningful to the Web site visitor.  For example, terms such as interactive media and on-line programs are part of the internal language of MIA staff, referring to collection-based on-line thematic programs (e.g., Modernism, Arts of Asia); these were sometimes confusing for Web site visitors. 

However, an important finding of the Web statistics evaluation of the Top 10 areas of the site reveals that nearly half of Web visitation is going to these very programs that appear to be ineffectively named in the menu.  Many of these programs are also available within the museum via the ILSs.  Thus, with its Web site, the museum is able to extend the reach of its rich collection into the homes of many more people than those directly served within the walls of the museum, and to reach multiple audiences with related program content.

Marketing Findings

When the 2002 baseline surveys indicated that satisfaction is relatively high for all the MIA's technologies and that awareness is relatively low, the What Clicks? project team turned its attention to marketing.  What could be done to increase awareness, and ultimately, use, for all the technologies?  Numerous marketing strategies and tactics were developed and implemented, including new or revised print and collateral materials, television and radio advertising, on-line advertising, a public relations campaign and training of staff to encourage visitor use of available resources.  

Marketing Findings for In-Museum Technologies: Despite the stepped-up marketing for the Museum Directory and Interactive Learning Stations, the overwhelming driver of awareness continues to be direct observation while on a museum visit.  There was a small but significant increase in users who became aware of the ILSs through Institute mailings and publications.  Eight out of 10 Directory users in both pre- and post- surveys became aware of the Directory on a museum visit.  This finding does not suggest, however, that marketing of in-museum technology is ill-advised.  Rather, because most of this advertising is free in the sense that the museum's communications would occur in any event, it is only a matter of including references to museum technologies.

Marketing Findings for the Web site: The Web marketing part of the What Clicks? study was particularly informative because MIA staff had had little experience with Web marketing prior to the project.  Using a test-retest approach, MIA marketing staff learned that on the Web, targeted messages to targeted audiences delivered the greatest results.  For example, local media sites, such as the local newspaper, delivered higher click-throughs than less targeted national sites.  Similarly, the highest coupon redemption from all on-line advertising came from direct, personalized e-mail messages to self-selected e-mail group lists.  Designing an ad for on-line advertising is similar to creating ads for other advertising media in that larger, eye-catching ads with compelling messages do best.  On the other hand, movement must be used carefully; too much movement may offend some viewers and movement may take up too much file space to be practical.  The efforts of What Clicks? to test on-line advertising failed to demonstrate that it is an effective way to drive museum attendance.

More in-depth information related to What Clicks? can be found at http://www.artsmia.org/what-clicks

Acknowledgements:

What Clicks? was funded by an Institute of Museum and Library Services (IMLS) National Leadership grant.

References:

Ockuly, Jim “What Clicks? An Interim Report on Audience Research “ Museums and the Web 2003  http://www.archimuse.com/mw2003/papers/ockuly/ockuly.html)