Papers: Audience Analysis in the Age of Engagement

When the Internet expanded from the private domain of a select group of scientists into the public domain, we entered new territory in information seeking and use. Museums that already had well-developed strategies for evaluating the experiences of on-site visitors were then required to develop different information delivery models and new ways to understand their on-line visitors’ needs.

Early museum Web sites generally took the form of on-line brochures, and were simply the first and most obvious way for museums to provide information by transforming existing materials for delivery via the new medium. Attempts to gain information about visitors to those sites tended to rely mainly on quantitative measurements and demographic research. However, Internet use has changed rapidly in the last decade, and while this presents new challenges for museums, current trends are simultaneously opening up great opportunities to those institutions willing to harness the power of the Internet. In attempts to name this time we’re in, the current age has variously been called the ‘age of engagement’ (Morgan Stanley, 2005), the ‘age of participation’ (Schwartz, 2005) and an ‘authorship society’ (Rushkoff, 2005). These titles all refer to a profound shift in emphasis from information delivery and seeking to a time where audiences participate, create value and demonstrate independence (Schwartz, 2005). While the information age was generally passive, the participation age is active.

The participation age requires new, polyvalent approaches to audience analysis, and organizations such as the Canadian Heritage Information Network (http://www.chin.gc.ca) (CHIN) are responding to this challenge. This paper will present an overview of research and initiatives CHIN has undertaken to determine its audiences’ overall engagement with the Virtual Museum of Canada (http://www.virtualmuseum.ca) (VMC), at both a group and an individual level. From the traditional, passive collection of information such as Web site statistics to a more systematic analysis of audience feedback, we will discuss how we are changing our models of audience analysis and opening dialogue with our audiences in the process.

In existence since March 2001, the VMC is a cultural gateway created by CHIN in collaboration with the Canadian museum community. The mission of the VMC is to engage audiences of all ages in Canada’s diverse heritage through a dynamic Internet service freely available to the public in French and English. In our early attempts to gain information about VMC visitors, we tended to rely on quantitative measurements, such as Web site statistics; however, traditional approaches to Web log analysis can be problematic. As Peacock (2002) documented in a paper previously presented at Museums and the Web, Web logs were initially used to measure traffic and the volume of page requests, with the ultimate aim of helping IT staff plan server capacity. However, the widespread, and somewhat misleading, use of hits contributed to the perception that Web statistics were neither accurate nor useful indicators of a site’s popularity and effectiveness. Although tracking visits, visitors, and page views over time can tell you about increases and decreases in traffic to your site, you are not ultimately engaged with the users themselves, or the reasons why they may be visiting more or less.

Therefore CHIN is now taking a more strategic approach that puts our Web stats in context, establishes a formal process for responding to identified issues, and correlates those issues with external factors. The output of this analysis is then summarized and packaged in a quarterly report called CHIN Monthly Indicators. At the core of this initiative is a desire to analyse Web log data in order to better understand groups of our users: we want to know who they are, how they are getting to our site, what they are most interested in, and how they interact with our content once they get there. For example, CHIN is hoping to better understand how visitors are using VMC content for educational purposes, and we are looking to Web log data to help us answer that question. By analysing domain names between January 1 and June 30, 2005, we were able to identify the type of educational institution from which users are accessing the VMC, by province in Canada,and by type of educational institution in the United States. Additionally, the September 2005 edition of CHIN Monthly Indicators took an in-depth look at the VMC’s section (or landmark) called the Teachers’ Centre in order to begin establishing benchmarks against which to measure the success of future CHIN initiatives aimed at teachers, such as the museum learning space to be discussed later in this paper. We found that search engines are responsible for referring most users to the Teachers’ Centre, and after Google, the second most popular search engine was Education Central in seeq.com, a search engine that consists of over 60 niche portals. Furthermore, an analysis of the top search phrases used to locate the Teachers’ Centre reveals that most users who arrive via search engines are looking for ‘lesson plans’ and ‘teachers resources.’ Not surprisingly, the most popular content in the Teachers’ Centre is the lesson plans page, followed by resources on Inuit history.

Looking at patterns and cycles of information-seeking behaviour also helps us know which content to promote at various times of the year. For example, in the weeks leading up to Remembrance Day (November 11) in Canada, the Teachers’ Centre landmark annually experiences an increase in searches for material related to World Wars I and II. In the past, CHIN has responded to this by creating a page with special Remembrance Day content which appears on the Teachers’ Centre interface in the weeks preceding this event. However, the impact of these and other on-going changes also needs to be analysed and evaluated in terms of their effectiveness. As such, a second component of CHIN’s strategic statistical analysis is the correlation of content and technology issues with external factors or ‘events.’

’Events’ in the analysis are defined as any activity, internal or external, which can potentially impact and affect visits and visitor behaviour. For example, an event could be a significant change in interface, an optimization of a virtual exhibit, a key word campaign, or the opening of a new national museum. Events are classified by type, in order to assess over time which ones have the greatest impact:

• conference
• content change
• external event (not financed or organized by CHIN)
• interface change
• international event
• marketing event (launch)
• on-line communication
• print communication
• search engine optimization strategy
• service delivery change

Events are documented and tracked in a log that identifies the event duration, audience, type, and expected impact. Monthly statistical information and feedback messages are then analyzed and compared with previous monthly and yearly figures to determine whether correlating events had the expected impact and to identify patterns in user behaviour.

For example, an interface change to the VMC home page in mid-April, 2005, to more prominently feature our news content resulted in a substantial increase in traffic to the In the News landmark. May 2005 saw a 558% increase in visits, from 628 in April to 4140. However, analysis of subsequent months’ visits demonstrated that this traffic would not be sustained if the content were not changed on a regular basis: a recommendation to do so was consequently put forth. It is expected that this approach to the analysis of Web site statistics will enable CHIN to develop a more strategic and holistic approach to our on-line service delivery, site structure, and design. Over the long term, we will also work toward using the statistics to measure our impact on society and the use of our content for capacity building, education purposes, helping the public better understand their communities, travel and tourism needs, furthering international diplomacy, and retracing family heritage.

Since the inception of the VMC, CHIN has kept a feedback database where visitors’ messages are coded according to message descriptions (message, date, location, language) and categories based on the content of the message (i.e. message tone, type of message, site feature mentioned in the e-mail). Comparison of these messages to visit statistics gives more information about our visitors, as we are now more engaged with the users as individuals rather than marks on a graph or numbers on a spreadsheet. Visitors make suggestions, draw attention to errors about content or navigation, share information about the subject matter and ask questions about the content of the Web site. Those visitors who submit messages to the feedback mechanism on the site tend to be a self-selected group of people (who may be particularly pleased or displeased), and the information provided is open-ended. Gathering this information in a systematic way gives us information about some of those visitors. From these messages, we can gather information about areas of the site that may need improvement, which sites users find exceptional, and how the VMC site is being used (school projects, personal research such as genealogy).

Additionally, visitors who send feedback messages are sent a follow-up survey to further engage them and find out more information. This follow-up survey consists of open-ended questions:

• For what purpose are you using this Web site?
• Did the site meet your needs?
• If not, what could be improved?
• If it did, what could be done to better meet your needs?
• From where (ex. Home, school, work, library, etc.) are you accessing this site?
• How often do you visit this site?

Through the VMC, CHIN is also recruiting interested members of the public to participate in an ‘advisory board’ where we can, from time to time, seek advice on various matters pertaining to VMC content. This allows visitors to feel that they have some say in directions for this federally funded site.

As expectations increase for interactivity, we must find new ways to understand visitors and engage them. It is important for museums to understand trends in technology, as they can impact delivery and success of programs. An environmental scan, or at least following a selection of ‘trend’ publications, can help us understand what our audiences expect from on-line content and help turn that into information to assist in developing our on-line offerings. Projects such as the World Internet Project (http://www.worldinternetproject.net/) can provide valuable information about Internet use. For example, a Pew Internet study (Pew / Internet, 2005) reports that fully 57% of teenager Internet users can be considered content creators through creation of blogs and Web pages, posting original artwork, photography, stories or videos on-line, and remixing on-line content into their own original creations. Furthermore, 44% of American Internet users are content creators. (Pew / Internet, 2004) If the youth segment is a key audience for a museum, this is important to take into account. Audiences are no longer satisfied with passively having information delivered to their screens: they want to remix, re-purpose, and re-use. This may make some museum professionals uncomfortable, as museums have traditionally held the role of interpretation for visitors. However, it is valid to consider that a visitor may re-interpret an artefact with a new meaning. It may not be the official, traditional meaning, but it will be valid for that person and possibly other people in the same demographic. Rushkoff (2005) says that those who are confident in their own core competency have nothing to fear from employees or customers with good ideas. This is equally valid for museums. Early attempts to allow users to create their own on-line museum content included applications such as My Personal Museum on the VMC site, Cybermuse on the National Gallery of Canada, as well as applications on the Metropolitan Museum of Art Web site and many more. There are different variations on this theme, from collections where only you see the content you created to ones where users can post the content they created (http://www.thinker.org/gallery/index.asp). There were generally some restrictions, though. An obvious audience for this type of material is the educational audience. As early as 1995, (McKenzie, 1995) some professionals could see the obvious potential for manipulation of on-line museum content in the educational environment.

CHIN is currently developing a museum learning space where visitors will be able to use museum content and create their own content for educational purposes. Eventually, this space will enable:

• museums to expand their educational outreach by interacting on-line with teachers and students, providing an interpretive dimension of the learning resources, and further engaging the public in Canada's heritage and artefacts
• teachers to develop and share lesson plans and learning scenarios, define learning outcomes, and invite museum educators to participate in interpretative (Web conferencing) sessions
• students to have access to learning resources that they can piece together, use as the basis to tell stories in their own words, share information with each other, and interact with museum educators.

As mentioned at the outset of this paper, early museum Web sites operated mostly in a ‘delivery mode’ where experts delivered authoritative material that the audience ‘consumed’ without necessarily interacting with it. In comparison to what is achievable with technology today, this was a fairly passive, one-sided type of communication. Projects such as the Museum Learning Space aim to address this.

Now some museums, such as the Tate for example, (http://www.tate.org.uk/netart/) have successfully initiated projects where users are invited into a dialogue, generally on a topic likely to invite comments. Other organizations, such as the BBC, are demonstrating innovation in their use of technology (http://www.bbc.co.uk/) allowing repurposing of their content, as well as having a ‘Have Your Say’ section, which they keep very timely by having the lead questions linked to world events of the day.

However, one of the challenges museums still face is how to effectively balance the organization’s need to retain its subject matter expertise with the users’ needs to interact. The authors of VMC - The Next Generation have suggested this dilemma could be resolved by

… developing a lively communications platform that provides access, communication and social spaces to meet audience needs to collect, relate, create and donate activities, without forgoing its commitment to collective memories; a platform that encourages individual points of view but provides access to usable authoritative information as the audience desires. (Dietz, Besser, Borda, and Geber with Lévy)

As such, CHIN has embarked on a research project to create a Virtual Museum of Canada Participatory Space. This interactive communications forum will enable users to engage in dialogue with museum experts and with other users about the content of VMC virtual exhibits. (These exhibits are created by CHIN member institutions whose proposals to the VMC Investment Program are recommended by the Program’s editorial board. For more information see: http://www.chin.gc.ca/English/Members/Vmc_Investment_Program/index.html). For this research project, CHIN is partnering with the National Gallery of Canada (NGC) to create a pilot participatory space for the NGC’s on-line exhibit, Drawing With Light (http://cybermuse.gallery.ca/cybermuse/youth/dwl/index_e.jsp).

Planned functionality for the space includes synchronous and asynchronous discussion tools - message boards, live text chats, and Web casts - as well as a calendar of events to which users can subscribe. Discussion in the space will be facilitated by a museum subject matter expert who will act as a ‘host’ as defined by Hamman (2005). The host will be responsible for establishing discussion topics, responding to inquiries, managing on-line conversations and bringing them to a logical conclusion. It is important to note that both the exhibit and the participatory space will exist as distinct entities on-line, thereby retaining the integrity of the authoritative content developed in the Drawing With Light exhibit (a link from the exhibit will be provided to the VMC Participatory Space and vice versa). However, it is expected that the participatory space will accommodate object-based interaction, thereby enabling discussion about a particular photograph from the exhibit and the opportunity to create links between the exhibit and discussion entity.

One of the success indicators for this research project will be whether there is an increase in visits to the Drawing With Light exhibit as a result of increased interest from the Participatory Space. Other quantitative measurements will include the number of visits to the VMC Participatory Space, the duration of time spent in the Space, and the number of repeat visitors. These figures will be examined in context. Since one of the key questions to be answered is whether a communications forum can facilitate deeper engagement with on-line heritage content, qualitative measurements will also be used to analyse the content and quality of the discussion that takes place. The content analysis coding scheme, discussed previously in this paper, that has been established to evaluate the VMC feedback messages will be adapted and applied to the evaluation of the VMC Participatory Space.

Current plans will see the VMC Participatory Space publicly available for a two-month period, during which data will be gathered. Following this period, CHIN will analyse the data, evaluate the project, and determine whether the project should be implemented on a wider scale to enable other creators of VMC exhibits access to the participatory space. Preliminary analysis of feedback messages sent in response to the VMC’s Community Memories landmark (a collection of local history exhibits) suggests that visitors are already very engaged with this resource and would welcome a Participatory Space in which to communicate and share information. An analysis of messages received since April 2003 shows that the intent of the messages can be broken down into the following categories: to communicate an error in the information (often comments such as “my grandfather’s name was Joseph”), to offer additional background information, to offer further content (photos, letters), and to submit requests for genealogical information. Additionally, CHIN’s redevelopment project for Artefacts Canada, CHIN’s national inventory database has a requirement to allow posting of comments about individual artefacts.

One final way museums can use new technologies is as a tool to monitor content and dialogue about the museum itself. Most museums follow traditional forms of media to track whether the museum is in the news, but blogs are gaining strength. Many corporations are now looking to blogs to see what the public is saying about them, and museums would do well to follow this practice. Blog search engines such as Technorati, Google’s Blogsearch, Blogpulse and others can inform a museum about what is being said about it in the on-line environment. The nearly 1 billion people (Hof, 2005) on-line worldwide with their shared knowledge, social contacts, on-line reputations, computing power, etc. are rapidly becoming a collective force of unprecedented power, and museums would be wise to be aware of it.

Technology has changed information delivery. Museums must utilize these new technologies to take advantage of the growing public interest in creating content and having an on-line voice.

References

Dietz, S. (2004) and H. Besser, A. Borda, and K. Geber with P. Levy. Virtual Museum (of Canada): The Next Generation, 37. http://www.chin.gc.ca/English/Members/Rethinking_Group/index.html.

Hamman, R. (2005). Moderation and Hosting - What? Who? Consulted October 8, 2005. http://cybersoc.blogs.com/cybersoc/2005/04/moderation_comp.html.

Hof, R. (2005). The Power of Us. BusinessWeek Online. , June 20, 2005. Consulted January 25, 2006. http://www.businessweek.com/magazine/content/05_25/b3938601.htm.

McKenzie, J. (1995). From Now On: A Monthly Electronic Commentary on Educational Technology Issues. Vol 5, No 5, January 1995.

McNealy, S. (2005). The Participation Age: A Letter from Scott McNealy. Consulted January 25, 2006. http://www.sun.com/aboutsun/media/features/participate.html.

Meeker, M. (2005). The Age of Engagement. Consulted January 25, 2006. http://www.morganstanley.com/institutional/techresearch/pdfs/age_of_engagement.pdf.

Peacock, D. (2002). Statistics, Structures & Satisfied Customers: Using Web log data to improve site performance. In D. Bearman and J. Trant (eds.). Museums and the Web 2002: Proceedings. Toronto: Archives & Museum Informatics, 2002. Consulted January 25, 2006. amphttp://www.archimuse.com/mw2002/papers/peacock/peacock.html.

Pew Internet & American Life Project (2004) Reports: Online Activities and Pursuits. Content Creation Online. Consulted January 25, 2006. http://www.pewinternet.org/PPF/r/113/report_display.asp.

Pew Internet & American Life Project (2004). Reports: Online Activities and Pursuits. Virtual Tours. Consulted January 25, 2006. http://www.pewinternet.org/PPF/r/143/report_display.asp.

Pew Internet & American Life (2005). Project Report: Teens and Technology. Consulted January 25, 2006. http://www.pewinternet.org/PPF/r/162/report_display.asp.

Rushkoff, D. (2005). Get Back in the Box Thought Virus #2: Open Source and the Authorship Society. Consulted January 25, 2006. http://www.rushkoff.com/2005/11/get-back-in-box-thought-virus-2-open.php.

Schwartz, J. (2005). The Participation Age. Consulted January 25, 2006. http://blogs.sun.com/roller/page/jonathan/20050404.

Cite as:

Carey S. and Jeffrey R., Audience Analysis in the Age of Engagement, in J. Trant and D. Bearman (eds.). Museums and the Web 2006: Proceedings, Toronto: Archives & Museum Informatics, published March 1, 2006 at http://www.archimuse.com/mw2006/papers/carey/carey

• Workshops
• Sessions
• Speakers
• Interactions
• Demonstrations
• Events
• Exhibits
• Best of the Web

• produced by
  
• Join our mailing list
• Search A&MI
  

published: March 27, 2006
analytics code updated: Oct. 28, 2010

Archives & Museum Informatics, 158 Lee Avenue, Toronto, Ontario, M4E 2P3 Canada
Telephone: +1 416 691 2516 | Fax: +1 416 352 6025 | E-mail:
Copyright © 2006 - Archives & Museum Informatics - All rights reserved.