Cruising Through History Wired

Patricia Gillard, User Insite Pty Ltd, Australia

Abstract

This paper presents the results of an evaluation of History Wired conducted at the National Museum of American History (NMAH), Smithsonian Institution, Washington, DC, during March and April 200l prior to its launch in August, 2001. The methods were based on ethnographic and constructivist approaches rather than heuristic or usability testing.  Eleven groups of visitors (31 individuals) viewed the History Wired site which was in development at the NMAH. Visitors were asked to use the site together for five to ten minutes, and then to answer some interview questions. Observation of their uses of navigation, as well as choices for more information and general movement around the site, were noted. Their comments during the session as well as the interview were taped for later analysis.

The results showed that all visitors were very engaged with the site, even if they had limited Web knowledge. The aims of designers, to give direct access to the user, and to provide many ways of finding and enjoying objects, were well realized. Visitors used the site in distinctive ways, including ‘reading’ from left to right, following subject interests and themes by use of the keywords, and navigating through objects. The study predicted the site’s success with audience groups and suggested wording changes to make the interface more descriptive of the actual contents. Once visitors used the Web site, they were likely to want many more objects to be available on-line and to make further enquiries at the Museum site.

Keywords: evaluation, ethnography, user research, audiences, interviews, interface design, usability testing

Introduction

This paper presents a Web site evaluation with a difference. Its approach builds on audience and communications research of the past fifteen years and can be described in broad terms as semiotic, ethnographic and constructivist. User Insite has been applying these approaches to evaluation of Web sites since 1997, but the ideas on which they are based have a longer history. In communications research there has been theoretical discussion of the ways the contents of television and new media create and position audiences and audiences negotiate these positions to construct their own meanings. Examples are found in Reception research (Charlton & Bachmair, 1990), Cultural Studies and Audience Ethnography (Livingstone & Bovill, 1999; Howard, 1998; Ang, 1996; Buckingham, 1993; Moores, 1993; Lull, 1990) and Naturalistic studies (Lindlof, 1987).

My own research began with close descriptive studies of children viewing television in the 1980s, (Palmer, 1986). Audiences have always been interactive in their behavior and active in constructing meanings, but this was usually downplayed by the media industries in the production process . Now we have a new medium where the relationships between audience members and contents must be made visible in the production process and ‘built in’ to the design. Research appropriate to the new Web medium seeks the patterns in users’ engagement with Web contents and relates this to the everyday contexts of users (sometimes referred to as audiences or visitors).

The Museums and the Web conference has been a leader in tracing the shape of developments in Web site evaluation. We are very pleased to be presenting our own work here, and continuing the discussion about evaluation research to improve museum Web sites.

Aims of the project

The aims of this research were:

• to describe the variety of ways that visitors to the physical museum engaged with the History Wired interface and its contents
• to seek visitor suggestions for changes or improvements
• to gauge reactions to the name
• to discern whether the intended aims were being realized
• to maximize the effectiveness of the program

HistoryWired: A few of our favorite things (http://historywired.si.edu) is described as ‘an experimental program through which you can take a virtual tour of selected objects from the vast collections of the National Museum of American History. Here you'll have an opportunity to look at hundreds of museum artifacts, most of which are not currently on exhibit’. The Web site uses a ‘tiled’ interface, designed by Smartmoney.com, to give direct access to descriptions of over 400 objects. The text and associated materials were especially written for the interests of broad audiences who are not necessarily museum visitors.

Because History Wired is database driven, changes could be made to the site to reflect the actions and interests of audience members, as they became known through their actual uses and comments.

This paper describes the major patterns that emerged in a close study of thirty-one participants in eleven sessions. These patterns would be expected to recur in many engagements with the Web site, but there may be additional forms of engagement that have not yet been observed.

Methods

Between 12 and 19 March 2001, research sessions were conducted with a variety of visitor groups to the NMAH. The purpose of the sessions was to observe the spontaneous interests of users rather than setting predetermined tasks. This way, we could discover how the Web site shaped a variety of responses, and how users created their own experience from the many possibilities of navigation and content. An open-ended interview was included to ask questions of importance to the Web site creators, as well as to gain comment about matters arising in the observation from the users’ perspective.

Eleven interviews were conducted with naturally occurring visitor groups. This included a total of 31 people. Participants were selected from those in attendance at the museum at different times of day. Participants were approached and asked if they would be willing to look at a new Web site that was being developed for the museum. If they agreed, they were taken a short distance to a conference room, the Lemelson Center, from which they could still see out to the public areas. The test site was available on a laptop computer, and participants made their own decisions about who would use the mouse.

The two distinctive methods were as follows:

• An observation session, where the researcher sat alongside visitors as they explored the Web site. The observer made notes of actions by participants and noted new content pages. She would answer questions and show features if visitors asked or stayed ‘stuck’ for some time.
• An open-ended interview, when questions were asked of the visitors about content, navigation, favorite objects, missing features, likely audiences and responses to the name.

Sessions varied from 13 minutes to one hour, with larger groups taking a longer time. The average session time was 25 minutes. More than half came from other states, on a holiday or on a school trip. Two groups lived nearby and were on a break from work.

The following groups were involved in the research:

• Father and adult son (20s), visiting to follow up father’s war experience
• Four friends from college, visiting during college break
• Six 8^th grade students (13/14 years old) on a school excursion, (2 boys, 4 girls)
• A retired couple and their adult daughter, a teacher
• Mother (40s) and son, (20s) visiting from Texas
• Three 8^th grade girls, friends, touring Washington (with school) from Albuquerque
• A couple, late 20s, Washington
• Family of 4 from Texas, first time in Washington. Son 15 or 16, daughter 11or 12.
• Male college lecturer and former high school teacher, social scientist
• Mother (30s) and daughter, 6^th grade
• Two female friends, (20s), health professionals

The observation sessions and interviews were taped and later transcribed. The transcriptions and observation notes together gave a full description of the session, and these accounts formed the basis of an interpretive analysis. A ‘history’ of each session recorded the actual pages visited by each group as part of the Web site browser activity, but this data has not been included in the analysis.

Observations

Initial reactions and general comments

In the observation stages, everyone began by moving the mouse across the map. Only one person tested the site by himself; the others were with friends or family. Perhaps as a result of this, comments and talk accompanied the use of the Web site. The talk was sometimes quiet and directed and other times noisy and expressive. Many unsolicited comments were made about the interface during the observations. From the beginning all the comments were positive, even glowing. Examples were:

In’t (sic) that cool?

That’s really neat

This is amazing it really is- you just open it all back up again

This is really something. Very impressive

In three of the sessions, people mentioned they could spend ‘a lot of time’ or ‘all day’ looking at the site.

In most sessions, one person would read out the titles of the boxes or images, or they would read them out together. When they saw an image, for example ‘Inside the Lone Ranger’s Mask’, or read a description, there was often laughter or personal comment. For example, in response to the knives displayed in a Yiddish shop sign: ‘Wild! Don’t think I can handle that’.

Distinctive forms of engagement with the Web site

As with all communications media, History Wired appealed to particular kinds of audiences. The creators of the site wanted to present and describe objects in ways that would appeal to broad and non-specialist audiences. The innovative nature of the Web site interface also raised questions about how it would be put to use by different audience groups.

The following were the distinctive patterns of use that were observed.

Using the mouse to travel left to right across the map, returning to the left and reading across again.

This approach mirrored conventional ‘reading’ and was only observed in one session. As her cursor moved, the user read pop-up labels but did not click and explore further images and information. When this possibility was pointed out towards the end of the session, she tried it a couple of times but mainly continued to read the map left to right . The woman was visiting the museum with older parents and was a grade school teacher, probably in her forties. She did not use the Internet at home, only at school in connection with teaching.

Quickly exploring the dimensions of the content, both the surface geography of the map and the information pages behind it

One group of six 8^th grade students did this with great enthusiasm and speed, with statements of ‘cool!’ accompanying their discoveries. The person with the mouse, a girl, began by exploring in circles within the main concept groups, in this case ‘Home/clothing’, then added selections of keywords at the top. They discovered there were categories within categories.

Once the back pages were discovered, which happened after a few minutes, they combined fast exploration of the map with clicking on back pages and further information pages whenever various members of the group indicated an interest. This group did not find the timeline or zoom features, but when they were told about them late in the session, they were so excited, they began another round of fast exploration.

Careful exploration of the Web site to answer particular interests.

Three of the visitor groups, who were with family members, including adult children, used the site in this way. One example of this was a father and son who looked at the site, then discussed an author, Stephen Ambrose, associated with military history. They spent most time on directed activity, selecting the keyword, ‘Military’ as well as moving around the relevant sections. They quickly discovered and used the back pages in these thematic areas. As they explored the site (the adult son used the mouse), they commented to each other about what they were finding and what they wanted. They didn’t find specific information about ‘101^st Airborne’, which was what they had come to the museum itself looking for that day, but they did find objects of interest to them.

A second family grouping who used the site in a similar way was a school-aged daughter and her mother. They were remarkable for the intense interest in reading every word of all the information about particular subjects such as the ‘Star-Spangled Banner’. They scrolled down to do this. Again, they were unhurried and talked about the subject matter as they went.

Experimenting with the navigation functions

A fourth approach was guided not so much by thematic content or personal interest as by selection of particular functions that people found interesting to use in themselves. One adult son with his mother explored the site mainly using the zoom function, the topic menu on the left hand side and the enlarge function. It took them some minutes at the beginning to find these and use them, but then they enjoyed the interface itself, choosing back pages for familiar content such as Kermit, Benny Goodman and the Gutenberg Bible.

Seeking the visual experience of objects

A fifth approach seemed to focus on inspection of the variety of visual images, yielding a close engagement with the objects. Two women friends in their twenties found the zoom and enlarge features early and systematically viewed objects of interest by reading out the titles in the zoom display, clicking for the information pages, reading the first paragraph (they didn’t scroll down) and then enlarging the image. They talked about the objects themselves as they did this.

The different patterns of engagement show that History Wired provides a variety of possibilities for different audiences. Even those who have little experience with the Internet can explore the site systematically, gaining pleasure from the thumbnail images and the labels on the map. The intention of the designers, to provide direct experience of a large number of objects and to place in the hands of users many different ways of driving their own experience, has met with very positive responses. Some enjoyed the driving itself while others were more focused on content, or they could hop from image to image. This is a feature which is not characteristic of most Web sites. While the content may vary based on user selections, sites rarely provide different ‘driving experiences’.

Some of the functions, such as the zoom, timeline and drop-down menu, were not used by many. However, they were all used by some, to support their exploration of the site. Confident users, experienced with scrolling and drop-down menus, had much more depth of experience, and it was then their purposes or personal preferences for image or written text that seemed to determine the ways they used the site.

The site was very flexible in providing different reading experiences. There was an immediate engagement with the written descriptions, though most users read the first few paragraphs only. There were no difficulties with the reading, and three groups read all of the text for some of the objects they chose.

Interviews

Overall reactions

Responses to the interview question, ‘What did you like best?’ confirmed the very positive comments made while people were exploring.

It’s like walking around the museum, exactly the same way, it’s also better ‘cos we’re not walking around

I enjoyed all of it

It’s keeping up with technology we have today

It’s really informative and original and it’d be helpful I think

One boy in 8^th grade who had not had a turn with the mouse made the observation:

I kind of like the way it was but I guess some people would want it simpler. My mother who’s completely computer illiterate… I think it would be hard for her.

Comparison with other Web sites

Interviewees liked the originality of the site and the associations it made.

Father: I've never seen a Web site like that. The format is different. Smaller or something.

Son: It’s the first Web site [I’ve] seen with so much stuff and a different way of finding it. Usually you just click. There’s a wide variety, saves a lot of time.

One person interviewed was an educator knowledgeable about Web site design:

If you’re doing research on a particular thing, you can't search immediately. But it’s like you throw a dart at a dartboard, you see other avenues in trying to get a more well-rounded collection of information about this particular thing. Nice.

A college student observed:

I’ve never seen another site that you can actually section up in little tiles like that before…see it before you even open up the page, see what it relates to.

Favorite features

A favorite feature mentioned in four interviews was the connection between the map and the keywords along the top, shown by pointer lines connecting them. Another was the timeline, and the ability to access all objects at once.

Daughter: I like the pictures and the timeline.

Mother: I like the multiple references up at the top, especially as you cursor around, the reference points back up to bar. That's pretty nifty.

I’m really impressed with the connectivity between choices, these lines coming down, works well for me, a kind of graphic orientation. The whole idea of highlighting boxes pertaining to category is very nice as well.

I like the way it’s set up where it’s all in one place. In a lot of places, like, you can still look at the site…it’s simpler, like it’s all more compact.

Favorite objects

People could name their favorite objects. Their answers revealed a connection between the favorites and their own interests. One grade 6 schoolgirl observed:

I liked the Log Cabin. Interesting. A bit of history. I like history a lot.

A nurse commented:

I liked the medicines. It's my job. Anything to do with medicine and medical history I like.

Pleasure in the objects was one of the major satisfactions of the Web site. This was observed rather than commented on in the interviews. As groups explored the front and back pages, they would read titles together, respond, comment and read further with interjections of ‘cool’. ‘neat’,’ Oh that’s nice’. Selection of the objects was successful in providing for the diverse interests of the group tested. The inclusion of some familiar objects enhanced their engagement with the site.

Improvements

There were criticisms and suggestions for improvement of the site. The suggestion made by the majority of visitors was for the addition of a search facility. [The current version of the program now has a search facility.]

I think it would be a better research tool if you had a search engine.

Search. Make it easier… What you are thinking might not be same as what the person who made this was thinking.

In three interviews, there was a request for additional information about objects: whether they were available to be seen; and a floor plan to show their exact locations.

Other suggestions included:

• emphasize the drop-down menu
• add categories such as ‘furnishings’ and ‘accessories’ in smaller lettering to the Home/clothing section on the map
• give more information about categories such as ‘Communication’ on the map
• define the lines around the sub-category areas more clearly
• change color when the box has been selected
• connect this Web site with the Smithsonian and NMAH web environments

Intended audience

Groups were asked who the Web site was especially made for. Half of those interviewed saw it as useful for those who were planning a trip to the museum, or following up afterwards:

You could find a bunch of items that you want to see. When we came we didn’t know what we wanted to see… we had trouble deciding what we wanted to see.

If you were interested in something here you could see if you could find some more information from the Web site.

They defined the following types of audiences:

• ‘Someone that’s out of town and looking to come to Washington’
• ‘Me!’ (a grade school girl)
• ‘Someone who is inquisitive’
• A grade 6 or junior high school student looking for a particular item or time period
• Classroom teachers who are going to give assignments to their students
• Older people because of the easy access

Its particular attraction for younger people was described:

There’s a cause and effect thing going on here. As soon as I move the cursor, a lot of things are happening which would be very engaging to school and high school age children.

History Wired was also thought to appeal to an audience who were interested in just exploring:

If ever you’re not looking for something in particular it would be fun to mess around with it.

The name

People were asked what they thought of the name and what it meant. This is not reported in detail because the original name will probably be used for future on-line exhibitions. It was discovered that the original name polarized the audience . Young people liked it, because it was dramatic and a bit daring. Older people rejected it because it could be misunderstood as an insult. It was uncanny that in one group the students thought up one of the potential names suggested by curatorial staff, and rejected it.

I think it’s cool because a lot of the names you see, it would be like ‘American Artifacts’. I think that would be really boring.

Voting

Participants could vote on how much they liked the description for each object that they viewed. A question about the voting option and whether participants would use it was asked in each interview because very few people used the voting function during the observations. Younger users seemed to like the voting option whereas those beyond school age were not interested, or said they would prefer to vote about something more meaningful like the quality of information.

I like it. You get to have an opinion on something. When you go to vote, it takes you to a screen about what percentage of people said yes and no. Will it do that?

Yes, I vote on Web sites. Like the pop culture and stuff. You vote if you think she's pretty or you think her dress is good. I feel like more of an individual

I’m not likely to do it. At this level it’s sort of a teaser…I would vote on the next level, the depth of information. I’ve not invested much time personally at this point.

Didn’t see it, I would vote if it referred to a description but not just if I like it.

In one family, the younger daughter said ‘yes’ but the father said ‘no’ and the adult son said he ‘didn’t care’.

Specific features

The interviews did not seek comments about every detailed feature of the Web site. However, where comments were made about specific features, they are listed below.

The map

­­First I was lost, confused but after a while it was cool, once you figured out how it worked

The pop-up menus are nice. See what you are looking at. Neat that it overlaps, gets darker

Pretty easy to understand

Regions

I was confused about the black lines cos I know it’s all broken into categories… is this all Communication?

Home and Clothing. (she demonstrates) This part is more like furnishings, such as bowls, and this one’s more like accessories. You have Lincoln’s hat then you have a charm bracelet…it would be better if it said what the divides mean.

Information pages

Oh, these secondary screens have a nice clean look. Meaningful options. This is a pretty substantive lot of information. Here is where I'm more likely to go.

Once I started to use the further information, I got the idea.

Zoom

That is so cool… It has different varieties of all the information. Moves around.

I like it, it’s not so busy. Like taking the map and simplifying it a lot.

Drop-down menu

Headings? Good because you can narrow it down to what your interest is. It makes it a lot easier to use.

Audience and content development

The questions about History Wired concerned the ways in which the Web site attracted particular kinds of audiences, and the ways these audiences engaged with the site. Over time, new audience relationships will be created by the site’s content and innovative design.

The detailed study of audience/content relationships suggests further strategies to develop content and extend the audiences for the Web site. Because these strategies are based on current interactions of audience members with the site, they are likely to succeed.

Audiences

Professionals in the image-related industries (media, publishing, architecture, graphic arts, etc)

NMAH currently receives many requests to librarians and curators about images of objects it possesses, to be used in publications and other communication. History Wired is likely to attract the interest of such professionals.

Genealogists and family historians

People interested in family history are a very large group with strong international research links and excellent Internet skills. They are usually over 45 years of age, with time to research and explore. Contents on History Wired about particular families, or about transportation connected with major immigration movements, are likely to attract this audience.

International audiences and those with limited skills in reading English

The site will reward those who cannot read English language text well, but who learn to navigate and find larger examples of the thumbnail images. The text, written for a general audience, will also be more engaging and will reward the effort of reading. This site is likely to attract an international audience, especially those interested in museums, historic objects, or American history who have previously not accessed NMAH’s on-line exhibitions because of language difficulties. The site will also be welcomed by users with an intellectual disability because the content is complex and interesting to adults but can be accessed with the movement of a mouse. Larger images can be found by responding to thumbprint images, without the need to read captions.

Children

Some children in grade school will enjoy features of this site, especially the way they can drive it, fast, with something happening visually in response. This evaluation included only one early grade-school girl who was intensely interested in the ‘Star Spangled Banner’ and seemed to read every word. It would be worthwhile to consider a special study of children as audiences for this site, so the site’s educational uses can be enhanced.

Collectors, clubs and special interest groups

A number of different groups will be drawn by specialist content which is closely related to their leisure or professional occupation and which opens up a resource of information and images not currently available. For example, auctioneers and retailers of antiques and historical objects may use History Wired to make comparisons with their own objects for sale.

Contents

Special collections and exhibitions

The drop-down menu function gives the possibility of creating special collections as part of History Wired, or featuring new content such as ‘mystery objects’. These items would attract existing collectors and new audiences. A ‘collectors’ topic could be defined in the drop-down menu, then key words could be used to highlight items as collectors define them; for example coins, stamps, football cards.

These parts of the site invite the development of discussion groups or on-line ‘donation’ and sharing of images and information. This might also be a location for creating special exhibitions or temporary collections.

Dialogue with curators and experts

With the current objects, it may be exciting to make available on-line the kind of expert advice from curators and collectors that visitors often seek in the physical museum. For limited periods of time, curators could be available to respond to e-mail questions and comments on particular museum objects. This would be highlighted on the information pages for each object. The interchange between curators and the public would also assist the NMAH to understand how users were engaging with the Museum in new ways through this on-line medium.

Conclusion

One of the distinctive qualities of History Wired is the provision of different means to access, view and engage with the large collection of objects. At the same time, there is a consistency of ‘address’ to the reader; one which tries to engage interest in the special stories or features associated with the object. An analysis of the content would have delineated the features of the writing, language , images and their interrelationship that created the possibilities for engagement that were taken up so enthusiastically by audience members.

Making the different audiences visible in this research has confirmed the importance of the careful work by producers and writers, as well as the significance of the experience that on-line visitors can create from material designed for them.

Research based on ethnographic approaches promises to contribute more to Web site evaluation than a diagnosis of the functioning of the site and a judgment about whether it is ‘user friendly’. Attention is paid to the emerging relationships that audiences create in their engagement with the site and its navigational and content possibilities. Relating this activity to the everyday contexts of audiences then suggests new developments such as extending the contents to satisfy specialist interests or to effectively address audiences whose uses were not originally predicted. The focus on audiences constructing meanings leads to interpretation work on-line that creates more extensive use and deeper understanding by Web site users.

Acknowledgements

This research was supported by an Australian Fulbright Professional Award. It was conducted within the Web Program at the NMAH with the excellent guidance and support of Dr. Judy Gradwohl, Manager, as well as Matthew MacArthur and Curtis Prather.

References

Ang, I. (1996). Living Room Wars. London: Routledge.

Buckingham, D. (Ed.) (1993). Reading Audiences. Manchester: Manchester University Press.

Charlton, M. and Bachmair, B, (Eds.) (1990). Media Communication in Everyday Life. Interpretative Studies on Children’s and Young People’s Media Actions. Munich: K.G . Saur.

Howard, S. (Ed.) (1998). Wired-up. Young People and Electronic Media. London: UCL Press.

Lindlof, T. (Ed.) (1987). Natural Audiences: Qualitative Research of Media Uses and Effects. New Jersey: Ablex.

Livingstone, S & Bovill , M. (1999). Young People New Media. London School of Economics and Political Science.

Lull, J. (1990). Inside Family Viewing. Ethnographic Research on Television’s Audiences. London: Routledge.

Moores, S. (1993). Interpreting Audiences. London: Sage.

Palmer, P. (now Gillard) (1986). The Lively Audience. Sydney: Allen & Unwin.