We're Building It, Will They Use It?  The MOAC II Evaluation Project

Anne Gilliland-Swetland, UCLA Department of Information Studies; Layna White, San Francisco Museum of Modern Art; Robin L. Chandler, Online Archive of California, The California Digital Library; USA

http://www.gseis.ucla.edu/~moac

Abstract

Museums and the Online Archive of California II User Evaluation (MOAC II) is a two-year study funded by the Institute of Museum and Library Services (IMLS) that seeks to evaluate the needs and behaviors of four key user constituencies of MOAC: 1) K-12 teachers; 2) university students; 3) academics in the humanities and social sciences; and 4) museum professionals, librarians, and archivists, with the goals of generating recommendations for improving the usability and value of MOAC as well as encouraging its use in research and learning contexts.  The study seeks to contribute to the continued enhancement of MOAC, first conceived in 1997, a museum/library collaboration that is building aggregated, publicly accessible collections of digitized museum content and their metadata.  This paper first reviews the development of MOAC, with a particular emphasis on the nature of its contents as well as its metadata infrastructure.  It then discusses the triangulated research design that is being applied by the study.

Keywords: research methodologies, user studies, digital libraries, Encoded Archival Description, metadata

I. Building it: Museums and the Online Archive of California (MOAC)

MOAC is the museum component of the Online Archive of California (OAC), a Web-based collaborative cultural information resource managed by the California Digital Library (CDL), designed primarily for users engaged in research and learning activities at every level.  The OAC comprises over 7,400 on-line finding aids and related digitized primary sources that have been contributed by a diversity of California archives, libraries, and museums.  The primary sources, of which there are over 120,000 digital surrogates and 50,000 pages of transcribed texts, include letters, diaries, manuscripts, legal and financial records, photographs, maps, architectural and engineering records, scientific logbooks, electronic records, sound recordings, moving images, oral histories, artifacts, ephemera, paintings and drawings, sculpture and ceramics, masks, textiles, cultural objects, and artists' books.

Finding aids provide detailed, highly-structured documentation of the context and contents of selected holdings from the over ninety museums, archives, and libraries participating in the OAC.  A finding aid generally takes a hierarchical approach that begins with an overall collection level description, followed by descriptions of each series or subset or thematic component within the collection, and finally, detailed item level descriptions within each series, as appropriate.  Examples of types of finding aids include collection indexes, inventories, registers, and guides.  MOAC partners create finding aids or collection guides and provide them to the OAC, which delivers them as part of the resources provided by the California Digital Library.

MOAC was first conceived in 1997, when several California museums, led by the Berkeley Art Museum/Pacific Film Archive (BAM/PFA), approached the CDL with the idea of forming the MOAC collaboration to build upon the newly developed Online Archive of California by integrating descriptions and digital images of museum collections drawn from partner institutions throughout the state.  Through the support of an Institute for Museum and Library Services (IMLS) National Leadership / Library and Museum Collaboration grant (1999-2002), the MOAC collaboration extended the potential of the OAC in several ways:

1. It tested the viability of using Encoded Archival Description (EAD) as a data structure standard for museum objects as well as archival materials;
2. The nature of collections of individual museum objects provided a rich basis for examining how well EAD could cope with item as well as collection level description and provide a navigation structure for both; and
3. It addressed issues involved with aggregating cultural materials described by museums, with a range of professional and disciplinary perspectives, with cultural materials collected and described by archives and libraries.

As mentioned above, EAD is the data structure standard used to provide the basic metadata infrastructure for all of OAC, including MOAC.  EAD is an Extensible Markup Language / Standard Generalized Markup Language Document Type Definition developed by and for the archival community.  The impetus behind EAD was principally to find a standardized way for archivists to develop finding aids that prior to this were frequently idiosyncratic and in paper form, and then to distribute and integrate them over the Web.  A less overt impetus was a desire to develop an archival descriptive structure that specifically addressed the characteristics of archival materials; in particular, their status as primary sources and their organically collective nature, beyond what was possible using library cataloguing approaches such as MARC.  MOAC's use of EAD to describe collections of museum objects was experimental (and remains so).  With support from IMLS, partners spent the first years of MOAC developing and contributing finding aids or collection guides to the OAC.  This required that, at the outset, partners reach agreement on parameters for representing museum descriptions in EAD: parameters that would be in keeping with practices and guidelines influencing partners in their regular, non-MOAC work, such as cataloguing new acquisitions.

The Society of American Archivists and the Library of Congress's Network Development / MARC Standards Office are the agencies that are chiefly responsible for the maintenance of EAD.  As part of their activities, they have developed a Tag Library of metadata elements and attributes that are included in EAD, together with notes on the definition and scope of each element and attribute and where it may be used in the hierarchical description.  By design, some of these data elements are not highly defined at this point, in order to accommodate a range of institutional practices.  This aspect has allowed MOAC partners to develop rules and guidelines for how certain data elements can be used in order to reflect museums' specialized descriptive practices and anticipated use of the materials they are contributing to the OAC.  While this approach accommodates what might be different about all or any museums' descriptive practices and holdings, it still facilitates the aggregation of museum and archival material in OAC because of the use of a common metadata structure (White, 2002).  The resulting MOAC EAD Guidelines are based on the OAC's EAD Retrospective Conversion Guidelines.  Because EAD is a data structure standard and not a data content standard, MOAC partners have also drawn upon relevant content standards such as REACH (Record Export for Art and Cultural Heritage) and the Categories for the Description of Works of Art.

Participation in MOAC is only one response, however, on the part of its partners to external expectations that digitized museum content will be available.  Each partner is also engaged in other public access projects and programs related to their permanent collections.  No partner could afford, financially or politically, to develop content in such a way that it could be used solely for MOAC.  Partners cull the information they need to include in their finding aids or collection guides from their local information systems.  In other words, they export to EAD the descriptive, administrative, and technical metadata specified by the MOAC project.  In developing MOAC, the museum partners did not test EAD to see whether it could play a role in primary object cataloguing, documentation, or management, since the metadata necessary to sustain these activities is much more complex and layered than that contributed to the OAC.

For MOAC, partners are assisted in the production phase by digitization management tools developed in part to automate conversion of metadata to the targeted EAD format.  Several museum partners, for example, use a digital assets management database first developed by the Berkeley Art Museum/Pacific Film Archive (BAM/PFA) to produce EAD collection guides (Waibel, 2000).  The tool makes it easy for partners to export collection and object data to OAC-compliant EAD.  MOAC partners also implemented the Making of America 2 (MOA2) standard and are preparing to implement the Metadata Encoding and Transmission Standard (METS) in order to present structured records for objects having more than one image; such as portfolios, scrolls, and artists' books (Rinehart, 2001).

The hierarchical design of EAD reflects the archival practice of organizing holdings into broader to narrower arrangements - that is, into collection (or other type of aggregation), series, and item level components.  Similarly, MOAC collection guides may follow the museum practice of placing objects into meaningful collective arrangements.  Unlike archival aggregations, however, which are dynamic to the extent that they may either be weeded or continue to accumulate over time but which never fundamentally shift from their intellectual arrangement according to provenance, museums move objects in and out of arrangements based on curatorial or programming interests, as evidenced by some collection guides contributed by partners to the OAC.  For example, the Grunwald Center for the Graphic Arts's Guide to Old Master Prints describes a selection of prints dating from 1500-1800, arranged by artist, nationality and period.  The Center acquired these prints from different donors and print dealers over many years; the prints in this collection do not share a common provenance.  EAD makes it difficult, if not impossible, for individual objects such as these prints to live in more than one collection at a time - which may be a limitation for museums, given the ephemeral nature of museum contexts.  Museums certainly acquire self-contained collections that may be described in an intellectual arrangement based on provenance: for example, BAM/PFA's Guide to the Theresa Hak Kyung Cha Collection is a collection guide which describes a large, single collection of mixed media by Cha, a conceptual artist.

In early 2003, the OAC launched a content management system (CMS) for searching and displaying METS digital objects, independent of finding aids.  Prior to release of the CMS, images and texts - METS digital objects - were accessible solely through browsing or searching the finding aid.  Implementation of EAD and METS prepares the way for users not only to search within a single finding aid and across the entire collection of finding aids, but also to search within the OAC's collection of METS digital objects (Fig. 1).  Initially, testing EAD as a standard for representing museum collections on the Web was one interest of MOAC partners.  Contributing finding aids or collection guides to the OAC to be aggregated with other digitized content was another interest.  Most partners assumed users would find such centralized access to library, archival, and museum content to be efficient or useful.

Fig. 1: EAD and METS provide multiple ways for users to get to content
(diagram courtesy of Günter Waibel)

As the project progressed, partners opened a point of access directly to MOAC digitized content from the project Web site, hosted by BAM/PFA.  Prior to Fall 2003 the MOAC Web site only presented MOAC finding aids or collection guides in the common interface used by the OAC.  In October, 2003, MOAC was the testbed for CDL's new access integration toolkit, developed to provide users with customized access to the METS digital object repository.  The underlying content contributed by partners to the OAC remains the same and is still delivered via the CDL.  Using the access integration toolkit, the MOAC Web site, however, now presents METS objects in a MOAC stylesheet, removed from their respective MOAC EAD collection guides.  In essence, the MOAC Website presents a snapshot of MOAC digitized content.  Employing the access integration toolkit, the MOAC stylesheet uses a specific query syntax to structure a search of the CDL's METS digital object repository.  Users starting searches from the MOAC Web site must link to the OAC if they wish to view the related EAD collection guides or search across the entire collection of finding aids.  In implementing the MOAC stylesheet, partners expressed an interest in providing multiple ways for users to get at their content.  The OAC supports discovery of museum content within the context of diverse institution types, while MOAC provides a Website rich in images, object-centric, and museum focused.

II: Using it: MOAC II User Evaluation

As described above, MOAC partners select collections and objects, prepare object descriptions and digital images, and package that into EAD collection guides and METS digital objects for contribution to the OAC, where they can be searched and also aggregated with digitized content from other libraries, archives, and museums.  However, we are very interested in knowing whether the current structure and content of MOAC are the most effective for use by our intended audiences, and, even more importantly, perhaps, whether those audiences find the material we have selected for inclusion to be interesting and of use.  With funds from a second IMLS National Leadership grant (2002-2004), MOAC partners are collaborating with researchers from the UCLA Department of Information Studies on a formal evaluation of MOAC - the MOAC II User Evaluation.  The goals of the study include the following:

• Conducting, analyzing, and documenting a formal evaluation of targeted MOAC user groups;
• Integrating what is learned from this evaluation into routine production and distribution activities;
• Developing strategies to encourage the application of cultural resources such as MOAC in research and learning activities;
• Recommending strategies to the California Digital Library for enhancing both MOAC and OAC; and,
• Recommending strategies to the museum, library, and archival communities for improving the value and use of their on-line resources, particularly in research and learning activities.

It is hoped that this evaluation will help MOAC partners understand which content and design decisions result in more, and more effective, use; and where improvements in content, presentation, and delivery might be made.  Such improvements might include tying selection of content more closely to particular subjects, themes, or curricular requirements; and tailoring descriptive metadata and also the presentation of collections and associated digital objects to different user needs and tasks: for example, through implementing task or community-specific navigation and retrieval mechanisms, as well as presentation, searching, and navigating interfaces.

III. Research design

Relatively little evaluative work has been done to date on access to and use of museum digital content by researchers, teachers, and learners, let alone on issues associated with aggregating museum content with library and archive content.  Bennett et al. (2002) report on the Digital Cultural Heritage Community Project, which built and tested an electronic database of historical information collected from participating museums, libraries, and archives and based in part around curricular requirements, then tested its efficacy as an educational resource in upper elementary school classrooms in three schools in Illinois.  The recommendations of this project, considerably smaller in scope than that of MOAC II, included the following:

• link digitized content to curricular standards: this was judged to be very valuable by teachers;
• ensure the robustness of the database;
• concentrate more on enabling teachers to utilize images and metadata off-line in the classroom and for assignments, and on high-quality, rather than high-quantity resources;
• undertake continuous evaluation of the database use: this is important;
• ensure that information in the database continues to be ‘trustworthy.'

Educators assigned a high value to the availability of ‘trustworthy' primary source information via the Web (Bennett et al., 2002).  A much larger and longer-term project that has been evaluated from various perspectives for several years is the Perseus Digital Library Project.  Perseus is a digital library on Greco-Roman antiquity that emanated out of humanities scholarship.  One focus of that project is on how different types of digital objects, published, unpublished, and artifactual, can interact with each other.  Crane et al. (2001), based upon their experience with Perseus, argue that well-structured information objects and an understanding of the nature of humanities research are key factors in building complex digital resources of heterogeneous cultural objects and in addressing item versus collection issues. MOAC II will contribute to these areas of research and generate benchmark data upon which future evaluations may draw.

Because there are both scope and temporal dynamics that need to be addressed in this study, as well as complex potential interactions between some of the variables under examination by MOAC II, the evaluation has multiple facets.  MOAC and OAC are not artificial test databases; they are live resources which continue to evolve - as Web-accessible resources do - with changes being made to the interface and new or revised content added regularly.  There is not just a single entry point into MOAC - it can be accessed from two locations on the Web (MOAC and OAC).  Each point of access has a distinct style or look; one point offers a broader, contextualized approach (in the EAD collection guides on the OAC), while the other offers a more limited but object-centric approach (using the METS digital objects on the MOAC Web site).  For the MOAC II study, therefore, we are gathering and triangulating both quantitative and qualitative data from several different sources.  Our goal is to obtain a better overall idea of the complexity of issues associated with providing on-line access to museum content in general, as well as to be able to tease apart aspects that relate to the design of MOAC in particular.  This triangulated approach is similar to one successfully employed in the Online Archive of California Evaluation Project (Gilliland-Swetland, 1998).

MOAC partners want to encourage sustained use of physical objects and their digital surrogates in research and education.  Therefore, MOAC II collaborators are examining the needs and behaviors of four key constituencies of MOAC: K-12 teachers, university students, academics in the humanities and social sciences, and museum professionals, librarians, and archivists.  The first three of these groups represent constituencies for whom it has been assumed a resource such as MOAC would be useful.  The fourth group, professionals working in repositories of cultural materials, are often an invisible constituency in the design of on-line resources such as MOAC.  However, they function on the front line in terms of activities such as reference and educational programming and other mediation activities between users and constituencies drawing upon those resources.

We expect the evaluation to elicit broad issues of concern associated with providing on-line access to cultural content, as well as detailed information about the relevance of MOAC to users, especially in terms of how they might want to use descriptions and images in their work, and the effectiveness of the on-line display and navigability aspects of the OAC and MOAC Web sites.  The following research questions, therefore, are guiding the evaluation of MOAC:

• Why do users use MOAC and how do they use what they retrieve?  In what ways does this vary across different user constituencies and disciplines?
• What do users need to understand about the nature of museum objects as information resources and to what extent does MOAC support and enhance that understanding through both description and presentation?
• In what ways can MOAC be used to enhance classroom teaching at K-12, undergraduate, and graduate levels?
• Do the existing structure, representation, and presentation of MOAC privilege certain users, needs, or search strategies over others?  If so, which ones, and in what way?
• Do user search strategies focus on MOAC, or do they extend across all of OAC and other CDL resources?
• What types of search strategies are users attempting, and how successful are they?  To what extent can failure be attributed to user error or system failure?
• What instruction do users need in order to be able to maximize successful use of MOAC, and how should this instruction be best provided (e.g., on-line, in the classroom, in repositories)?

IV. Data sources

MOAC II data sources comprise transaction logs, pre-existing use data, feedback forms, high-level questionnaires, and in-depth interviews with participants (discussed below in further depth).  We are developing the research instruments in phases because we expect data collected in the first phase of user evaluation — for example, using high-level questionnaires — will affect task-oriented instruments used for the second phase of evaluation — the in-depth interviews with participants.  Once gathered, data will be analyzed in light of the goals and research questions guiding this study.

The Website for the MOAC II study has three main functions: 1) to communicate information about MOAC II to any interested parties; 2) to provide a restricted, collaborative research space for the research team; and 3) to serve as a data gathering tool.  The latter, for example, includes making questionnaires available on-line to evaluation participants and collecting the data submitted by questionnaire respondents.

a. Transaction logs

Existing transaction logs, which generally yield quantitative data at a fairly generic level, will be re-instrumented by the research team in an attempt to capture more specific data about the volume and geographic spread of traffic on the two Web sites.  We will use transaction logs for the MOAC Web site hosted by BAM/PFA to collect data for users starting inquiries there; we will also examine transaction logs for the OAC, in order first to track users starting at the OAC Web site, and then to track MOAC users directly.  We expect transaction logs to provide data about how people get to MOAC content, what kind of session trajectories they make when inside the sites, and where people go on the Web, after leaving MOAC.

We will not seek to identify individual users through the transaction logs; rather, we will investigate activities at the broad user level.  We are not concerned about whether one or many individuals are logging in from the same IP address, but are interested in identifying gross patterns that occur from within the same domain: for example, the volume, peak hours of use, and use patterns of users at an IP address, especially if we know that the address corresponds with a school or university site.  We may be interested to know how much traffic is coming from the K-12 community and at what times (e.g., during the school day or in the evening).  That said, we will use transaction log software to track the individual sessions of participants, in conjunction with in-depth interviews.  Quantitative data from transaction logs will be compared against qualitative data collected in high-level questionnaires and in-depth interviews.

b. Pre-existing use data

What happens after a user visits the MOAC Web site?  Do we know if people are currently using MOAC content in their research and learning activities?  To get a picture of the current usage of MOAC content, we asked museum and library partners to forward any statistical or qualitative pre-existing data relating to use that they have gathered since contributing content to the OAC.  This data might suggest that access to MOAC content has affected use of physical collections (e.g., a change in inquiries about collections or objects may be tied to the implementation of MOAC).  Partners were asked to pass whatever data they had, in whatever form, to the research team, excluding any personally identifiable information on individual users, such as patron or visitor names.

Responses to this request suggest there may be differences between museums and libraries in their readiness for evaluation.  Most museum partners did not have pre-existing use data readily available, whereas MOAC's library partner had statistical and qualitative data at the ready.  Museum professionals may be less likely to keep track of how patrons discover or learn that the museum owns an object.  When such data is collected by museums it may be anecdotal and, even then, it could be difficult to tie the effect of MOAC to inquiries about physical collections because a museum's Web presence may not be limited to MOAC.

c. Feedback forms

The research team developed a generic feedback form to ensure that anyone and everyone (including those outside the four targeted user constituencies) is given an opportunity to comment on MOAC.  The feedback form is used to collect fairly general information about a person's experience using MOAC.  For example, is there enough content?  Were you able to move around the Web site without getting lost?  Were the results satisfactory?  We anticipate receiving feedback from two extremes: from frustrated users lodging complaints, and from satisfied users extending complements.  Partners plan to continue making the feedback form available after the close of the MOAC II project, as a means of collecting data from users on a regular basis.

For MOAC II, we provide a passive link to the feedback form on the OAC and MOAC Web sites.  The form is packaged in Websurveyor, the on-line surveying service used by the CDL.  We are also making the feedback form available in hardcopy next to public workstations at OAC repositories.  Few MOAC museum partners, however, have public workstations around which to place copies.  Museums that do have public workstations may use them to provide access to the museum's Web site or its local collection database - not necessarily to provide access to MOAC or the OAC.  Given the limited experience of museum partners in evaluating these kinds of activities, one set of recommendations coming out of MOAC II might be about how to improve the collection and evaluation of use data by museums.  Such data could help museums learn what, if any, impact the availability of the on-line resources in which they are investing (like MOAC) is having upon collections and services.

d. High-level questionnaires

The feedback form provides one means of collecting information from users about their experiences with MOAC.  We will seek additional, more detailed information from the targeted user constituencies through high-level questionnaires, also administered in Websurveyor.  Audience-specific questionnaires have been developed for each group, and selected participants will be directed to the questionnaire appropriate for their group.  From responses to these questionnaires, we are hoping to probe what is important to individuals about Web-accessible resources like MOAC.  For example, is consistent presentation important?  Is breadth and diversity of content important?

The high-level questionnaires are made available on the MOAC II project Web site to selected evaluation participants via password login.  Participants in this phase, therefore, do not have to travel to the researchers, nor vice versa, and the processes of administering and completing the questionnaire is also likely to be faster.  At some point after the research instruments have been administered, we will make the instruments, including the questionnaires, more widely available (primarily to other researchers).

Museum and library partners in MOAC II have provided considerable feedback about the research questions and instruments, and their involvement has revealed the importance of collaboration in evaluation.  As practitioners, partners bring an understanding of audiences and content to bear when validating the research methodology - making the methodology applicable for this particular project and community.  In developing the questionnaire for the information professionals user group, for example, the research team learned - rather, confirmed - from partners that terminology used by museum professionals differs from that used by library and archive professionals. For instance, the words collection and context mean different things to different professions, and the closest approximation of the concept of the finding aid as understood by archivists is, in the museum environment, the collection guide.  As a result, the research team adjusted the language in the questionnaire to match, as nearly as possible, that used by professionals.

e. In-depth interviews

We plan to invite individuals who have completed a high-level questionnaire to participate further through an in-depth, structured interview lasting approximately two hours.  Interviews will be conducted individually wherever practicable, but the research team may choose to conduct some interviews in small focus groups of participants from the same area: for example, a focus group of high school teachers in Northern California.  Focus groups would be used only in situations where the team may have to travel to an interview site and interviewee availability is limited, or in situations where the team believes participants might be more forthcoming or thoughtful when interviewed in a group setting.

The interviews will include a ‘think-aloud' session, with the research team working one-on-one with individuals in controlled sessions, engaging with MOAC and the OAC.  Individuals will be asked to conduct two on-line searches.  The research team will predetermine one search topic for these sessions. In addition, we will ask individual participants to conduct a second search based on a topic of their choosing.  We are interested in identifying the more successful search strategies (that is, successful in terms of whether participants can find objects and can find information about the objects, as well as which on-line resources they use in searching and how efficiently).  Transaction log software will be loaded on local machines and instrumented to capture individual user transactions and to help us learn which content participants are trying to locate and how, and in turn, what tools they need to achieve those objectives.  The controlled ‘think aloud' sessions will be videotaped to show what is happening on the monitor and with the participant.

Because these sessions require much more time from the participant and research team, the number of participants may need to be limited by the team, or may be limited naturally by the number of participants who volunteer to participate.  Research has shown, however, that a small number of participants will uncover the majority of usability issues (Neilson, 2003).  The research team is using non-intrusive data collection methods - transaction logs and the generic feedback form - to balance the difficulties of recruiting and keeping participants.

 V. Research subjects

MOAC content, in the form of EAD collection guides and METS digital objects, might be expected to appeal to educators, students, professionals, and independent researchers - or people with a self- or teacher-motivated interest in cultural materials.  MOAC II partners are examining and comparing the following potentially overlapping or complementary user groups.

a. K-12 teachers

The research team is identifying teachers from a range of grade levels and curricular subjects in two ways.  First, we are approaching teachers who have participated in two other University of California projects that involved teachers working with primary source materials held by the University.  The first of these projects is the UCLA Institute for Primary Resources, a collaborative project between the UCLA Seeds University Elementary School and UCLA Department of Special Collections.  This project brings Southern California teachers to UCLA to work with special collections and integrate them into their course units.  The second project is the Interactive University Cultural Heritage Pilot Project based at UC Berkeley which has worked with a somewhat smaller group of teachers, introducing them to the Berkeley on-line archival collections.  In general, the research team would like to attract to the study people who have used resources like MOAC in the past as well as people who are new to such projects and tools.  To that end, the team asked staff responsible for educational programming and K-12 outreach in each partner institution to identify teachers with whom they have already worked so that we may contact them about participating in the MOAC II study.

Recruiting teachers may be challenging since they are often overwhelmed with their regular work.  The research team has some funds available to support substitute teachers, during the time teachers are engaged in the research activities.  We will also arrange the in-depth interviews with teachers in locales and at times that best suit the teachers' work schedules.  Museum partners may also be able to offer tours and activities in their museums for students while the teachers are participating in the interviews.

In building MOAC, museum and library partners confirmed differences in how museums and libraries describe and represent objects.  For example, museums and libraries use different vocabularies to describe their objects, and these differences become more apparent when content is aggregated on-line.  Added to this, museum and library partners are aware that scholars may expect descriptions of objects and collections to use scholarly terms, while other users may need non-expert terms that match language or subjects used in textbooks and curricula.  Do the finding aids and collection guides privilege scholars over others?  Do people want to bypass the finding aid and go straight to the objects?  MOAC partners are interested to know from teachers how a tool such as MOAC can be packaged to make it easy for them to use.

b. Students in the humanities and social sciences

Methodologically, and because of human subjects' constraints, it is difficult to identify students, especially at the undergraduate level, to participate in research of this kind.  We are identifying students for MOAC II in three ways:

1. by asking faculty members in whose courses students have previously been enrolled or who have advised masters' or doctoral students using archival resources;
2. by asking staff in participating OAC repositories to distribute an invitation to participate in the research to any students who fall into the targeted student group;
3. by a call for participation posted within humanities and social science departments on campuses. 

While we are aware that such methods of selecting students are unlikely to yield a sufficiently large sample to be representative of all possible student uses and needs, we believe that this aspect of the evaluation will nevertheless provide us with valuable insight into how students currently or potentially might integrate MOAC into their school activities and workflow.  It will also provide valuable insight into student search strategies and aspects of MOAC that they might find intriguing, difficult, or frustrating.

c. Academics in the humanities and social sciences

Although MOAC could be valuable to scholarly users in an almost infinite number of disciplinary areas, it is beyond the scope of this project to carry out such exhaustive evaluation.  Instead, we have chosen to focus on those academics who are likely to constitute the most frequent or substantive users of a resource such as MOAC.  Accordingly, we are drawing a systematic sample of faculty across the University of California, in the humanities and social sciences, that represents a range of disciplinary areas.

Taking a systematic survey requires great effort and, in the end, might not result in a large number of participants.  Therefore, the team will identify a second group of faculty through their known use of museum resources in their scholarship and teaching.  Museum and library partners have been asked to distribute an invitation to participate in the study to any faculty researchers who fall into the targeted faculty group.  These may be faculty with whom partners are already in contact, such as repeat visitors to a collection.  We may also draw information about the usefulness of MOAC from existing faculty advisory committees at partner institutions: for example, soliciting comments from committee members about when and where MOAC resources might be used in teaching and research.  Depending upon resources and the true timeframe of the MOAC II project, partners may work with faculty to get MOAC content integrated into classroom instruction so that it can be tested in a live setting.

d. Museum professionals, librarians, and archivists

While professionals are heavy users of on-line resources, it is uncommon for user evaluations to focus on them.  The research team will ask information professionals to identify what about MOAC works and what does not for them and their patrons.  One recommendation coming out of MOAC II might be to find ways to better publicize resources like MOAC.  In preliminary work, we are learning that even professionals, often the intermediaries between resources and patrons, may not know about MOAC.  Part of the difficulty is that MOAC content is not readily apparent on the OAC and CDL Web sites - keeping in mind that those two sites offer a vast amount of content and services.

We have asked MOAC partners and other contributors to the OAC to identify staff members and colleagues who work in capacities such as reference, education, and collection management.  The names of these staff members will be compiled into a list from which we will draw a systematic sample representing each type of institution.  We will then invite the selected staff to participate in the evaluation of MOAC.

Summary

MOAC II is about discovering what needs to happen for access to digitized museum content to be useful, usable, and used.  From results of the study, MOAC II partners will develop recommendations for improving on-line resources, particularly for education and research activities.  Museum and library partners will also assess the practicality of integrating what users want into routine production and distribution activities - the implication being that partners will integrate what is learned from evaluation into their regular work.  One other expectation is that the project will make recommendations for best practices regarding how to evaluate the use, usage, and usability of on-line systems like MOAC over time.  While the content evaluated in this project is specific to MOAC and the OAC, the research model and analysis should be of interest to the broader cultural heritage community, especially in planning and implementing public access projects - when using it is equally important as building it.

Acknowledgements

The authors gratefully acknowledge funding from the Institute of Museum and Library Services that has supported the work discussed in this article.  The authors would also like to thank the MOAC II partners and graduate research assistants Carina MacLeod and Kathleen Svetlik for their assistance with this paper.

References

Bennett, N., B. Sandore, and E. Pianfetti (2002). Illinois Digital Cultural Heritage Community - Collaborative Interactions Among Libraries, Museums and Elementary Schools. D-Lib Magazine Volume 8, Number 1. Consulted January 29, 2004. http://www.dlib.org/dlib/january02/bennett/01bennett.html

Crane, G., R. Chavez,  A. Mahoney,  T. Milbank,  J. Rydberg-Cox , D. Smith,  and C. Wulfman (2001).  Drudgery and deep thought: Designing a digital library for the humanities. Communications of the Association for Computing Machinery, 44(5), 35-40. Consulted January 29, 2004. http://www.perseus.tufts.edu/Articles/cacm2000.pdf

Gilliland-Swetland, A. (1998). Evaluation Design for Large-Scale, Collaborative Online Archives: Interim Report of the Online Archive of California Evaluation Project. Archives & Museum Informatics 12, nos. 3-4, 177-203.

Neilson, J. (2003). Ten Usability Heuristics. consulted January 29, 2004. http://www.useit.com/papers/heuristic/heuristic_list.html

Rinehart, R. (2001). Museums and the Online Archive of California: Museum Consortia, Digital Library Projects, Standards for Complex Multimedia Objects, and More Fun than a Barrel of Monkeys. SPECTRA 28, no. 1, 20-27.

Waibel, G. (2000). Produce, Publish and Preserve: A Holistic Approach to Digital Assets Management. SPECTRA 26, no. 7, 38-44.

White, L. (2002). Museum Implementation of Encoded Archival Description. Art Documentation 21, 1, 15-20.