The KDN team consisted of the DCM project manager and experienced KDE facilitators (including engineers, scientists, and former classroom teachers), the independent evaluator, and the contracted Web site developer's programmer and project director. This group worked closely together for 2 years to develop KDN.

DCM Employees

In-house members of the KDN team included former classroom teachers, engineers, scientists, and exhibit and program designers. They were experienced in conducting effective science programming for elementary children, as well as expert in the development of open-ended problem-solving activities. Other DCM employees involved included facilitators of school and museum floor programs, art specialists, exhibit developers, and DCM management. They provided fresh eyes to review all aspects of KDN, and were a ready source of Web site testers. Since these people were not involved in the day-to-day development, they provided a different perspective on everything from Web site graphics to navigation to teacher guidance to ensure KDN was consistent with DCM's mission.

Advisory Board

DCM regularly uses community-based advisory boards when developing new programs and exhibits. KDN's advisory board was especially important in providing philosophical and practical guidance in the early stages of development. School officials on the advisory board helped insure that KDN would be useful to and usable by classroom teachers. Principals and curriculum coordinators helped pilot test KDN in their school's classrooms. Computer experts gave advice on available technology. Experts in school-university-professional partnerships provided information on existing collaborative networks, how these collaborations worked, and potential pitfalls KDN might face. Engineers helped provide Challenges and advice on how to staff the on-line engineer positions. Since engineers would be using the Web site, their input on the components necessary for effective communication were invaluable.

Students and Teachers

It is part of the culture of DCM that all programs and exhibits are pilot tested first in small groups and then in the setting w successful development. Because teachers are the primary users of KDN, a strong emphasis was placed on meeting their needs. Our ehere they will be used. Feedback from the users is absolutely necessary tovaluation methods needed to identify teacher concerns and ways to alleviate them. We found that teachers' opinions/reactions to KDN after using it with children were markedly different from their reactions before using it. For example, teachers greatly underestimated students' enthusiasm for talking with the engineer and the value of the building process. We therefore tested KDN in classrooms and computer labs with entire classes of children. We held preliminary briefing sessions with teachers, observed and talked with them while their students were using KDN, and whenever possible, talked with them afterwards. Ultimately, 40 classroom teachers participated in the development of KDN. Their input was a critical factor in development.

We knew in advance that elementary teachers have very little discretionary time and are under constant pressure to demonstrate that their students perform well on standardized tests. We needed to ensure that teachers could see the benefits of KDN in achieving educational goals. We also needed to ensure teachers were given enough information to feel comfortable using KDN on their own. Many teachers are not accustomed to using hands-on activities in their classrooms, and many are not adept at using computers.

Pilot testing with children is an absolute necessity because of their widely varying skill levels, interests, and experiences. Sometimes our hypotheses were confirmed in the tests, but frequently we were surprised. Testing with large numbers of children is critical for a program intended for use in classrooms. Children function differently in large classroom settings than they do in small groups in or out of school. Tests in schools also gave important information on Web site performance on school computers with many children (up to 30) using the site's interactive features at once. Every part of KDN was extensively child tested. with over 900 children eventually involved.

Web Site Developers and Programmers

When KDN development began, DCM had no in-house Web site development expertise. Many potential contractors were considered, with emphasis on identifying one which could provide interactive, innovative and engaging programming for children. Educational Web Adventures, LLP (Eduweb) was chosen. Eduweb worked closely with other members of the KDN team in developing the evaluation plan as well as the Web site. In addition to normal extensive preliminary testing of Web site components, Eduweb's programmer and project director were in the classroom observing children for the first full-scale test of the KDN. Subsequently, they frequently served as on-line engineers, getting first hand understanding of how the Engineer chatboard works. The programmer continues to periodically monitor engineer chat sessions and trouble-shoot as needed.

Independent Evaluator

When given the choice, many museums conduct front-end and formative evaluation in-house and save their budgets to bring in an independent evaluator at the summative stage when the project is complete. DCM brought in an independent evaluator at the very early stages of KDN development. This designated one person as the focus for evaluation, and the evaluator became an integral part of the KDN team. She helped define the evaluation methods used, interpreted the data, and provided detailed interim and final reports needed to satisfy grant requirements. Involvement from the beginning of the project gave her valuable longitudinal perspective on development. She offered many suggestions that were implemented as KDN progressed. Had she been brought in at the conclusion of the project (after the grant money was spent), DCM would not have been as able to take advantage of her input.

Working with the KDN project manager, the independent evaluator developed a thorough evaluation plan very early in the KDN development process. The plan detailed her role in the process, the activities she would conduct, the data collection methods to be used, and a topical framework outlining the questions her evaluation would address. The topical framework was used to focus and guide the evaluator's observations and conversations with respondents, but as important new issues emerged, these were often explored as well.

The independent evaluator and DCM staff employed a naturalistic methodology to collect and analyze data (Lincoln and Guba, 1985). Naturalistic inquiry is particularly useful when seeking to understand real life settings and the situations and ways in which people experience them. It begins with a focus of study, and thorough systematic collection and analysis of the data identifies and reports on issues as they emerge. Data is collected from a variety of sources. This "triangulation " of data yields the completest possible description of the setting and program. Naturalistic inquiry often includes the respondents' own language as a way of painting a rich, detailed, and meaningful picture of their experience.

Ultimately, evaluation became a very collaborative and continuous effort. Although the independent evaluator was not present at all tests, she was present for most of the tests where major revisions to the program were assessed. She was also present at the meetings where the team took stock of the program's status and charted the course for the next stage of development and evaluation. The project manager kept her constantly informed via e-mail and phone. Information and opinions freely flowed to and from the evaluator, project manager, and Web designers.

Although DCM already had an e-mail based math program on the Web and was experienced in conducting engineering programs for children, KDN was a venture into uncharted territory. Because there were so many aspects of the program to evaluate, it was helpful to isolate components and design evaluation to answer specific questions. Most components were assessed individually first and then incorporated into a trial of the whole KDN experience. For example, when the Slowly Moving Marble Challenge was being developed, DCM employees' children in the museum on a school holiday were the first to try their hands at designing and building a pathway to slow down a rolling marble. These children didn't use the computer; they simply built their pathways. A few months later, a 3^rd-grade school class (taught by an KDN Advisory Board member) took on the marble challenge as part of the total KDN experience. Web site components were also evaluated in isolation first. Numerous versions of the Engineer chatboard were tested by the Web designers, then DCM staff, before children ever encountered them. The first children to see the chatboard were the project manager's children visiting the museum. Many iterations later, the first classroom trials of the chatboard occurred.

At the outset, three overriding questions were identified. Evaluation would be geared toward answering these questions:

These questions and how we answered them are discussed below. Methods are summarized in Table 2.

Table 2: Evaluation methods used during KDN development.

Is KDN technologically feasible?

The first question to be answered was whether live two-way communication was even technically possible in the school setting. Three primary concerns were quickly identified:

Would kids and teachers be interested in talking to the engineer?

Assuming the technology issues were worked out, we still needed to know if students and teachers would find talking to the engineer on-line worth their time and effort. "Ask the expert" Web sites other than KDN are based on e-mail communication where children have plenty of time to formulate questions which are reviewed by teachers or parents who correct grammar and edit to ensure clarity. Would children be able to communicate well enough to gain usable advice about their gadgets? Finding out if students thought talking to the engineer helped them meet the challenge was a major focus of the first two field studies and a lesser focus in following studies.

Because children's access to KDN, and especially the engineer, depended on the cooperation of teachers and parents, the grownups' perception of the value of talking to the engineer was important to assess. Would teachers see enough value that they would be willing to negotiate the potential maze of scheduling computer time and learning to use the Web site?

What, exactly, should the content of the Web site be?

The initial Web site concept included the challenge presented in short story format, a printable student journal, a student-engineer chatboard with only text chat, and an on-line bulletin board where pictures of gadgets could be posted. The inadequacy of these features became apparent very early in development. Identifying what was necessary for effective student-engineer communication was a focus early in the project. Programming and testing the solutions ultimately took a year longer and $70,000 more than was originally planned.

Evaluation techniques originally planned (based on the actual grant application)were:

During the first year of the project, two small field studies were conducted at schools with which DCM had a strong relationship. The purpose was to assess students' reaction to the concept of an on-line engineer, the necessary components for effective communication, the appropriate age for KDN, and the appropriateness of the first challenge. Teachers chose a few students to participate in KDN. The museum brought in their own computers and set up a closed link between the engineer's computer and students' computer. Communication was via a FileMakerPro database made to resemble a Web page with two-way text chat in the first field study. The second study utilized a CU-SeeMe program to provide video chat with an engineer via a local area network.

Much was learned about evaluation during these studies. Written surveys, so frequently used in evaluation, yielded minimal valuable information. Parents and teachers did not take time to record the detailed observations that are needed for comprehensive assessment. The value was also limited because the people completing the surveys (parents and teachers) were not present when children were working with KDN. Direct observation of students and conversations with them yielded the most information. Student journals were collected and an extensive video record of the sessions was made. Video and journals proved less valuable for evaluation purposes, but very valuable for bringing a new project manager up to speed a few months later.

A third small group trial aimed predominantly at testing a new challenge was held at the museum. No engineer was involved in this trial. A Girl Scout leader was asked to gather a group of students for this test. Children were asked to read two versions of the Puppet Challenge aimed at different age groups. The children were asked questions to see if they understood the story and its vocabulary, and if they found it interesting. The team wanted to assess the story format, the particular versions of this story, whether the challenge provided an appropriately open-ended engineering problem solving experience, and to revisit the question of the appropriate age for doing KDN. The session was videotaped and the children's journals were collected. The independent evaluator was not present. Once again, personal conversations with and observations of the children provided the most useful data.

These very preliminary field studies answered some basic questions and posed new ones:

Results of the field studies were presented to the Advisory Board. They felt that the potential impact of KDN on science and engineering education was huge, and that there were still a lot of questions to be answered. Extensive discussions occurred about the best form of communication best (email, video, live text chat, voice, fax, or phone), the focus of KDN (science, computers, or engineering), and the importance of addressing teachers' needs.

A big meeting was held that included the newly hired Web site designers (Eduweb), the independent evaluator, the old and new project managers, and most of the DCM staff who had worked on KDN to date. A free flowing exchange of ideas made sure everyone was on the same page regarding where the project had been and what needed to happen in the near future. Eduweb presented information of the most appropriate technology available. An understanding of two overarching types of problems emerged from the meeting: technology problems and people problems.

Eduweb focused on finding technology that could provide the best communication, could tunnel through school firewalls, and was compatible with the typical aging computer systems common in elementary schools. Making the site user-friendly - in both graphic design and navigation - and compatible with school computers also became a focus. DCM would review everything and try out all interactive components of the site.

DCM focused on the people problems. Most of the team had already concluded that 1^st and 2^nd graders were too young to use KDN effectively. However, the Museum felt that additional data was needed to support this conclusion for the Illinois State Board of Education (the funder). It was important to make the teacher, parent and child user interested in and comfortable with KDN. This would be addressed through both the content of the Web site, and Eduweb's work on the graphic design and navigation. Finding and training on-line engineers continued to be of concern. DCM was also responsible for coordinating with schools which had volunteered to participate in the second round of field studies.

Over the next few months the first version of the KDN Web site was developed in preparation for the 1^stfull-fledged classroom tests. During this period, the DCM project manager served as the internal evaluator. Eduweb posted pages on the Web for the project manager to review. She typically reviewed the first few iterations of a Web page or design element, and then asked for opinions from other DCM staff. Preliminary assessment of firewall tunneling technology was tested on a few local school computers before being incorporated into the site. Any DCM staff children visiting the museum were pressed into service as voluntary evaluators. They tried out the functionality of the Engineer chatboard and drawing program, critiqued the graphics, and were usually the first to try their hands at a new challenge.

Because the site was to be accessed by entire classes of children at once, DCM staff participated in group tests of the drawing program and the Engineer chatboard. We found again and again that even if it works on the programmer's computer and even the project manager's computer, that does not mean that 10 or 20 or 30 computers could draw or use the chatboard at once. The programmer would access error logs and make adjustments, and we'd try again another day.

Phase 2: Classroom Tests

Scheduling classroom tests proved to be complicated; school calendars and schedules are generally very rigid. An extraordinary amount of accommodation was necessary on the part of the school. The whole school was often affected because computer lab time and frequently library time were adjusted so that the classes using KDN had access to the computers.

What we did right in these trials

Administrators at the participating schools were on the KDN Advisory board, so they were familiar with the concept behind KDN. These schools and administrators had, for the most part, been selected because DCM conducted many programs there and had a good relationship with the teachers and administrators.

Students from 1^st through 4^th grade were involved in this phase of testing, allowing collection of data on the age issue. Also important were the responses of teachers and students of different ages to the challenges and their presentation, the KDN student journals, and students' desire, ability, and willingness to talk to the engineer.

To address 1^st and 2^nd graders' communication difficulties, a drawing program and whiteboard with chat component (Engineer chatboard) had been added to the site to provide a visual component. Children and engineers could both see the child's design and mark on it. Cell phones were used with and without the whiteboard to talk to the engineer as the children were building.

Three very different schools were chosen: a suburban school in a middleclass neighborhood (School 1), a rural, smalltown school (School 2), and a suburban school with a large Spanish-speaking population in a low income neighborhood (School 3).

DCM provided a Spanish-speaking facilitator for the bilingual classes at School 3. This allowed us to ensure students understood KDN as well as allowing DCM to receive feedback from the students.

Although facilitation was by experienced DCM personnel, classroom teachers were present for all KDN sessions and actively participated in the KDN experience. DCM and the independent evaluator chatted with teachers before and during KDN sessions to find out what teachers wanted and needed before using KDN. Informal conversations allowed us to see how revolutionary open-ended, child-centered problem solving is to some teachers and what type of background information on the philosophy behind KDN would be needed on the Web site. For example, one 2^nd grader making a bell puppet asked a DCM facilitator for help early in the process, and was later helped by her teacher. At the end of the session, the DCM facilitator asked her how her bell turned out. The girl said, "It's not a bell anymore. My teacher changed it." It was clear the Web site needed specific guidance on how teachers could help children solve their problems instead of taking over their project.

Eduweb's project manager and the computer programmer were present at the first school where testing occurred. Although both are experienced specialists in children's Web sites, they agreed there was no substitute for seeing children actually trying to use the site en masse in the school. It gave them a new appreciation for the state of technology in the elementary school, and of the problems of Web site functionality with young children.

Eduweb's computer programmer and project manager both played the role of engineer several times during this phase of testing. This gave them first hand knowledge of the difficulties of using the engineer's communication tools. The programmer reported that this experience was extremely valuable. For example, being limited to only the student whiteboard chat for communication, the engineer didn't know what was happening in the classroom during a long period of inactivity. Had the students all left for lunch? The solution was to add a separate chat component enabling a private channel of communication with the teacher.

Major revisions to the drawing program and the Engineer chatboard were made after each school participated.

The independent evaluator was present for most of the session at all three schools. She carefully observed all aspects of the sessions, chatted with students and teachers during the sessions, and asked follow-up questions at the end. For those sessions not attended, the project manager provided detailed notes and comments. Following the evaluator's model, the project manager asked students and teachers follow-up questions at the end of their KDN experience.

After School 1 tests, the team completely revised the challenge format to make it clearer. After in-house evaluation at DCM, the new format was tested at School 2, revised slightly based on teacher comments, and tested again at School 3. This format is still used today.

What we did wrong in these trials

The schools had been chosen because their administrators were on the KDN Advisory Board, but without regard for the state of their computer systems. A few weeks before the first tests, we found out School 1 had very old computers that simply could not handle even the most basic aspects of the KDN site. Eventually, the district computer expert borrowed a portable iBook lab from the junior high and set it up at the elementary school for the KDN tests. Consequently, the children were learning to use a new computer and the KDN Web site at the same time.

Because the administrators, rather than the teachers, had made the decision to participate in KDN development, many teachers were less than enthusiastic. Many, but not all, were not adept at computer use and did not see the value of hands-on activities. However, one bilingual teacher at School 3 enthusiastically reported that KDN had sparked a whole class discussion on how "this concept of keep trying until you get it right" applies to the rest of life, too. He said that the open-ended problem solving of KDN helps teachers see a new way of doing things, too.

School 3 had no computer lab at all until two months before the KDN trial. Most of these children had no home computers and consequently very little computer expertise. This group of students, however, proved to be very excited about KDN and vividly illustrated the potential impact of KDN on traditionally underserved populations. The students thrived when building their gadgets because there was no language issue with the hands-on activity. The students easily learned to use the Web site's interactive features. They were thrilled to talk to the engineer, and the girls especially were excited to learn that the engineer was a woman. However, many of their teachers struggled with the computer. These teachers, many of whom had never used computers in their classrooms, illustrated the importance of thorough, clear guidance in using KDN and user-friendly navigation.

Because of individual school scheduling constraints, the first trials were with 1^st and 2^nd graders, the group we expected to have the most trouble with KDN in general. A more refined version of the Web site would have given us a truer picture of the children's abilities without the complications of computer bugs. However, the tests still demonstrated conclusively that these young children were not developmentally ready for KDN.

Follow-up questionnaires were given to the teachers. Due to the poor return rate on written teacher questionnaires, the evaluator scheduled group de-briefing sessions. This was a more efficient use of teachers' time, and the group dynamic yielded more useful information. For example, a teacher would raise an issue, and several others would agree. This helped us see which issues resonated most with teachers. These discussions provided valuable, specific data that was used by the project manager and programmers to make major changes to the Web site and program content.

Web Site Usability Tests

After the classroom tests at the third school, the KDN team met and concluded we basically had a workable Website, albeit still with many bugs. A primary concern was that the site had become so big that navigation was a mess. The home page was overloaded with information, and there was a general lack of user-friendliness to the site, largely due to its complexity. Because the team was so accustomed to using the site, a new pair of eyes was needed for assessment. Two new evaluation strategies were adopted: teacher usability studies and review by independent Web site developers.

Teacher Usability Studies

To identify gaps in information and get teachers' perspectives, two elementary classroom teachers were brought in to look at the Web site in two usability studies. We gave the teachers very little orientation to the site because we wanted to observe them the first time they saw the Web site. Using a "think-aloud" protocol, the evaluator, project manager and Web developer took careful notes as each teacher navigated the site while talking aloud. This gave us specific information on where navigation and information needed revising. Following their time exploring the site, we held debriefing sessions during which teachers explained and expanded upon some of their comments. These usability studies were among the most useful techniques used in the entire evaluation.

After the first teacher reviewed the site, major revisions were made. Then a second teacher was observed using the same "think-aloud" protocol. After a morning of learning about the site, we observed as the teacher directed six children, who were unfamiliar with KDN, in using the site, including the drawing program and talking to the engineer. As the focus of this exercise was on the Web site itself, the children did not construct their gadgets. One of the most striking observations was that, prior to observing the children, the teacher did not think they would be interested in talking to the engineer. She stated she was very surprised to see how enthusiastic they were when talking to the engineer, and this changed her opinion of KDN considerably.

Both teachers pointed up a paradox the KDN team had long been wrestling with: they thought there was too much information on the site, but they wanted more information. The KDN team discussed this problem and concluded the solution must lie in presenting the information in a more eye-appealing way. Once we had a good idea what teachers needed on the site, we needed an uninitiated Web designer's opinion on how to give it to them.

Independent Web site Developer Reviews

After extensive revisions based on the teacher usability studies, two Web site developers unfamiliar with KDN were asked to review the site. We were particularly interested in their ideas on the overall look and feel of the site - especially graphic design and navigation. Although there were some common themes in the reviewers' comments - especially the need to simplify the navigation and put less information on each page - the reviewers came up with significantly different solutions to these problems. The KDN team evaluated the reviewers' comments and used them as a starting point for major revisions to the site navigation, page layout, and site organization.

Analysis and Redesign

Much discussion occurred between Eduweb and DCM, and many prototypes were reviewed before a basic design evolved. We were, in effect, reinventing the wheel with four new visions on how to go about it - DCM's, Eduweb's, and the two reviewers'. Text on every page was revised, a new graphics shell was developed, site content was reorganized and an entirely new navigation scheme developed, colors were constantly changed, a new log-in sequence was developed, and major revisions - both functional and cosmetic - to the drawing program and engineer chatboard were implemented. Revising the KDN home page was a major effort. The home page used in Phase 2 classroom trials is shown in Figure 2, and the current home page is shown in Figure 3. There were at least a dozen interim versions. The resulting current version of the site bears little resemblance to the reviewers' suggestions or the version they reviewed, but based on feedback from teachers who have since used the site, most of the problems have been resolved.

Figure 2: KDN home page used in Phase 2 classroom trials.

This page was too long, too busy and generally notuser-friendly.

Figure 3: The current KDN home page (http://www.dupagechildrensmuseum.org).

The solution to too much information was the creation of a KDN "Quick Tour" that takes visitors through a virtual tour explaining how KDN works; and a "What is KDN?" section providing additional details.

Phase 3 Classroom Trials

The plan for Phase 3 Classroom Trials was for classroom teachers to facilitate KDN with only e-mail and phone support from DCM. Two teachers, one in a small town near Chicago and one in an inner city Chicago school, found KDN on the Web on their own. They provided the first real test of the extensive information in the Teacher's Guide section of the Web site, and the newly revised Web site. The schools' proximity to DCM allowed the KDN Project Manager and, in one case, the independent evaluator to visit the schools and observe as students built their gadgets and talked to the engineer. The evaluator and project manager conducted an extensive interview with the teachers. Their feedback was used to further refine the Web site.

We were relieved to see that these children's experience was remarkably similar to KDN activities facilitated by DCM staff. The teachers were able to glean the philosophy behind KDN from the Website, navigate the site, and use its information to produce a hands-on, minds-on learning activity for their classes. Each teacher had added an individual twist to the challenge to emphasize different concepts, but the child-centered, creativity and problem solving concepts were paramount. The teachers had successfully used KDN exactly as was intended.

However, there were still problems with the engineer chatboard, most notably stemming from a modification that allowed the engineer to simultaneously chat with several students. Ultimately, this feature was disabled because it did not function reliably.

Conclusion

Although the basic concept behind KDN has remained constant, KDN as originally envisioned by DCM was very different from today's version. From the early stages of development, the team placed emphasis on continuous evaluation and a willingness to make revisions. The collaboration in evaluation between professional evaluator, Web site developers, museum personnel and engineers resulted in a fluid development process that mimicked the engineering problem solving process KDN is designed to teach. This iterative nature of engineering design as applied to KDN development ultimately produced a user-friendly experience consistent with the Museum's mission. Our efforts were rewarded with a Gold Muse for Programs Emphasizing Two-way Communication from our peers at the American Association of Museums. However, the best rewards come from the children who say "WOW! This is so cool! I wanna be an engineer when I grow up!"

Lincoln, Y.S. and E.G. Yuba, 1985. Naturalistic Inquiry. Sage Publications, Beverly Hills, CA.