Evaluation Design - Prospects and Problems
The general philosophy that has guided the development of the Hydroville Curricula was overlaid onto the evaluation design and used as the basis for developing a set of measurement concepts to detect change in important student outcomes such as problem solving style, value of information, personal involvement, and self-efficacy. These factors are strongly related to the process aspect of science as a contributor to environmental health and safety through the perception and assessment of environmental risk and the decision making behaviors that manage those risks. It is an intention of the Hydroville Curriculum Project (HCP) to instill in students important skills that will help them achieve a comfort level with environmental science such that they can participate in environmental problem solving, much as occurs in local communities as well as national debates about environmental issues.
The overall approach to implementing an evaluation in the context of high school classrooms involves a set of design decisions, and necessitates a comprise between what is ideal with respect to evaluation methodology and what is achievable and feasible given the constraints placed on teachers and students in the classroom. In all cases, and over the four evaluation years, classroom time was consistently at a premium, and the task of engaging teachers in using the curriculum, including adapting it to classroom needs, consumed much of their available classroom resources. The evaluation attempted as much as possible to impose few additional burdens, while at the same time providing useful information to the HCP staff and to teachers concerning the gains students’ make by their involvement with Hydroville Curricula.
To accomplish this overall objective, the evaluation was streamlined and modularized into a structured evaluation instrument that called upon students to complete a pretest and posttest protocol at the beginning and again at the end, respectively, of the classroom segment during which a curriculum module was implemented. Generally, this was a nine-week period, though this varied from one curriculum to another and from one teacher to another depending upon how the classroom teacher chose to apply the curriculum in their particular circumstances. The evaluation protocol was administered by teachers in the classroom. Students completed the protocol by providing structured responses on a Scantron mark-sensing form.
An initial evaluation protocol was developed prior to the implementation of the Pesticide Spill (PS) curriculum in the 2001-2002 school year. This implementation provided an opportunity to field test the evaluation protocol and to identify problems and challenges associated with having teachers administer the protocol and having students use the Scantron form to provide their responses.
Recruitment of schools, teachers and students involved a great deal of effort on the part of HCP staff. No opportunities existed in this context for random assignment of teachers, classrooms or students to treatment conditions. The recruitment and development of appropriate control groups also proved problematic. Though control groups were sometimes available, their numbers were consistently small, and there was no assurance that they matched treatment subjects in terms of important variables likely to affect student outcomes (e.g., attendance, achievement). As a result, the evaluation design relied on a pretest/posttest approach whereby each student acted as their own control. Since the treatment condition rarely spanned a period of more than nine weeks, it was judged unlikely that student maturation effects would account for any pre/post changes in student outcomes.
Student absenteeism and attrition proved to be a significant problem for the evaluation design. Typical attendance rates for schools in the program were on the order of 70% to 80% (Based on Average Daily Attendance or ADA) Therefore, the likelihood that a given student present for the pretest would also be present for the posttest was on the order of 50% to 60%, resulting in a large attrition of students from the pretest to the posttest. Actual rates were probably higher given that students were also eliminated from the evaluation if they moved classroom or transferred out of the school. Obtaining exact figures on student attendance during the Hydroville implementation period proved infeasible. Some teachers worked to accommodate and adjust for attendance problems by allowing students to “make up” the evaluation if they were within a day or so of the pretest or posttest administration. However, not all teachers were able to do this and their rules or procedures for doing so were inconsistent.
Using Scantron sheets to obtain students responses to the evaluation instrument proved feasible but not without problems. Randomized responding, patterned responding, double marking, incomplete protocols and extraneous marks all compromised the protocols of many students. HCP staff undertook a visual inspection of Scantron sheets prior to data processing and eliminated those that suffered from too many errors, etc., to provide useful information. As the evaluation progressed over subsequent years, teachers were provided more training with regard to spotting Scantron problems and addressing them at the classroom level.
The length of time required for students to complete the evaluation protocol proved to be a problem in the first implementation year (Pesticide Spill, 2001-2002). The overall protocol for that year contained 72 structured response items and was estimated to require an hour of student time, or about a classroom period. Teachers commented that the evaluation placed too great a burden on them and the students, and could not be completed in the time allowed. For the second year (Mysterious Illness Outbreak, 2002-2003), the protocol was reduced significantly to 32 items. Although this step eliminated most of the administration problems in the classroom, it also provided less information that was needed to adequately assess student outcomes. A compromise approach was developed before implementing the evaluation in the third and fourth implementation years (Indoor Air Quality, 2004-2005; Water Quality, 2005-2006). As a first step, the evaluation instrument was reviewed and refined in light of HCP experience with the first two implementations. Evaluation objectives for student outcomes were focused on key concepts relating to problem solving attitudes, self-efficacy, teamwork, and perception of risk. The evaluation instrument was expanded to 46 items to accommodate this review.
In addition, during the last two evaluation cycles teachers were given additional background in the evaluation design, as well as the meaning and importance of collecting information on student outcomes Teachers received focused presentations on the evaluation design at the Summer Institutes held prior to each of the final two implementation years. These steps proved to be effective and eliminated many, if not all, of the problems associated with administering a structured-response evaluation design under the constraints of time normally experienced in a classroom. By the third and fourth years, we were able to field an effective evaluation approach that demonstrated the impact of the Hydroville experience on student outcomes.
