Qualitative Measures and Results
By Nam-Hwa Kang, Ph.D, Assistant Professor
Science and Mathematics Education, Oregon State University
Introduction
In contemporary science and technology-based societies, most people have contact with science through socio-scientific issues, which requires citizens to be scientifically literate (AAAS, 1989; NRC, 1996). “Science is no longer the specialized activity of a professional elite” (Wilson, 1998, p. 2048). But it has become part of world culture and available to all as a way of learning about the material world. The modern citizen encounters not only the successes of science but also the risks caused by science which can lead to uncertainty about science. Moreover, science in real life does not come in mutually exclusive disciplines; rather, real science comes in interdisciplinary form in which causal explanations in one discipline depend on those in other disciplines including various sciences, politics, and economics (Rudolph, 2005). Such uncertainty and interdisciplinarity are the nature of science with which the contemporary citizen must deal. Therefore, science education should address the complex nature of science in preparing scientifically literate citizens.
Current science reform promotes scientific literacy for all students (AAAS, 1989; Millar & Osborn, 1998). Scientifically literate people should be able to understand and respond critically to media reports of science-related issues; be able to hold and express a personal point of view on science-related issues; appreciate the process of science through an understanding of the ways to establish scientific explanations (DeBoer, 2000; Shamos, 1995). These capacities cannot be fully developed through didactic teaching. Students need to experience how science works in real life through participating in authentic inquiry activities (Resnick, 1987; Roth, 1995). Moreover, exposure to real science facilitates students’ application of school learning to everyday life (Bransford et al., 2000).
Funded by the National Institution of Environmental Health Science, a series of high school science curricular modules were developed under the title of “Hydroville” and refined through five year-long pilot implementation. The curriculum goals were in alignment with the goal of current science education reform, i.e., developing scientific literacy by emphasizing development of problem-solving and decision-making capabilities through increased awareness of environmental effect on human health, science content knowledge and inquiry skills (Krajcik, et al., 1998). The Hydroville curricula are based on real-world problem solving through investigations.
Purpose
The purpose of this study was to examine students’ learning outcomes from their experience with an environmental health science curriculum in terms of inquiry capabilities. These capacities included inquiry questioning and ideas about approaches to inquiry through students’ own questions and perceived learning outcomes. Research questions included: (1) What are the types of questions students ask and how do these questions change through the experience of the curriculum? (2) What are students’ ideas about approaches to inquiry into their own questions and their changes? (3) How do students characterize the curriculum and what are their perceived learning outcomes?
Methods
Participants
The students in this study included 344 9th and 10th graders in three high schools in the US. Two of the high schools were located in suburban areas. A small number of students were from an alternative program in an urban school. In academic year 2005-2006, Mrs. L and Mr. D taught the Water Quality unit in a course entitled Inquiry Science (n=130) in the same school. In year 2006-2007, Mrs. L moved to a different school and taught the Indoor Air Quality unit to 9th grade general science students and the Water Quality unit to 10th grade biology students (n=214). Mrs. K in an alternative program taught the Indoor Air Quality unit to 10th grade science students (n=6).
Data Collection
In order to examine students’ learning through the curriculum, multiple sources of data were used. For data on student inquiry capabilities, student written responses in pre- and post-journal prompts were collected. These prompts required students to respond to an environmental health scenario. Using environmental health scenarios in probing students’ ideas seems to be authentic to the goal of science education for scientific literacy. We tried to assess students’ learning through their response to environmental health issues just like regular citizens encounter science through media such as TV news or newspapers.
In the pre-journal prompt (administered at the beginning of the unit), students watched a videotape of a scenario about a water quality or indoor air quality issue and answered several questions including defining the problem, posing questions, and suggesting ways to answer the questions. In the post-journal prompt (administered at the end of the unit), students were given a new environmental health issue in faux newspaper reports in which contaminated chicken feed allegedly caused the death of chickens and mutations of chicken offspring (Boston University Center for Interdisciplinary Research in Environmental Exposures and Health, 2007). Students answered the same questions regarding the new case. We also collected control group data by administering the same assessment to the students of Mr. D and an additional teacher, Mr. B during the second year when the teachers did not use the curriculum but taught the similar population, i.e., the same grade in the same school and no significant differences in the state standardized test score between the first year and the second year students (p=0.9379).
During the second year of the study, data on students’ characterization of the curriculum and perceived learning outcomes were collected through group or individual interviews of selected students of Mrs. L (n=13) and Mrs. K (n=6). Students for interviews were randomly selected on the basis of availability among those who agreed to participate in the study. The interviews took from 20-40 minutes and were video recorded and transcribed for analysis.
Data Analysis
All the students’ written responses were typed in MS Word. Three researchers individually read and coded the data. For initial coding, we agreed on nine subcategories of types of student questions that were later grouped into three and four categories of approaches to answering the questions. We had about 95% agreement in coding. Then we discussed inconsistencies and reconciled differences. Coded data were scored for statistical analysis as shown in the findings section. The first author initially coded the qualitative data and discussed with the second author for triangulation.
Trustworthiness
In order to triangulate, multiple data sources were used and multiple researchers were involved in the analysis of the study (Patton, 2001). Triangulation procedures were to find convergence of data across contexts as well as to find different patterns emerging from different contexts to broaden our understanding of student learning through the curriculum (Mathison, 1988).
Findings
Student Inquiry Capabilities
Types of student questions. The number of questions was not significantly different between pre- and post-journal responses (about 2 questions per student in both pre- and post-journal prompts). Initially, nine different types of questions were identified in the data as the students asked about eight different aspects of each case and/or reacted to each case with a question. The nine categories provided initial insight into the pattern of student questions. However, we simplified the nine types of questions into three by combining questions of similar nature that could indicate different levels of students’ inquiry skills (Table 1).
Table 1. Categories of Student Questions
| Categories |
Type of Questioning |
Cognitive Level |
| Generic Questions (GQ) |
Random information |
Simple, not related to inquiry understanding |
| Active Inquiry Questions (AIQ) |
Data analysis, problem cause, correlations among factors, hypothesis based |
High, related to inquiry understanding |
| Socio-political Questions (SPQ) |
Social and political context of the case |
Understanding from a broader perspective |
Questions about phenomena, solution, random information and unspecified method for searching solutions, and simple reaction to the case were grouped as “Generic Questions (GQ)” because these questions did not indicate any commitment to solving the problem that is actively looking for solutions; rather, these questions seemed to be asked either for the sake of asking a question or from the perspective of uncritical, passive consumers. In addition, these questions required cognitively simple procedures to answer. Therefore, the expected learning outcomes from answering the questions were low. Questions that consisted of questions about data analysis, cause of the problem and correlations among factors in the case were grouped together as “Active Inquiry Questions (AIQ)” because they indicated students’ understanding of the problem from a scientific inquiry perspective and/or a commitment to problem solving as an inquirer. Moreover, these questions required high-level cognitive processes to answer. Therefore, the expected learning outcomes from answering the questions were high. Finally, questions about socio-political contexts of the case were separated from the other types of questions because those questions were different from scientific inquiry skills but central to understanding of the problem from a broader perspective.
In order to control teacher effect, we first compared students of the same teacher (Mr. D) who learned the curriculum with those who didn’t learn the curriculum. The group comparison demonstrated statistically significant differences between the groups in terms of student asking more AIQ when learned the curriculum (p<0.005). Once we found the differences made by the curriculum, we conducted within group pre and posttest comparison. Compared with the pre-journal responses, students asked about 11% more inquiry questions in the post-journal (p<0.05).
Student ideas about approaches to problem solving. Not all students provided specific ideas about how to answer their questions. Many students provided generic responses such as “investigate the problem,” or “gather more information.” These types of answers were excluded in the final analysis because they lacked information on students’ knowledge about how to do inquiry. In so doing, about 80% of student responses in both pre- and post-journals were included in the final analysis.
Students’ ideas about approaches to answering their own questions were coded based on the sources of knowledge/information including text media, asking experts, collecting field data and collecting data with clear goals or hypotheses.
When teacher effect was controlled, students who learned the curriculum demonstrated statistically significant difference in ideas about approaches to problem solving (p<0.0005). We then conducted within group pre and posttest comparison. Compared with pre-journal responses, students suggested more hypothesis-based, goal-oriented approaches in post-journal responses (about 17% increase, p<0.0005) while utilizing experts was reduced. This comparison indicated that the students became more specific in coming up with ideas about how to answer their question and took a more active inquirer’s position rather than a passive consumer of science position.
Perceived Learning Outcomes
We first examined how the students characterized the curriculum assuming its relevance to students’ perceived learning outcomes. The students interviewed in this study characterized the curriculum in various ways. However, their characterization was mostly in two dimensions: pedagogical dimension and content dimension. The former included worksheet driven, problem-solving driven, hands-on driven, investigation driven, group-work driven and/or self-teaching. Other students characterized the curriculum in terms of the content: studies about water contaminants or air quality, environmental effect on health, practical information, and/or flow of underground water or measure of air quality.
The characterization of the curriculum was, however, not closely related to students’ perceived learning outcomes. When asked about what they learned throughout the curriculum 11 out of 19 students mentioned increased awareness of “the effect of air quality” “water contaminants”, or “water problem.” For example, one of Mrs. K’s students mentioned, “It pretty much opens up everybody's mind. . .beyond pollution. . . so many different things are wrong. . . . I think it helps us. . .what to look for and how to keep our places clean” (TK). Others mentioned content learning by providing detailed contents covered in class while only one mentioned inquiry skills without prompt. Although those who characterized the curriculum in terms of its content tended to mention specific contents as their learning outcomes, there was no clear pattern between students’ perceptions of the curriculum and their learning outcomes.
Discussion and Implications
The study has shown that after experiencing the Hydroville curriculum more students became analytical by asking more questions about data analysis and explanations in terms of correlations or causal relations in the case. These changes indicated students’ increased understanding of environmental health issues from the perspective of science (Dori, & Herscovitz, 1999). Moreover, more students took on a researcher or investigator role as they suggested collecting and analyzing data to answer their questions rather than asking experts for answers. At large, students seemed to transform their disposition towards more critical about the issues.
The qualitative data confirms the quantitative data that demonstrated students’ increased scientific disposition toward environmental health issues but falls short of confirming students’ learning inquiry skills. Although many students characterized the curriculum as problem-solving oriented or investigation oriented they rarely mentioned their inquiry skills as learning outcomes. Considering that students need to engage in a metacognitive process to become aware of their learning of inquiry skills while doing inquiry, the findings suggest that the curriculum needs to provide students with opportunities to explicitly discuss the inquiry processes that they practice.
Further analysis of actual performance and understanding how some elements of the curriculum helped the students to transform their positions will provide some insight into curriculum development and classroom enactment.
References
American Association for the Advancement of Science. (1989). Science for all Americans. New York: Oxford University Press.
Boston University Center for Interdisciplinary Research in Environmental Exposures and Health (2007). Dioxin-contaminated chicken: An environmental health disaster scenario. Retrieved February 27, 2007, from http://www.bu.edu/bahec/story.html
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school (Expanded ed.). Washington, D. C.: National Academy Press.
DeBoer, G. E. (2000). Scientific literacy: Another look at its historical and contemporary meanings and its relationship to science education reform. Journal of Research in Science Teaching, 37, 582-601.
Dori, Y. J., & Herscovitz, O. (1999). Question-posing capability as an alternative evaluation method: Analysis of an environmental case study. Journal of Research in Science Teaching, 36, 411-430.
Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., Fredricks, J., & Soloway, E. (1998). Inquiry in project-based science classrooms: Initial attempts by middle school students. Journal of the Learning Sciences, 7, 313-350.
Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13-17.
Millar, R., & Osborne, J. (Eds.). (1998). Beyond 2000: Science education for the future. London: King’s College London.
National Research Council. (1996). National science education standards. Washington, D. C.: National Academy Press.
Patton, M. Q. (2001). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: Sage.
Resnick, L. B. (1987). The 1987 presidential address: Learning in school and out. Educational Researcher, 16(9), 13-20.
Roth, W.-M. (1995). Authentic school science: Knowing and learning in open-inquiry science laboratories. Boston, MA: Kluwer.
Rudolph, J. L. (2005). Inquiry, instrumentalism, and the public understanding of science. Science Education, 89, 803-821.
Shamos, M. H. (1995). The myth of scientific literacy. New Brunswick, N.J.: Rutgers University Press.
Wilson, E. O. (1998). Integrated science and the coming century of the environment. Science, 279(5359), 2048-2049.
Contact:
Nam-Hwa Kang, Ph.D
Assistant Professor
Science and Mathematics Education
255 Weniger Hall
Oregon State University
Corvallis, OR
Phone: 541-737-9891, Fax: 541-737-1817
E-mail: kangn@science.oregonstate.edu
