Order matters: pre-assessments and student generated representations
© Halverson et al.; licensee Springer. 2013
Received: 31 May 2013
Accepted: 31 May 2013
Published: 19 July 2013
People have preconceived notions about how the world works from personal experiences. When these notions are scientifically inaccurate they can encourage unintended learning outcomes. Thus assessing students’ prior knowledge is important, allowing instructors to target misconceptions and optimize learning. However, the structure/administration of pre-assessments can influence students’ achievement and potentially mask errors in understanding.
We investigated the influence of pre-assessment task order on students’ tree thinking responses. We investigated student responses on a two-tiered pretest and an associated tree building task from 133 participants.
Differences in the styles of student generated representations were significantly related to task order.
This influence creates the challenge of diagnosing student ideas and facilitating conceptual change, while not accidentally provoking misconceptions in the process.
According to Novick and Catley (), substantial current college biology curriculum is driven by the use of phylogenetic trees. However, it is clear that most students do not interpret trees in the same manner as evolutionary biologists (Baum et al. ; Gregory ; Halverson et al. ; Meir et al. ). Students’ alternative conceptions, such as folk taxonomy groupings, can cause individuals to generate a variety of inaccurate responses when interpreting and explaining scientific concepts. If a person’s initial knowledge is inaccurate, when confronted with new, scientifically accurate explanations, they may not accept the new idea because it does not fit into their prior understanding (Bransford et al. ; Chi and Roscoe ; Duit & Treagust ; Posner et al. ).
Current quantitative assessment instruments (e.g., Baum et al. ) may uncover categories of student errors, such as those described by Gregory (), but they do not uncover the underlying reasoning patterns. Thus, assessing and diagnosing students’ prior knowledge is important in classroom settings to allow teachers to target alternative conceptions and optimize learning. However, the structure or even the administration of a pretest itself can influence students’ achievement and potentially mask errors in understanding (Solomon ). The purpose of this study was to investigate how student generated representations changed based on when they were given a tree building task during a pre-assessment (e.g., before or after tree thinking questions).
Learning can be described as a progression of change from the learner’s prior naïve conceptions to intermediate conceptions to scientific conceptions (Niedderer and Goldberg ). In this process, the individual constructs personal meaning by interpreting new information through a lens of prior knowledge; in other words, new knowledge builds upon prior knowledge (Bransford et al. ). Many educators use pretests as a method of assessing students’ prior knowledge and diagnosing alternative conceptions in a particular content area (Berry ). It has been suggested that taking a pretest can influence students’ perceptions of content, influencing student performance on the assessment (Solomon ). This can be explained as a form of intuitive heuristics. Maeyer and Talanquer () describe intuitive heuristics as a process students use to answer questions through simplification of their reasoning process. Students use less information than scientists to determine scientific relationships and use a series of potentially arbitrary rules when deciding where to look for information and how to answer questions based on the information gathered (Maeyer and Talanquer ). Students may answer assessment questions using short-cuts: (1) students’ recognition of the material; (2) the similarities they see between subject material (e.g., similarities between taxa); and (3) external cues that promote choosing one answer over another (e.g., one answer is more in line with everyday thinking than accurate scientific thought or reading the trees across the tips) (Maeyer and Talanquer ). Students may consciously or unconsciously use these rules or short-cuts when presented with tree building or tree reading assessments.
Tree building (generating a phylogenetic diagram) is a cognitively more difficult task than tree reading (interpreting and comparing phylogenetic diagrams) and builds upon a student’s tree reading ability (Halverson ). One common problem individuals have is incorrectly reading phylogenetic trees across the tips, looking only at extant taxa rather than correctly interpreting the nodes (Crisp and Cook ). Maeyer and Talanquer () found a similar problem in chemistry where students used arbitrary trends in the periodic table to answer questions when comparing individual atoms.
Researchers have used a two-tiered pretest to assess student tree thinking by presenting multiple styles of phylogenetic trees for students to interpret and compare (see Halverson et al. ). A traditional two-tiered pretest consists of two parts: the first is a multiple-choice, the second requests the students explain their answer choice. On this pretest students were also asked to generate a visual representation illustrating the relationships among 14 plant, fungi, and animal taxa and to explain the relationships drawn. While we accept that task order can influence students’ learning, it is unknown if task order on a single pretest influences students’ responses on the given assessment.
In order to identify how student generated representations changed based on when they were given a tree building task during a pre-assessment, we asked the following three research questions: 1) What styles of representations did students generate when given a tree building task?; 2) How do student justifications of representations differ when students are given a tree building task before versus after a tree reading assessment?; and 3) What is the connection between student generated representations and student tree reading?
Participants came from two public research universities across four semesters, for a total of 133 students. Students were enrolled in one of two upper-level biology courses. Both courses used phylogenetic trees heavily during instruction, highly emphasizing tree thinking. Each course ranged from about 30 to 50 students and was designed for life sciences majors. There was no significant difference between the sample populations at each university for the last two semesters (t(73) = .608, p = .545). Thus, we considered the two populations as one for the purpose of this study. We gave students enrolled in the first two semesters a tree thinking pretest. The last question of this pretest was a tree building task. We changed the order of pretest questions for students enrolled in the last two semesters. For these students the tree building task was the first question on the pretest.
Data came from student responses on a two-tiered pretest and associated tree building task (see Halverson et al. ). The pretest assessed students’ tree reading skills. The tree building task asked students to draw a visual representation that illustrated their understanding of how given taxa were related to one another. Students also provided written descriptions explaining their representation.
For the qualitative portion of our data analysis, we used a deductive method to code the types of representations students generated and students’ justifications for these images, on the pretest. We identified four categories of student generated representations: accurate tree representations, alternative tree representations, alternative tree-like representations, and alternative non-tree-like representations (see Halverson ). We further identified the nature of the student generated representations as appearing tree-like in nature (accurate tree representations, alternative tree representations, and alternative tree-like representations) or non-tree-like in nature (alternative non-tree-like representations). To improve the reliability of our findings, each researcher coded all student responses. We compared codes and had 100% inter-rater reliability.
As a secondary analysis, we grouped student representations into categories of diagrams according to the justifications students provided for their images. Phylogenetic diagrams were the only representations for which students used scientific explanations to justify accurate evolutionary relationships among taxa. Students justified relationships in ecological diagrams by relying on factors such as habitat, trophic level, etc. Students explained that they viewed evolution as progressive or that organisms evolved into one another when they generated diagrams representing alternative evolutionary principles.
For the quantitative portion of our data analysis, we scored the accuracy of students’ tree thinking at the beginning of the semester by giving students one point for selecting the appropriate multiple choice response and one point for providing an accurate explanation for why they selected their answer (maximum of two points per question). We then assigned numeric values to the style of representations students generated on the tree building task. In order to determine the relationship between task order and its influence on student generated phylogenetic trees, we conducted a Pearson’s Chi-square test. For this analysis we grouped student generated representations into four categories: phylogenetic diagrams, ecological diagrams, diagrams representing alternative evolutionary principles, or no representation. We compared these styles of representations with respect to when the task was administered, before or after the pretest. We used Yates’s continuity correction in place of χ2 to account for the expected cell count violation. Statistical significance was assigned when p < 0.05. We also analyzed the likelihood of students generating tree-like diagrams in relation to task order by calculating tree-like and non-tree-like frequency counts.
Results & discussion
We organized our findings into three major sections. First, we describe the frequency and types of representations students generated. Second, we describe the representation style differences related to tree building to task order. Finally, we describe the relationship between the representation styles generated and students’ tree reading.
Styles of representations generated
Generating a phylogenetic tree involves isolating, interpreting, and using data as evidence of evolutionary relationships. There are many different styles of representation an expert could generate if asked to draw a visual representation illustrating the relationships among the following organisms: bat, onion, dolphin, parrot, oak tree, fern, pine tree, slug, daisy, human, trout, algae, mushroom, turtle, crab, and fly. Of the varying representation styles, scientifically accurate representations share the following features: relationships are grouped based on evolutionary histories and common ancestry, all organisms are considered to be related and are connected within a single representation, taxa are placed at the terminal tips assuming there are hypothetical ancestors at the nodes, and consensus nodes are used when relationships are unknown and result in the tree being non-bifurcating at that location. This can cause the tree to have three or more branches coming off a single node.
Styles of student generated representations
Number of students
Changes in physical representation styles in relation to task order
Tree building task given after pretest
Tree building task given before pretest
Number of students
Number of students
Student justification differences
Change in content of representation styles in relation to when tree building task was given
Tree building task given after pretest
Tree building task given before pretest
Number of students
Number of students
Alternative evolutionary principles
Student representations and tree reading
We compared the styles of students’ representations to their tree reading pretest scores using an independent group t-test and found there was a statistically significant relationship (t(131) = 2.191, p = .03) between the tree-like nature of the representation styles and students’ tree reading scores. Although we found that Levene’s Test for Equality of Variance was not significant, the significant results of the t-test may be a factor of the category size differential rather than differences in student scores. This unknown is due to the non-tree-like category being 45% larger than the tree-like category.
Sum of squares
Planned contrast table
Value of contrast
Preassessments are often used to evaluate students’ prior knowledge at the beginning of a course (Berry ). This study provides evidence that assessments used to diagnose students’ prior ideas can actually present information that influences their understanding. In some cases of this study, the assessments presented new information to some students as they had no prior experience or no recollection of seeing phylogenetic trees. Maeyer and Talanquer () found that errors and bias may arise because students are looking at the face value of an assessment. By looking only at the surface of the assessment rather than attempting to understand the content, students are not making a connection between the tree reading questions and the tree building task the way the expert biologists would. Students may use shortcut strategies to answer assessment questions regarding the biological relatedness of organisms. The students who took the assessment with the tree building task as the last question may have been influenced by the tree reading questions and used the shapes found in those questions to pattern the physical shape of the representations they generated. Additionally, the arbitrary trend of reading a phylogenetic tree based on the relative position of the organisms across the tips of the trees (Halverson et al. ) is reflected in how students justify the representations they generated. Thus, students with higher tree thinking ability are more likely to generate tree-like representations when given the tree building task after the pretest than those students given the tree building task before the pretest.
Using a tree reading pretest and a tree building task in tandem can be used to gain an understanding of students’ prior knowledge (Halverson et al. ). However, we found that task order influenced student responses on a single tree building task. The connection between task order and pretest scores indicates that using an evaluative activity as a pretest question can serve as a prior knowledge diagnostic tool for courses that explore evolutionary relationships. Exposure to the phylogenetic diagrams helps students think about biological relatedness more in terms of evolution, even if inaccurate, rather than in ecological terms which is more consistent with everyday experiences (Staub et al. ). However, simply exposing students to phylogenetic diagrams will not increase their tree thinking skills; this can only be accomplished through explicit instruction.
The influence of task order can create a challenge for diagnosing student ideas. Educators must remember that task order on preassessments can influence student responses and plan assessments with this knowledge in mind. While these diagnostic tools can inform future instruction, they may mask additional alternative understandings. By expanding this study to include additional samples from more universities, educators will develop a deeper understanding of the relationship between task order and how students represent their ideas.
Our study provides evidence that assessments used to diagnose students’ prior ideas can present new information that influences their current understanding. We found that task order influenced student responses on a single tree building task. Exposure to the phylogenetic diagrams helped students think about biological relatedness more in terms of evolution, even if inaccurate, rather than in ecological terms as is more consistent with everyday experiences.
- Baum DA, Smith SD, Donovan SSS: The tree-thinking challenge. Science 2005, 310: 979–980. 10.1126/science.1117727View ArticlePubMedGoogle Scholar
- Berry T: Pre-test assessment. American Journal of Business Education 2008, 1: 19–22.Google Scholar
- Bransford JD, Brown AL, Cocking RR: How people learn: brain, mind, experience, and school. Washington D.C: National Academy Press. Expanded ed.; 2000.Google Scholar
- Chi MTH, Roscoe RD: The process and challenges of conceptual change. In Reconsidering conceptual change: issues in theory and practice. Edited by: Limon M, Mason L. Dordrecht: Kluwer; 2002:3–27.View ArticleGoogle Scholar
- Crisp MD, Cook LG: Do early branching lineages signify ancestral traits? Trends in Ecology & Evolution 2005, 20: 122–128. 10.1016/j.tree.2004.11.010View ArticleGoogle Scholar
- Cobern WW, Gibson AT, Underwood SA: Conceptualizations of nature: an interpretive study of 16 ninth graders’ everyday thinking. Journal of Research in Science Teaching 1999, 36: 541–564. 10.1002/(SICI)1098-2736(199905)36:5<541::AID-TEA3>3.0.CO;2-1View ArticleGoogle Scholar
- Driver R, Squires A, Rushworth P, Woods-Robinson V: Making sense of secondary science: research into children’s ideas. London: Routledge; 1994.Google Scholar
- Duit R: Students’ conceptual frameworks: consequences for learning science. In The psychology of learning science. Edited by: Glynn SM, Yearny RH, Britton BK. Earlbaum: Mahwah, NJ; 1991.Google Scholar
- Duit R, Treagust DF: Conceptual change: a powerful framework for improving science teaching and learning. International Journal of Science Education 2003, 25: 671–688. 10.1080/09500690305016View ArticleGoogle Scholar
- Gregory TR: Understanding evolutionary trees. Evolution: Education and Outreach 2008, 1: 121–137. 10.1007/s12052-008-0035-xGoogle Scholar
- Halverson KL: Improving tree-thinking one learnable skill at a time. Evolution: Education and Outreach 2011, 4: 95–106. 10.1007/s12052-010-0307-0Google Scholar
- Halverson KL, Pires JC, Abell SK: Exploring the complexity of tree thinking expertise in an undergraduate plant systematics course. Science Education 2011, 95: 794–823. 10.1002/sce.20436View ArticleGoogle Scholar
- Maeyer J, Talanquer V: The role of intuitive heuristics in students’ thinking: ranking chemical substances. Science Education 2010, 94: 963–984. 10.1002/sce.20397View ArticleGoogle Scholar
- Meir E, Perry J, Herron JC, Kingsolver J: College students’ misconceptions about evolutionary trees. American Biology Teacher 2007, 69: 71–76. 10.1662/0002-7685(2007)69[71:CSMAET]2.0.CO;2View ArticleGoogle Scholar
- Niedderer H, Goldberg F: An individual student’s learning process in electric circuits (Paper presented at the NARST Annual Meeting). CA: Anaheim; 1994.Google Scholar
- Novick LR, Catley KM: Assessing students’ understanding of cladograms (Paper presented at the National Association for Research in Science Teaching Annual Meeting). MD: Baltimore; 2008.Google Scholar
- Posner GL, Strike KA, Hewson PW, Gertzog WA: Accommodation of scientific conception: toward a theory of conceptual change. Science Education 1982, 66: 211–227. 10.1002/sce.3730660207View ArticleGoogle Scholar
- Solomon R: An extension of control group design. Psychological Bulletin 1949, 46: 137–150.View ArticlePubMedGoogle Scholar
- Staub NL, Pauw PG, Pauw D: Seeing the forest through the trees: helping students appreciate life’s diversity by building the tree of life. The American Biology Teacher 2006, 68: 149–151. 10.1662/0002-7685(2006)68[149:STFTTT]2.0.CO;2View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.