Effect of Computer-Based Preparation
For the California High School Exit Exam
Sandy Jaquish
California State University, Sacramento
April 2006
Abstract
This research involved the creation of an interactive curriculum covering some of the math portion of the California High School Exit Exam (CAHSEE). The original intention of the research was to compare student responses to a web-based CAHSEE preparation course to the workbook based approach currently in use. When it became apparent that there was no established web-based resource available, the scope of the project was expanded to include the creation of a sample of such a resource. The web-based curriculum developed for this research covered two standards within the Number Sense strand, and employed a variety of interactive technologies including animated characters doing the “teaching”, audio and visual feedback for correct and incorrect responses, and optional hints for students to access when needed. This online curriculum was then presented to 20 students enrolled in an urban independent study high school. Data collected included scores on pre and post-assessments covering the specific math standards included in the software, a written questionnaire pertaining to students’ subjective attitudes about the experience, and a personal interview upon completion of the trial. Results were very positive with a 20% average increase in post-assessment scores over pre-assessment scores, 85% of the students reporting that the web-based curriculum was a positive experience, and 45% of the students reporting a strong preference for a web-based study course for CAHSEE preparation if it were available.
Introduction
The use of technology-based tools in the classroom is an undeniable upward trend. In fact, a large portion of the estimated $600 billion educational market in the United States includes an element of technology (Buchen, 1999). Specifically, the Benton Foundation Communications Policy Program reported that the country invested over $37.9 billion between 1990 and 2000 to bring educational technology and internet access to schools through the E-Rate telecommunications discount and other state and federal funding (Great expectations: Leveraging America's investment in educational technology, 2002).
Some of the benefits of technology in the classroom are obvious. Technology enables students and teachers to access a universal field of information at the click of a mouse. Multimedia technology supplements written text books with memorable visual images. Prescriptive software can pinpoint a student’s gap in understanding and provide the necessary remediation. This amazing progress in the development of computer-based tools was aptly described by Seymour Papert when he said, “…this is not simply a story of the development of a technology. It is better described as the development of a culture” (Papert, 1997, p. 78).
Concurrent with the trend toward the use of technology in the classroom is the trend toward the development of national academic standards to serve as a basis for measuring learning and attaching consequences to teacher and student performance (Weiss, 2000). Since the measurement of learning standardized material requires standardized assessment, there has been a corresponding increase in standardized “high-stakes” testing over the past 15 years. One such test is the high school exit exam covering minimum proficiencies deemed to be necessary to obtain a high school degree. By the spring of 2004 twenty states had implemented high school exit exams as a graduation requirement, with five additional states (including California) phasing in exit exams by spring of 2006 (Wise, 2005).
Setting and Participants
The setting of this action research was an independent study high school in an urban low income area in northern California. This school provides an educational alternative for students who, in most cases, have not been successful in traditional classroom settings. The twenty students involved participated voluntarily as partial fulfillment of their weekly attendance requirements. Grade levels ranged from grades 8 through 12. The five students in grades 8 and 9 had not yet taken the CAHSEE exam. Of the remaining 15 students, only one had already passed both the Math and English sections of the exam.
The research was conducted in the school’s math lab, which is amply equipped with personal computer stations with high speed internet access. Each student was given access to a computer station in the math lab and a headphone to use for the activity. Prior to the beginning of the activity the URL of the online curriculum was brought up on the computer screen.
Methodology
The creation of the web-based curriculum began with the selection a few of the standards within the Number Sense Math Strand ((CDE), 2005). I then created a series of illustrations, animations and practice problems using Flash. Concepts were initially presented by breaking them down into understandable components presented by animated “teachers” in the form animals. The explanations were followed by animated illustrations showing how to apply the concept to solve a given math problem. The curriculum presented a graduated progression beginning with showing the student how the problem is done, then having the student do some parts of the problem with visual aids and hints available through the software, to the final phase of having the student work several problems on their own.
Content for the program was gleaned from two mathematics workbook sources: the workbook provided by California Department of Educationto all 10 grade students ((CDE), 2004a, , 2004b), and a workbook written through University of California, Davis (Solovitch, 2006). Both workbooks are intended to be used in a traditional classroom setting where concepts are explained and illustrated by the teacher or a trained tutor.
Description of the Computer Application
The design of this application included the incorporation of animated characters that fulfilled the role of “teachers” throughout the program, “show me again” options, audio and visual feedback for correct or incorrect answers, audio and visual hints, and occasional background music.
Animal characters as teachers.
An important objective of this curriculum was to serve as an effective CAHSEE preparation tool without requiring the presence of a teacher or tutor. The “teacher” role in this curriculum was filled by the creation of four unique personalities embodied by animals who could speak and project emotions through facial expressions. This cast of characters, including three male personalities and one female personality, made it possible to convey significant amounts of conceptual information in the absence of an actual teacher without requiring the student to read a large amount of screen text.
The rationale for the characters.
Four unique animal character personalities were created as visual agents of communication – or “teachers”. With the help of my 24 year old daughter and a local radio disc jockey who plays to an urban high school target market, personalities and voices were created and scripts were recorded. Each character fulfilled a different role and, by design, was geared to appeal to a different audience. In this way it was hoped that the cast of characters, as a whole, would appeal to a broader audience. The four characters and their respective “roles” were as follows:
1. Romeo the Cat was our primary teacher. His personality was modeled after the infamous “Puss N Boots” character. Romeo is very calm and has the persona of “smooth talker.” His Spanish accent is over-emphasized for the purpose of humor.
2. Dot the Dalmatian was our second most prominent teacher. Dot presents the persona of a very smart female who is a bit sassy but to smart to argue with.
3. Snap the Lion was our “keeping it real” character. Snap sounds like he was raised in an urban neighborhood and knows just what to say to break something down to its most basic form.
4. Wellington the Koala Bear plays the role of the character you love to hate. Wellington comes across as a slightly haughty Australian who delivers his messages arrogantly but not negatively.
Basing characters upon learning styles.
Not only did these characters add warmth and leadership to the learning experience, they also increased the array of “intelligences” to which the program appealed by increasing the elements of sight, sound and movement. Further value was added by assisting those students who were not strong readers by delivering the message in a format beyond that of written text only (Gardner, 1983).
Show-Me-Again Functions.
Each illustration in the software program was equipped with a button which allowed the student to replay the animated illustration of a concept as often as they chose. This function added an advantage over a typical classroom environment where time would not allow such a luxury, or even if time were unlimited, a student may not request repeated demonstrations due to the possible embarrassment factor.
Feedback for Correct and Incorrect Answers.
Each component of the software requesting a student’s input was equipped with a button by which a student could choose to have feedback as to whether their answer was correct or incorrect. This feedback was in the form of character with a voice appearing on the screen. In most cases, a written message also appeared on the screen simultaneously.
Audio and Visual Hints.
Hints were presented in various forms throughout the program. In some cases, the hint would be presented in text form as part of the “incorrect feedback” function. In other cases, the hints were identified on the screen by “hint” button which would convert to a specific message relevant to that problem when the curser was held over the button. In this way, students could choose whether or not to seek out the assistance of the hints as they progressed.
Description
Students were initially briefed on the purpose of the activity and the fact that they would not be graded on their responses. They were also told that the results of the data would be anonymously compiled to be used for the author’s master’s degree research, and that they were free to have their data excluded if they or their parents so chose.
Pre-assessment.
Students were then provided with a pre-assessment consisting of five questions similar to questions used in past CAHSEE exams. Three of the questions pertained to adding or subtracting fractions. Two of the questions pertained to finding the prime factored form of a least common denominator.
On-line curriculum trial.
After completing the pre-assessment students were seated at one of the computer stations and given a brief explanation as to the format of the online curriculum. At that time it was explained to each student that their opinions about the program itself, what they liked and what they didn’t like, would be sought. At no time were the students told that they must complete the entire program.
Post-assessment.
When each student stated that they were finished with the online program they were then given a post-assessment consisting of five questions, all of which were parallel in content to the five questions on the pre-assessment. A 16 question questionnaire was attached to the post-assessment page.
Questionnaire.
The questionnaires contained a variety of questions primarily geared toward gauging the subjective responses of the students to the online program. The format of the questionnaire allowed the student to circle the response that best fit their feelings and ranged from very negative, slightly negative, slightly positive and very positive. The three content categories in the questionnaire included their general feelings about the online program, their feelings about the animated characters in the program, and their feelings about using computers in general. In addition to these subjective questions, additional questions regarding the student’s gender, grade, their access to a computer at home and their access to the internet at home were included.
Interview.
The final phase of the process for each student was a brief informal interview with me. I asked each student to say what they liked most and what they liked least about the online program. Some students went into detail; others were very brief in their responses. The level of detail in a student’s response was accepted at face value. No additional probing or questioning was done.
Limitations of This Study
The web-based curriculum developed for this research covered only a small portion of the standards included in the CAHSEE, so it is not possible to generalize these results across the entire exam content base. Additionally, the participants in this study are a unique group of students who are not necessarily a representative sampling of students who are preparing for the CAHSEE, so these results are not necessarily generalizable to other student groups in different educational environments.
Timeline
Developmental work on the web-based curriculum was a slow process because I didn’t have any experience with Flash prior to this project. I began teaching myself Flash in November, 2005 through the use of textbooks and online tutorials. By December I was able to design the initial illustrations used in the course to show the concept of adding fractions, and finally completed the technical components of the curriculum by the end of February. During those last few weeks the personalities of the animal characters were conceptualized and scripts were written and recorded. The audio scripts were then used applied to animal models and given animated lip movements through the use of Crazy Talk software.
My action research with the students took place over a three day period in March, 2006, during the first week of the school’s optional math lab hours for CAHSEE preparation. Data was analyzed during the following month.
Presentation of the Data
The analysis of data included a comparison of the pre and post-assessment scores, and an analysis of the questionnaire responses and the interview responses.
Pre-Assessment vs. Post-Assessment Scores
Average performance improved by 20%.
The post-assessment scores reflected an average of a 20% improvement over the pre-assessment scores. The average score on the pre-assessment was 41% (or 2.06 correct answers out of five). The average score on the post-assessment was 61% (or 3.06 correct answers out of five).
Individual range of performance.
When pre and post-assessment score comparisons were broken down on an individual student basis, the results showed that 59% of the students improved their scores. Specifically, 10 of the 17 students improved their scores somewhere between 20% and 60%, while 6 students’ scores remained unchanged, and one score decreased on the post-assessment. (See Table 1)
Table 1
Individual Range of Performance Change on Pre and Post Assessments
% Improvement |
% of Students |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
-20% |
6% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0% |
35% |
|
|
|
|
|
|
|
|
|
 59% of students showed improvement |
20% |
23.50% |
|
|
40% |
23.50% |
|
|
60% |
12% |
|
|
|
|
Questionnaire Results
Most students said they liked the curriculum.
The student responses relating to questions about how much they liked or dislike the online curriculum were generally very positive. Of the six questions in this category, 85% of the students’ answers were in the “slightly positive” or “very positive” categories. (Table 2)
Table 2
85% of Responses Regarding Level of Enjoyment of Curriculum were Positive
Interview Results
Responses to the software were defined, in part, by responses to the animal characters.
Each interview began with the question, “What did you like best and least about this study software?” Interestingly, 13 of the 20 students immediately answered as if I had asked, “Who did you like best and least?” Without hesitation, most of these students had strong feelings about the characters, as if the characters themselves defined the experience for them. Feelings about certain characters differed widely as shown in the following excerpts:
1. “I liked Snap because he had a hip accent.”
2. “I liked the way the animals’ mouths talked – especially Romeo.”
3. “The Koala bear made me laugh out loud every time he laughed.”
4. “I hated that annoying Koala bear!”
5. “I liked the lion because he used language that is appropriate for high school kids.”
6. “I thought the cat was annoying but he explained things well.”
Some students reported breakthroughs in math concepts.
Two students answered the “what did you like best” question by pointing out a specific
math concept that the curriculum helped them understand for the first time. “This is the first time I’ve understood improper fractions,” reported one 10 th grade female. Another student showed me the notes she had taken on a notepad while going through the curriculum and then, smiling proudly, proceeded to demonstrate to the vice-principal how to find the lowest common denominator of two fractions. (This same student, a female 11 th grader, was one of the few students who reported that they did not have a computer at home. When she initially sat down at the computer to try out the curriculum she told me this was the first time she had used a computer with a mouse.)
Validity
The triangulation of methods in this research revealed several points of consistency between student responses. For example, there was virtually 100% consistency between the students’ character preferences as stated in the interview compared to their questionnaire responses. Another consistent element was the fact the only student who expressed strongly negative feelings about the software in the questionnaire was also the only one who expressed strongly negative feelings in the interview. He is also the only student whose score was lower on the post-assessment than the pre-assessment. (This student happens to be the highest scorer on the assessments with nine out of ten correct answers, revealing that he does not represent the target audience for this software from an academic point of view.)
Recommendations and Reflections
The students in this research remained voluntarily engaged in the computer program from 30 minutes to over an hour, evidencing the fact that computerized delivery of information can be engaging and instructive. The student’s responses to the animated characters in this program exemplifies the value of incorporating a “teaching agent” to assist in conveying concepts in online curricula, to connect with various personality types, and to provide an additional element of entertainment.
The results of this study provide support the case for an interactive online curriculum for CAHSEE preparation. Although this group of participants does not statistically represent a sampling of students who have not yet passed the CAHSEE in California, these findings appear to have some generalizability to other students for whom traditional classroom instruction has not been entirely successful. The fact that almost half of these students said that a web-based CAHSEE study program would be their “preferred” method of study indicates that a web-based program offers something that their current study options do not. One possible factor may be the time flexibility, allowing students with after-school jobs or sports commitments the opportunity to study from home when their schedules allow. Given the fact that 95% of these students reported that they like using computers in general, an online CAHSEE course fits into a medium for which the students already have a general preference.
Conclusions
A high-quality interactive online CAHSEE study program would potentially solve other problems currently being encountered by school districts trying to assist their students, such as:
1. The subject matter is primarily content from grades 7 through 10. Many high school teachers do not have classroom experience teaching content standards of earlier grades.
2. Many CAHSEE study programs are held after school. Students depending on bus transportation or who must hold down jobs to assist their families financially often cannot attend after-school courses.
3. Current work-book teaching methods are usually less interactive and may not engage the interest of the student who has not been successful in a traditional classroom environment.
4. Each district is required by legislation to provide assistance to students who are not showing progress toward passing the exam, but no specific funding is allocated for this service (The California High School Exit Exam, 2003). A web-based CAHSEE course may reduce the cost to districts that do not have available funding to hire qualified teachers for classroom-based CAHSEE preparation.
Challenges of offering an entire CAHSEE curriculum to every California student include the necessity of having access to a computer and the internet. An additional challenge is the ticking of the clock. Within a very few weeks our first group of students being denied their high school diploma due to CAHSEE requirements will have their lives changed. They will need to choose between difficult options such as returning to high school for another year, finding a community college program which will offer classes to bridge the learning gap and ultimately prepare the student to pass the CAHSEE exam, or pursue an alternate graduation certificate such as the GED.
Hopefully, the data provided by this research will assist in building support for the funding and implementation of an effective online study alternative to better meet the needs of California high school students.
References
(CDE), C. D. o. E. (2004a). CAHSEE: English-Language Arts Teacher Guide.
(CDE), C. D. o. E. (2004b). Preparing for the California High School Exit Examination: A Mathematics Study Guide.
(CDE), C. D. o. E. (2005). Mathematics Framework for California Public Schools Kindergarten Through Grade Twelve: Adopted by the California State Board of Education, March 2005
Buchen, I. H. (1999, 1999). Business Sees Profits in Education: Challenging Public Schools. The Futurist, 38 - 44.
The California High School Exit Exam. (2003).): EdSource.
Gardner, H. (1983). Frames of Mind: The Theory of Multiple Intelligences. New York: Basic Books.
Great expectations: Leveraging America's investment in educational technology. (2002).). Washington DC: Benton Foundation Communications Policy Program.
Papert, S. (1997). Educational Computing: How Are We Doing? THE Journal, 24(11), 78.
Solovitch, S. (2006). Cahsee on Target: Number Sense.
Weiss, S. (2000). The Progress of Education Reform (No. vol. 1, no. 5): Education Commission of the States.
Wise, B., Harris, Taylor, Johnstone, and Miller. (2005). Independent Evaluation of the California High School Exit Examination (CAHSEE): 2005 Evaluation Report (Commissioned Report by Human Resources Research Organization for the California Department of Eduation No. FR-05-43). Alexandria, VA: Human Resources Research Organization (HumRRO).
