LA-) PS 3S67 POLICY RESEARCH WORKING PAPER 3 057 Teaching Adults to Read Better and Faster Results from an Experiment in Burkina Faso Helen Abadzi The World Bank Operations Evaluation Department Sector and Thematic Evaluation Division May 2003 POLICY RESEARCH WORKING PAPER 3057 Abstiract Two cognitively oriented methods were tested in Burkina treatments, and 72 percent of the measurements in favor Faso to help illiterates learn to read more efficiently. of treatments were statistically significant. The evidence These were (a) speeded reading of increasingly larger suggests that phonological awareness training is word units and (b) phonological awareness training to particularly effective in situations where the training help connect letters to speech. Learners were given period was short, and that rapid reading was more reading tests and a computerized reaction time test. advantageous in longer training situations. Overall, the Although the literacy courses were shortened by the results are indicative of the potential that scientifically arrival of rains and government delays, the piloted backed methods have in making adult literacy instruction methods helped adults read better than those in the more effective. However, due to the short duration of standard "control" classes. Learners enrolled in the the classes (3-4 months) learners apparently did not experimental classes performed better on the outcome receive sufficient practice to consolidate skills. Literacy tests than did learners enrolled in control classes. Ninety skills may still be prone to being forgotten if readers do percent of the possible comparisons between treatment not learn to read automatically and if opportunities to classes and control classes favored classes receiving read are few. This paper-a product of the Sector and Thematic Evaluation Division, Operations Evaluation Department-is part of a larger effort in the department to assess the effectiveness of adult literacy. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact HelenAbadzi, room H3-307, telephone 202-458- 0375, fax 202-522-3123, email address habadzi@worldbank.org. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at habadzi@worldbank.org. May 2003. (19 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the senes is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findtngs, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessanly represent the view of the World Bank, its Executive Directors, or the countries they represent. Produced by Partnerships, Capacity Building, and Outreach Teaching Adults to Read Better and Faster: Results from an Experiment in Burkina Faso Helen Abadzi, OEDST CAAS Computer-based Academic Assessment System IDA International Development Association INA Institut National d'Alfab6tisation msec milliseconds OSEO Organization Suisse d'Entraide Ouvriere UNESCO United Nations Educational, Scientific and Cultural Organization Contents Executive Summary In tro d u ctio n ...................................................................................1.............................1 Research Design ...................... Research Participants .................................1...... , , , .. Development and Pilot-Testing of Materials .2 A. Baseline Measurements and Comparisons .2 B. Developing and Pilot-Testing Literacy Interventions .4 Designing the Intervention and Training Teachers ..........................................................,. 5 C. Implementing Instruction .6 Subjects.6 Classroom Observations ......................................................... ,.7 Evaluating the Impact of the Instructional Interventions-Data Challenges .8 Results of the Intervention Efforts .8 Questionnaire Results .10 Discussion.1. . Appendix 1. Tests and Questionnaire .13 Appendix 2. Supplemental Tables .15 References .19 Figures Figure 1. Reaction speed and accuracy of U.S. and Burkinabe Students .3 Figure 2. Effect Sizes of Various Treatment Combinations .10 Tables Table 1. Research Design .6 Table 2. Learners' Demographic Information .................................,,,,,,7 Executive Summary The research carried out in Burkina Faso is a preliminary example of how a cognitive methods approach, supported by previous scientific work, helps improve literacy class outcomes on those crucial skills of learning how to read and write. (See Abadzi 2003a, 2003b for a review.) The research took place from May 2000 to August 2001. First, baseline measurements were taken and compared with U.S. norms. The comparisons indicated that student scores in basic anthmetic additions and subtractions approximated U.S. scores of the appropriate grades. However, the Burkmabe adults completmg one phase of adult literacy training were generally performing at a level lower than that attained by 2nd grade students in both Burkina Faso and the United States. Learners in the process of completing two years of nonformal adolescent and adult classes read too slowly (about 2.2 seconds per word) and inefficiently (80-87 percent correct); given the limitations of human memory, they were probably not able to use reading extensively. Graduates of past years who became literacy teachers have become more efficient readers with time and approximated the speed and accuracy of formally educated literacy teachers and of secondary rural school students. Some of the tested literacy teachers, however, had scores only slightly above those of their learners. Subsequently, the research team tested the alternative hypotheses that (a) speeded reading of increasingly larger word units would improve learners' readmg scores over those of a control group and (b) phonological awareness traming would improve reading scores over those of a control group. Two cognitively oriented instructional methods were implemented in 15 adult literacy centers of Burkina Faso comprising about 240 learners: phonological awareness and rapid reading of increasingly difficult words. The literacy centers were operated by a Swiss nongovernmental organization (Organisation Suisse d'Entraide Ouvriere - OSEO) and the governmental National Institute of Literacy (INA). One class was held in each center. Literacy centers were assigned at random to receive phonological awareness, rapid readmg, a combination of the two, and no special treatment (control group). Learners were given reading tests and a computerized reaction time test. Although the literacy courses were shortened by the arrival of rains and government delays, the piloted methods helped adults read better than those in the standard, "control" classes. Learners enrolled in the experimental classes performed better on the outcome tests than did learners enrolled in control classes. Ninety percent of the possible comparisons between treatment classes and control classes favored classes receiving treatments; and 72 percent of the measurements in favor of the treatments were statistically significant. The evidence suggests that phonological awareness training is particularly effective in situations where the training period was short, and that rapid reading was more advantageous in longer training situations. The findings indicate that even with the short literacy class duration and the various test administration problems in the field, effects could be discerned for both cognitive methods that were tried out. However, due to the short duration of the classes (34 months) learners apparently did not receive sufficient practice to consolidate skills. A year after instruction was completed, informal observations indicated that a number of these rural neoliterates had forgotten the sounds associated with some consonants. More exposure to print and better use of class time may be necessary for stable skills acquisition. INTRODUCTION Skilled reading is dependent on the development of fast and accurate reading of small units: letters (or ideographs), syllables, and words.' The modest levels of performance in many adult literacy classes worldwide raise the possibility that learners may not reach the necessary proficiency in the low-level skills required to read these small units automatically. Another reason for low perfornance, aside from inaccurate reading, may be that learners have limited phonological awareness, that is, awareness of how words are structured. Instruction that moved from phonological awareness training to rapid processing of syllables, words, and sentences, and then to activities designed to foster reading comprehension would be particularly beneficial to adults. These techniques could be integrated into existing instructional methods. (See Abadzi 2003a, 2003b for a review of the Bank's experiences in literacy and the cognitive-neuropsychological research related to adult literacy.) An opportunity to understand better the reading processes of neoliterates arose when the Swiss Development Cooperation agreed to finance, through its World Bank Trust Fund, evaluative research of Swiss-supported programs in Burkina Faso. This country, which has a literacy rate of only about 20 percent,2 was chosen because it has established a tradition of literacy activities, often onented toward income-generating groups. Literacy is taught by the governmental National Literacy Institute (INA) as well as by many nongovernmental organizations (NGOs). Notable among them is the Organization Suisse d' Entraide Ouvnere (OSEO), which pledged its help in carrying out the research. Because of weather patterns and agricultural work, literacy classes in rural areas are carried out only from January to June of every year. The research took place from May 2000 to August 2001. The research involved: (a) baseline measurements and comparisons with U.S. norms, (b) designing interventions and assessment procedures, (c) implementing interventions, and (d) evaluating the outcomes. A follow-up phase, assessment of forgetting and relapse into illiteracy was to be carried out in the spring of 2002, but time permitted only informal observations. RESEARCH DESIGN A research team was assembled, consisting of Dr. James M. Royer, educational psychologist at the University of Massachusetts - Amherst, and Burkinabe researcher Dr. Jules Kinda, professor of linguistics at University of Ouagadougou, along with three Burkinabe assistants3. The Burkinab6 research team was given background trainig on the literacy issues to be investigated and was trained in the use and administration of materials as well as on practical test development and data analysis. Research Participants The participants in the research were learners and graduates of literacy classes in Manegdbzanga, a rural area located about 45 kilometers outside Ouagadougou. Area residents have become known for their dedication to group formation and adult literacy activities4 where groups learned to read in the 1980s and 1990s in order to increase their agricultural and trading productivity. Representatives of the donor community frequently visit the villagers' association of Nomgana. In this area of limited formal education, literacy classes are well attended. In addition to adult literacy I Royer 1997. 2. UNESCO Statistical Yearbook 1999-2000. 3. Prof Pierre Magloubi and graduate students Laura Tahita and Sarata Dialo. 4. Easton 1998. 2 and income generation, OSEO has supported a successful Moore-French program in the local schools since about 1990. (The schools of Burkina Faso teach exclusively in French.) These children learned first to read in Moore. Several cohorts of them have completed primary school and gone on to French-speaking secondary schools. For literacy instruction and testing, it was decided to use the Moore language, which is spoken in Ouagadougou and its environs, includmg Manegdbzanga. The participants ranged in age from 7 to people in their 40s who did not know their exact age. The researchers initially tested various groups of participants to establish baseline data. Then the reading methods were applied to 15 classes of about 250 learners, whose average age was 20. Seventy-eight percent of them had not been to school at all. Development and Pilot-Testing of Materials Computer-assisted measuirements. Central to the measurement of reading performance was the Computer-based Academic Assessment System (CAAS). This system records the speed and accuracy of readers in letters, syllables, words, or sentences, and was originally developed for dyslexic children. A stimulus appears on the screen (such as a letter, word, or number) and the examinee responds into a microphone. The vocal response stops a clock in the computer and the examiner scores the data for accuracy. The speed and accuracy data, along with learner names and characteristics, are stored in a file on the computer and are later retrieved for statistical analysis. Measurements on speed and accuracy can be reported separately, or combined into an index. Three notebook computers with extra batteries were purchased and carried to Ouagadougou for this purpose. Stimuli for the CAAS were prepared in Moore by the team of linguists. Tests were developed for letters, syllables, common words, simple sentences, numerals, simple additions, and simple subtractions. Each test consisted of 20 stimuli, and their development presented some challenges. Moore is a tonal language, which uses accents on many vowels as well as some letters taken from Greek; developing fonts for the computer displays required some work, as did the development of criteria regarding which responses would be considered acceptable, given the tonality of the language. Paper and pencil achievement tests. Paper and pencil tests that measured letter identification, syllable identification, word identification, and understanding of sentences were developed in Moore. These were multiple-choice tests, where the learners were asked to circle the correct word. Sirmlarly, a multiple-choice test was developed for phonological awareness tasks. These were to be used as backup tests in case the electronics of CAAS failed and to double-check the CAAS results. (See details in Appendix 1.) The learners were not tested on reading aloud and listening comprehension because of the complexities involved in testing for readability of texts. Quesfionnaire. A questionnaire was developed m Moore and was administered to participants orally to obtain data on demographic characteristics, including attendance in fornal or Koranic school, frequency of reading use, and reasons for taking the literacy course. (See details in Appendix 1). A. BASELINE MEASUREMENTS AND COMPARISONS In this phase, the research team wanted to find out how well various groups of people read mn the area and to establish baselines. Comparisons were made with U.S. students, for whom data were available. Thus it would be possible to make comparisons between a literate country and a sample of Burkinabe school children and adult graduates of literacy courses. Also, the research team would thus evaluate the extent to which literacy class graduates were proficient in performing low-level reading 3 skills. Another group of interest consisted of Burkinabe neoliterates who had become literacy teachers. In May 2000, the researchers tested groups of residents with the CAAS and obtained speed and accuracy data for basic reading and math tasks. Results were compared with children who were (a) students in grades 1-3 of the bilingual schools; (b) adolescents attending nonformal schools for adolescents, largely financed by UNICEF; (c) secondary school students, who had completed the bilingual education primary school program; (e) graduates of literacy classes; (f) graduates of literacy classes who had become literacy teachers; (g) literacy teachers with formal education; and (h) secondary school students. The results were compared with norms of U.S. students. The comparisons indicated that the Burkinabe adults completing one phase of adult literacy training were generally performing at a level lower than that attained by 2nd grade students in both Burkina Faso and the United States (Figure 1). More specifically: * Students at all levels tested (rural bilingual grades 2, 3, 8, adolescent and adult nonformal classes) read more slowly and less accurately than a sample of second grade students in the United States. * Learners in the process of completing two years of nonformal adolescent and adult classes read too slowly (about 2.2 seconds per word) and inefficiently (80-87 percent correct); given the limitations of human memory, they were probably not able to use reading extensively. * Graduates of past years who became literacy teachers have become more efficient readers with time and approximated the speed and accuracy of formally educated literacy teachers and of secondary rural school students. Some of the tested literacy teachers, however, had scores only slightly above those of their learners. * Student scores in basic arithmetic additions and subtractions approximated U.S. scores of the appropriate grades. These results supported the hypothesis that adult literacy training programs may not be developing adequate low-level reading skills. Suitable methods were sought to find out if it was possible to increase the speed and accuracy of the next cohorts of literacy learners. Figure 1. Reaction speed and accuracy of U.S. and Burkinabe Students CAAS Speed (in seconds) of Burkinabe CAAS Accuracy (percent correct) of and US Students on Word Recognition Burkinabe and US Students on Word Task Recognition Task 3 ~ O 95 O m l r > 90> , | Ir~~~9 2 Omde 4 1 I L D o Aduh irAd 80 I Aduh Iad Burkinabe U.S. Burkinabe U S. 4 CAAS Accuracy (percent correct) on CAAS Speed (in seconds) of Burklnabe Burkinabe and US Students on Sentence and US Students on Sentence Recognition Task Recognition Task 16_ 100 12 O Gr 3 80 1TTM = | E:lGr 3 10- ~ ~~~~~G4 OG4 108 O = [:]GAdult yr1 60I I _ uAyr4 62 DAdul Grad 2 OAulGr 6 - 40 2- 20 - Burkinabe U.S. Burkinabe U.S. IB. DEVELOPING AND PILOT-TESTING LITERACY JENTERVIENTIONS From October 2000 to February 2001, the Burkinabe research team operated a "literacy laboratory" at the Center for the Handicapped and Orphans of Sector 28 in Ouagadougou. They invited illiterate or semi-literate area residents to come for aftemoon classes, in which various methods were tried and developed. About 30 people attended on a regular basis. The researchers pilot-tested various methods, observed and videotaped results, and asked the participants for their opinions regarding effectiveness. The following methods were tried: Phonological awareness exercises. These teach illiterates the structure of the language and are related to reading effectiveness. Participants learned to find the initial and final letters of words, make rhymes, and determine the number of syllables in words. Speeded reading tasks of progressive diffineulty. This task was to help learners read automatically as many common words as possible, so that their working memory would not be overloaded with letter-by-letter reading. Learners received packs of 20 cards, each with one word, which they were asked to shuffle and read to each other as fast as possible. The task involved two people; one was reading, and the other was timing the reader with an inexpensive plastic stopwatch. The learners were taught to read the results and place them every day on a graph paper. Every day, the readers read the pack a little faster. After the improvement had leveled off, they were given packs with longer words. Learners went through a total of four packs that had words of three, four, or more letters. Groupimg of similar letters and systematic pairing of consonant and vowel combinations. This technique was developed by the biostatistician Goteti Bala Krishamurthy, who had applied it extensively in south India. Learners were asked which consonants looked similar to them, and they learned these in groups of 4-5, in hopes of learning to discriminate among them. To help learners deduce the reading strategy by themselves, each of the consonants was paired systematically with all vowels. After learning a group of new letters, the learners were told to search for them in a local newspaper. The learners, many of whom had briefly attended school or other literacy classes, found the activities attractive. Variability and work in small groups helped learners focus on the material much of the time. It was gratifying to see (and capture on video) the reactions of learners who through phonological exercises understood the structure of language. Speeded tasks created competitions, with learners reading as fast as possible while others measured them with a stopwatch. More potential learners dropped by the center every day. Sometimes they were so excited, they did not want to leave at the end of the class. 5 However, problems were also noted. Many learners made errors when reading to each other, and these could go uncorrected when a monitor was not available nearby. Though chronometers incited learners' interest, registration of reaction times on graph paper was often incorrect. Some people, particularly women, could not read them well. Readmg from cards was in some respects hard and time-consuming, since each pack had to be kept separate and not nmxed up with others. Reading from sheets of paper was simpler, but then the order of the words stayed the same, and readers could memorize it. Phrases that required filling in the blanks were tried, but learners could not perform effectively, possibly because their working memory was too short to keep all the material. It was found that reading individual letters was not useful, and the lowest units presented to learners were syllables. Though wnting seemed necessary, it was not emphasized in this research study. After four months of working with the learners, the team developed appropnate tasks for phonological awareness in Moore and feasible tasks for timing and graphing progress of progressively harder speeded reading tasks. The Krishnamurty technique for Indian languages did not perform as expected with the Roman alphabet; learners were confused rather than helped by learning the b, d, p, and q together, and the researchers realized that they did not understand what degree of similanty helps and what degree impedes learning. So, this method was set aside. In addition to these activities, the CAAS and questionnaire were administered to instructors and neoliterates of centers in Ouagadougou. The tests were used for practice, and data were analyzed for possible significant insights to the automaticity process. However, the performance levels of the participants were nearly 100 percent in accuracy, and they did not provide important information. Designing the Intervention and Training Teachers To test the efficacy of the phonological awareness tasks and speeded reading, a quasi- expenmental design was developed. It involved four interventions: (a) phonological awareness training, (b) rapid reading training, (c) both phonological awareness and rapid reading, and (d) a control group involving only traditional teachmg techniques, that is, one person reading and the group repeating what was read. These were carried out in literacy centers in (a) Manegdbzanga, where OSEO finances implementation during the dry season every year (January to June) and (b) Windyam, also outside Ouagadougou, where the governmental National Literacy Institute is active. Although the assignment of learners in classes was not random (due to geographic considerations), the 15 centers were randomly assigned to one of the four interventions. With nine centers sponsored by the government and nine by OSEO, 18 treatment combinations were created. It was anticipated that each of the classrooms would enroll approximately 25 learners, making a total of 250 learners participating in the study. The original design was to have every participant complete a questionnaire and the paper- and-pencil tests at the end of the study. Because CAAS is time-consumng and it was not possible to charge multiple batteries m rural areas without electncity, only a sample of learners could be tested with it. Thus, the plan was to randomly select five learners from each school to complete the CAAS tasks both pnor to beginning instruction and upon completion of the first phase of schooling. This plan would result in CAAS assessments for 90 learners (Table 1). 6 Table i. lResearch Design Approach Treatments Classes Leamers CAAS INA Phonological Awareness (PA) 2 50 10 random (Government) (25 per class) (5 each class) Rapid Reading (RR) 2 50 10 random (25 per class) (5 each class) RR + PA 2 50 10 random (25 per class) (5 each class) Control 3 75 15 random (25 per class) (5 each class) OSIEO Phonological Awareness (PA) 2 50 10 random (Swiss NGO) a (25 per class) (5 each class) Rapid Reading (RR) 2 50 10 random (25 per class) (5 each class) RR + PA 2 50 10 random (25 per class) (5 each class) Control 3 75 15 random (25 per class) (5 each class) Total 18 250 90 Teacher training. The first challenge was to train 24 teachers in carrying out the methods. Training was done over four days (January 8-12, 2001) at the Manegdbzanga center of Nomgana. The teachers were paid an honorarium for their attendance as well as a small stipend for implementing the experimental methods. The rationale for all methods was explained, and videotapes were shown, which had been recorded during the operation of the "literacy laboratory." Then the teachers, many of them themselves graduates of literacy classes, were asked to perform the tasks. They found the training very interesting but also very brief. A refresher training would have been useful, but for logistical reasons, it was never carried out. It would have been helpful to measure teacher performance and use it as explanatory variable in the treatment and control groups, but time did not permit this procedure. A random sample of learners was pretested through the CAAS. It was determiined that most could not read at all, although a few recognized a few letters. Eighty-one percent of them had not been to school at all, while most of the remaining 18 percent reported attending for one year. Thus, no pretest paper-and-pencil test was applied. Also, the assumption was made that experimental and control groups were similar at the beginning of treatment, allowing a comparison of the post-test results only. The number of years of school correlated only .09 with final reading scores. Because Burkina Faso has a literacy rate of only about 20 percent, it was unlikely for participants to learn reading outside formal or nonformal classrooms. C. IMPLEMENTING INSTRUCTION Subjects The subjects for the experimental part of the study were 425 participants of literacy classes. There were more females (295) than males (130) enrolled in the classes and there tended to be a wide range in ages with the youngest enrollee 7 years of age and the oldest reportedly 50 years of age. The mean reported age of the learners was 20. Forty-two percent of the learners were married and 32 percent were single. One hundred thirty five of the learners reported they had at least one child. The number of children the learners reported having ranged from 1 to 9. About 78 percent had never been to school, and about 22 percent of the learners had some previous formal schooling. The shortest time attendance penod was 1 year, and the longest was 3 years (Table 2). Twenty-eight learners reported 7 having gone to a Koranic school, and may have known some Arabic reading. Their competency in Arabic was not tested. Since literacy is a desirable activity in Burkina Faso and a form of social gathering for rural residents, more people attended than expected. The subjects were rural residents who found out about the classes and decided to come and thus were self-selected. (No surveys were done to determine which residents had decided not to come and for which reasons, therefore, the criteria for self- selection are unknown.) All were from farming families, although their specific socioeconomic status was not known. Parental formal education was not known, but was probably non-existent. Table 2. Learners' Demographic Information Gender: 30 male (30.5% 295 female (69.5) Age: 20.07 years (SD = 8.48); range 7-50, mostly were 11-21; mode 15 Marital Status: 79 married (42%), 246 single (58%) Number of children: 289 (68%) had no children; modal was 2, range 1-9 Primary school attendance: None 326 (78%) One year 93 (22%) Two years 6 (0.1%) Three years 1 (0.02%) Koranic school attendance 28 (6%) Modal age at attendance was 10 (N=5) Other types of training 47 (11%) Classroom Observations Subsequently, the researchers regularly observed and helped the teachers. The classes were observed at least once a week by the research team. The teachers largely implemented the methods as expected and were given help when they did not. The researchers only corrected issues related to the implementation of the methods; they did not offer suggestions for the instructional improvement of the literacy class cumcula, although several examples were presented. For example, one teacher initially instructed learners to read slowly, although the goal of literacy classes should be rapid, effortless reading. The research team provided materials for the classes to carry out the two treatments: (a) word lists for phonological awareness and sheets for learners to register what differences they heard in words; (b) stopwatches, graph sheets, packs of cards with words of varying difficulties as well as sheets with lists of words. These were printed and local print shops. Stopwatches (one for every two learners) were bought for 2000 CFAF (US$2.50) each at the local market. Classes met for two hours daily, and learners practiced the experimental methods for about 30 minutes in class each day. For the remaining time, learners practiced decoding skills and took turns reading aloud while other learners repeated. This methodology does not use class time well, but it is widely used, and the researchers did not intervene to change it. Again, the researchers observed that the activity level required by progressive speeded tasks was a welcome departure from the usual read-and-repeat method. Generally, learners became able to time their classmates, read stopwatches, and record time on simple graphs. The phonological awareness tasks also worked well, but they were time-consuming as presented, and the method needed more refinement. 8 Because the methods were applied in different villages that had some distance among them, it is highly unlikely that there were contamination effects from one classroom to another. EvaRuadtng the llmpact of the linstructioimal lInterventions-lData Challenges Because of delays and administrative difficulties, only 15 centers could be used in the study, but the number of participants was much larger than the 250 planned (Appendix 2, Table 1). Reading tests and a questionnaire were administered to a total of 425 learners. Of these, 199 were enrolled in classrooms conducted by OSEO staff and 227 were enrolled in classrooms conducted by INA staff. Field research in relatively remote rural areas inevitably runs into the unexpected. OSEO classes were underway in early February, but government-financed (INA) classes did not start until the end of March. The INA leamers received only three months of instruction by the time the rains arrived in June, while the OSEO leamers had five months of instruction. Five leamers from each class were to be chosen at tested at random using CAAS. Although three batteries were provided for each notebook frequent electrical blackouts and perhaps the 450 C heat in the villages (where there was no electricity) contributed to the deterioration of the batteries. Each would last 20 minutes or so, and they often ran out in the middle of a test, losing the results. Thus, researchers could not test the number of leamers expected. Overall, only 37 learners could be assessed through the CAAS in pre- and post-tests, and these were not distributed evenly in the various classes. Therefore, the analyses of these data were not included in this report. Near the end of the course, learners answered a questionnaire that collected demographic information (see Table 2 for a summary), and they completed tests (in their native language) that measured their competence at identifying letters, syllables, words, and sentence length material. Obtaining demographic data from leamers also posed some problems. About 30 learners stated that they had never before attended literacy class or school, but displayed rudimentary reading skills during the pretest. Apparently literacy classes are a social function in the villages, and residents did not want to be excluded. Because the experimental conditions were imperfect, the data were "noisy" with more error variance than expected. Efforts were made to understand the effects of the treatments and disentangle them from related variables. The observational data collected (including videotaped sessions) provided qualitative information to supplement the statistical analyses. Results of the linterventdon Efforts Analyses of variance were carried out, comparing the various treatment groups in OSEO, INA, and control classes. Performance on the tests administered near the end of phase one training indicated that learners enrolled in courses sponsored by OSEO performed better than did learners enrolled in government-sponsored courses. Given that the government courses had lasted a shorter period, this result was not surprising. The results on the paper-and-pencil tests also showed that the learners receiving any of the new instructional interventions tended to perform better on the tests than did the learners enrolled in the control classes. There were 20 possible comparnsons between treatment and control classes; the treatment groups scored higher than the control group in 18 of the 20. Moreover, in 13 of these 18 comparisons the advantage for the treatment group over the control group was statistically significant. In the two comparisons favoring the control group over the treatment group, the difference was not statistically significant. Specifically, analysis of variance results were: 9 * Letter identification. A two-way analysis of variance indicated that the advantage for the OSEO groups over the INA group was statistically significant, F(1,241) = 10.19, p < .01. The effect for treatment group was also significant, F(3,241) = 4.21, p < .01. The interaction between sponsor and treatment was not significant, F(3,241) = 2.35, N.S. * Syllable identification. A two-way analysis of variance indicated that there was a statistically significant effect of sponsor, (F1,229) = 32.63, p < .01, and treatment type, F( 3, 229) = 4.66, p < .01. The Interaction between sponsor and treatment type was also significant, F(3, 229) = 3.85, p < .05. * Word identifi cation. The two-way analysis of variance for the word identification test indicated that sponsor was a significant source of variance, F(1,207) = 24.89, p < .01. Treatment condition was also a significant source of variance, F(3, 207) = 5.99, p < .01. The phonological awareness treatment group in INA classrooms did not complete the word identification test because the test was too difficult for them. The test of the interaction is uninterpretable given the lack of data for the phonological awareness INA group * Sentence identification. The statistical analysis for the sentence identification and understanding test was conducted only on the OSEO treatments. This analysis indicated that treatment group was not a significant source of variance, F(3,188) = 2.66, p < .05. The reader will note that all three of the treatment groups outperformed the control group on the sentence understanding test. Post hoc contrasts indicated that the rapid reading group and the PA + RR group performed significantly better that did the control group. The impact of the treatment conditions differed according to the agency sponsoring the courses. Learners enrolled in government courses tended to benefit most from phonological awareness training, whereas learners enrolled in OSEO courses tended to benefit most from rapid reading or the combination training. The probable reason for this differential impact is that phonological awareness training is most likely to be useful early in instruction whereas rapid reading training is most likely to be beneficial after some skill in phonological awareness has been acquired. The shorter government- sponsored training allowed the benefits of phonological awareness to be felt before the benefits of rapid training became evident. In contrast, the longer OESO courses, brought out the benefits of rapid reading training. There were gender differences in performance. Males tended to perforn better than females in the tests (Appendix 2, Table 10). However, there was no evidence that the different treatments affected the performance of the two sexes differently. As mentioned earlier, some tests could not be completed. The phonological awareness tests were computerized, and when the computers failed, they could not be scored. Also results from the computer-based CAAS assessments were not interpretable, since they could be administered to only a small subset of the learner population; only 37 learners could be tested on a pre and post basis, and they were not randomly or evenly distributed among classes. Regression analyses were carried out, but the error variance due to the CAAS problems was large, and outcomes were confusing. Differences may be statistically significant, but how practically useful are they in determining whether literacy courses should include phonological awareness and rapid reading? Educational research often uses effect size, a measure of the differences between treatment and control groups in 10 standard deviation (z score) units.5 Most effect sizes (shown in Appendix 2, Tables 2-9, Figure 2) are moderate to large. Figure 2. Effect Sizes of Various T1reatment Combinations Percent Correct on Syllable Identification: Percent Correct on Word Identification: Effect Sizes at Top of Bar, t At .05. Effect Sizes at Top of Bar, c" At .05. 292-- 26D 337 397 85 197 asi. S- 04-- |1 5 124L - 75'1X , .. M = . 75' 9 _ 45 4 _ |_ 1 0 - 2 5J' l l - PA RR PA+RR Control PA RR PA+RR Control Percent Correct on Letter Identfication: Percent Correct on Sentpulaio enrtofication: Effect Sozes at Top of far, llAt.05. Effect Sizes at Top of Bar, At .05. ass. 369-- 323-- ss 391 472-- 603-- 75'1 l, 75 PA RR PA+RR Contro PA RR PA+RR Contrd[ Q;uestionnaire Results The data fr7om the questionnaire provided a portrait of the population enrolling in the literacy programs. Some of the pertinent findings follow. Few questionnaire variables showed significant relationships with reading scores. The only learner characteristics significantly related to performance on the paper-and-pencil test were the number of years of prior schooling and the age at which some learners attended Koranic school; those who had attended Koranic schools had better performance if they had attended at a younger age. However, only 6 percent of the learners had attended Koranic schools, and the correlation should be interpreted with caution (Appendix 2, Table 12). 5. An effect size is the mean of the treatment group mnus the mean of the control group divided by the standard deviation of the control group. Effect sizes may show how the average treatment group student would score IF the student were in the control group. Sizes of 0.4-0.6 are considered small, 0.6-0.8 moderate, and above 0.8 large. For example, an effect size of 0.5, located at a statistical table of cumulative normal probabilities (area of the normal curve encompassed by various z score values), would be found to encompasses 69% of the normal distribution, hence the average treatment group student would score at the 69th percentile IF they were in the control group. An effect size of 1.5 means that the average treatment group student would score at the 93rd percentile of the control group. 11 There was no relationship between self-ratings of learners and reading scores. There was also no relationship between learners' evaluation of their own reading skills, reported reading frequency, and reading performance on the paper and pencil tests (Appendix 2, Table I1). It would have been expected for literates to have some insight about how well they were reading. This lack of relationship brings into question the validity of self-ratings of competence among literacy programs that use them, such as REFLECT. Similarly, a sample of learners tested in Bangladesh after course completion showed 92 percent pass, but a later retest showed only 32.3 percent of learners passing. Nevertheless, a tracer study of the same population using self-reports found that 85 percent of participants still wrote from time to time and 97 percent still read two years after program completion.6 It was noted that learners who had attended the rapid reading courses tended to rate themselves as better readers. It is possible that the lively reading activities promoted by this method created in the minds of the participants illusions of competence or comprehension, which are known to happen when learners are exposed to dynamic teachers and challenging learming events.7 About 135 graduates were interviewed and retested in May 2002, but time and funding did not permit reliable research. Most had not had anything to read for the previous year. Several had forgotten individual consonants and could read with difficulty. They did remember numbers.and still could do phonological awareness exercises. Many expressed pleasure at the rapid reading and phonological exercises and some went again to the literacy class hoping that the new methods would be again applied. Given the informal observation conditions, it was not possible to estimate the incidence of relapse into illiteracy. Due to the short duration of the classes (34 months) learners apparently did not receive sufficient practice to consolidate skills for the long term. Discussion The findings indicate that even with the short literacy class duration and the vanous test admmistration problems in the field, effects could be discerned for both cognitive methods that were tried out. The alternative hypotheses that these two methods improve reading scores above those of control classes using "traditional" method were accepted. The research team recommended to the National Literacy Institute that these methods be adopted, and teacher training videotapes were made available to the agency. The results of the reported research are best viewed as a pilot effort that demonstrates the potential benefit of weaving aspects of modern reading instruction into the fabric of traditional adult literacy training in developing countries. The treatments evaluated in the project were mostly "add- ons" to traditional procedures rather than tightly integrated instructional methods. Even so, the learners receiving the supplemented instruction outperformed their control school counterparts. Possibly some of the effects of rapid reading may be due to the increased engagement and time on task that this method requires. Instead of learners watching others read inactively, they were engaged in reading with their partners much of the time. If new methods were coupled with increased time on task, literacy outcomes might be even more beneficial. Future research should evaluate this expectation. It should also move beyond techniques targeted solely at low-level reading skills. It should evaluate reading comprehension methods that have proved effective in developed countries. Instruction that moves from phonological awareness training, to rapid processing of syllables, words, 6. Ahmad and Lohani 2001. 7. Bjork 1994, 1999. 12 and sentences, and then to activities designed to foster reading comprehension skill would be particularly beneficial to adults. These techniques could be integrated into existing instructional methods, thereby maintaining the cultural relevance of the instructional procedures. To recommend that these methods become a routine part of literacy courses in other countries, more tnals must be carried out in environments different from those of rural Burkina Faso. Issues to study further in future research and tryouts might be: o A brief neuropsychological assessment should be carried out before and after the teaching program (digit span, verbal memory, workmng memory conceptual knowledge, visuospatial abilities), as shown by the Mexican Neuroalfa literacy program. Thus, it would be possible to find out the extent to which the cognitive processes of the graduates have improved. o The phonological awareness exercises should be more extensive and gradually increase in complexity. They could include syllable matching and rhyme comparisons with words and non-words. It is also important to do pair discriminations and manipulate the phonemic distance between the consonants and the vowels. o Improvements must be made in the low-level reading task logistics. The reading pack always contained the same words, so even with shuffle, learners might have memonzed the range of words contained therein, since there were only 20, and these were sounded out as a reader and as a reader-timer. Perhaps longer packs should be tned. O The amount of practice needed for the sound-letter linkages to stabilize in the memory of neoliterates must be assessed. Clearly, 3- and 4-month courses are too short for rural populations with limited daily exposure to print. With literacy software available to supplement the fast reading tasks, an experimental use of computers could be applied in various centers. Reaction time data could be collected. Additional software, if it becomes available, could offer practice in visual scanning tasks, discnmination tasks, reading, lexical decision tasks. Data would give information about learning process and development of automaticity. Neoliterate adults may achieve automatic reading only after a very large number of pairings of visual and auditory stimuli. Such a large number of trials is not feasible in classes, but it is feasible through computers. 13 Appendix 1. Tests and Questionnaire The development of assessment procedures occurred in workshop dunng Dr. Royer's January 2001 visit to Burkina Faso. Because CAAS assessments were time-consuming and dependent on technology, it was decided that additional paper-and-pencil measures that could be group administered would need to be developed. All of the assessment instruments were developed in Moore. The instruments and their contents were: * Phonological awareness. The test was administered by havinig the examiner read two words. In some cases the learner was asked to circle yes if the words rhymed and no if they did not. Other variations of the test were circling yes if the words started with the same sound and no if they did not, or to circle yes if the words ended with the same sound, and no if they did not. * Letter recognition. This test was administered by having the examiner read a letter name. The leamer then selected one of four letters on the test page and circled it as the correct answer. The leamer was asked to identify 20 letters. * Syllable recognition. This test was administered by having the examiner read a syllable and the leamer then selected one of four syllables on the test page and circled it as the correct answer. The leamer was asked to identify 20 syllables. * Word recognition. Thls test was administered by having the examiner read a word and the leamer then selected one of four words on the test page and circled it as the correct answer. The leamer was asked to identify 20 words. * Sentence recognition. This test was administered by having the examiner read a sentence and the leamer then selected one of four sentences on the test page and circled it as the correct answer. The leamer was asked to identify 20 sentences. Learner questionnaire. A vanety of factors in addition to instructional effectiveness could influence the amount of skill acquired in adult literacy training, and the questionnaire identified some of the possible factors. The questionnaires were read to individual leamers in Moore, and the examiners coded the answers given by the leamers. The following information was requested: * The leamer's gender * The leamer's age * Whether the leamer was single or married * The number of children the leamer had * The leamer's native language * The language of reading instruction * The number of years of previous schooling the learner had received * Whether or not the learner attended formal school * The age of the leamer when he/she first attended formal school * The number of years the leamer attended formal school * The level of instruction previously completed by the learner * The number of years since the leamer had attended formal school * Whether or not the leamer attended Koranic school * The age of the leamer when he/she attended Koranic school * Whether or not the leamer attended any other type of school, and if so, what kind 14 o The learner's evaluation of their own reading skills (e.g., very good =1, good = 2, fair = 3, poor=4, none=5) o The number of days during the academic period that the learner was absent o The learner's motivation for learning to read o The learner's intended use for literacy skills o How often does the learner read (coded with 1 the most frequent) o What does the learner read o How many days during the instructional period that the instructor was absent 15 Appendix 2. Supplemental Tables Table 1. Sites and Treatments Administered Site Agency Instructional Treatment Number of learners Goey OSEO Phonological Awareness (PA) 17 Sogpeelse INA Phonological Awareness 66 Zaongo SEO Phonological Awareness 27 Godre OSEO Rapid Reading (RR) 25 Kamsaooge OSEO Rapid Reading 18 Nagreongo INA Rapid Reading 28 Samande INA Rapid Reading 22 Koaada OSEO PA+ RR 31 Koasenge INA PA + RR 28 Kulkeka INA PA + RR 23 Waanvuus OSEO PA + RR 35 Kulkeka/jeunes [NA Control 33 Nongstenga OSEO Control 17 Tanlaorgo OSEO Control 28 Wurgu INA Control 27 Table 2. Effect Sizes for Letter Identification Test Sponsor Phonological Rapid PA + RR Awareness Reading OSEO 025 369** 323** INA 879** 439** .726** Table 3. Percent Correct Means and Standard Deviations On the letter Identification Test for Treatment Groups and Sponsors Sponsor Phonological Rapid Reading PA + RR Control Totals Awareness OSEO 69.43 79 76 78 40 68.68 74.79 (31.55) (24.03) (26.56) (30 01) (28 03) INA 73 67 58 75 68 50 43 80 60.14 (27 55) (36 40) (29 45) (33 98) (33 42) Totals 70 70 73 86 74.82 58 14 (30 19) (29.29) (27 88) (33.82) Table 4. Effect Sizes for Syllable Identification Test Sponsor Phonological RapidPA+R Awareness Reading PA + RR OSEO - 197 292** 260 INA I 04** 851** 912** 16 Table 5. Percent Correct Means and Standard DeviatAons On the Syflabile fIdentificatAon Test for Treatnemt Giroups aind Sjponsoirs Sponsor Phonological Rapid PA +M Control Totals SPonor °Awareness Reading OSEO 77.57 89.02 8827 82.19 85 17 (26 89) (21.48) (23 00) (23 38) (23 92) INA 74.00 67.65 69 67 39 62 61.14 (29 89) (29.64) (28.22) (32 92) (32 90) Totals 76 54 82.76 81 71 55 83 (27 54) (25 82) (26 36) (36 04) Table 6. Effect Sizes for Word ]dentiffication Test Sponsor Phonological Rapid PA + M Awareness Reading OSEO -.124 337 397 INA Xxx .910°° 912°° Table 7. Percent Correct Means and Standcard DevD ations Onm the Woird Ji dentificationm Test foir Treatment Giroups Anmd Sjponsoirs Sponsor Phonological Rapid PA +M Control Totals Awareness Reading OSEO 76.03 8707 88 52 79.00 83 89 (30.87) (24.42) (22.60) (23 92) (25.93) INA xxx 70.88 70.95 44.63 61 14 (30 32) (32.43) (28 85) (32.90) Totals 76 03 82 33 83 60 56 90 (30 87) (27 05) (26.71) (31 64) Table 8. Effect Sizes for Sentence Understamding Test Sponsor Phonological Rapid PA + M Awareness Reading OSEO 391 .472°° .608°° INA Xxx Xxx xx Table 9. Percent Correct Means and Standiard Deviatonms On the Sentence Understanding Test for TreatmentGrosips and Spomsors Sponsor Phonological Rapid PA + RR Control Totals Awareness Reading OSEO 73 92 75 98 79 44 64 00 75.51 (23.49) (20.80) (20.96) (25 37) (22.27) INA xxx 57.19 67.10 xxx 63 72 (23.87) (22.54) (23 23) Totals 73 92 70 70 74 94 64.00 (23 49) (23 12) (22.23) (25 37) 17 Table 10. Performance on the Literacy Tests as a Function of Gender and Treatment Approach and Treatment Sex Letter Syllable Word Sentence OSEO Male 62.3 692 61 15 700 Phonological Awareness 11 12 13 12 (29 7) (30.4) (37 5) (26.6) Female 72 7 816 83 5 75 8 24 25 26 25 (32 4) (24 6) (24 5) (22 2) Male 84 6 95 0 93 7 83 7 Rapid Reading 12 12 12 12 (206) (11 9) (172) (198) Female 77.8 86 5 84.3 72 7 29 29 29 29 (25.4) (24 1) (26 6) (20 7) PA+RR Male 884 95 6 93 9 84 3 22 23 23 22 (148) (104) (136) (201) Female 71 3 83 0 84 5 76.1 31 32 31 32 (30 7) (27 8) (22 6) (21 2) Control Male 95.0 100 100 70 l l l l Female 67 9 810 775 63 6 33 15 14 14 (30 1) (23 7) (24.1) (26.3) INA Male 817 80 0 Phonological Awareness 6 6 (14 0) (26.8) Female 68 3 700 9 9 (33 5) (32 7) Male 68 6 714 82 1 61 4 Rapid Reading 7 7 7 7 (42 0) (34 2) (28.8) (26 1) Female 51 1 650 63 0 53 9 9 10 10 9 (31.8) (27 6) (30 2) (23.0) PA+ RR Male 88 3 82 1 90 0 762 6 7 6 8 (14 0) (33 6) (24 5) (25 7) Female 63 5 65 9 63.3 63 9 24 23 15 23 (304) (26 1) (32.7) (21 0) Control Male 55.5 48 6 52 3 10 11 11 (34 8) (35 4) (28 8) Female 36 0 33 0 39 4 15 15 16 (32.2) (30 5) (28.6) Table 11. Learner Self-Perceptions of Reading Skills Evaluation Sponsor Awareness Rapid PA + RR Control Good/very good OSEO 69% 81% 53% 77% INA 57% 64% 52% 64% Fair/bad/can't OSEO 29% 19% 47% 21% read INA 42% 36% 48% 35% 18 Tablbe 12. Coirrellallons B3etween Learneir ReadEing erfoirmnnaice and Learmer CbanrcteirMscs (Co˘rreRation, Signifecance, N) 1 2 3 4 5 6 7 8 9 Reading Performance 1.0 (1) Learner Age (2) .012 1 0 .874 170 Mantal Status (3) .048 -.644** 1.0 .529 .000 172 422 Number of Children -.127 .835* - 638** 1.0 (4) .098 .000 .000 171 422 424 Age first attended .277 -.017 -.142 040 1.0 Formnal school (5) .063 881 195 .715 46 84 85 85 Years attended .093 .145°° -.096 .047 .298 1.0 Formal school (6) .225 003 .048 .334 .006 171 418 421 420 84 Age attended Koranic .-7312 132 078 .001 - - 1.0 school 002 .513 .699 .995 - 15 27 27 27 -- -- OwnEval.Of -121 -.045 011 032 -.161 -.425*0 -091 10 Reading skill (8) .133 366 .827 515 .147 .000 .665 157 406 408 407 82 82 25 Howoften Learner -061 -.014 -.041 .011 -.147 .050 -.091 092 10 reads 438 .783 411 824 196 314 665 .070 165 403 406 405 79 405 25 391 19 References Abadzi, Helen. 2003a. Improving Adult Literacy Outcomes: Lessons from Cognitive Research for Developing Countries. Operations Evaluation Department. Washington, D.C.: World Bank (forthcoming). 2003b. Adult Literacy. A Review of Implementation Experience. Operations Evaluation Departnent. Washington, D.C.: World Bank (forthcoming). Aftab, T. 1994. Fighting Illiteracy: What Works and What Doesn 't: A Case Study of Female Literacy in Pakistan. Convergence 27: 25-34. Ahmed, Manzoor and Shiv Lohani. 2001. NFE in Bangladesh: Synthesis of Experience and Future Directions. Dhaka: Directorate of Nonformal Education. Bjork R. A. and E. L. Bjork. 1992. "A New Theory of Disuse and an Old Theory of Stimulus Fluctuation." In A. Healey S. Kosslyn, and R. Shiffrin (Eds.) From Learning Processes to Cognitive Processes: Essays in Honor of William K Estes. Vol. 2, p. 35-67. Hillsdale, N.J.: Erlbaum. Bjork, R.A. 1994. "Memory and Metamemory Considerations in the Training of Human Beings." In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing about Knowing (p. 1 85-205). Cambndge, MA: MIT Press. Easton, Peter. 1998. Decentralization and Local Capacity Building in the Sahel. Results of the PADLOS Study. OECD/CILSS/Club du Sahel. Royer, J. 1997. "A Cognitive Perspective on the Assessment, Diagnosis, and Remediation of Reading Skills." In Phye, G. (Ed.) Handbook ofAcademic Learning. Academic Press. Royer, J. M., Abadzi, H., & Kmnda, J. 2003. "The impact of phonological awareness and rapid reading training on the reading skills of adolescent and adult neoliterates in Burkina Faso." International Review of Education (forthcoming). UNESCO: 1999-2000 Statistical Yearbook. Policy Research Working Paper Series Contact Title Author Date for paper WPS3043 The Incentive-Compatible Design Thorsten Beck May 2003 K Labrie of Deposit Insurance and Bank 31001 Failure Resolution: Concepts and Country Studies WPS3044 Impregnated Nets Cannot Fully Mead over May 2003 H. Sladovich Substitute for DDT. Field Bernard Bakote'e 37698 Effectiveness of Malaria Prevention Raman Velayudhan in Solomon Islands Peter Wilikai Patricia M. Graves WPS3045 Causes and Consequences of Civil Klaus Deininger May 2003 M. Fernandez Strife Micro-Level Evidence from 33766 Uganda WPS3046 Migration, Spillovers, and Trade Ross Levine May 2003 E Khine Diversion. The Impact of Sergio L. Schmukler 37471 Internationalization on Stock Market Liquidity WPS3047 Comparing Mortgage Credit Risk Robert Buckley May 2003 0 Himid Policies. An Options-Based Approach Gulmira Karaguishiyeva 80225 Robert Van Order Laura Vecvagare WPS3048 Targeted Transfers in Poor Countries: Martin Ravallion May 2003 P Sader Revisiting the Tradeoffs and Policy 33902 Options WPS3049 Hidden Impact? Ex-Post Evaluation Shaohua Chen May 2003 P Sader of an Anti-Poverty Program Martin Ravallion 33902 WPS3050 The Changing Financial Landscape Wafik Grais May 2003 R Vo Opportunities and Challenges for Zeynep Kantur 33722 the Middle East and North Africa WPS3051 Regional, Multilateral, and Unilateral Glenn W Harrison May 2003 P Flewitt Trade Policies of MERCOSUR for Thomas F. Rutherford 32724 Growth and Poverty Reduction in David G. Tarr Brazil Angelo Gurgel WPS3052 Long-Run Impacts of China's WTO Kym Anderson May 2003 S. Lipscomb Accession on Farm-Nonfarm Income Jikun Huang 87266 Inequality and Rural Poverty Elena lanchovichina WPS3053 Economic Impacts of China's Elena lanchovichina May 2003 S. Lipscomb Accession to the World Trade William Martin 87266 Organization WPS3054 The New Comparative Economics Simeon Djankov May 2003 G Sorensen Edward Glaeser 37088 Rafael La Porta Florencio Lopez-de-Silanes Andrei Shleifer WPS3055 The Global Growth of Mutual Funds Deepthi Fernando May 2003 A Yaptenco Leora Klapper 31823 Viktor Sulla Dimitri Vittas Policy Research Working Paper Series Contact Title Author Date for paper WPS3056 Avoiding the Pitfalls in Taxing Patrick Honohan May 2003 A. Yaptenco Financial Intermediation 31823