BACKGROUND

IL&FS Education piloted ReadToMe in 10 schools in Uttar Pradesh in the academic year 2017 – 18. EnglishHelper conducted assessments designed as Baseline – End line studies, in 5 of the schools. This would allow measurement of learning outcomes achieved in one academic year among schools using ReadToMe.

The assessments were conducted among students of grades 6, 7, and 8. The Baseline Assessments were conducted in November 2017. End line assessments were conducted in February 2018. Some observations that are important to note here are:

a. Typically, Baseline – End line assessments are conducted at the beginning and at the end of the academic year allowing for at least 6 to 9 months to gauge impact. However, in the case of Uttar Pradesh, the assessments were separated by 3 months. The program was implemented for an average of 6.8 weeks (ranging between 5 and 8 weeks across the schools).

b. Additionally, EnglishHelper recommends at least 75 sessions of ReadToMe in an academic year. The schools in the pilot in Uttar Pradesh conducted between 8 and 32 sessions (averaging 15 sessions) of ReadToMe in the academic year 2017-18. This is 20% of the recommended number of sessions for ReadToMe. The results of the assessments have been interpreted in light of these constraints.

Some of the key questions that were sought to be answered for an evaluation of the pilot were:

• • Does the RightToRead program have an impact on the learning outcomes of English learners?
• • Can the program impact learners from different grades?

ABOUT RIGHTTOREAD

EnglishHelper enables technology – based reading and comprehension improvement for learners across all age groups. Since launching in India in 2011, EnglishHelper has successfully implemented its reading and comprehension solution, ReadToMe, in public and private schools across the country.

The RightToRead program was launched by EnglishHelper in 2013 based on its multi-sensory technology platform for reading and comprehension called ReadToMe.

The goal of RightToRead is to demonstrate that reading and comprehension technology when integrated with the given school curriculum can make a material difference in literacy. Working on the tenets of Minimum Change and Sustainability, the following actions are practiced:

• • The class text book and state prescribed syllabus is digitized and made available on the reading software. No additional or new study material is introduced to the students.
• • The software is integrated into the school time-table with the regular English class period. Students are not required to devote additional time for ReadToMe classes.
• • The existing teachers are trained and empowered to use ReadToMe.

SAMPLE DESIGN

Assessments were undertaken among five out of the ten schools enrolled for the pilot. A total of 249 students were assessed in the Baseline and the End line in the three grades. The tables below present the distribution of the students by grade.

ASSESSMENT METHODOLOGY

Assessments were carried out using tablets at all five schools by IL&FS personnel to ensure unbiased delivery of the assessments. On completion of the assessments at each school, the student submissions were available on the assessment app as a ‘read only – protected file’ which were uploaded by the field personnel. Each file was uniquely identified by school name and school code. Subsequently, STAMP (Skill Training Assessment Management Partners Ltd.) extracted the data from these files, processed it on their proprietary assessment engine and shared outcomes with EnglishHelper. The delivery of the assessments in schools, processing of outcomes and reporting is the combined effort of STAMP and EnglishHelper.

ASSESSMENT INSTRUMENTS

The assessment instruments were designed to test the constructs of Word Structure, Vocabulary and Reading Comprehension. The instruments encompassed single-response close-ended items presented as multiple-choice responses with a single correct option.

The assessment instruments were mapped to the Common European Framework of Reference (CEFR) scale. The CEFR allows definition of proficiency such that like-to-like comparisons of outcomes can be made across grades, states, cohorts and any learner group.

The CEFR describes language proficiency (related to listening, speaking, reading and writing) on a six-level scale:

• • A1-A2 for Basic User
• • B1-B2 for Independent User
• • C1-C2 for Proficient User

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do” statements. These are presented below:

It also allows for branching and defining sub – competencies, such as A1.1 and A1.2. The questions in the assessment instruments range from the CEFR levels A1 to B1. A low proportion of B1 questions was included and these have been classified with A2 questions under the heading, “A2 and higher.”

The composition of questions in the assessment instruments is as follows:

ASSESSMENT OUTCOME

The test instruments were composed of dichotomous items. Learners’ responses to every question was scored a 1 or 0, depending upon whether the learner answered the question correctly or not. No partial credits were allowed. The sum of the learner’s responses to all questions in the test was scaled to 100; providing the learners’ score. Scores from the entire sample were aggregated and the averages analyzed. (Scores were not analyzed at the individual school level owing to low population and sample sizes). We examine the outcomes for every grade separately. The metric “Improvement” is measured as the difference of the End line score from the Baseline score, as a percentage of the Baseline score.

Learners from all three grades witnessed improvement of outcome in End line over Baseline levels. Learners in Grade 6 improved scores by 7.79% over Baseline levels. Grade 7 witnessed improvement of 4.14%. Grade 8 had improvement of 10.55% over Baseline levels.

OUTCOMES MAPPED TO THE CEFR

As stated under the section ‘Assessment Instruments’, the test instruments for all three grades comprised mainly of A1- and A2-level questions. Learners’ proficiency at each of these levels was examined and learners were categorized as being at A1 or A2 level. Additionally, depending upon the proficiency at each of these levels, proficiency was further classified as being at A1.1 or A1.2 (or, A2.1 or A2.2, as the case may be) levels. The number of learners and hence the distribution of learners at each of these levels was examined. Baseline distribution of learners’ levels was then examined with End line distribution.

Almost 82% of learners were at the CEFR level A1.1 in the Baseline. By the End line, only a little over 75% remained at this level; learners were seen to have transitioned to a higher CEFR level of A1.2.

The difference in the movement at the A2.1 level is not statistically significant. This advantage in learning has been reached despite learners’ exposure to fewer than the recommended 75 sessions of ReadToMe.

CONCLUSION

The impact analysis conducted among learners of Grades 6, 7, and 8 in the five pilot schools in Uttar Pradesh has indicated that improvement in learning outcomes can be gained through deployment of ReadToMe. However, given that the number of sessions fall quite short of the recommended number and that the program was implemented for fewer than 7 weeks, it is recommended that the positive impact be validated on a larger scale in the state of Uttar Pradesh.

Large-scale assessments of scaled deployments have proven the positive impact of ReadToMe deployed under the RightToRead program. There is evidence of a strong correlation between usage of ReadToMe and improvement in outcomes, proven over time. Based on this proof of concept, learners from the schools of the state of Uttar Pradesh can derive the benefit of RightToRead through a state-wide deployment of the program.

ACKNOWLEDGEMENTS