Background

EnglishHelper provides technology-based reading and comprehension platforms for students across all age groups. Since launching in India in 2011, EnglishHelper has successfully implemented its multi-sensory reading and comprehension solution, ReadToMe®, in public and private schools across the country.

The RightToRead Program is an effort to demonstrate that reading and comprehension technology can play a key role in solving the country’s reading crisis.

Supported by the United States Agency for International Development (USAID) under the USAID–IPP initiative, EnglishHelper deployed RightToRead in government and aided schools during the academic years 2016 to 2018. This large-scale rollout of RightToRead covered over 1 M students in grades 3 to 8 and touched 15,000 teachers in 5,000 schools across 8 States – Punjab, Delhi, Gujarat, Maharashtra, West Bengal, Tamil Nadu, Telangana and Karnataka. EnglishHelper implemented RightToRead in the states of Maharashtra and West Bengal in partnership with IL&FS Education.

In partnership with Skill Training Assessment Management Partners (STAMP) and IL&FS Education, EnglishHelper conducted impact assessments spanning two academic years among students of Maharashtra (Grades 5-6 and Grade 3) and West Bengal (Grades 5-6). The assessments were implemented by IL&FS Education. STAMP provided the technology platform for conducting the assessments.

This document presents the details and outcomes of these assessments.

About RightToRead                                                                

The RightToRead program was launched by EnglishHelper in 2013. Under this program EnglishHelper deploys its multisensory technology platform for reading and comprehension, ReadToMe®, in government and government-aided schools.

The goal of RightToRead is to demonstrate that reading and comprehension technology when integrated with the given school curriculum can make a material difference in literacy. Working on the tenets of Minimum Change and Sustainability, the following operating model is implemented:

• The class textbook and prescribed syllabus is digitised and made available on the reading and comprehension software, ReadToMe®.

• ReadToMe® is integrated into the school timetable with the regular English class period. Teachers and students are not required to devote additional time for ReadToMe® enabled English classes.

• The existing teachers are empowered to use ReadToMe® in class.

Assessment Design

To assess the impact of RightToRead, longitudinal Baseline, Midline and End line assessments were administered in a sample of the schools where RightToRead was deployed. In Maharashtra, the Baseline, Midline and End line assessments were conducted in October 2016, March 2017 and March 2018, respectively. For Grade 3, the Midline and End line were conducted among the same cohort of students – students graduating from Grade 3 in 2018. The Baseline was conducted among the previous cohort, i.e. students graduating from Grade 3 in 2017. Grade 5-6 assessments followed the same cohort of students across the two years. The West Bengal Grades 5-6 assessments also followed the same cohort of students in January 2016, December 2016 and December 2017 as the academic year in West Bengal follows the calendar year.

The assessment followed a standardised rubric measuring the constructs of letter recognition, word recognition, vocabulary, and reading and comprehension. The assessments weres also mapped to the CEFR and outcomes were also analysed based on the CEFR levels achieved by the students.

A Treatment-Control design approach was adopted for Grades 5-6 across both states, allowing for measurement of a difference-in-difference of scores, thus delineating improvement attributable to the RightToRead program. (The improvement in scores of the Treatment group are compared with the improvement in scores of the Control group, to understand the change in outcome that is brought about through RightToRead as different from the change in outcome that could ordinarily be expected over the two years of measurement).

Presented below is the total (End line) count of students in each cohort.

	Treatment	Control
Maharashtra – Grade 3	745	–
Maharashtra – Grade 5-6	5,399	599
West Bengal – Grade 5-6	2,114	361

Assessment Methodology

Assessments were carried out using ‘tablets’ at all schools to ensure unbiased delivery of the tests. On completion of the assessments at each school, the student submissions were available on the assessment app as a ‘read only – protected file’ which were uploaded to the cloud data-warehouse of the assessment partner, STAMP (Skill Training Assessment Management Partners Ltd). Each file was uniquely identified by school name and school code. Subsequently, STAMP extracted the data from these files, processed it on their proprietary assessment engine and shared outcomes with EnglishHelper. The delivery of the assessments in schools, processing of outcomes and reporting is the combined effort of STAMP and EnglishHelper. For Grade 3, assessments were carried out using pencil and paper; student submissions were captured on paper and transferred to a spreadsheet. All subsequent steps followed the same procedure as with tablets.

Assessment Outcome

The test instruments were composed of dichotomous items. Students’ responses to every question were scored as 1 or 0, depending upon whether the student answered the question correctly or not. No partial credits were allowed. The sum of students’ responses to all questions in the test was scaled to 100; providing students’ score. Scores from the entire sample were aggregated and the averages analysed. These are presented in Figures 1 and 2. (For the purpose of conciseness, in the figures, Grade 5-6 has been referred to as Grade 6.)

The Treatment group shows remarkable increases from Baseline scores. Increases in the Control group are less remarkable with Maharashtra Grade 6 showing a decline in scores in the Midline.

The metric “Improvement” is measured as the difference of the Midline / End line score from the Baseline score, as a percentage of the Baseline score. 1-year Improvement is the improvement of the Midline; 2-year Improvement is the change of the End line score over Baseline score.

As an extension of the observation on Mean scores, improvements in the Treatment group are consistently significant. 1-year improvements in the Treatment group are high (and higher than the corresponding Control group); the improvement is sustained over the 2-year period of assessment. The difference-in-difference from corresponding Control ranges from 11% to 22%. These findings validate that the use of ReadToMe® allows for significant improvement over Control which is sustainable over a two-year period.

The CEFR Analysis

The assessment instrument has been mapped to the CEFR (Common European Framework of Reference for Languages) scale. The CEFR is a common framework to grade proficiency and enables like-to-like comparison across student groups.

The questions in the assessment instruments range from the CEFR levels A1.1 to B1. (The CEFR also allows for branching and defining sub-competencies, such as A1.1 and A1.2.) Since there are very few B1 questions in the assessment, the A2 and B1 levels have been categorised as A2 and above. The overall adapted framework for the assessments reads as follows:

The English Profile Project (www.englishprofile.org) funded by Cambridge University Press and Cambridge English Language Assessment, among others, has compiled a list of words with their associated CEFR levels and a list of grammatical forms that are used by students at various CEFR levels. These are called the English Vocabulary Profile (http://www.englishprofile.org/wordlists) and the English Grammar Profile (http://www.englishprofile.org/english-grammar-profile), respectively. These have been used as the fundamental guiding principles when assigning a CEFR level to questions that satisfy the Letter Recognition, Word Recognition and Vocabulary constructs in the assessment instrument.

All Reading Comprehension questions in the assessment instruments were assigned a CEFR level using the “Can do” statements presented in Table 2.

The figure below presents the percentage of students who are at the CEFR levels A2 and above at each assessment phase for all cohorts of students (Treatment and Control).

The Treatment group shows consistent and significant increases in the percentage of students at CEFR level A2 and above, in the Midline and the End line. Increases in the Control group are not as remarkable and there is significantly less incremental gain in proficiency of the students at a cohort level.

The table below shows the distribution of the students across CEFR levels for each cohort in the Baseline, Midline and End line. Apart from observations and findings of the Mean score, which prove trends on an average level, the distribution help identify movements of smaller magnitude within the different proficiency levels.

In all three cohorts of the Treatment group, we observe a reversal of distribution of the proficiency levels of students by the End line: the percentage of students in the higher CEFR level – A2 and above – is comparable to or higher than the percentage of students in the lowest CEFR level – A1.1. This is due to significantly higher percentages of students shifting to higher proficiency levels in each academic year. The shift to the higher CEFR levels is sustained over the two-year period of assessment. In comparison, the Control group shows only limited movement. This bears out that the use of ReadToMe® brings about sustained improvement over proficiency levels of students across cohorts, over a two-year period.

Conclusion

Longitudinal assessments for the RightToRead program conducted across three cohorts, Grade 3 of Maharashtra, Grades 5-6 of Maharashtra and Grades 5-6 of West Bengal, were analysed to evaluate the impact of ReadToMe® across cohorts (grades and states). These were designed as Baseline- Midline-End line studies.

The analysis validates that ReadToMe® has positive impact on English reading and comprehension outcomes of students belonging to different cohorts. Proficiency improved significantly and in comparison with Control. These improvements sustain over the two-years measured.

The CEFR analysis also revealed that increasing numbers of students achieved higher proficiency levels after undergoing the RightToRead program. A significantly higher percentage of students were at levels A1.2 and above by the End line. Control group reflected a much lower shift to the A1.2 and above CEFR levels.

In conclusion, the analysis validates that use of the ReadToMe® platform has a significantly positive impact on English reading and comprehension of students of different states and grades undergoing the RightToRead program, measured over a two-year period.