BACKGROUND

The South Delhi Municipal Corporation (SDMC) piloted ReadToMe in 20 schools in SDMC in the academic year 2017 – 18. EnglishHelper conducted assessments in a sample of 7 schools designed as Baseline – End line studies. This will allow measurement of learning outcomes achieved in one academic year among schools using ReadToMe.

The assessments were conducted across grades 3 to 5. A Control-Treatment design was adopted for comparison of outcomes. Learners who underwent technology-enabled reading under the RightToRead program through ReadToMe were classified as the Treatment group. Learners who were not exposed to technology-enabled English learning constituted the Control group. The design allows analysis of outcomes attributable mainly to the program, compared with learning that may be observed in a defined academic period in the absence of the program.

The Baseline Assessments were conducted in August 2017. End line assessments were conducted in February/March 2018. Some observations that are important to note here are:

a. Typically, Baseline – End line assessments are conducted at the beginning and at the end of the academic year allowing for at least 6 to 9 months to gauge impact. However, in the case of the SDMC schools, the assessments were separated by just about 6 months. It is recommended that the program be conducted for the entire academic year to obtain full benefits; in SDMC the program was implemented for only 22 weeks, on average.

b. Additionally, EnglishHelper recommends at least 75 sessions of ReadToMe in an academic year. The schools in the pilot in SDMC conducted, on an average, 56 sessions (75% of the recommended sessions) in the academic year 2017-18. The results of the assessments have been interpreted in light of these constraints.

Some of the key questions that were sought to be answered for an evaluation of the pilot are:

• • Does the RightToRead program have an impact on the learning outcomes of English learners?
• • Does the program impact learners from Hindi and/or English medium of instruction?
• • Does the program impact learners at all levels of proficiency?
• • Can the program impact learners from different grades?

ABOUT RightToRead

EnglishHelper enables technology-based reading and comprehension improvement for learners across all age groups. Since launching in India in 2011, EnglishHelper has successfully implemented its reading and comprehension solution, ReadToMe, in public and private schools across the country.

The RightToRead program was launched by EnglishHelper in 2013. Under this program, EnglishHelper deploys its multi-sensory technology platform for reading and comprehension called ReadToMe in government and aided schools.

The goal of RightToRead is to demonstrate that reading and comprehension technology when integrated with the given school curriculum can make a material difference in literacy. Working on the tenets of Minimum Change and Sustainability, the following operating model is implemented:

• • The class textbook and state-prescribed syllabus is digitized and made available on the reading software. No additional or new study material is introduced to the students.
• • The software is integrated into the school timetable with the regular English class period. Teachers and students are not required to devote additional time for ReadToMe classes.
• • The existing teachers are trained and empowered to use ReadToMe.

SAMPLE DESIGN

Assessments were undertaken among seven schools enrolled for the pilot and two schools where the program was not undertaken (7 Treatment and 2 Control schools). Learners enrolled in the Hindi medium and in the English medium were sampled for the assessment. A total of 3939 tests were administered between the Baseline and End line across the three grades. The tables below present the distribution of the students by grade, medium of instruction, and gender.

Adequate sample was achieved by medium of instruction and by gender in each of the Baseline and End line, to allow for meaningful comparisons of scores.

ASSESSMENT METHODOLOGY

Assessments were carried out using ‘tablets’ at all nine schools to ensure unbiased delivery of the assessments. On completion of the assessments at each school, the learner submissions were available on the assessment app as a ‘read only – protected file’ which were uploaded by the field personnel. Each file was uniquely identified by school name and school code. Subsequently, STAMP (Skill Training Assessment Management Partners Ltd) – the assessment partner – extracted the data from these files, processed it on their proprietary assessment engine and shared outcomes with EnglishHelper. The delivery of the assessments in schools, processing of outcomes and reporting is the combined effort of STAMP and EnglishHelper.

ASSESSMENT INSTRUMENTS

The assessment instruments were designed to test the constructs of Word Structure, Vocabulary, and Reading Comprehension. In addition, learners of Grade 3 were also tested on Alphabet recognition. The instruments encompassed single-response, close-ended items presented as multiple-choice responses with a single correct option. The instrument for each grade has a high internal consistency with high-reliability measures (Cronbach’s alpha > 0.7).

The assessment instruments were mapped to the Common European Framework of Reference (CEFR) scale. The CEFR allows the definition of proficiency such that like-to-like comparisons of outcomes can be made across grades, states, cohorts, and any learner group. The CEFR describes language proficiency (related to listening, speaking, reading, and writing) on a six-level scale:

• • A1-A2 for Basic User
• • B1-B2 for Independent User
• • C1-C2 for Proficient User

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do” statements. These are presented below:

It also allows for branching and defining sub-competencies, such as A1.1 and A1.2. The questions in the assessment instruments range from the CEFR levels A1 to B1. A low proportion of B1 questions was included and these have been classified with A2 questions under the heading, “A2 and higher.”

The composition of questions in the assessment instruments is as follows:

ASSESSMENT OUTCOME

The test instruments were composed of dichotomous items. Learners’ responses to every question was scored as 1 or 0, depending upon whether the learner answered the question correctly or not. No partial credits were allowed. The sum of the learners’ responses to all questions in the test was scaled to 100; providing the learners’ score. Scores from the entire sample were aggregated and the averages analyzed. (Scores were not analyzed at the individual school level owing to low population and sample sizes). We examine the outcomes for every grade separately. The metric “Improvement” is measured as the difference of the End line score from the Baseline score, as a percentage of the Baseline score.

GRADE 3 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores obtained by the Treatment group and the Control group to understand the improvement (if present) exhibited by the learners that can be attributed to ReadToMe, overall, and for each medium of instruction. Further, we perform similar comparisons among the subsets of females and males.

Examination of Mean Scores

In the graph that follows, the mean score obtained by the Control group (the two bars on the left) is compared with the mean score obtained by the Treatment group (the two bars on the right). As mentioned earlier, mean scores have been scaled to 100%. The improvement of the relevant group from the Baseline to the End line has also been presented. A negative sign (-) preceding the improvement number indicates a lower mean score in the End line compared to the Baseline.

Learners in the Treatment group have improved by 53.71% over their Baseline scores. The Control group shows a decline in scores by 3.78%. The increase in score for the Treatment group is statistically significant at the 95% Confidence Level (two-tailed). This indicates that ReadToMe has a positive impact on learners’ reading and comprehension ability.

The Treatment and Control groups in the sample were also matched on the medium of instruction for learners. In the graph that follows we examine the mean scores of learners from each of the media of instruction. This helps us delineate impact (where present) in each sub-group of learners.

The two pairs of bars on the left present scores obtained by learners who use Hindi as the main medium of learning and hence learn English as a second language. Again, the comparison of the Treatment group with the Control group helps us isolate the impact of ReadToMe on learners’ proficiencies. The two pairs of bars on the right are for groups using English as the medium of learning.

As seen from the graph above, learners from both Hindi and English media of instruction have benefited from ReadToMe: Treatment group score for Hindi medium has improved by 56.61% while that of the corresponding Control group has reduced by almost 5%; score for the English medium Treatment group improved by 43.33% while that of the corresponding Control group saw a reduction by over 3%. None of the differences in the Control group was statistically significant while all the differences in the Treatment group are significant at a 95% Confidence Level (two-tailed test).

The improvement is poignant given that the Baseline scores in both sets of Treatment groups were lower than those of the corresponding matched Control groups. This indicates that ReadToMe has a positive impact on learning levels, irrespective of the medium of instruction (Hindi or English).

In the Control group, we see that scores of males have dropped and that of females has improved marginally. Change in scores in the Control group were not statistically significant (at 95% Confidence Level – two-tailed). Females and males from the Treatment group, on the other hand, showed marked improvement with the increase in scores being statistically significant (at 95% Confidence Level – two-tailed). Thus ReadToMe was seen to impact learners from both cohorts, significantly.

Examination of Achievement by Quartile

A quartile analysis rank-orders learner based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

Quartile bounds in the End line for the Treatment group in Grade 3 have consistently increased from the Baseline levels, despite having started at the same level as or lower than the Control levels. In addition, we see an improvement in the minimum and maximum scores. Thus, learners at every learning level are positively impacted by the program.

GRADE 4 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores obtained by the Treatment group and the Control group to understand the improvement (if present) exhibited by the learners that can be attributed to ReadToMe, overall, and for each medium of instruction. Further, we perform similar comparisons among the subsets of females and males.

Examination of Mean Scores

In the graph that follows, the mean score obtained by the Control group (the two bars on the left) is compared with the mean score obtained by the Treatment group (the two bars on the right). As mentioned earlier, mean scores have been scaled to 100%. The improvement of the relevant group from Baseline to the End line has also been presented. A negative sign (-) preceding the improvement number indicates a lower mean score in the End line compared to the Baseline.

Learners in the Treatment group have improved by 45.47% over their Baseline scores. The Control group shows a decline in scores by 1.39%. The increase in score for the Treatment group is statistically significant at the 95% Confidence Level (two-tailed). This indicates that ReadToMe has a positive impact on learners’ reading and comprehension ability.

The Treatment and Control groups in the sample were also matched on medium of instruction for learners. In the graph that follows we examine the mean scores of learners from each of the media of instruction. This helps us delineate impact (where present) in each sub-group of learners.

The two pairs of bars on the left present scores obtained by learners who use Hindi as the main medium of learning and hence learn English as a second language. Again, comparison of the Treatment group with the Control group helps us isolate the impact of ReadToMe on learners’ proficiencies. The two pairs of bars on the right are for groups using English as the medium of learning.

As seen from the graph above, learners from both Hindi and English media of instruction have benefitted from ReadToMe: Treatment group score for Hindi medium has improved by 49.35% while that of the corresponding Control group has reduced by more than 7%; score for the English medium Treatment group improved by 34.75% while that of the corresponding Control group saw improvement of 24.31%. (Treatment group exhibited an additional 10.44% improvement). Except for the Control group undergoing instruction in Hindi, all the differences are statistically significant at a 95% Confidence Level (two-tailed test).

This indicates that ReadToMe has a positive impact on learning levels, irrespective of the medium of instruction (Hindi or English). Improvement in the Control group is seen only among English medium learners.

In the Control group, we see that scores of males have dropped and that of females has improved significantly (at 95% Confidence Level – two-tailed). The improvement in the Control group is hence concentrated on one cohort of learners: females. Females and males from the Treatment group, on the other hand, showed marked improvement with the increase in scores being statistically significant (at 95% Confidence Level – two-tailed). Thus ReadToMe was seen to impact learners from both cohorts, significantly.

Examination of Achievement by Quartile

A quartile analysis rank-orders learner based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

Quartile bounds (other than the Minimum) in the End line for the Treatment group in Grade 4 have consistently increased from the Baseline levels. Thus, learners at every learning level are positively impacted by the program. The Control group exhibits a plateauing or a decline of bounds (except the Minimum where it shows an increase).

GRADE 5 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores obtained by the Treatment group and the Control group to understand the improvement (if present) exhibited by the learners that can be attributed to ReadToMe, overall, and for each medium of instruction. Further, we perform similar comparisons among the subsets of females and males.

Examination of Mean Scores

In the graph that follows, the mean score obtained by the Control group (the two bars on the left) is compared with the mean score obtained by the Treatment group (the two bars on the right). As mentioned earlier, mean scores have been scaled to 100%. The improvement of the relevant group from Baseline to the End line has also been presented. A negative sign (-) preceding the improvement number indicates a lower mean score in the End line compared to the Baseline.

Learners in the Treatment group have improved by 30.41% over their Baseline scores. The Control group shows a slight decline in scores. The increase in score for the Treatment group is statistically significant at the 95% Confidence Level (two-tailed). This indicates that ReadToMe has a positive impact on learners’ reading and comprehension ability.The Treatment and Control groups in the sample were also matched on medium of instruction for learners. In the graph that follows we examine the mean scores of learners from each of the media of instruction. This helps us delineate impact (where present) in each sub-group of learners.

The two pairs of bars on the left present scores obtained by learners who use Hindi as the main medium of learning and hence learn English as a second language. Again, comparison of the Treatment group with the Control group helps us isolate the impact of ReadToMe on learners’ proficiencies. The two pairs of bars on the right are for groups using English as the medium of learning.

As seen from the graph above, learners from both Hindi and English media of instruction have benefitted from ReadToMe: Treatment group score for Hindi medium has improved by 34.21% while that of the corresponding Control group has reduced by nearly 7%; score for the English medium Treatment group improved by 22.29% while that of the corresponding Control group saw improvement of 17.17%. (Treatment group exhibited an additional 5.12% improvement). All the differences are statistically significant at a 95% Confidence Level (two-tailed test). Thus the Control group exhibits non-uniform improvement in learning ability while ReadToMe has a positive impact on learning levels, irrespective of the medium of instruction (Hindi or English).

The improvement is poignant given that the Baseline scores in both sets of Treatment group were lower than those of the corresponding matched Control groups.

In the Control group, we see that scores of males have dropped and that of females has improved marginally. Neither change in score is statistically significant (at 95% Confidence Level – two-tailed). Females and males from the Treatment group, on the other hand, showed marked improvement with the increase in scores being statistically significant (at 95% Confidence Level – two-tailed). Thus, ReadToMe was seen to impact learners from both cohorts, significantly.

Examination of Achievement by Quartile

A quartile analysis rank-orders learner based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

OUTCOMES MAPPED TO THE CEFR

As stated under the Section ‘Assessment Instruments’, the test instruments for all three grades comprised mainly of A1- and A2-level questions. Learners’ proficiency at each of these levels was examined and learners were categorized as being at A1 or A2 level. Additionally, depending upon the proficiency at each of these levels, proficiency was further classified as being at A1.1 or A1.2 (or, A2.1 or A2.2, as the case may be) levels. The number of learners and hence the distribution of learners at each of these levels was examined. Baseline distribution of learners’ levels was then examined with End line distribution.

The Treatment group saw a movement in distribution of learners from the lower CEFR level of A1.1 to the higher CEFR levels assessed. Just over 82% of learners were at the CEFR level A1.1 in the Baseline. By the End line, fewer than 48% remained at this level; learners were seen to have transitioned to higher CEFR levels. This is a statistically significant movement. Statistically higher numbers of learners were seen to have transitioned to each of the A1.2, A2.1 and A2.2 levels in the End line compared to the Baseline. This indicates an improvement in learners’ proficiency at and to all levels upon use of the program. (Wherever statistically significant at 95% Confidence Level, the higher cell has been marked with an *. Comparisons have not been made in those pairs where one of the columns has 0%).

As the table above depicts, significant increase in proficiency is observed in the distribution among learners from both Hindi as well as English media of instruction. It is interesting to note that the End line distribution of learners from the Hindi medium indicates that these learners are at a higher proficiency than were the learners from the English medium at the Baseline.

This pattern of responses was observed across grades, with Grade 5 showing most proficiency compared to the other grades. The table below presents the distribution of learners across CEFR levels for each of the grades. Baseline and End line percentages for each level have been compared within each grade. Wherever statistically significant at 95% Confidence Level, the higher cell has been marked with an *. Comparisons have not been made in those pairs where one of the columns has 0%.

The tables above validate that there is increase in proficiency across grades and across the two media of instruction within each grade. Let us examine the above three tables in more detail. In Table 14, let us compare the percentage of learners in Grade 3 who are at A1.1 CEFR level at the End line with the percentage of learners from Grade 4 who were at A1.1 at the Baseline. We find that compared to the Baseline of Grade 4, a significantly lower percentage of students from Grade 3 are at the A1.1 level by the End line. We can hence conclude that the learners of Grade 3 in the Treatment group have surpassed their counterparts from the higher grade when they entered Grade 4. Such a conclusion can be drawn between every grade. This also holds true when we compare within a medium and across media of instruction. In fact, learners who learn English as the second language (Hindi medium of instruction) are at higher proficiency levels than when their English-medium higher grade counterparts entered the next grade (Baseline levels of higher grade).

Let us now examine these results in light of the timing of the assessments. The Baseline assessments were conducted in October while the End line assessments were conducted in March of the academic year. This indicates that learners in Grade 3 in March are at higher proficiency levels than the previous batch was in Grade 4 in October. Considering that there are between 5 and 6 academic months separating March from October in the academic year, we can conclude that learners in Grade 3 are already ahead of the previous batch by more than 5 to 6 academic months. This advantage in learning has been reached despite learners’ exposure to fewer than the recommended 75 sessions of ReadToMe.

A similar conclusion can be drawn when comparing the proficiencies of Grade 4 at the End line with the proficiencies of Grade 5 at the Baseline.

Thus, it can be concluded that ReadToMe has a positive impact on the level of proficiency as well as the time it takes to help learners reach a pre-determined proficiency level.

CONCLUSION

The impact analysis conducted among learners of Grades 3, 4 and 5 in the nine pilot schools in South Delhi Municipal Corporation has validated that:

• • Use of ReadToMe significantly improves outcomes in reading and comprehension proficiency
  • o While Control group scores have declined from Baseline to End line, Treatment groups exhibits improvement of between 30% and 54%
  • o Treatment group exhibits between 30% and 50% more improvement than Control Group
• • Improvement in outcomes is derived by learners at all learning levels and across grades
• • Improvement was evidenced among both female and male learners across all the three grades
• • Improvement in outcome also positively impacts the CEFR proficiency of learners
• • Use of ReadToMe can reduce the time to attain proficiency levels
• • Improvement in outcomes has been obtained despite the fact that 75% of the recommended number of sessions of ReadToMe having been conducted

In conclusion, the assessments validate that the use of the ReadToMe platform has a positive impact on English reading and comprehension of learners. Large-scale assessments of scaled deployments have also proven the positive impact of ReadToMe deployed under the RightToRead program. There is evidence of a strong correlation between usage of ReadToMe and improvement in outcomes, proven over time. Based on this proof of concept, learners from the schools of the South Delhi Municipal Corporation can derive the benefit of RightToRead through a wider deployment of the program.

ACKNOWLEDGEMENTS