BACKGROUND

IL&FS Education piloted ReadToMe in 11 schools in Telangana in the academic year 2017 – 18. EnglishHelper conducted assessments designed as Baseline – End line studies, in 9 of these schools. This will allow measurement of learning outcomes achieved in one academic year among schools using ReadToMe.

The assessments were conducted among students of grades 6, 7, and 8. The Baseline Assessments were conducted in October 2017. End line assessments were conducted in February 2018. Some observations that are important to note here are:

a. Typically, Baseline – End line assessments are conducted at the beginning and at the end of the academic year allowing for at least 6 to 9 months to gauge impact. However, in the case of Telangana, the assessments were separated by 4 months. It is recommended that the program be conducted for the entire academic year to obtain full benefits; in Telangana the program was implemented for only 14 weeks.

b. Additionally, EnglishHelper recommends at least 75 sessions of ReadToMe in an academic year. The schools in the pilot in Telangana conducted, on an average, 32 sessions (42% of the recommended sessions) in the academic year 2017-18. The results of the assessments have been interpreted in light of these constraints.

Some of the key questions that were sought to be answered for an evaluation of the pilot are:

• • Does the RightToRead program have an impact on the learning outcomes of English learners?
• • Does the program impact learners at all levels of proficiency?
• • Can the program impact learners from different grades?

ABOUT RIGHTTOREAD

EnglishHelper enables technology – based reading and comprehension improvement for learners across all age groups. Since launching in India in 2011, EnglishHelper has successfully implemented its reading and comprehension solution, ReadToMe, in public and private schools across the country.

The RightToRead program was launched by EnglishHelper in 2013 based on its multi-sensory technology platform for reading and comprehension called ReadToMe.

The goal of RightToRead is to demonstrate that reading and comprehension technology when integrated with the given school curriculum can make a material difference in literacy. Working on the tenets of Minimum Change and Sustainability, the following actions are practiced:

• • The class text book and state prescribed syllabus is digitized and made available on the reading software. No additional or new study material is introduced to the students.
• • The software is integrated into the school time-table with the regular English class period. Students are not required to devote additional time for ReadToMe classes.
• • The existing teachers are trained and empowered to use ReadToMe.

SAMPLE DESIGN

Assessments were undertaken among nine schools enrolled for the pilot. (Implementation details for two of the eleven schools were still not finalized during assessment planning and hence, they were not included in the assessments). Learners enrolled in Telugu medium were sampled for the assessment. A total of 1261 tests were administered between Baseline and End line across the three grades. The tables below present the distribution of the students by grade and and gender.

Adequate sample was achieved by gender in each of the Baseline and End line, to allow for meaningful comparisons of scores.

ASSESSMENT METHODOLOGY

Assessments were carried out using tablets at all nine schools by IL&FS personnel to ensure unbiased delivery of the assessments. On completion of the assessments at each school, the learner submissions were available on the assessment app as a ‘read only – protected file’ which were uploaded by the field personnel. Each file was uniquely identified by school name and school code. Subsequently, STAMP (Skill Training Assessment Management Partners Ltd.) extracted the data from these files, processed it on their proprietary assessment engine and shared outcomes with EnglishHelper. The delivery of the assessments in schools, processing of outcomes and reporting is the combined effort of STAMP and EnglishHelper.

ASSESSMENT INSTRUMENTS

The assessment instruments were designed to test the constructs of Word Structure, Vocabulary and Reading Comprehension. The instruments encompassed single-response close – ended items presented as multiple – choice responses with a single correct option. The instrument for each grade has a high internal consistency with higher than acceptable reliability measures (Cronbach’s alpha > 0.7).

The assessment instruments were mapped to the Common European Framework of Reference (CEFR) scale. The CEFR allows definition of proficiency such that like-to-like comparisons of outcomes can be made across grades, states, cohorts and any learner group. The CEFR describes language proficiency (related to listening, speaking, reading and writing) on a six-level scale:

• • A1-A2 for Basic User
• • B1-B2 for Independent User
• • C1-C2 for Proficient User

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do” statements. These are presented below:

It also allows for branching and defining sub – competencies, such as A1.1 and A1.2. The questions in the assessment instruments range from the CEFR levels A1 to B1. A low proportion of B1 questions was included and these have been classified with A2 questions under the heading, “A2 and higher.”

The composition of questions in the assessment instruments is as follows:

ASSESSMENT OUTCOME

The test instruments were composed of dichotomous items. Learners’ responses to every question was scored a 1 or 0, depending upon whether the learner answered the question correctly or not. No partial credits were allowed. The sum of the learner’s responses to all questions in the test was scaled to 100; providing the learners’ score. Scores from the entire sample were aggregated and the averages analysed. (Scores were not analysed at the individual school level owing to low population and sample sizes). We examine the outcomes for every grade separately. The metric “Improvement” is measured as the difference of the End line score from the Baseline score, as a percentage of the Baseline score.

GRADE 6 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores to understand the improvement exhibited by the learners. Further, we perform similar comparisons among the subsets of girls and boys.

Examination of Mean Scores

Overall, the learners have improved scores by 21.20% over Baseline levels. Boys exhibit higher improvement than girls; girls’ mean score remains slightly higher than that of boys. The mean score in the End line is statistically higher than that in the Baseline, tested at a two-tailed 95% Confidence Level. The End line mean scores of the subsets of girls and boys are also statistically higher than the corresponding Baseline mean scores.

Examination of Achievement by Quartile

A quartile analysis rank orders learners based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

Quartile bounds in the End line for Grade 6 have consistently increased from the Baseline levels. In addition, we see an improvement in the minimum and maximum scores. Thus, learners at every level are positively impacted by the program.

The graph below presents the distribution of learners’ scores in the Baseline and the End line. (The End line graph is inverted to enable a superimposed visualisation). We see a shift in the scores to the right (increasing scores on the X-axis). The distribution in the Baseline exhibits a tail on the left end of the X-axis (lower scores), while the End line distribution exhibits a right tail (higher scores). These are also indicators of positive impact of the program.

GRADE 7 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores to understand the improvement exhibited by the learners. Further, we perform similar comparisons among the subsets of girls and boys.

Examination of Mean Scores

Overall, the learners have improved scores by 10.08% over Baseline levels. Girls exhibit higher improvement than boys; girls’ mean score is higher than that of boys by the End line. The mean score in the End line is statistically higher than that in the Baseline, tested at a two-tailed 95% Confidence Level. The End line mean score of girls is also statistically higher than the corresponding Baseline mean scores; boys’ mean score has also increased and is directionally indicative.

Examination of Achievement by Quartile

A quartile analysis rank orders learners based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

Learners at or above the 50th Percentile in Grade 7 appear to have been most impacted by the program as exhibited in the increase in the Median and the 75th Percentile bound.

The graph below presents the distribution of learners’ scores in the Baseline and the End line. (The End line graph is inverted to enable a superimposed visualization). We see a shift in the scores to the right (increasing scores on the X-axis). This is indicative of positive impact of the program.

GRADE 8 OUTCOMES

In the graphs that follow, we first examine the mean scores achieved by learners in the Baseline and the End line. We then compare the scores to understand the improvement exhibited by the learners. Further, we perform similar comparisons among the subsets of girls and boys.

Overall, the learners have improved scores by 25.51% over Baseline levels. Girls exhibit higher improvement than boys. The mean score in the End line is statistically higher than that in the Baseline, tested at a two-tailed 95% Confidence Level. The End line mean scores of the subsets of girls and boys are also statistically higher than the corresponding Baseline mean scores.

Examination of Achievement by Quartile

A quartile analysis rank orders learners based on their scores, segmenting the learners into four groups or quartiles. The quartile limits are labelled as 25th Percentile, Median (50th Percentile), and 75th Percentile. The minimum score forms the lower bound of the lower quartile and the maximum score forms the upper bound of the upper quartile.

To understand the spread of improvement in students’ scores, we examine the change in value of the quartile bounds. When all quartile bounds increase in value, we can infer that improvement in learning outcome has been achieved at all levels, irrespective of the Baseline score of the learner. In such a case, the program can be considered to be effective at all levels.

Quartile bounds in the End line for Grade 8 have consistently increased from the Baseline levels. In addition, we see an improvement in the minimum and maximum scores. Thus, learners at every level are positively impacted by the program.

The graph below presents the distribution of learners’ scores in the Baseline and the End line. (The End line graph is inverted to enable a superimposed visualization). We see a marked shift in the scores to the right (increasing scores on the X-axis). This is indicative of positive impact of the program.

OUTCOMES MAPPED TO THE CEFR

As stated under the Section ‘Assessment Instruments’, the test instruments for all three grades comprised mainly of A1- and A2-level questions. Learners’ proficiency at each of these levels was examined and learners were categorized as being at A1 or A2 level. Additionally, depending upon the proficiency at each of these levels, proficiency was further classified as being at A1.1 or A1.2 (or, A2.1 or A2.2, as the case may be) levels. The number of learners and hence the distribution of learners at each of these levels was examined. Baseline distribution of learners’ levels was then examined with End line distribution.

Almost 70% of learners were at the CEFR level A1.1 in the Baseline. By the End line, only a little over 53% remained at this level; learners were seen to have transitioned to higher CEFR levels. This is a statistically significant movement. Statistically higher numbers of learners were seen to have transitioned to each of the A1.2, A2.1 and A2.2 levels in the End line compared to the Baseline. This indicates an improvement in learners’ proficiency at and to all levels upon use of the program.

This pattern of responses was observed across grades with Grade 8 showing most proficiency compared to the other grades. The table below presents the distribution of learners across CEFR levels for each of the grades. Baseline and End line percentages for each level have been compared within each grade. Wherever statistically significant at 95% Confidence Level, the higher cell has been marked with an *.

Let us examine the above table in more detail. Let us compare the percentage of learners in Grade 6 who are at A1.1 CEFR level at the End line with the percentage of learners from Grade 7 who were at A1.1 at the Baseline. While a higher percentage of learners are at the A1.1 level in the End line of Grade 6 compared to the Baseline of Grade 7, this difference is not statistically significant at the 95% Confidence Level. We can hence conclude that the percentage of learners at A1.1 level in Grade 6 End line is marginally higher than that in Grade 7 Baseline.

Let us now examine these results in light of the timing of the assessments. The Baseline assessments were conducted in October while the End line assessments were conducted in March of the academic year. This indicates that learners in Grade 6 in March are at similar levels as the previous batch was in Grade 7 in October. Considering that there are between 5 and 6 academic months separating March from October in the academic year, we can conclude that learners in Grade 6 are already ahead of the previous batch by 5 to 6 academic months. This advantage in learning has been reached despite learners’ exposure to fewer than the recommended 75 sessions of ReadToMe.

A similar conclusion can be drawn when comparing the proficiencies of Grade 7 at the End line with the proficiencies of Grade 8 at the Baseline.

Thus, it can be concluded that ReadToMe has a positive impact on the level of proficiency as well as the time it takes to help learners reach a pre-determined proficiency level.

CONCLUSION

The impact analysis conducted among learners of Grades 6, 7 and 8 in the nine pilot schools in Telangana have validated that:

• • Use of ReadToMe significantly improves outcomes in reading and comprehension proficiency
• • Improvement in outcomes is derived by learners at all learning levels and across grades
• • Improvement in outcome also positively impacts the CEFR proficiency of learners
• • Use of ReadToMe can reduce the time to attain proficiency levels
• • Improvement in outcomes has been obtained despite the fact that 42% of the recommended number of sessions of ReadToMe having been conducted

In conclusion, the assessments validate that the use of the ReadToMe platform has a positive impact on English reading and comprehension of learners. Large-scale assessments of scaled deployments have also proven the positive impact of ReadToMe deployed under the RightToRead program. There is evidence of a strong correlation between usage of ReadToMe and improvement in outcomes, proven over time. Based on this proof of concept, learners from the schools of the state of Telangana can derive the benefit of RightToRead through a state-wide deployment of the program.

ACKNOWLEDGEMENTS