Impact Evaluation Of ReadToMe® Among Students in Mo Schools, Odisha

Background

ReadToMe® is a multi-sensory reading and comprehension software that helps enhance students’ reading skills and increases teacher effectiveness.

The Mo School Abhiyan Parichalana Sangathan, a Government of Odisha Initiative under the School & Mass Education Department has commissioned EnglishHelper to deploy ReadToMe® Solutions in 30 schools for Grade 9. As part of the project, Baseline-End line assessments were conducted among the students of a sample of 2 schools to measure the impact of the program. Baseline assessments were conducted on October 18 and 19, 2022; End line assessments were conducted at the end of the three-month pilot on January 11 and 12, 2023.

Note: Initially, 3 schools were chosen for assessments. However, by the End line, one school conducted fewer than 15 sessions due to the following constraints:

– Computer lab availability

– Large class sizes, thus combining sections was not feasible

– Less than 20 class days during the period between the Baseline and End line.

In the two schools where the Baseline-End line were conducted, due to limited IT infrastructure capacity, some Grade 9 sections were combined to conduct ReadToMe®-enabled classes. These combined sections often had more than 60 students per ReadToMe®-enabled class.

For the purposes of evaluation of improvement, the set of students who were present in both the Baseline and the End line assessments has been considered (n=156). Presented below are the number of ReadToMe®-enabled classes conducted in the schools and the students in the assessment sample:

Assessment Instruments

The assessment instruments test the constructs of Word Structure, Vocabulary, Sentence Sequencing, Reading Comprehension, and Sentence Structure. The assessments comprise Multiple Choice Questions (MCQs) with only a single correct response. The assessments are designed based on a standardized rubric and are mapped to the Common European Framework of Reference for Languages (CEFR). The questions in the assessments range from the CEFR levels A1 to C1.

The CEFR describes language proficiency on a six-level scale:

– A1-A2 for Basic User

– B1-B2 for Independent User

– C1-C2 for Proficient User

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do”

statements. The framework for Overall Reading Comprehension used to design the instruments is as follows:

The English Profile Project (www.englishprofile.org) funded by Cambridge University Press and Cambridge English Language Assessment, among others, has compiled a list of words with their associated CEFR levels and a list of grammatical forms that are used by students at various CEFR levels. These are called the English Vocabulary Profile (http://www.englishprofile.org/wordlists) and the English Grammar Profile (http://www.englishprofile.org/english-grammar-profile), respectively. These have been used as the fundamental guiding principles when assigning a CEFR level to questions that satisfy the Word Structure and Vocabulary constructs in the assessment instruments.

All questions in the assessment instruments were assigned a CEFR level using the “Can do” statements presented in Table 1.

Assessment Outcome

The assessment instruments were composed of dichotomous items. Students’ responses to every question were scored as 1 or 0, depending upon whether the student answered the question correctly or not. No partial credits were allowed. The sum of the students’ responses to all questions in the test was scaled to 100, providing the students’ scores. The arithmetic mean of these scores is presented as the Mean Score of the cohort.

Improvement in Mean Score

A mean Score of 62% was achieved across all the students from the two assessment schools in the End line. This is significantly higher than the mean Score of 39% achieved in the Baseline. Students have exhibited a significant improvement of 55% in scores.

Significant improvement in Mean Score was observed among students in each school, despite the high-density classes due to combining multiple sections.

Multiple factors are expected to have contributed to the high improvement of the students within the short duration of the pilot (3 months):

– To derive significant benefit from ReadToMe®, the assessment schools conducted the requisite number of classes.

– These classes were conducted at the required regularity within the three months, allowing for concerted learning gains.

These results validate outcomes consistently observed from large-scale, multi-regional studies that prove ReadToMe-enabled classes lead to significant learning gains.^[1,2]

Improvement in CEFR Proficiency

Students of Grade 9 are required to be proficient at CEFR level B1-B2 to successfully read and comprehend their textbooks. In the Baseline, 85% of the students were below CEFR level B1. By the End line, a significant number of students bridged the proficiency gap. 58% of the students were at CEFR level B1, compared to 15% in the Baseline.

The graph below shows the distribution of students across the CEFR levels.

Improvement Across Domains

Baseline scores highlighted that students particularly need help with Reading and Comprehension, Sentence Structure, and Sentence Sequencing, especially with interpreting short texts. By the End line, students improved significantly across all domains tested, especially in Reading and Comprehension which was the lowest scoring domain in the Baseline.

Conclusions and Recommendations

– Students exhibited a significant improvement of 55% in learning outcomes after exposure to ReadToMe- enabled classes.

– A significantly high number of students (58%) were at the desired grade-level proficiency of CEFR B1 in the End line compared to 15% in the Baseline.

– Students improved across all domains of English and especially recorded marked improvement in reading and comprehension.

– It is recommended that the schools continue to conduct ReadToMe-enabled classes and also consider expansion to other grades.

– Introduction of ReadToMe® in earlier grades will enable students to bridge proficiency gaps and accelerate the acquisition of advanced English proficiency.³

---

^1. Improving reading and comprehension in K-12: Evidence from a large-scale AI technology intervention in India https://www.sciencedirect.com/science/article/pii/S2666920X21000138

^2. ReadToMe® – a Study of Impact of Multi-Sensory Reading and Comprehension on Learning Outcomes – a Consolidated Analysis https://www.englishhelper.com/resources/research-center/2018/05/24/readtome-a-study-of-impact-of-multi-sensory-reading-and-comprehension-on- learning-outcomes-a-consolidated-analysis/

^3. Impact Analysis of ReadToMe in South Delhi Municipal Corporation Pilot Schools | 2017 – 2018 https://www.englishhelper.com/resources/research-center/2018/05/29/impact-analysis-of-readtome-in-south-delhi-municipal-corporation-pilot-schools- 2017-2018/

---

For more information, please visit https://www.englishhelper.com/resources/research-center/