Background

ReadToMe® is a multi-sensory Reading and Comprehension software which helps enhance student’s reading skills and increases teacher effectiveness.

The Delhi Government has piloted ReadToMe® Solutions in two schools. As part of the project, assessments were conducted among the students of these schools as Baseline-Endline studies to measure the impact of ReadToMe® on the proficiency of students. The Baseline assessments were conducted in the first week of March 2022 for grades 6, 7 and 8. These assessments were conducted at the end of the academic year, a month before the summer vacations. Endline assessments were conducted in the first week of August 2022 for the same set of students who are now in grades 7, 8 and 9, when schools re-opened right after the summer vacations. Throughout this document, the most recently completed grade has been mentioned (i.e. grades 6, 7 and 8 mentioned rather than the new respective grades 7, 8 and 9).

Typically, Baseline – Endline assessments are conducted at the beginning and at the end of the academic year after a full year’s intervention to gauge the impact of ReadToMe® on student learning outcomes. Additionally, EnglishHelper recommends at least 75 sessions of ReadToMe® in an academic year per class. However, in the case of Delhi, the pilot was designed to be of short duration. Consequently, over a period of 5 months (3 months excluding summer vacation), students were exposed to only 41% of the recommended number of ReadToMe®-enabled classes delivered over the course of an academic year. Despite the substantially lower exposure, students have demonstrated significant improvement as described later in the document.

The table below shows the sessions conducted in the two schools across grades.

School Name	Grade 6	Grade 7	Grade 8	Total
Sarvodaya Bal Vidyalaya, Rouse Avenue	38	27	49	114
Govt. Girls Sr. Sec. School, Sultanpuri	17	28	26	71
Total	55	55	75	185

Some of the key questions that were sought to be answered for evaluating the pilot are:

– Does ReadToMe® have an impact on the learning outcomes of students?
– Does the program impact students at all levels of proficiency?
– Can the program impact students from different grades?

Sample Design

The assessments were conducted among students from the section under the Hindi medium of instruction. The table below presents the distribution of students in each grade.

Grades	Number of Students in Baseline	Number of Students in Endline
Grade 6	77	72
Grade 7	65	67
Grade 8	74	67
Total	216	206

An adequate sample was achieved in each of the Baseline and Endline assessments to allow for meaningful comparisons of scores. 

Assessment Instruments

The assessment instruments test the constructs of Word Structure, Vocabulary and Reading Comprehension. In addition, Grade 8 students are assessed on Sentence Structure. The assessments comprise Multiple Choice Questions (MCQs) with only a single correct response. The assessments are mapped to the Common European Framework of Reference for Languages (CEFR). The questions in the assessments range from the CEFR levels A1 to C1. A low proportion of B1, B2, and C1 questions were included.

The CEFR describes language proficiency (related to listening, speaking, reading and writing) on a six-level scale: 

– A1-A2 for Basic User 
– B1-B2 for Independent User 
– C1-C2 for Proficient User 

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do” statements.  The framework for Overall Reading Comprehension used to design the instruments is as follows:

	OVERALL READING COMPREHENSION
C2	Can understand and interpret critically virtually all forms of the written language including abstract, structurally complex, or highly colloquial literary and non-literary writings.   Can understand a wide range of long and complex texts, appreciating subtle distinctions of style and implicit as well as explicit meaning.
C1	Can understand in detail lengthy, complex texts, whether or not they relate to his/her own area of speciality, provided he/she can reread difficult sections.
B2	Can read with a large degree of independence, adapting style and speed of reading to different texts and purposes, and using appropriate reference sources selectively. Has a broad active reading vocabulary, but may experience some difficulty with low-frequency idioms.
B1	Can read straightforward factual texts on subjects related to his/her field and interest with a satisfactory level of comprehension.
A2	Can understand short, simple texts on familiar matters of a concrete type which consist of high-frequency everyday or job-related language.   Can understand short, simple texts containing the highest frequency vocabulary, including a proportion of shared international vocabulary items.
A1	Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.

The English Profile Project funded by Cambridge University Press and Cambridge English Language Assessment, among others, has compiled a list of words with their associated CEFR levels and a list of grammatical forms that are used by students at various CEFR levels. These are called the English Vocabulary Profile and the English Grammar Profile, respectively. These have been used as the fundamental guiding principles when assigning a CEFR level to questions that satisfy the Word Structure and Vocabulary constructs in the assessment instruments.

All Reading Comprehension questions in the assessment instruments were assigned a CEFR level using the “Can do” statements presented in Table 3. 

The composition of higher CEFR level questions in the assessment instruments increases from Grade 6 to Grade 8.

Table 4 shows the distribution of different CEFR questions across grades 6, 7 and 8.

CEFR	Grade 6	Grade 7	Grade 8
A1	60%	53%	36%
A2	38%	44%	47%
B1	2%	2%	11%
B2	–	–	4%
C1	–	–	2%

Students in Grade 6 are expected to be at the CEFR level A2; students in Grades 7 and 8 are expected to be at the CEFR levels B1/B2.

Assessment Outcome

The test instruments were composed of dichotomous items. Students’ responses to every question were scored as 1 or 0, depending upon whether the student answered the question correctly or not. No partial credits were allowed. The sum of the students’ responses to all questions in the test was scaled to 100; providing the students’ score. The arithmetic mean of these scores is presented as the Mean Score of the cohort.

Figure 1 shows the mean scores of the students across all grades in Baseline and Endline Assessments.

Students in the Delhi pilot schools achieved a mean score of 33% in the Baseline Assessments. The same set of students have a mean score of 42% in the Endline Assessments. Overall, the students have shown a score improvement of 27% on average (ranging from 21% to 37% across grades) over Baseline scores.

CEFR Proficiency

Students of grades 6 to 8 are required to be in the proficiency range of A2-B2 to successfully read and comprehend their textbooks.

The graph below shows the distribution of students across the CEFR levels in Baseline and Endline Assessments.

In the Baseline assessments, 88% of the students assessed across grades were at CEFR level A1, lower than the A2 requirement for Grade 7 and only 12% of the students were at A2 and Higher CEFR proficiency levels.

When these students were assessed in the Endline, 14% of these students have been observed to move to higher CEFR levels.

Conclusion

The impact analysis conducted among students of Grades 6, 7 and 8 in the two pilot schools in Delhi have validated that:

– Use of ReadToMe® significantly improves outcomes in reading and comprehension proficiency.
– Improvement in outcomes is derived by students at all learning levels and across grades.
– Improvement in outcome also positively impacts the CEFR proficiency of students.
– Use of ReadToMe® can reduce the time to attain proficiency levels.
– Improvement in outcomes has been obtained despite the fact that during the short-duration pilot, 41% of the recommended number of sessions per academic year of ReadToMe® have been conducted per grade.

In conclusion, the assessments validate that the use of the ReadToMe® solutions has a positive impact on the English reading and comprehension of students. There is evidence of a strong correlation between usage of ReadToMe® and improvement in outcomes, proven over time. Based on this proof of concept, students from the schools of Delhi can derive the benefit of ReadToMe® through full deployment of the program.