Background

ReadToMe® is a multi-sensory reading and comprehension software that helps enhance students’ reading skills and increases teacher effectiveness.

The Mo School Abhiyan Parichalana Sangathan, A Government of Odisha Initiative under the School & Mass Education Department has commissioned EnglishHelper to deploy ReadToMe® Solutions in 30 schools for Grade 9. As part of the project, assessments were conducted among the students of a sample of 3 schools to measure the proficiency of students at the start of the pilot project. The assessments were conducted in the third week of October, 2022.

The assessments were conducted among students from all the sections of the sampled Mo Schools. A total of 359 students from the 3 sampled schools were part of the baseline assessment. An adequate sample was achieved to allow for meaningful comparisons of scores. 

Assessment Instrument

The assessment instruments test the constructs of Word Structure, Vocabulary, Sentence Sequencing, Reading Comprehension, and Sentence Structure. The assessments comprise Multiple Choice Questions (MCQs) with only a single correct response. The assessments are mapped to the Common European Framework of Reference for Languages (CEFR). The questions in the assessments range from the CEFR levels A1 to C1. A low proportion of B1, B2, and C1 questions were included.

The CEFR describes language proficiency (related to listening, speaking, reading, and writing) on a six-level scale: 

– A1-A2 for Basic User 
– B1-B2 for Independent User 
– C1-C2 for Proficient User 

The CEFR defines specific competencies of a language learner at each of these levels in the form of “Can do” statements. The framework for Overall Reading Comprehension used to design the instruments is as follows:

	OVERALL READING COMPREHENSION
C2	Can understand and interpret critically virtually all forms of the written language including abstract, structurally complex, or highly colloquial literary and non-literary writings. Can understand a wide range of long and complex texts, appreciating subtle distinctions of style and implicit as well as explicit meaning.
C1	Can understand in detail lengthy, complex texts, whether or not they relate to his/her own area of speciality, provided he/she can reread difficult sections.
B2	Can read with a large degree of independence, adapting style and speed of reading to different texts and purposes, and using appropriate reference sources selectively. Has a broad active reading vocabulary, but may experience some difficulty with low-frequency idioms.
B1	Can read straightforward factual texts on subjects related to his/her field and interest with a satisfactory level of comprehension.
A2	Can understand short, simple texts on familiar matters of a concrete type which consist of high-frequency everyday or job-related language Can understand short, simple texts containing the highest frequency vocabulary, including a proportion of shared international vocabulary items.
A1	Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.

The English Profile Project (www.englishprofile.org) funded by Cambridge University Press and Cambridge English Language Assessment, among others, has compiled a list of words with their associated CEFR levels and a list of grammatical forms that are used by students at various CEFR levels. These are called the English Vocabulary Profile (http://www.englishprofile.org/wordlists) and the English Grammar Profile (http://www.englishprofile.org/english-grammar-profile), respectively. These have been used as the fundamental guiding principles when assigning a CEFR level to questions that satisfy the Word Structure and Vocabulary constructs in the assessment instruments.

All Reading Comprehension questions in the assessment instruments were assigned a CEFR level using the “Can do” statements presented in Table 1. 

Assessment Outcome

The test instruments were composed of dichotomous items. Students’ responses to every question were scored as 1 or 0, depending upon whether the student answered the question correctly or not. No partial credits were allowed. The sum of the students’ responses to all questions in the test was scaled to 100; providing the students’ scores. The arithmetic mean of these scores is presented as the Mean Score of the cohort.

A mean score of 38% was achieved across all the 3 sampled Mo schools. On average, students of all the schools were able to answer 1 in 3 questions correctly.

1.1 CEFR Proficiency

Students of Grade 9 are required to be proficient at CEFR level B1-B2 to successfully read and comprehend their textbooks. However, 83% of the students assessed across Mo Schools are still at the lowest proficiency CEFR level A1.

The graph below shows the distribution of students across the CEFR levels.

This gap in proficiency calls for immediate remediation.

1.2 Areas for Improvement

Across all the Mo Schools, students were able to answer fewer than 60% of questions correctly in each domain tested. Students particularly need help with Reading and Comprehension, Sentence Structure, and Sentence Sequencing, especially with interpreting short texts.

Key Takeaways

– Students of Grade 9 were able to answer 1 in 3 questions correctly, exhibiting low ability.

– Students particularly need help with Reading and Comprehension, Sentence Structure, and Sentence Sequencing.

– More than 80% of students across all three schools are at A1 CEFR proficiency lower than the desired proficiency.

Recommendations

Students are expected to show significant improvement in learning outcomes with regular use of ReadToMe® in schools and at home. It is recommended that:

– The schools should conduct at least 3 classes a week of ReadToMe® in class.
– In addition, students should download and revise their textbooks (at least 5 minutes each day) on ReadToMe Student Edition.

Consistent use of ReadToMe® has shown a positive impact on English reading and comprehension ability of students over time[1].

---

[1] Impact Analysis of ReadToMe in South Delhi Municipal Corporation Pilot Schools | 2017 – 2018, https://www.englishhelper.com/resources/research-center/2018/05/29/impact-analysis-of-readtome-in-south-delhi-municipal-corporation-pilot-schools-2017-2018/

For more information, please visit https://www.englishhelper.com/resources/research-center/