An investigation of Illinois state assessment system high school scoring patterns
This study investigates the origins of an apparent divergence of WorkKeys Reading, a component of the Prairie State Achievement Examination (PSAE), from other components of the assessment. The issue was noted by a group of schools and districts concerned with the scoring patterns of the 2006 and 2007 cohorts of students. They noticed that students in 2007 scored much lower on WorkKeys Reading than the 2006 cohort, while 2007 students scored higher on all other components of the PSAE. The group was particularly concerned by an increase in ACT Reading, another PSAE component. ACT produces concordance tables linking ACT Reading with WorkKeys Reading, making the divergence seem unlikely.
The full population of student-level data was provided by ACT and the Illinois State Board of Education (approximately 130,000 cases per year). As a first step, the divergence was verified at the state level. This step confirmed that schools and districts noting the divergence had not mistakenly interpreted their data. It also helped ensure that the divergence was not isolated to only a few schools or districts. Finally, because the federal No Child Left Behind
(NCLB) legislation (2002) requires that schools report percent proficient, it was necessary to verify that the divergence represented a mean shift in student scores rather than a statistical artifact created by the calculation of percent proficient. All of these potential explanations for the divergence were quickly discarded by simply calculating the means and standard deviations for each PSAE component test. WorkKeys Reading means, for the state student population, did indeed decline, while all other components improved from 2006 to 2007.