We’ve heard for nearly a year that the Ohio Local Report Card Data would be a mess due to the new (one-time-only) assessments, new cut scores, etc. Patrick O’Donnell from Cleveland explained some of the problems last week, the day before the majority of the grades were released:
The state will be issuing its grades tomorrow for how well kids met math and English expectations and for how much they learned over the 2014-15 school year – the first year that Ohio tested students on the new multi-state standards.
Just don’t expect any neat and tidy answers – only lower scores across the state.
Several factors make it hard to compare results on this report card to previous years, while also raising questions about just what the data really means.
Among them: Increased expectations of what kids should know; constantly-shifting definitions of what makes a student “Proficient;” issues with taking tests online for the first time; widespread “opt-outs” from state tests; and dissatisfaction with tests from PARCC, the multi-state Common Core testing consortium Ohio used before the legislature fired it in July. (February 24, 2016, The Plain Dealer)
Here’s an example of the most significant change. The Performance Index Score, which is essentially a composite of student performance on the state’s standardized tests, was expected to be lower this year. It would have been even lower if legislator’s didn’t completely freak out upon receiving preliminary results and adjust the definition of “proficient” on the fly. Anyway, of Ohio’s 609 school districts on ODE’s report for 2014-15, 607 of them saw a drop on their overall Performance Index score.
For those who like percentages, that means that 99.67% of school districts dropped from the 2013-14 school year (quick shout out to Stryker Local & Bridgeport Exempted Village, the only two districts in the state that did not drop). The average drop across the districts was just over 7.7 points (out of a possible total of 120 points).
For comparison purposes, 244 of the 609 saw a drop between 12-13 & 13-14, 40% of Ohio’s school districts, but the average change in PI between those school years was actually PLUS 0.27 points.
So yea, it’s a mess if you simply look at the grades and numbers and don’t dig deeper into the REAL data underneath it all.
But I digress…
If it wasn’t already bad enough that everyone is trying to figure out what the new grades mean in the context of previous years and changing metrics, the Ohio Department of Education’s Report Card website isn’t doing anyone any favors. Let’s look at the website.
A key part of a school or district’s report card that people look at is the number of “indicators” that were met by calculating student achievement scores. Here’s a screenshot from one district:
You’ll see that this district received a D grade for meeting only 21 out of the 34 indicators that this district was subject to (36 is the most a district can have). Sure, that seems clear enough, I guess, but look at the caption at the top explaining how a district received credit for meeting the specific indicator.
So, I log in to look at my district’s results and I see a D. I’m likely disappointed, but I’m also suddenly confused, too. I see that my district received fancy green check marks for areas that are nowhere near 80%. What gives? I click on all the tabs and read all of the pages, but no answer. WTH?
Not one to give up so easily, we dug into the Ohio Department of Education website to try to find the answer. And dig we did, until we found it. You can see the trail of breadcrumbs at the top of the screenshot.
What we also found are the “real” answers to how a school or district met the indicator (see red arrow above over the column of percentages) and the explanation of why all of the percentages are different [emphasis-added]:
The board decided to maintain the 80 percent for tests established before the 2014-2015 school year. These tests include the Ohio Achievement Assessment in third grade reading and the Ohio Graduation Tests for students who took them as 10th-graders in the 2014-2015 school year. The board also maintained 85 percent for students who were in 11th grade during the 2014-2015 school year and took one or more OGTs.
Acknowledging that 2014-2015 was the first year of new tests in English language arts, mathematics, science and social studies, the State Board’s decision is to use the state percentage of students scoring proficient or higher for each new test.
Sweet. So the Kasich-friendly State Board of Education basically graded schools on a curve this year. That’s some solid, airtight education policy right there…..
Here’s how it’s done: Wait until you get back all the scaled scores of all of the tests back before setting the cut scores (i.e., grading scale) to determine what scores made a student proficient or not (or in one of the other categories; making sure there is a fairly equal distribution of students in each category), and then use the percentages you’ve created (curved) to determine if a school or district met the newly-created criteria.