This was written by Larry Cuban, a former high school social studies teacher (14 years, including seven at Cardozo and Roosevelt high schools in the District), district superintendent (seven years in Arlington, VA) and professor emeritus of education at Stanford University, where he has taught for 20 years. His latest book is “As Good As It Gets: What School Reform Brought to Austin.” A version of this post appeared on his blog .

By Larry Cuban

Test scores are the coin of the educational realm in the United States. No Child Left Behind demands that scores be used to reward and punish districts, schools, and teachers for how well or poorly students score on state tests. In pursuit of federal dollars, the Obama administration’s Race to the Top competition has shoved state after state into legislating that teacher evaluations include student test scores as part of judging teacher effectiveness.

Numbers glued to high stakes consequences, however, corrupt performance. Since the mid-1970s, social scientists have documented the untoward results of attaching high stakes to quantitative indicators not only for education but also across numerous institutions. They have pointed out that those who implement policies using specific quantitative measures will change their practices to insure better numbers.

The work of social scientist Donald T. Campbell and others about the perverse outcomes of incentives was available and known to many but went ignored. In “ Assessing the Impact of Planned Social Change ,” Campbell wrote:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor” (p. 49).

Campbell drew instances of distorted behavior when police officials used clearance rates in solving crimes, the Soviet Union set numerical goals for farming and industry, and when the U.S. military used “body counts” in Vietnam as evidence of winning the war.

That was 40-50 years ago. In the past decade, medical researchers have found similar patterns when health insurers and Medicare have used quantitative indicators to measure physician performance. For example, Medicare requires — as a quality measure — that doctors administer antibiotics to a pneumonia patient within six hours of arriving at the hospital.

As one physician said: “The trouble is that doctors often cannot diagnose pneumonia that quickly. You have to talk to and examine the patient and wait for blood tests, chest X-rays and so on.” So what happens is that “more and more antibiotics are being used in emergency rooms today, despite all-too-evident dangers like antibiotic-resistant bacteria and antibiotic-associated infections.”

He and other doctors also know that surgeons have been known to pick reasonably healthy patients for heart bypass operations and ignore elderly ones who have 3-5 chronic ailments to insure that results look good.

Here are some more examples:

TV stations charge for advertising on the basis of how many viewers they have during “sweep” months (November, February, May, and July). Nielsen company has boxes in 2 million homes (representative of the nation’s viewership) that register whether the TV is on and what families are watching during those months. They also have viewers fill out diaries. Nielsen assumes that what the station shows in those months represents programming for the entire year (see 2011-2012-Sweeps-Dates). Nope. What do TV networks and cable companies do during those “sweeps?” They program new shows, films, extravaganzas, and sports that will draw viewers so they can charge higher advertising rates. They game the system and corrupt the measure (see p. 80).

And, ripped from the headlines of the daily paper, online vendors secretly ask purchasers of their products to write reviews and rate it with five stars in exchange for a kickback of the price the customer paid. Another corrupted measure.

Of course, educational researchers also have documented the link between standardized test scores and narrowed instruction to prepare students for test items, instances of state policymakers fiddling with cut-off scores on tests, increased dropouts, and straight out cheating by a few administrators. (see Dan Koretz, “Measuring Up ”)

What Donald Campbell had said in 1976 about “highly corruptible indicators” applies not only in education but also to many different institutions.

So why do good policy makers use bad indicators?* The answer is that numbers are highly prized in the culture because they are easy to grasp and use in making decisions.The simpler the number — wins/losses, products sold, profits made, test scores — the easier to judge worth. When numbers have high stakes attached to them, they then become incentives (either as a carrot or a stick) to make the numbers look good. And that is where indicators turn bad as sour milk whose expiration date has long passed.

The best policymakers, not merely good ones, know that multiple measures for a worthy goal reduce the possibility of reporting false performance.

So, here’s a quote from a News Journal Story from 1/12/12, after a pivot from schoolwide scores for non tested subject areas, Dr. Lowery then proclaims this:

Officials alleviate teachers’ concerns | The News Journal | delawareonline.com

She also said the state received a special one-year U.S. Department of Education exemption from rules that require student test scores to be a component of teacher evaluations.

“It will not in any way impact a teacher’s grade or trigger teacher discipline,” she said.

Then she writes this: HERE in which there is the following quote:

For the 2011/12 school year, educator summative ratings will be determined without use of Component V except as used to identify highly effective teachers, as noted below. Using Components I – IV, an educator’s summative rating will be determined as follows:

• 0 or 1 satisfactory components = Ineffective summative rating
• 2 satisfactory components = Needs Improvement summative rating
• 3 or 4 satisfactory components = Effective summative rating

Only teachers of DCAS-tested subjects (math, English Language Arts in grades 3-10) will receive a Component V score for this year. They will be eligible for a Highly Effective summative rating and therefore eligible for incentive and retention bonuses. Details on the incentive/retention program will be finalized and announced this spring and will be voluntary at the local level.

So, I attended the hearing that the NJ piece is based on as did several teachers. It couldn’t have sounded more clear to me: non tested subject teachers were not going to be subjected to component 5 at all, schoolwide scores or otherwise and component 5 will not be used in teacher performance conclusions even if the teacher got a rating their would be no punitive actions. Now, a quick read may suggest that since they are only offering perverse incentives for the ELA and Math teachers who score well, that they have satisfied the sensibility of no punitive actions….but, the scoring rubric above clearly suggests that a non ELA/Math teacher who gets 4/4 is effective only. Compared to an ELA/Math teacher who gets 5/5 and can be HIGHLY effective. So, Arts, Social Studies, Phys Ed, Foreign Language, Science teachers, no matter how good they are, cannot be rated highly effective this year? Seems so, and it seems to me like we’ve been lied to, again: non ELA/Math teachers will be punished by having an entire tier of ratings made inaccessible to them for OUTSTANDING, SUPERLATIVE work. I feel bad for all teachers in the system: leadership is failing them in so many ways and from so many different places. Keeping highly effective ratings from those who deserve, and more importantly EARN, them will poison any reasonable evaluation system with toxic doses of bad morale, jealousy, anger, and resentment: DPASS-II(R) will be no exception….

Cant wait to see the DDOE PIO spin machine on this load of crap.

My take: Governor Markell and Dr. Lowery are trying to sprinkle in performance bonuses and incentives in an attempt to get some measure of positivity in the system (getting teachers who get bonuses to talk up the program, etc) in an election year…… Here’s the two fold mistake:

1) They have been saying all year that it will not count, so this has become a lie. Not surprising from our state ed leadership, but just another example of dishonesty piled on top of many others.

2) Performance incentives do not work. Period. There is extensive research on this subject, but our DOE has chosen to ignore it because the Feds ignore it. I believe all involved will rue the day we ignored this fact.

Well, I can’t wait to start getting phone calls from my constituents and teachers complaining that we have created class-ism inside the DE teaching profession. Unlike, Governor Markell and Dr. Lowery, I’ll handle it rather than kick the can.

Albeit for different reasons, mark this day down, I agreed with Rodel, twice!

Partnership Zone Plans : More of the Same | Rodel Foundation of Delaware

Partnership Zone Plans : More of the Same

By brett on January 25, 2012 in Delaware Schools, Family & Community Engagement, Race to the Top, Turnaround Schools

The Delaware Department of Education approved plans at five of six second round Partnership Zone schools (Laurel Middle’s plan is due in February), which will wrap up the state’s Race to the Top commitment to turnaround ten persistently low-performing schools.

Although the plans represent significant effort on the part of all stakeholders to come together and negotiate a path forward for these schools, they seem to focus on incremental gains without an overarching, clear, and coherent vision for these schools – and a willingness to fundamentally change conditions for schools that we know need more than a “light touch”.

Instead, they feel like a mash up of various, and potentially conflicting, initiatives, which frustrates all involved and leads to the disenchantment and fatigue already felt by many.  However, scanning the plans, there are components that leave me optimistic about the possibility of these to produce positive results for kids:

• Incorporating Technology: While I question the use of these funds to purchase necessary hardware, it’s heartening to see schools explicitly state the potential of technology to help students through such innovative programs as Dreambox at Marbrook Elementary;
• Extended Learning Time: A couple schools are adding a fairly significant chunk of time onto the school day and year in order to catch students up on essential ELA and math skills, such as at Stanton Middle; and
• Family and Community Engagement: Schools are strengthening or adding on to the initiatives laid out in their Race to the Top plans, such as family home visits by teachers before the start of school at Dover High.

This is a stark contrast to other turnaround efforts across the country, such as Houston’s Apollo 20 program, which has five tenets guiding their efforts, including effective teachers and leaders, extended learning time, high-dosage tutoring, data-driven instruction, and a culture of high expectations for all students.  This program, in its second year of implementation, has yielded significant initial results for their students.

The ESEA flexibility process provides an opportunity to strengthen the state framework around low-performing schools, and not just the bottom ten. Other states have taken advantage of this opportunity to change the policy context, funding conditions, and incentives for low-performing schools. Delaware’s waiver is due February 6^th.

Myside bias in deciding “what to think” about research results–(S)extrapolation II

Myside bias in deciding “what to think” about research results–(S)extrapolation II

By Sherman Dorn on January 12, 2012

This morning, the New York Times carried a column by Nicholas Kristof talking about the import of the Chetty, Friedman, and Rockoff paper; later, Kevin Carey wrote a blog entry telling us what to think about the study.¹ To be honest, I’m shocked it took more than half a week for folks to use Friday’s Times story by Annie Lowrey as a springboard for public policy discussions. Maybe the quick responses by Bruce Baker and Matthew Di Carlo played a role in delaying the inevitable.² What was most surprising about the Kristof column is not that he bought the weakest part of the paper as a shiny bright object (as did Carey) but that he first cited (and linked to) Di Carlo’s comments and then entirely ignored Di Carlo’s cautions about the extrapolatory analysis on young-adult effects.

What we’re seeing in both comments is confirmation bias: Carey and Kristof (but especially Kristof) are using the study to confirm preexisting policy preferences. Neither acknowledge any weakness in the extrapolations made by the study authors, even though there are several items that should raise red flags for a reader reasonably well-educated in statistics. Carey even makes the (surprising) mistake of confusing statistical significance with effect size (note: see discussion of this item in comments).³

There’s a short passage in Carey that conflates two separate issues and requires some explanation in rebuttal:

Academics complain all the time that policy is insufficiently informed by evidence, and as a general proposition that’s true. But these complaints are themselves often informed by a vague or naive view of how standards of evidence properly translate to policy choices…. For CFR to conclude from their research that present policies ought to be more strongly weighted toward the possibility of going with someone else … [is] a case of academic researchers fulfilling their responsibility to make their findings meaningful on behalf of society.⁴

The issues here⁵ are whether it was appropriate for this study to identify useful policy consequences and, quite separately, the burden of proof in using research evidence.

1) Is the classroom-aggregate income claim a responsible effort by the researchers at outreach to policymakers? I don’t know if Carey read my comments on the paper, but I specifically pointed out clear policy consequences I saw from the stronger parts of the paper (specifically, the method CFR used to test potential bias effects on value-added measures from within-school student assignment). But exaggeration of policy implications annoys me as a reader, and that’s what I saw in the section Carey likes. He quoted a clear example of Statistical Bull Shiitake from the paper:

Replacing a teacher in the bottom 5% with an average teacher generates earnings gains of $9,422 per student, or $267,000 for a class of average size… (underlining of non-zero figures added)

I spent half a decade as a journal editor dreading the occasional discussion with authors on the number of non-zero figures that made sense in research results, and the basic lesson is that just because SAS prints out 16 digits doesn’t mean it’s impressive or justifiable to use all of them; in general, it shows one’s statistical ignorance (or a temptation to imply too-great accuracy) instead. In this case, the study authors estimate the income effects of a 1 standard-deviation change in teacher effects on the order of 0.9%-1.1%. So how did they get from two significant figures in the underlying parameter estimate to three or four in the dollar amounts? When I see that sort of nonsense, my first impression is that the measure is “merely corroborative detail to add verisimilitude to an otherwise bald and unconvincing narrative,” as Poo-Bah from The Mikado put it. Let me state clearly that the estimate of long-term outcomes by itself is fine as research. It’s the packaging of extrapolation as a soundbite that is irresponsible, and CFR have to perform some interesting contortions to come up with anything that doesn’t look like the moderate effects they found.⁶

2) What is the proper presumption stance on incorporating research/who has the burden of proof in arguing policy? Carey argues that there’s no such thing as a nonchoice in policy–or, more specifically in the case of the Hanushek argument about “deselection” of teachers with low value-added measures, the opposite of “deselection” or any proposed policy is not a policy vacuum. At one level, I agree that the lack of explicit comparisons makes it difficult to argue ethically that one should wait for conclusive research findings before ever changing policy. But that’s a false dichotomy. One need not keep singing A Study’s About To Begin to know there’s a substantial difference between a nihilist approach to policy change (what Carey essentially accuses me of having) and cautious reception of a single study, no matter how interesting.⁷ And, speaking of explicit comparisons, the relevant section of the Chetty, Friedman, and Rockoff paper has no explicit modeling of opportunity costs. That’s the central conceptual engine in economics, and the lack of a full comparative analysis in their paper doesn’t mean the classroom-aggregate estimates are evil, just that they create a sketch and nothing more. Not much to hang policy on, Kristof and Carey’s protests notwithstanding.

Notes

1. Yes, he used “what to think” in the entry title. Carey’s mind-control machinery isn’t working tonight, at least down here in Florida. I suspect he reversed the polarity of his neocortical reticulator. Or he bought the model that was shown in the 2011 Consumer Electronics Show. Didn’t he know not to trust that stuff? [↩]
2. I think it notable that Baker, Di Carlo, and I generally agreed both on where the paper overreach was but also on what the significant contribution of the paper was. We don’t always agree, and the convergence with separate readings should say something to those inclined to take the paper at face value or to dismiss it offhand. [↩]
3. Given the quantity of records the study uses, it’s very easy to see statistical significance with a minimal effect. All statistical significance tells you is the likelihood that the existence of an effect–not its magnitude, but just the existence–could have been generated by random variation in the data. [↩]
4. The full quotation of the second sentence from Carey, without ellipses, is “For CFR to conclude from their research that present policies ought to be more strongly weighted toward the possibility of going with someone else isn’t the academic equivalent of staging a fake wedding for Entertainment Tonight and pocketing the profits, it’s a case of academic researchers fulfilling their responsibility to make their findings meaningful on behalf of society.” He’s responding to my quip that for Lowrey to spend more than 10% of the article on the authors discussing the need to fire teachers based on value-added measures is a waste of column inches when she did not get Jesse Rothstein’s response to CFR’s bias-testing method. Carey is confusing my criticism of CFR with my criticism of Lowrey. [↩]
5. Apart from Carey’s misunderstanding my Kardashian quip. [↩]
6. I could distort the findings to minimize the effect of schools on the lives of students, but that would be equally irresponsible. The demonstration of both a deliberate minimization and its methodological irresponsibility is left as an exercise for the reader. [↩]
7. As far as I am aware, neither Bruce Baker nor Matthew Di Carlo is on record as being a policy change nihilist. [↩]

Original Artist

Begin show

End Show

Interview with Welcome Back Kotter Star Bob Hegyes AKA Juan Epstein