Stop Ignoring the Facts: In New York City, Controversial Education Program Lives Despite Rocky Data
Public officials often disregard educational research, but it's hard to catch them red-handed. They don't reach positions of influence without learning to obfuscate, to redirect. Rarely does a policymaker as much as say, "Screw the data, I'm doing what I want."
Last week, one did.
New York City Schools Chancellor Carmen Fariña said she supports a greater number of New York City schools using balanced literacy as a reading program. (To clarify, balanced literacy describes a general philosophy of teaching reading, but Fariña was talking about a particular balanced literacy program that has been implemented in New York City. So I'm talking here about that particular program, not the general approach.) The approach is a combination of several approaches to reading and writing, in which students often work in small groups, instead of listening to lectures, and are encouraged to write about personal experiences and social issues. Novels, plays, and fluency are favored in lieu of textbooks and spelling and grammar – students get to choose what they want to read.
Fariña has called for more schools to adopt the program, despite its rocky history. The City mandated the approach in 2003, but began to back away in the late 2000s when accumulating data failed to support its effectiveness. To his credit, then-Chancellor Joel Klein, who had overseen the adoption and championed the program, paid attention to student data and began to look for alternatives.
One alternative Klein thought worth exploring was the Core Knowledge Foundation's new reading program, and a pilot study was set up: 3 years, 1,000 students, 20 schools (10 using the new Core Knowledge program, 10 using whatever they typically use, which for most was the city's balanced literacy approach). Under Core Knowledge, curriculum emphasizes vocabulary skills, content and nonfiction texts. Whereas the impetus behind balanced literacy is to get students engaged by allowing them more freedom in learning, the philosophy behind Core Knowledge is that basic knowledge in core subjects is key to student success.
Three years later in 2012, the results were in: the kids in the Core Knowledge schools were reading better. The advantage was indisputable.
Fariña was reminded of these results last week: how could she justify calling for a return to balanced literacy when another program had shown superior? And the response of the city education department was the same given by Fariña when the study was first published: the study is too small as a basis for policy decisions.
In one sense, she's got a point. When you see that Core Knowledge kids scored better, what that means is that the difference between Core Knowledge kids and control kids was so large, it was very unlikely to be due to chance. But what if the 10 Core Knowledge schools (or the 10 control) happened to be quirky in some way? Well, the statistical tests offer some protection; when numbers of schools compared is small (as in this case) you set a higher bar for how big the difference in kids' scores must be before you conclude "very unlikely to occur by chance." But that protection isn't perfect. It could be that some of the schools were atypical and a researcher would worry about that.
What a researcher would NOT do is respond as Fariña did -- just ignore the data.
It reminds me of a story I used to tell when I taught Introductory statistics. A quality control inspector works a ball-bearing factory, and for every lot of ball-bearings manufactured, he's supposed to test 100 and make sure that no more than 1 is faulty. It's boring job, because there's always either 0 or at most 1 faulty ball in each lot. But one day he finds 5 faulty balls. "That's funny," he thinks. "I must have picked a weird set of 100." So he picks another 100 and four are faulty. That also seems unusual, so he picks another hundred and this time only 1 is faulty. "There," he thinks. "That's better."
The Core Knowledge study isn't large enough, but that doesn't mean the chancellor ought to ignore it, or worse, act in a way that seems counter to the conclusion. Instead she should say "Gosh, there's a possibility here that we could help the kids of this city read better. Let's do another, larger study."
But policymakers too often feel free to ignore research. This is why we can't have nice things.
Disclosure: Daniel Willingham is on the Board of Trustees of the Core Knowledge Foundation.