Charter School Success: Are They Really As Outstanding As the Figures Show?

Charter School Success: Are They Really As Outstanding As the Figures Show?
X
Story Stream
recent articles

Michael Perkins, center, waits with classmates as they line up to graduate from BART charter high school at Massachusetts College Of Liberal Arts in North Adams, Mass., Saturday, June 7, 2014. (AP Photo/The Berkshire Eagle, Stephanie Zollshan)

RCEd Commentary

Last week Education Next published an easy-to-digest account (West et al., 2014) of a study examining oversubscribed charter schools in Boston. (The original study by Finn et al. had been published last March.) The main finding is that attending one of these charter schools is associated with improved scores on the Massachusetts state tests, but attendance is not associated with levels of fluid intelligence. These results are fascinating, but they are also easy to misinterpret.

Here’s how the study was done. Researchers collected data from a large group (1,367) Boston 8th graders. Some had been attending an oversubscribed charter school since 5th grade. Because the schools are oversubscribed, entry is determined by lottery. Thus, researchers could also test students whose parents had tried to get them into the charter school, but had lost out on the lottery. This is a standard procedure to attempt to control for the fact that parents with enough get-up-and-go to try to get their child to a charter school may differ from other parents in a variety of ways, and it may be these parenting/home factors that are important, not the effect of the charter school.

The researchers administered a battery of tests to eighth graders, the most of important of which they generically refer to as “cognitive skills.” In one, students must use a key to rapidly find the digit that is associated with a meaningless symbol. In another, they briefly remember colored shapes from amongst an increasingly heavy load of irrelevant visual stimuli. And in a third they were to find order amongst a set of complex geometric forms.

Note that there’s not really knowledge you can bring to these tasks to help you. That’s by design. They are meant to tap fluid intelligence. Fluid intelligence refers to a basic ability to manipulate information in the mind. It contrasts with crystallized intelligence, which is basically stuff you know. In the most commonly accepted models of intelligence, being smart (and performing well on IQ tests) relies on a combination of crystallized and fluid intelligence.

Now, to the interesting findings of the study.

The researchers had access to reading and math scores from the Massachusetts state exam, the MCAS from the fourth and eighth grades. The goal was to describe who showed big gains and who showed small gains, using other variables.

They found that, after accounting for demographic variables (gender, race, age, free- and reduced-priced-lunch status, limited English proficiency, and special-education status) and fourth grade MCAS math scores, the school that students attended was a significant predictor of eight grade MCAS math scores. Each year in an oversubscribed charter school led to a .13 standard deviation edge in test score improvement.

But a parallel analysis indicated that attending a charter school was not associated with higher fluid intelligence scores. Once researchers controlled for demographic variables and fourth grade MCAS scores, the difference between the charter schools and regular schools on the eight grade MCAS was negligible.

That’s notable, because fluid intelligence and mathematics proficiency are themselves correlated.

The tensions between these two findings is nicely illustrated in this figure showing the relationship of fluid cognitive skills and test score gains.

Each dot is a school, and the size of the dot represents the number of students participating in the study. The line shows the positive relationship between cognitive skill and math gains. Most of the dots are fairly near the line. But note that all of the green dots are above the line; students at the charter schools are scoring better on the math test than their fluid intelligence predicts they ought to.

There are a couple of ways that one could go wrong in interpreting these results.

First, these results don’t exactly mean “charter schools don’t improve fluid intelligence.” It could be that all schools are improving intelligence, but they do so to an equivalent extent. The data show that there’s not an effect on fluid intelligence of attending one school or another.

That’s what we can conclude from the data, but it’s far from a crazy guess to suggest that schools are not moving fluid intelligence a whole lot during this four year period. Other data indicate that schooling improves crystallized intelligence, but not fluid intelligence (Carlsson et al, 2012). And researchers who have tried focused training on fluid intelligence have had mixed success (Shipstead et al, 2012). It may be that fluid intelligence is simply hard to improve.  

A more serious potential source of confusion is the meaning of what the article calls “cognitive skill.”

In this context, cognitive skill does not mean something like “being able to solve problems,” or “being resourceful when you encounter difficulties,” or “a tendency to evaluate arguments critically.” It’s a step or two more primitive than that. It really means raw horsepower in your ability to mentally move information around. So it’s inaccurate to interpret these results as showing that charters are making kids good at scoring well on math tests, but they haven’t really taught them math because they haven’t improved their cognitive skill.

Successful charters are improving kids’ math scores by teaching them a lot of math, which is probably what any successful school does. Fluid intelligence may move slowly (or not at all) but we know crystallized intelligence is trainable, and cognitive performance is a product of the two. When a fourth-grader solves a long-division problem, she may encounter the need to multiply 8 and 4 as a substep of that problem. She can either compute that product in her head (which would place a high demand on working memory, a component of fluid intelligence) or she may have memorized the fact that the product is 32. The crystallized fact obviates the need for fluid resources.

So in the end, the “how charters do it” aspect of the findings is good to know, but doesn’t seem shocking. West et al. call for more study of how to improve fluid intelligence, presumably so that we could implement techniques in schools.  No argument, but that sector of research seems plenty vigorous to me, fueled by financial incentives.


Reference:

Carlsson, M. Dahl, G. B. & Rooth, D-O. (2012). The Effect of Schooling on Cognitive Skills. NBER Working Paper No. 18484 October 2012.

Finn, A. S., Kraft, M. A., West, M. R., Leonard, J. A., Bish, C. E., Martin, R. E., & Gabrieli, J. D. (2014). Cognitive Skills, Student Achievement Tests, and Schools. Psychological Science, 25, 736-744.

 Shipstead, Z, Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138, 628-654.

 

West, M. R., Gabrieli, C. F. O., Finn, A. S., Kraft, M. A., & Gabrieli, J. D. E. (2014). What effective schools do. Education Next, 14, 

Comment
Show commentsHide Comments

Related Articles