SACRAMENTO, CA - Educators who consider themselves progressive argue that factual knowledge and getting the right answer aren't as important as higher-order thinking, i.e., the analytical skills used in problem solving. However, a new study that claims to support the effectiveness of higher-order thinking methods fails to answer key research questions.

The study, co-sponsored by the Educational Testing Service (ETS) and the Milken Family Foundation and
authored by Harold Wenglinsky of ETS, finds that, “In math, students whose teachers emphasize higher-order
thinking skills outperform their peers by about 40 percent of a grade level.” While this sounds impressive, there is much less to this finding than meets the eye.

First, the study covers only a single grade -- the eighth. Second, Wenglinsky's conclusion is not based on
longitudinal data, i.e., data over a period of time. Rather, his conclusion is based on cross-sectional
data, i.e., data at a single moment in time. In this case, Wenglinsky looked at eighth-grade test scores on
the 1996 National Assessment of Educational Progress (NAEP) exam and correlated these scores with teachers
who answered a NAEP questionnaire asking whether they had taken a professional-development course that
emphasized higher-order-thinking practices. East Tennessee State University education psychologist J.E.
Stone, one of the nation's leading authorities on teaching methodologies, says that Wenglinsky's use of
cross-sectional data is seriously flawed.

Wenglinsky himself admits that, “The disadvantage of cross-sectional studies, such as this one, is that the
outcomes occur at the same time as the factors that apparently influence them, raising the possibility that
the outcomes influence the factors rather than the other way around.” Stone warns that, “This caution
needs to be stamped in red ink on the cover of this report.” Stone points out that correlation does not
equal causality. Eighth-grade teachers of higher-achieving students may use higher-order methods,
but that doesn’t mean that the methods are responsible for the higher achievement. It’s more likely, says
Stone, that the students’ higher achievement permitted teachers to use teaching methods that went beyond
acquisition of basic knowledge.

Stone’s observations point to perhaps the biggest flaw with the Wenglinsky study. Because it doesn’t use
longitudinal data, it’s impossible to know whether the higher-achieving eighth-grade students in the study
received their early-grades’ instruction through a practice-and-learn-the-basics method or a higher-order-thinking method. Jeanne Chall, the late famed Harvard education professor, noted that most research shows that students who first learn lower-level basic knowledge and skills ultimately do better in higher-order problem-solving. Also, researchers at Carnegie Mellon University have found that real competence only comes with lower-level practice and drill.

Thus, the achievement of the eighth-graders in the study may be due, not to the higher-order-thinking
practices of their eighth-grade teachers, but to the more basic methods of teachers in earlier grades. The
study, therefore, doesn’t prove that higher-order-thinking methods are mainly responsible for higher achievement in the eighth grade.

The Wenglinsky study is dangerous because it seems to give a quantitative basis for methods that, despite
their label, do not help students to progress. The existing experimental and quantitative data overwhelmingly favor traditional teaching methods. Those are the methods that policymakers should encourage.

- Lance T. Izumi