Banner
    Test Scores Are Meaningful? Even If They Go Up, Cognitive Abilities Don't
    By News Staff | December 11th 2013 01:07 PM | 7 comments | Print | E-mail | Track Comments

    There is culture cold war in America over education. One side says American kids are under-performing because teachers are not using agreed upon criteria and so students don't do as well as some other countries on international standardized tests. The other side says American kids are under-performing because the government wants to 'teach to the test' so students do better on international standardized tests.

    Both sides are manned by teachers, educational institutions and unions.

    Do tests adequately predict academic success? Not really. When American students took the first international standardized test in the early 1960s, they came in next-to-last. But since then, those same students have dominated worldwide science output and Nobel prizes.

    Such tests are designed to measure the knowledge and skills that students have acquired in school — what psychologists call "crystallized intelligence." However, schools whose students have the highest gains on test scores do not produce similar gains in "fluid intelligence" — the ability to analyze abstract problems and think logically — according to a new study from MIT neuroscientists working with education researchers at Harvard University and Brown University.

    In an analysis of nearly 1,400 eighth-graders in the Boston public school system, the researchers found that some schools have successfully raised their students' scores on the Massachusetts Comprehensive Assessment System (MCAS). However, those schools had almost no effect on students' performance on tests of fluid intelligence skills, such as working memory capacity, speed of information processing, and ability to solve abstract problems.

    "Our original question was this: If you have a school that's effectively helping kids from lower socioeconomic environments by moving up their scores and improving their chances to go to college, then are those changes accompanied by gains in additional cognitive skills?" says John Gabrieli, the Grover M. Hermann Professor of Health Sciences and Technology, professor of brain and cognitive sciences at MIT.

    Instead, the researchers found that educational practices designed to raise knowledge and boost test scores do not improve fluid intelligence. "It doesn't seem like you get these skills for free in the way that you might hope, just by doing a lot of studying and being a good student," says Gabrieli, who is also a member of MIT's McGovern Institute for Brain Research.


    Measuring cognition



    This study grew out of a larger effort to find measures beyond standardized tests that can predict long-term success for students. "As we started that study, it struck us that there's been surprisingly little evaluation of different kinds of cognitive abilities and how they relate to educational outcomes," Gabrieli says.

    The data for the Psychological Science study came from students attending traditional, charter, and exam schools in Boston. Some of those schools have had great success improving their students' MCAS scores — a boost that studies have found also translates to better performance on the SAT and Advanced Placement tests.

    The researchers calculated how much of the variation in MCAS scores was due to the school that students attended. For MCAS scores in English, schools accounted for 24 percent of the variation, and they accounted for 34 percent of the math MCAS variation. However, the schools accounted for very little of the variation in fluid cognitive skills — less than 3 percent for all three skills combined.

    In one example of a test of fluid reasoning, students were asked to choose which of six pictures completed the missing pieces of a puzzle — a task requiring integration of information such as shape, pattern, and orientation.

    "It's not always clear what dimensions you have to pay attention to get the problem correct. That's why we call it fluid, because it's the application of reasoning skills in novel contexts," says Amy Finn, an MIT postdoc and lead author of the paper.

    Even stronger evidence came from a comparison of about 200 students who had entered a lottery for admittance to a handful of Boston's oversubscribed charter schools, many of which achieve strong improvement in MCAS scores. The researchers found that students who were randomly selected to attend high-performing charter schools did significantly better on the math MCAS than those who were not chosen, but there was no corresponding increase in fluid intelligence scores.

    However, the researchers say their study is not about comparing charter schools and district schools. Rather, the study showed that while schools of both types varied in their impact on test scores, they did not vary in their impact on fluid cognitive skills.

    The researchers plan to continue tracking these students, who are now in 10th grade, to see how their academic performance and other life outcomes evolve. They have also begun to participate in a new study of high school seniors to track how their standardized test scores and cognitive abilities influence their rates of college attendance and graduation.


    Implications for education



    Gabrieli notes that the study should not be interpreted as critical of schools that are improving their students' MCAS scores. "It's valuable to push up the crystallized abilities, because if you can do more math, if you can read a paragraph and answer comprehension questions, all those things are positive," he says.

    He hopes that the findings will encourage educational policymakers to consider adding practices that enhance cognitive skills. Although many studies have shown that students' fluid cognitive skills predict their academic performance, such skills are seldom explicitly taught.

    "Schools can improve crystallized abilities, and now it might be a priority to see if there are some methods for enhancing the fluid ones as well," Gabrieli says.

    Some studies have found that educational programs that focus on improving memory, attention, executive function, and inductive reasoning can boost fluid intelligence, but there is still much disagreement over what programs are consistently effective.



    Upcoming in Psychological Science.  

    Comments

    "Do tests adequately predict academic success? Not really. When American students took the first international standardized test in the early 1960s, they came in next-to-last. But since then, those same students have dominated worldwide science output and Nobel prizes."

    The conclusion doesn't follow from the example. A large number of students take the standardised test, whereas only very few get Nobels or dominate worldwide science. Basically what it shows is America has a low mean and a long tail of outliers.

    Hank
    Yes, yes, American education sucks. The self-loathing fix has been in for decades - but when anyone tries to fix it, they get told they shouldn't tell teachers how to teach.

    Which country were you raised in, where they educated only the elites and therefore the mean stayed high?
    I'm from the UK. Our education system isn't perfect but I think it's ok.

    The biggest problem is that we have multiple competing boards who set the main exams. This means there is an incentive to make exams easier, as schools are free to pick whichever board they want, so generally go for the boards which are seen as easiest. Grades therefore inflate, and have to be periodically reset (as is happening right now).

    Next biggest is trying to make as many people as possible get degrees, which get people in a lot of debt (maybe even £30k), and miss out on 3 years of career development, for not a lot of benefit.

    There are some excellent schools which are extremely pricey, and draw in a lot of foreign students which brings in a lot of cash, but actually some of the very best in the country are completely free - they take students based on attainment and how local they are.

    In my opinion the biggest advantage we have in the UK over the USA is that we don't make such a big thing of specialising so early, and so the average person has better and broader general knowledge. Last time I was in America I knew more about US history than a lot of the people I spoke to! And as I'm sure you'll agree the science standard is very low in the states.

    Hank
    I don't agree on that last part - it is a modern mythology. America leads the world in adult science literacy and Americans don't teach to the test in school so going by test scores - which dooms some students to mediocrity in the UK - isn't the thing we do. Unless someone is trying to drum up more money, then they criticize test scores.

    I can find ways to be critical of US education, of course, just like you can find flaws in the UK. But criticizing American education has become its own well-funded industry and enough marketing money works. That just doesn't change reality.
    miles
    Do tests adequately predict academic success? Not really. When American students took the first international standardized test in the early 1960s, they came in next-to-last. But since then, those same students have dominated worldwide science output and Nobel prizes.


    As stated above, ("but since then".  When was this?) There is time gap between the international standardized test and the achievements in science outputs and Nobel prizes, so that, there is a question of whether there were something between the gaps  that became factors of the achievements?
    Hank
    1964 was the first test and given the average age of when people do the work that eventually gets them Nobel prizes, it means those terrible students on that first test have dominated science output.
    miles
    Sad to admit as it may, the Philippines is at the bottom of both the test and the output. The two may seem to correlate (no figures provided in the article so don't bother on the exactness of this statistical term). Since it is different with other countries (America), It looks then that culture, economic status and motivation are factors. 
    But, then I agree that that international standardized test score is not a measure of general intelligence, in which both fluid and crystallized intelligence are parts of.  What is really the objective of the international standard test? Is it intended to measure general intelligence or how well educational system is doing? Objectives are important in assessment for it enlightens one on the validity of the test.