When VaxGen announced the results of the first-ever efficacy trial of an AIDS vaccine on 24 February, a few sharp-eyed scientists questioned the statistical analyses used by the Brisbane, California, biotech company. Now Marc Gurwith, an infectious disease specialist who heads the company's clinical trials, concedes that VaxGen should have presented its analysis differently, calling into question the controversial claim that the vaccine worked in some racial groups--most dramatically in blacks.
From the outset, the company reported that the vaccine failed to show protection (ScienceNOW, 24 February). But VaxGen scientists claimed that analyses revealed a statistically significant efficacy in 66.8% of blacks, Asians, and people of mixed race, and 78.3% in blacks alone. Subsequently, Bette Korber, as AIDS researcher at Los Alamos National Laboratory, and biostatistician Steven Self of the University of Washington, Seattle, independently ran calculations that convinced them of a key flaw.
Each time researchers dissect out a subset of their dataset for separate analysis, they increase the likelihood that a given result will be due to chance. To correct for this, statisticians lower the confidence interval--which says with a certain degree of certainty, or "p value," that the result is not due to chance. Both Korber and Self concluded that VaxGen, despite explicit declarations to the contrary by company scientists, did not adjust for the subgroup analyses. Now VaxGen acknowledges that a misunderstanding occurred between the company's statistician and other scientists there. "The p values that were in the press release were not adjusted," Gurwith told ScienceNow.
Gurwith emphasizes that statisticians do not have a single way to do these adjustments. "It's fairly complicated because what the proper adjustments are is not so obvious," he says. And he stresses that the differences in vaccine efficacy seen by racial groups do exist.
However, VaxGen originally claimed a p value of less than .02 for the black subgroup. (Biologists typically use a p value of .05, indicating 95% confidence that a result is not due to chance, as the threshold for statistical significance.) Gurwith says they did nine substudies based on race, which, with a straightforward, conservative adjustment known as the Bonferroni correction, changes the p value to roughly 0.18. "So it wouldn't be significant," he acknowledges. He says by the same test, though, the finding of significance would remain when combining blacks with Asians and people of mixed race.
But Cornell University's John Moore, a longtime critic of the vaccine and HIV antibody expert, finds this reasoning absurd. "Blacks and Asians lumped together is biological rubbish," says Moore. "They might as well do a subgroup analysis based on signs of the Zodiac." Self further questions the accuracy of VaxGen's claim about statistical significance in even that one group. "There's some marginal effect [of the vaccine] and it's worth going after, but it's not worth overblowing. It's a hypothesis-generating result," he says.
Related site
VaxGen
)
)
)
)
)
)
)
)
)
)
)
