“Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation.” – Doug Altman, BMJ 1994
Doug Altman was a statistician in Oxford who spent much of his career trying to improve the quality of research. His motivation was the waste outlined in the quote above, and in particular the fatal flaws being introduced that led to the research becoming not just useless, but often misleading. In 1994 he wrote a famous essay in the British Medical Journal entitled “The Scandal of Poor Medical Research” (from which the quote above is taken) which he begins by saying: “We need less research, better research, and research done for the right reasons.“
In a previous blog, we looked at quantifying Altman’s “huge sums” by looking at five stages of research: the questions asked; the appropriateness of the design; the conduct of the research; the accessibility of a report; whether that report was unbiased and usable. From the second fourth and fifth stages of that we could calculate an 85% avoidable waste arising from 50% avoidable design flaws, 50% non-publication, and 50% unusable report. What does this “waste” look like in individual research projects, and what can we do to reduce it? I recently spoke about these issue to the College of Surgeons, describing the issues, some examples, and what might be done (10 minute video here – Increasing Value, Reducing Waste in clinical research ).
As one example, in a study of the impact of giving flaxseed the authors found 27 men with cardiovascular risk factors, took blood at baseline and again at 42 days – not a trivial task. But their analysis was incorrect (rather than compare to the control group, they compared before-and-after in a single arm). So all their hard work was not only wasted, it was misleading. Yet, as Altman and Bland have repeatedly pointed out, this is a common error which often slips past reviewers and editors.
This huge research waste may sound a little depressing. But it is also a huge opportunity – for the effort (and funds) that we already put in we could be getting so much more out of our research system. That is true for individual researchers, for research organisations, and for countries. We could publish all the unpublished studies; train people in good research practices; routinely do rapid systematic reviews before new research; reduce the inefficiencies and overheads in the conduct of research; and improve the infrastructure for research quality. By just spending a fraction of research time focusing on how the research system and processes could work a little better, we would gain both research productivity and trustworthiness, with the ultimate beneficiaries being patients and the public.
Healthy Evidence 