Mar

21

John P.A IoannidisThis is a sad conclusion for scientific research:

Why Most Published Research Findings Are False

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

paper by John P. A. Ioannidis

Pitt Maner writes:

Dr. Ioannidis puts up good argument to criticisms of his paper, which is frequently downloaded. One wonders if you can extend his arguments to economic and business research — are there too many "paper mills" out there?

"Scientific investigation is the noblest pursuit. I think we can improve the respect of the public for researchers by showing how difficult success is. Confidence in the research enterprise is probably undermined primarily when we claim that discoveries are more certain than they really are, and then the public, scientists, and patients suffer the painful refutations."

His credentials appear acceptable, so he may be worth listening to.

Dr. Ioannidis continues to push for better use of and improvements in the use of statistics in medical studies. So on the face of it he appears to fill an important role in maintaining high standards for published medical research. Perhaps those in the profession can give a better assessment of Dr. Ioannidis' methods and the validity of his criticisms. Here is the overview of a paper titled Limits to forecasting in personalized medicine:

Biomedical research is generating massive amounts of information about potential prognostic factors for health and disease. However, few prognostic factors or systems are robustly validated, and still fewer have made a convincing difference in health outcomes or in prolonging life expectancy. For most diseases and outcomes, a considerable component of the prognostic variance remains unknown, and may remain so for the foreseeable future. I discuss here some of the main problems in medical forecasting that pose obstacles to personalized medicine. Their recognition may help identify solutions to improve personalized prognosis, or at least understand and cope with the component of the future that we cannot predict. Much prognostic research is stuck at generating "publishable units", without any interest in conclusively proving their worth, let alone moving them into real life applications. Information is reported selectively and reporting is deficient. The replication record of prognostic claims is poor. Even among replicated prognostic effects, few are convincingly shown to add much information besides what is already known through more simple, traditional measurements. There are few efforts to systematize prognostic knowledge. Most prognostic effects are subtle when traced to the molecular level, where most current research operates. Many researchers, clinicians, and the public are not appropriately educated to interpret prognostic information. We still have not even agreed on what the important health outcomes are that we want to predict and intervene for, and some subjectivity may be unavoidable. Finally, without concomitant effective, affordable, and non-harmful interventions, prognosis alone is of questionable value, and wrong prognosis or a wrong interpretation thereof can be harmful. The identification of these problems also suggests a roadmap on what could be done to amend them. Solutions include a systematic approach to the design, conduct, reporting, replication, and clinical translation of prognostic research; as well as the education of researchers, clinicians, and the general public. Finally, we need to recognize that perfect individualized health forecasting is not a realistic target in the foreseeable future, and we have to live with considerable residual uncertainty.


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search