American Journal of Ophthalmology
Volume 146, Issue 5 , Pages 638-640, November 2008

Not for Your Eyes: Information Concealed through Publication Bias

Editorial Offices of the american journal of ophthalmology, archives of ophthalmology, and ophthalmology

Accepted 22 July 2008.

Article Outline

 

Peer-reviewed medical journals strive to present new information for their readers' consideration and judgment—information that will guide their medical practice and will benefit their patients. Vigorous efforts are made by the editors to control and minimize factors that may inhibit or impair the impartiality of the data and the conclusions they warrant. Toward the end, potential conflicts of interest on the part of the authors are revealed, the quality of data is examined, and control groups are required. In clinical trials, when possible, actual treatment and sham or placebo treatments are masked. These practices are carried out to eliminate or minimize the possibility of bias.

We want to address here three less obvious, but important, forms of bias: 1) submission bias, 2) methodologic bias, and 3) acceptance bias. These are key among the various forms of publication bias that can limit or distort the interpretation of journal contents for clinical care.1, 2 The real world of research, funded by government grants, foundations, private industry and subsequently scrutinized by institutional review boards, is far different than the world of research reflected in reports in the journals. A black hole exists between the real world of clinical investigation and the world of published studies, and into that hole disappear those studies with “insignificant” results; those studies with strong positive results are submitted and published preferentially, whereas those with negative or inconclusive results are often filed away, unsubmitted or when submitted, are rejected, and remain hidden from public view.3, 4, 5

Both investigators and journal editors share responsibility for submission bias. But study sponsors, frequently pharmaceutical companies, often are the culprits for data suppression and methodologic flaws or bias.6 Methodologic bias refers to poor study design and faulty execution of studies.1 Two broad classes of methodologic bias are selection bias and observation bias, both of which are (sometimes intentional) systematic errors in studies that cause or encourage one outcome over another.1, 7 Individual analyses and meta-analyses consistently have found a significant association between industry sponsorship and proindustry conclusions.8, 9, 10, 11 We agree with Lesser and associates that there is a natural tendency for industry preferentially to fund those studies that favor their products or that disparage their competitors' products and exclude products in which the company has no economic interest.12 Investigators are encouraged to formulate hypotheses, to design studies, or to analyze data in ways that are consistent with the financial interests of sponsoring companies.12 It has been demonstrated that authors' financial competing interests may bias the conclusions of clinical trials and that the bias typically favors the sponsors funding the studies.12, 13

Sponsors frequently have the capability of obstructing, delaying, or completely preventing publication of trials or can impose practical or legal obstacles to publication, at times with the acquiescence of United States medical schools, in violation of the International Committee of Medical Journal Editors guidelines.14 Consequently, pharmaceutical company sponsorship of economic analyses of new drugs has been associated with a reduced likelihood of reporting unfavorable results.15 Research funded by drug companies is less likely to be published than research funded by other sources; in fact, approximately half of all completed trials go unpublished.3, 10, 16 And, as the public media have reported, industry and investigators have been shown to delay or not to publish findings that have negative implications for the sponsor's product. Another form of methodologic bias seen in industry-sponsored studies is the selective reporting of favorable outcomes. Moreover, the reporting of trial outcomes often is not only incomplete but may be distorted and inconsistent with protocols.17

Acceptance bias can be defined as the tendency of reviewers and editors excessively to favor and accept manuscripts on the basis of the so-called direction or strength of study results.18 Regarding direction, studies with positive results clearly are more likely to be published than those with negative or inconclusive results.18, 19, 20 Stated otherwise, studies showing high efficacy are more likely to be reported than those in which the observed efficacy is average or poor.21 Publication bias based on the strength of the findings refers to the favoring of studies with strong statistical significance. Conversely, studies lacking statistical significance are published less frequently.4, 22, 23, 24 It is heartening, however, that little evidence can be found that publication bias occurs when high-quality manuscripts are submitted to the best medical journals. It seems in these cases that the chances of being published are not significantly different for trials with positive results and trials with negative results.4

The danger with submission bias, methodologic bias, or acceptance bias, of course, is that the literature becomes skewed toward studies with positive outcomes or with strong statistical results. This in turn leads to overestimation, or at least misrepresentation, of the efficacy or the absence of adverse effects of a diagnostic or therapeutic method and can result in a failure to institute proper clinical guidelines.2, 7

Subsequent scientific reviews, especially by industry-selected authors, do not remedy publication bias because the search and interpretation may be selective.12 Studies suggest that readers should interpret industry-supported reviews with caution and should rely on, for example, the more transparent Cochrane reviews.25 Meta-analysis or systematic analysis of multiple small trials only amplifies the bias, because only the reported (positive) trials are included.17

Publication bias is a recognized economic and health policy issue. The Health Technology Assessment Program of the National Health Service recently published a 115-page monograph devoted entirely to exploring the implications of publication bias.26 Unreported clinical studies essentially become lost data and represent wasted resources. An ethical imperative is squandered as well because there is typically a hint to research subjects that “they may help others by participating.” If the knowledge is not shared, that help is not imparted. The mandatory clinical trial registration guidelines for data registration in a centralized searchable database in a publicly available website6 helps to address publication bias, and most reliable journals will not publish results from unregistered studies. To improve transparency, access to the protocols also should be made available, such as those published by the Cochrane Library. Further, data monitoring committees should be completely independent. Journals should request a disclosure of both the authors' roles and the sponsor's role in the study and should affirm that the principle author had access to all the data and controlled the decision to publish. Deviations from trial protocols should be described in the published articles to assess the potential for bias.17 Journal editors should require that original protocols and subsequent amendments be submitted with the manuscript or at least be available for the benefit of the peer-reviewers17; admittedly, proprietary issues need to be considered.

Clinical trial registration, however, does not ensure submission of trial results. Neither does publication ensure that the research has been conducted with accuracy and objectivity and with a meaningful hypothesis. Details regarding interventions that fail in clinical trials, however, should be available publicly. The responsibility to submit these studies for publication lies not only with the investigators, but also with research ethics committees (institutional review boards) and funding bodies.

In summary, institutional review board-approved studies should not be buried when the results are indecisive or negative, because all resulting information is important if the study has been carried out properly.12 Editors must guard against basing the decision to publish on the significance of a study's results. Rather, they should prioritize manuscripts on the basis of the clinical question addressed, the quality of the research methods, and findings that will impact subsequent treatment. In fact, the inconclusive or negative studies provide prospective and balance against the seductive power of positive data in the literature. These steps will assure both editors and readers that the aggregate information presented is accurate and reliable and will enable journals to reflect the real world of research.

Back to Article Outline

 

The authors indicate no financial support or financial conflict of interest. All of the authors were involved in design and conduct of study; data collection; analysis and interpretation of data; and preparation, review and approval of the manuscript.

Back to Article Outline

References 

  1. Sica GT. Bias in research studies. Radiology. 2006;238:780–789
  2. DeMaria AN. Publication bias and journals as policemen. J Am Coll Cardiol. 2004;44:1707–1708
  3. Veitch E. Tackling publication bias in clinical trial reporting—PLoS announces the launch of a new online journal. PLoS Med. 2005;2:e367
  4. Olson CM, Rennie D, Cook D, et al. Publication bias in editorial decision making. JAMA. 2002;287:2825–2828
  5. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results (Follow-up of applications submitted to two institutional review boards). JAMA. 1992;267:374–378
  6. Abaid LN, Grimes DA, Schulz KF. Reducing publication bias through trial registration. Obstet Gynecol. 2007;109:1434–1437
  7. Gluud LL. Bias in clinical intervention research. Am J Epidemiol. 2006;163:493–501
  8. Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: A systematic review. JAMA. 2003;289:454–465
  9. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?. JAMA. 2003;290:921–928
  10. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170
  11. Liss H. Publication bias in the pulmonary/allergy literature: effect of pharmaceutical company sponsorship. Isr Med Assoc J. 2006;8:451–454
  12. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Med. 2007;4:e5
  13. Kjaergard LL, Als-Nielsen B. Association between competing interests and authors' conclusions: epidemiological study of randomized clinical trials. BMJ. 2002;325:249
  14. Gøtzsche PC, Hróbjartsson A, Johansen HK, Haahr MT, Altman DG, Chan AW. Constraints on publication rights in industry-initiated clinical trials. JAMA. 2006;295:1645–1646
  15. Friedberg M, Saffran B, Sinson TJ, Nelson W, Bennett C. Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA. 1999;282:1453–1457
  16. Dickersin K. How important is publication bias? (A synthesis of available data). AIDS Educ Prev. 1997;9:15–21
  17. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DA. Empirical evidence for selective reporting of outcomes in randomized trials comparison of protocols to published articles. JAMA. 2004;291:2457–2465
  18. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385–1389
  19. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998;280:254–257
  20. Callaham M, Wears RL, Weber EJ. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002;287:2847–2850
  21. Begg CB. A measure to aid in the interpretation of published clinical trials. Stat Med. 1985;4:1–9
  22. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significances. Am Stat Assoc J. 1959;54:30–34
  23. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867–872
  24. Harris IA, Mourad MS, Kadir A, Solomon MJ, Young JM. Publication bias in papers presented to the Australian Orthopaedic Association Annual Scientific Meeting. Aust N Z J Surg. 2006;76:427–431
  25. Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006;333:782
  26. Song F, Eastwood AJ, Gillbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess. 2000;4:1–115

PII: S0002-9394(08)00586-2

doi:10.1016/j.ajo.2008.07.034

American Journal of Ophthalmology
Volume 146, Issue 5 , Pages 638-640, November 2008

Article Tools