PUBLICATION BIAS UNDERMINES EVIDENCE BASED MEDICINE

                                     Mark Bennett MD, FRCPC 

The bias towards publishing positive results in research is well known, yet despite this nearly half of all clinical trial data goes unpublished.  The consequences are severe; biasing clinicians, harming patients, and affecting health care costs.

Without access to all clinical trial data harmful effects of treatments can go undetected. Examples of this include the COX-2 inhibitors and the diabetes drug rosiglitazone.  In the case of rofecoxib cases of myocardial infarction and causes of death were missing from clinical trial reports, and the association between the drug and heart attacks was undetected, until the drug was eventually pulled from market¹.   

Meta-analysis that included unpublished data examining the cardiac effects of celecoxib also found an association with myocardial infarction². In 2004, a legal settlement required the makers of rosiglitazone to make available all clinical trial results. A meta-analysis with access to this data, showed a significant association with the drug to MI

and cardiovascular death³. 

 

Publication bias misleads clinicians and ultimately their patients by inflating effect size estimates of the treatments they provide. This effects the risk-benefit discussion clinicians have with their patients.   

Turner et al.⁴ demonstrated this in a meta-analysis that accessed all phase 2 and 3 trials for 12 antidepressant agents approved by the FDA between 1987-2004, accessed through Freedom of Information Act requests.  Of these 74 trials, 38 were deemed by the FDA to support the efficacy of the drug.  Of those 37 were published in academic journals.  The remaining 36 trials were deemed negative or questionable by the FDA. Of those 22 were unpublished while the others were published as positive conflicting with the FDA’s assessment.  When effect size estimates from these trials do not include unpublished trial data, they are overestimated⁴.  One might think that trials were unpublished due to methodological flaws, however unpublished trials were not found to differ significantly in terms of sample size or methodology.  Systematic review and meta-analysis were revolutionary in terms of their impact on modern medicine, and they are undermined by publication bias⁴. 

Perhaps the example of publication bias that has gathered the most media attention is that of the anti-viral oseltamivir. Billions of dollars of the drug have been stockpiled by governments across the globe in preparation for influenza outbreaks, generating approximately 18 billions dollars in sales for its maker.  These decisions were made without approximately 60% of the clinical trial data.  

However following a five-year campaign by Cochrane reviewers and the British Medical Journal, the full data set was made available, and the revised meta-analysis suggested the small benefits provided by Tamiflu do not outweight the harms⁵.

Why does publication bias occur?  It’s commonly believed that medical journals are more likely to publish clinical trials if they are positive.  However the largest systematic review of studies investigating publication bias did not find this to be the case⁶.  In a pooled analysis of four prospective studies following clinical trials submitted for publication, the direction of the results did not predict publication⁶. 

 Inception cohort studies however, where trials are followed to publication from their design, found that trials were approximately three times more likely to be published if positive.  This suggests that it is the investigator’s decision to publish, more so than editorial decisions, that leads to publication bias⁶. 

When authors are surveyed regarding their reasons for not submitting articles for publication, most common reasons include, lack of time or low priority, results not seeming important enough, and fear of journal rejection. Finally industry sponsorship has been repeatedly associated with publication bias, with positive industry trial found be three to four times more likely to be published if positive⁶. 

Not only are the consequences of publication bias severe, methods in detection, such as funnel plots, are often ineffective⁷.  Efforts in prevention have focused primarily on clinical trial registries.  Clinicaltrials.gov and similar online clinical trial registries in Europe were created in the early 2000’s. In 2005 the International Committee of Medical Journal Editors (ICMJE) required, in a nonbinding agreement, that all clinical trials should be indexed in a clinical trial registry to qualify for publication in a journal.  

Perhaps, the legislation with the most potential for increasing access to clinical trial data was the FDA Amendment Act (2008).  This legislation required that all clinical trial results be disseminated on clinicaltrials.gov within one year of the studies completion.   However Gopal et al⁸ evaluated the rate of compliance with the act and found only 19.1% of trials had available results in 2007-8 and 10.8% in 2008-9.  A more recent systematic review of interventions against publication bias demonstrated that although the use of prospective trial registration has been increasing, the over representation of positive results in the published literature persists⁹.  

Within psychiatry, Scott et al.¹⁰ reviewed the registry information of all published clinical trials in five journals adhering to the ICMJE agreement, including The American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, The Journal of Child and Adolescent Psychiatry, and the Journal of Clinical Psychiatry; 181 trials in total.  It was found that only 15% of trials were registered prospectively, without any changes to their primary outcome measure. 

Advances in evidence-based medicine, such as systematic review and meta-analysis have shaped clinical practice in dramatic ways.  With new technologies and decision making tools evidence is more accessible to clinicians to inform their patients better.  These tools however rely on access to all the clinical trial data to truly be effective; otherwise they may be leading physicians and their patients astray.

Public awareness of the impact of publication bias is increasing, much in thanks to the work of the AllTrials campaign¹¹, an international organization founded by psychiatrist and epidemiologist Dr. Ben Goldacre, which calls for all clinical trials in humans to be registered and results be reported. Systems are in place to achieve this but monitoring and enforcement are lacking.  There is an urgency to implement these measures to improve the health and safety of our patients. 

References

11.  Topol EJ. Failing the Public Health — Rofecoxib, Merck, and the FDA. N Engl J Med 2004; 351:1707-170.

2 2. Caldwell B, Aldington, S, Weatherall M, Shirtcliff, P, Beassley R. J R Soc Med 2006, 99: 132-140

33.   Nissen SE, Wolski K. N Engl J Med 2007, 356:2457-2471

44.   Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal.  N Engl J Med 2008; 358(3):252-60.

55.   Jefferson T, Jones M, Doshi P, Spencer EA, Onakpoya I, Heneghan CJ BMJ 2014;348:g2545.

66.   Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Dissemination and publication or research finding: an updated related biases. Health Technology Assessment 10 (9); 2010.

77.   Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005.

88.  Gopal RK, Yamashita TE, Prochazka AV. Research without results: Inadequate public reporting of clinical trial results. Contemp Clin Trials 2012; 33 (3):486-491. 

99.   Thaler K, Kien C, Nussbaumer B, Van Noord MG, Griebler U, Klerings I, Gartlehner G. Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol 2015; 68(7):792-802.

110. Scott A, Rucklidge J, Mulder R.  Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PloS One 2015:e0133718.

111. alltrials.net

 

BETWEEN DSM AND RDoC: THE SYNAPSES TO SYSTEMS PARADIGM

ALAN EPPEL

Psychiatry is a dynamic and rapidly evolving medical speciality. It is now at a transition point in its history. The limitations of the DSM and ICD systems have become apparent while at the same time the promise of the neuroscience paradigm has not yet been realised.

DSM

The introduction of  DSM III in 1980 has had a profound impact on the practice of psychiatry. DSM III was an improvement from earlier classifications which consisted of imprecise narrative descriptions mixed in with psychoanalytic concepts.

DSM III was atheoretical adopting a phenomenological Kraepelinian approach based on observable surface features. However an objective-descriptive system has intrinsic limitations. Without an explication of the underlying pathophysiology it lacks explanatory power and construct validity. A criterion based system leads to overlap of syndromes and a lack of sharp boundaries between diagnostic entities. This leads to the artificial  over diagnosis of “ comorbidities which are identified in 35 to 45% of patients. A model which generates such a high rate of false comorbidity creates difficulty both for research and for clinical practice.

Also the reliance on symptoms and signs, has resulted in a narrowing of clinical focus and an overemphasis on psychopharmacology. The interpersonal and sociocultural frameworks became downgraded.

RDoC

The high rates of comorbidity and the heterogeneity within diagnostics undermine validity and have  made research difficult. Consequently over the past seven years the National Institute of Mental Health has developed the Research Domain Criteria Project. RDoC is based on dimensions of observable behavior linked to neurocircuits.

The purpose of RDoC is to build from the ground up based upon research in molecular biology, genomics, brain imaging and neurobiology. NIMH has identified key domains related to fear, approach, motivation, reward, cognitive systems, social processes (including attachment separation and social dominance) arousal and regulatory systems. The fundamental premise of this approach is that psychiatric disorders are disorders of neurocircuits. This is a long-term open-ended project that by its very nature does not propose a unifying heuristic conceptual model.

There are five Domains in the RDoC matrix:

Negative Valence Systems. Positive Valence Systems. Cognitive Systems.

Systems for Social Processes. Arousal and Regulatory Systems.

RDoC is actually not a classification system  but rather an process of research that could lead to a new classification and understanding of psychiatric disorders. RDoC does not have any a priori criteria for defining disorders.

RDoC then is not a system ready for clinical application. It also does not contain any linkages to the broader determinants of health,  dimensions of social and environmental factors, that impact on the development of psychiatric disorders.

SYNAPSES to SYSTEMS

In response to the conceptual limitations of DSM and RDoC  the psychiatry residency program at McMaster University  has introduced a framework known as Synapses to Systems. This consists of a multi-dimensional conceptual model that has application to training and clinical practice.

 

The Synapses to Systems paradigm identifies four levels of discourse: Synapses, Self, Systems, and Society.

These domains are used to understand psychiatric disorders and to guide therapeutic approaches.

Relationship of Synapses to Systems and RDoC

SYNAPSES to SYSTEMS DOMAINS	RDoC	RDoC DOMAINS
SYNAPSES Neurosciences, Neurocircuitry,      Neuroimaging Genomics   Molecular Psych.	Genes Molecules Cells Neurocircuits	Negative & Positive Valence systems Arousal & Regulatory systems
SELF Child Development, Attachment, Psychodynamics	Affiliation & Attachment Perception & Under- standing of Self &others	Systems for Social Processes
SYSTEMS AND SOCIETY Populations, Culture Determinants of Health, Global Mental Health Families, and Social Systems	Social Communication	Systems for Social Processes Cognitive Systems No domains for broader social factors

   

The Synapses to Systems approach is a bridge linking the categorical DSM and dimensional RDoC frameworks. It is designed to conceptualize psychiatric disorders, therapeutic strategies and recovery that better equips trainees and practitioners for the future practice of Psychiatry.