Hope: The Mother of Bias in Research

I realized the other day that underlying every slanted report or overly-optimistic interpretation of a trial's results, every contorted post hoc analysis, every Big Pharma obfuscation, is hope.  And while hope is generally a good, positive emotion, it engenders great bias in the interpretation of medical research.  Consider this NYT article from last month:  "Dashing Hopes, Study Shows Cholesterol Drug Had No Effect on Heart Health."  The title itself reinforces my point, as do several quotes in the article.

“All of us would have put money on it,” said Dr. Peter Libby, a Harvard cardiologist. The drug, he said, “was the great hope.”

 Again, hope is wonderful, but it blinds people to the truth in everyday life and I'm afraid researchers are no more immune to its effects than the laity.  In my estimation, three main categories of hope creep into the evaluation of research and foments bias:

1. Hope for a cure, prevention, or treatment for a disease (on the part of patients, investigators, or both)
2. Hope for career advancement, funding, notoriety, being right (on the part of investigators) and related sunk cost bias
3. Hope for financial gain (usually on the part of Big Pharma and related industrial interests)

Consider prone positioning for ARDS.  For over 20 years, investigators have hoped that prone positioning improves not only oxygenation but also outcomes (mostly mortality).  So is it any wonder that after the most recent trial, in spite of the 4 or 5 previous failed trials, the community enthusiastically declared "success!"  "Prone Positioning works!"  Of course it is no wonder - this has been the hope for decades.

But consider what the most recent trial represents through the lens of replicability:  a failure to replicate previous results showing that prone positioning does not improve mortality.  The recent trial is the outlier.  It is the "false positive" rather than the previous trials being the "false negatives."

This way of interpreting the trials of prone positioning in the aggregate should be an obvious one, and it astonishes me that it took me so long to see the results this way - as a single failure to replicate previously replicable negative results.  But it hearkens to the underlying bias - we view results through the magnifying glass of hope, and it distorts our appraisal of the evidence.

Indeed, I have been accused of being a nihilist because of my views on this blog, which some see as derogating the work of others or an attempt to dash their hopes.  But these critics engage, or wish me to engage in a form of outcome bias - the value of the research lies in the integrity of its design, conduct, analysis, and reporting, not in its results.  One can do superlative research and get negative results, or shoddy research and get positive results.  My goal here is and always has been to judge the research on its merits, regardless of the results or the hopes that impel it.

(Aside:  Cholesterol researchers have a faith or hope in the cholesterol hypothesis - that cholesterol is a causal factor in pathways to cardiovascular outcomes.  Statin data corroborate this, and preliminary PCSK9 inhibitor data do, too.  But how quickly we engage in hopeful confirmation bias!  If cholesterol is a causal factor, it should not matter how you manipulate it - lower the cholesterol, lower cardiovascular events.  The fact that it does appear to matter how you lower it suggests that either there are multiplicity of agent effects (untoward and unknown effects of some agents negate some their beneficial effects in the cholesterol causal pathway) or that cholesterol levels are epiphenomena - markers of the effects of statins and PCSK9 inhibitors on the real, but as yet undelineated causal pathways.  Maybe the fact that we can easily measure cholesterol and that it is associated with outcomes in untreated individuals is a convenient accident of history that led us to trial statins which work in ways that we do not yet understand.)

Enrolling Bad Patients After Good: Sunk Cost Bias and the Meta-Analytic Futility Stopping Rule

Four (relatively) large critical care randomized controlled trials were published early in the NEJM in the last week.  I was excited to blog on them, but then I realized they're all four old news, so there's nothing to blog about.  But alas, the fact that there is no news is the news.

In the last week, we "learned" that more transfusion is not helpful in septic shock, that EGDT (the ARISE trial) is not beneficial in sepsis, that simvastatin (HARP-2 trial) is not beneficial in ARDS, and that parental administration of nutrition is not superior to enteral administration in critical illness.  Any of that sound familiar?

I read the first two articles, then discovered the last two and I said to myself "I'm not reading these."  At first I felt bad about this decision, but then that I realized it is a rational one.  Here's why.

Sell Side Bias and Scientific Stockholm Syndrome: A Report from the Annual Meeting of the American Thoracic Society

What secrets lie inside?

Analysts working on Wall Street are sometimes categorized as working on either the "buy side" or the "sell side" depending on whether their firm is placing orders for stocks (buy side, such as institutional investors for mutual funds) or filling orders for stocks (sell side, which makes commissions on stock trades).  Sell side bias refers to any tendency for the sell side to "push" stocks via overly optimistic ratings and analyses.

Well, I'm at the American Thoracic Society (ATS) meeting in San Diego right now, and it certainly does feel like people - everyone - is trying to sell me something.  From the giant industry sponsored banners, to the emblazoned tote bags, to the bags of propaganda left at my hotel room door every morning, to the exhibitor hall filled with every manner of new and fancy gadgets (but closed to cameras), to the investigators themselves, everybody is trying to convince me to buy (or prescribe) something.  Especially ideas.  Investigators have a promotional interest in their ideas.  And they want you and me to buy into their ideas.  I have become convinced that investigators without industry ties (that dying breed) are just about as susceptible to sell side bias as those with industry ties.  Indeed, I have also noted that the potential consumer of many of the ideas himself seems biased - he wants things to work, too, and he has a ready explanation for why some ideas didn't pan out in the data (see below).  It's like an epidemic of scientific Stockholm Syndrome.

The first session I attended was a synopsis of the SAILS trial by the ARDSnet investigators, testing whether use of a statin, rosuvastatin, in patients with sepsis-incited lung injury would influence 60 day mortality.  The basis of this trial was formed by observational associations that patients on statins had better outcomes in this, that, and the other thing, including sepsis.  If you are not already aware of the results, guess whether rosuvastatin was beneficial in this study.

Billions and Billions of People on Statins? Damn the Torpedos and Full Speed Ahead

Absolutely Relative
Risk is in the Mind of the Taker

Among the many editorials providing background and backlash about the new cholesterol guidelines is this one:  More Than a Billion People Taking Statins? by John Ioannidis, which echoes the worries of others that the result of the guidelines (which changed the 10-year risk threshold for treatment from 10% to 7.5%) may be that many more people (billions and billions?) will be prescribed statins.  But the title is a curious one - if statins are beneficial, should we lament their widespread prescription and adoption or is it just unfortunate that heart disease is so prevalent? Whose side are we on, the cure or the disease?

Are the premises of the guidelines flawed leading to flawed extrapolations, or are the premises correct and we just don't like the implications?  Let's look at the premises - because if they're flawed, we may find that other premises we have accepted are flawed.

The Cardiologist Giveth, then the Cardiologist Taketh Away: Revision of the Cholesterol Guidelines

There has been quite a stir this week with the publication of the newest revision of the ACC/AHA guidelines for the treatment of cholesterol.  The New York Times is awash with articles summarizing or opining on the changes and many of the authors are perspicacious observers:

• 3 Things to Know About the New Cholesterol Guidelines by Harlan Krumholz of Yale
• Don’t Give More Patients Statins by authors from Harvard and UCSF
• Experts Reshape Treatment Guide for Cholesterol
• New Cholesterol Advice Startles Even Some Doctors
• Questions For A New Class of Cholesterol Drugs

As the old Spanish proverb states, "rio revuelto, ganancia de pescadores" - when the river is stirred up, the fishermen benefit.  I will admit that I'm gloating a bit since I consider the new guidelines to be a tacit affirmative nod to several posts on the topic of the cholesterol hypothesis (CH).  (More posts here and here and here, among several others - search for "cholesterol" or "causal pathways" on the Medical Evidence Blog search bar.)

The Cholesterol Hypothesis on the Beam: Dalcetrapib, PCSK9 inhibitors, and "off-target" effects of statins

The last month has witnessed the publication of three lines of research that could tip the balance of the evidence for the cholesterol hypothesis depending how things play out.  Followers of this blog know that I have a healthy degree of skepticism for the cholesterol hypothesis which was emboldened by studies of torcetrapib (blogged here and here) and anacetrapib that have come to light along with the failures of vytorin (ezetimibe; blogged here and here and here) and the addition of niacin to statins to improve cardiovascular outcomes in parallel with improvements in cholesterol numbers.

I think it's finally time to bury the CETP inhibitors. The November 29th NEJM (published online on November 5th) reports the results of the dal-OUTCOMES trial of dalcetrapib in patients with a recent acute coronary syndrome. Almost 16,000 patients were enrolled in this study of high risk patients, providing the study with ample power to detect meaningful improvements in cardiovascular outcomes - but alas, none were detected. The target is HDL, so the LDL hypothesis is not debunked by these data, but I think it is challenged nonetheless.