Underperforming the Market: Why Researchers are Worse than Professional Stock Pickers and A Way Out

I was reading in the NYT yesterday a story about Warren Buffet and how the Oracle of Omaha has trailed the S&P 500 for four of the last five years.  It was based on an analysis done by a statistician who runs a blog called Statistical Ideas, which has a post on p-values that links to this Nature article a couple of months back that describes how we can be misled by P-values.  And all of this got me thinking.

We have a dual problem in medical research:  a.)  of conceiving alternative hypotheses which cannot be confirmed in large trials free of bias;  and b.) not being able to replicate the findings of positive trials.  What are the reasons for this?

Sepsis Bungles: The Lessons of Early Goal Directed Therapy

On March 18th, the NEJM published early online three original trials of therapies for the critically ill that will serve as fodder for several posts.  Here, I focus on the ProCESS trial of protocol guided therapy for early septic shock.  This trial is in essence a multicenter version of the landmark 2001 trial of Early Goal Directed Therapy (EGDT) for severe sepsis by Rivers et al.  That trial showed a stunning 16% absolute reduction in mortality in sepsis attributed to the use of a protocol based on physiological goals for hemodynamic management.  That absolute reduction in mortality is perhaps the largest for any therapy in critical care medicine.  If such a reduction were confirmed, it would make EGDT the single most important therapy in the field.  If such reduction cannot be confirmed, there are several reasons why the Rivers results may have been misleading:

• As I have blogged in the case of intensive insulin therapy, Single center studies inflate treatment effects when compared to multicenter studies for reasons that are unclear, but which may be related to bias especially in unblinded studies.  (The revelation that Rivers was an investor in one of the devices used in the trial raised additional concerns about bias.)
• Regression to the mean may lead to reduced effect sizes when trials are repeated, especially when the index trial has a very large effect size.  In a similar vein, since large absolute mortality reductions are statistically unlikely in critical care medicine, Bayesian inference means that trials reporting large reductions are likely to represent type I statistical errors.

There were other concerns about the Rivers study and how it was later incorporated into practice, but I won't belabor them here.  The ProCESS trial randomized about 1350 patients among three groups, one simulating the original Rivers protocol, one to a modified Rivers protocol, and one representing "standard care" that is, care directed by the treating physician without a protocol.  The study had 80% power to demonstrate a mortality reduction of 6-7%.  Before you read further, please wager, will the trial show any statistically significant differences in outcome that favor EGDT or protocolized care?

Goldilocks Meets Walter White in the ICU: Finding the Temperature (for Sepsis and Meningitis) that's Just Right

In the Point/Counterpoint  section of the October issue of Chest, two pairs of authors spar over whether fever should be controlled in sepsis by either pharmacological or external means.  Readers of this blog may recall this post wherein I critically appraised the Schortgen article on external cooling in septic shock that was in AJRCCM last year.  Apparently that article made a more favorable impression on some practitioners than it did on me, as the proponents of cooling in the Chest piece hang their hats on this article (and their ability to apply physiological principles to medical therapeutics).  (My gripes with the Schortgen study were many, including a primary endpoint that was of little value, cherrypicking the timing of the secondary mortality endpoint, and the lack of any biological precedent for manipulation of body temperature improving mortality in any disease.)

Reading the Point and Counterpoint piece (in addition to an online first article in JAMA describing a trial of induced hypothermia in severe bacterial meningitis - more on that later) allowed me to synthesize some ideas about the epistemology (and psychology) of medical evidence and its evaluation that I have been tossing about in my head for a while.  Both the proponent pair and the opponent pair of authors give some background physiological reasoning as to why fever may be, by turns, beneficial and detrimental in sepsis.  The difference, and I think this is typical, is that the proponents of fever reduction:  a.) seem much more smitten by their presumed understanding of the underlying physiology of sepsis and the febrile response; b.) focus more on minutiae of that physiology; c.) fail to temper their faith in application of physiological principles with the empirical data; and d.) grope for subtle signals in the empirical data that appear to rescue the sinking hypothesis.

Why Most Clinical Trials Fail: The Case of Eritoran and Immunomodulatory Therapies for Sepsis

The experimenter's view of the trees.

The ACCESS trial of eritoran in the March 20, 2013 issue of JAMA can serve as a springboard to consider why every biological and immunomodulatory therapy for sepsis has failed during the last 30 years.  Why, in spite of extensive efforts spanning several decades have we failed to find a therapy that favorably influences the course of sepsis?  More generally, why do most clinical trials, when free from bias, fail to show benefit of the therapies tested?

For a therapeutic agent to improve outcomes in a given disease, say sepsis, a fundamental and paramount precondition must be met:  the agent/therapy must interfere with part of the causal pathway to the outcome of interest.  Even if this precondition is met, the agent may not influence the outcome favorably for several reasons:

• Causal pathway redundancy:  redundancy in causal pathways may mitigate the agent's effects on the downstream outcome of interest - blocking one intermediary fails because another pathway remains active
• Causal factor redundancy:  the factor affected by the agent has both beneficial and untoward effects in different causal pathways - that is, the agent's toxic effects may outweigh/counteract its beneficial ones through different pathways
• Time dependency of the causal pathway:  the agent interferes with a factor in the causal pathway that is time dependent and thus the timing of administration is crucial for expression of the agent's effects
• Multiplicity of agent effects:  the agent has multiple effects on multiple pathways - e.g., HMG-CoA reductase inhibitors both lower LDL cholesterol and have anti-inflammatory effects.  In this case, the agent may influence the outcome favorably, but it's a trick of nature - it's doing so via a different mechanism than the one you think it is.