RCT Autopsy: The Differential Diagnosis of a Negative Trial

At many institutions, Journal Clubs meet to dissect a trial after its results are published to look for flaws, biases, shortcomings, limitations.  Beyond the dissemination of the informational content of the articles that are reviewed, Journal Clubs serve as a reiteration and extension of the limitations part of the article discussion.  Unless they result in a letter to the editor, or a new peer-reviewed article about the limitations of the trial that was discussed, the debates of Journal Club begin a headlong recession into obscurity soon after the meeting adjourns.

The proliferation and popularity of online media has led to what amounts to a real-time, longitudinally documented Journal Club.  Named “post-publication peer review” (PPPR), it consists of blog posts, podcasts and videocasts, comments on research journal websites, remarks on online media outlets, and websites dedicated specifically to PPPR.  Like a traditional Journal Club, PPPR seeks to redress any deficiencies in the traditional peer review process that lead to shortcomings or errors in the reporting or interpretation of a research study.

PPPR following publication of a “positive” trial, that is one where the authors conclude that their a priori criteria for rejecting the null hypothesis were met, is oftentimes directed at the identification of a host of biases in the design, conduct, and analysis of the trial that may have led to a “false positive” trial.  False positive trials are those in which either a type I error has occurred (the null hypothesis was rejected even though it is true and no difference between groups exists), or the structure of the experiment was biased in such a way as that the experiment and its statistics cannot be informative.  The biases that cause structural problems in a trial are manifold, and I may attempt to delineate them at some point in the future.  Because it is a simpler task, I will here attempt to list a differential diagnosis that people may use in PPPRs of “negative” trials.

Coffee Drinking, Mortality, and Prespecified Falsification Endpoints

A few months back, the NEJM published this letter in response to an article by Freedman et al in the May 17, 2012 NEJM reporting an association between coffee drinking and reduced mortality found in a large observational dataset.  In a nutshell, the letter said that there was no biological plausibility for mortality reductions resulting from coffee drinking so the results were probably due to residual confounding, and that reductions in mortality in almost all categories (see Figure 1 of the index article) including accidents and injuries made the results dubious at best.  The positive result in the accidents and injuries category was in essence a failed negative control in the observational study.

Last week in the January 16th issue of JAMA Prasad and Jena operationally formalized this idea of negative controls for observational studies, especially in light of Ioannidis' call for a registry of observational studies.  They recommend that investigators mining databases establish a priori hypotheses that ought to turn out negative because they are biologically implausible.  These hypotheses can therefore serve as negative controls for the observational associations of interest, the ones that the authors want to be positive.  In essence, they recommend that the approach to observational data become more scientific.  At the most rudimentary end of the dataset analysis spectrum, investigators just mine the data to see what interesting associations they can find.  In the middle of the spectrum, investigators have a specific question that they wish to answer (usually in the affirmative), and they leverage a database to try to answer that question.  Prasad and Jena are suggesting going a step further towards the ideal end of the spectrum:  to specify both positive and negative associations that should be expected in a more holistic assessment of the ability of the dataset to answer the question of interest.  (If an investigator were looking to rule out an association rather than to find one, s/he could use a positive control rather than a negative one [a falsification end point] to establish the database's ability to confirm expected differences.)

I think that they are correct in noting that the burgeoning availability of large databases (of almost anything) and the ease with which they can be analyzed poses some problems for interpretation of results.  Registering observational studies and assigning prespecified falsification end points should go a long way towards reducing incorrect causal inferences and false associations.

I wish I had thought of that.

Added 3/3/2013 - I just realized that another recent study of dubious veracity had some inadvertent unspecified falsification endpoints, which nonetheless cast doubt on the results.  I blogged about it here:  Multivitamins caused epistaxis and reduced hematuria in male physicians.

Troponin Predicts Outcome in Heart Failure - But So What?

In today's NEJM, Peacock and others (http://content.nejm.org/cgi/content/short/358/20/2117 ) report that cardiac troponin is STATISTICALLY associated with hospital mortality in patients with acute decompensated heart failure, and that this association is independent of other predictive variables. Let us assume that we take the results for granted, and that this is an internally and externally valid study with little discernible bias.

In the first paragraph of the discussion, the authors state that "These results suggest that measurement of troponin adds important prognostic information to the initial evaluation of patients with acute decompensated heart failure and should be considered as part of an early assessment of risk."

Really?


The mortality in patients in the lowest quartile of troponin I was 2.0% and that in the highest quartile was 5.3%. If we make the common mistake of comparing things on a relative scale, this is in an impressive difference - in excess of a twofold increase in mortality. But that is like saying that I saved 50% off the price of a Hershey Kiss which costs 5 cents - so I saved 3 cents! As we approach zero, smaller and smaller absolute differences can appear impressive on a relative scale. But health should not be appraised that way. If you are "buying" something, be it health or some other commodity, you shouldn't care about your relative return on your investment, only the absolute return. You have after all, only some absolute quantity of money. Charlie (from the Chocolate Factory) may find 3 cents to be meaningful, but we are not here talking about getting a 3% reduction in mortality - we are talking about predicting for Charlie whether he will have to pay $0.05 for his kiss or $0.02 for it, and even if our prediction is accurate, we do not know how to help him get the discounted kiss - he's either lucky or he's not.

Imagine that you are a patient hospitalized for acute decompensated heart failure. Does it matter to you if your physician comes to you carrying triumphantly the results of your troponin I test and informs you that because it is low, your mortality is 2% rather than 5%? It probably matters very little. It matters even less if your physician is not going to do anything differently given the results of that test. Two percent, 5 percent, it doesn't matter if it can't be changed.

Then there is the cost associated with this test. My hospital charges on the order of $200 for this test. Consider the opportunity costs - what else could that $200 be spent on, in the care of American patients, and perhaps even more importantly in the context of global health and economics? Also consider the value of the test to a patient who might have to pay out of pocket for it - is it worth $200 to discriminate within an in-hospital mortality range of 2-5%?

This study, while meticulously conducted and reported, underscores the important distinction between statistical significance and clinical significance. With the aid of a ginormous patient registry, the authors clearly demonstrated a statistically significant result that is at least mildly interesting from a biological perspective (is it interesting that a failing heart spills some of its contents into the blodstream and that they can be detected by a highly sensitive assay?) But the clinical significance of the findings appears to be negligible, and I worry that this report will encourate the already rampant mindless use of this expensive test which, outside of the context of clinical pre-test probabilities, already serves to misguide care and run up healthcare costs in a substantial proportion of the patients in whom it is ordered.