The Study of the Week Looks Back and Asks One Question
What is going on with medical science?
I am worried about the state of medical science and our journals.
At the risk of sounding naive, I want to believe that people who choose academic medicine do so to answer important scientific questions.
I’ve traveled the world and met academics. Many of them impress me as serious people trying to advance the field.
But that observation is hard to square with some of the studies that make it into big journals—and then into the lay press.
Today let’s do a bit of meta-analysis of recent examples.
Example 1:
Last week, Professor James McCormack described a subgroup analysis of a trial studying a new cholesterol-lowering drug called bempedoic acid.
A subgroup analysis of a trial involves comparing two types of patients within the main trial group—for instance, diabetic/non-diabetic, male/female, older than 55/younger than 55, etc.
JAMA published the subgroup analysis of primary/secondary prevention patients. But. But. They only presented the data for the primary prevention arm—which happened to have the stronger results.
Here is the thing: everyone knew this was a problem. The authors of the paper knew it. The peer reviewers and editors and statisticians also knew it. But they did it anyways. And more than 80 media outlets covered the study. One small caveat: JAMA editors at least allowed a critical editorial.
Example 2
I wrote yesterday about the MONITOR HF trial of an expensive device placed in the pulmonary artery to help manage patients with heart failure.
The trial was designed without a proper placebo arm. As an open-label trial, one group knew that they got the procedure and fancy device and the other group knew that they did not. The primary endpoint was subjective: how patients score their health on a questionnaire.
Here is the thing: everyone knew this was wrong. The authors knew that a subjective endpoint required proper blinding. The editors and peer reviewers also knew it. But there it is, published in the top journal, The Lancet.
Along with that fatally flawed study, was a glowing editorial from an author who declared strong financial relationships with the company that makes the device.
Example 3
This winter, I wrote about the ATLAS trial comparing the subcutaneous ICD to a regular transvenous ICD. The S-ICD is a more expensive device that is heavily marketed to doctors in my field.
Get this. To compare these two devices, the authors chose an endpoint with 5 bad outcomes from the implant. But. But. Four of the five endpoints could only occur in the transvenous ICD arm.
When the positive results came out, so did the marketing materials.
Here is the thing: the authors of this study come from one of the world’s top research organizations. They knew. The editors and peer reviewers of the Annals of Internal Medicine also knew the fatal design flaw. But there it is, published in a prominent journal.
Example 4
Of all journals, we expect the utmost from the New England Journal of Medicine. They hold the loftiest reputation. One reason is that they rarely publish observational studies that attempt to make causal claims.
But in November of 2022, the NEJM published an observational comparison of masking in schools. This was an attempt to compare—in retrospect without randomization—two groups of individuals. I have written often about the problems with this technique, namely, without randomization, you can’t tell if the groups were similar.
Here at Sensible Medicine, Tracy Beth Hoeg, MD, PhD, offered a point-by-point rebuttal of the limitations of this paper. Instead of this sort of critical appraisal, NEJM published a supportive editorial.
In this case, it is possible that the authors actually thought their analysis was correct. But surely the neutral scientists at NEJM knew otherwise. And if so why did they publish such a flawed analysis?
I could go on with more examples. Let me close with two big thoughts.
First is that NEJM, JAMA, The Lancet and the Annals of Internal Medicine are the top journals in all of medicine.
If such biased and flawed papers can not only reach this level, but do so with outsized conclusions and glowing editorials, what are consumers of medical evidence to think of the adjudication process of science?
We all agree that someone or some process has to judge science. But if this can happen, how can we trust all the other papers in these journals? The system seems broken.
The second thought is worse.
In most (if not all) of these examples, the authors knew the fatal flaw in their study but ignored it.
I may be wrong; I hope I am wrong, but if they knew the design and analysis was flawed and passed it off anyways, then…whoa. I don’t want to go there. It makes me extremely sad.
I am happy for anyone in the comments to help me feel better about our current climate of medical evidence. Or suggest a better way.
I would also urge you to think past pure financial conflicts of interest. Money is obviously a driver of conflicts, but it can’t be the only issue.
As an Engineer I have frequently been appalled by the poor levels of evidence, weak analysis, and over the top promotion of medical interventions. I used to hold my medical colleagues in high esteem and rarely checked their work. My journey began with dietary claims when I discovered the poor or outright false statements on that subject. During the pandemic I began reading medical papers. I was stunned by the obvious attempts to denigrate somewhat effective and cheap treatments for the virus. I was still deluded in thinking that the vaccines had been properly tested. But over the intervening months I have learned just how artificial and erroneous the initial reports were. I also discovered how mendacious the authors were. They knew that they were lying about effectiveness and were definitely obscuring the abysmal safety profile of these vaccines. I no longer trust the medical community in general. I trust a few doctors who I know to be honest. But now I check everything.
John, Sensible Medicine is on a roll. I have published many papers -- I know you have as well. I have been an editor (still am). But the real problem is with your final sentence. In almost all of these situations money is the ONLY issue THAT MATTERS...the only other one is some variation of POWER.
The entire coverup of the covid debacle (the NPIs uniformly did not work as described [or work at all], the spikeshots were woefully undertested and mostly given to those who demonstrably had more harm than good taking them, etc.) showed better than at least I had realized how corrupt the entire process had become. I could give you many examples as egregious as the ones you have noted -- while important and accurate articles are being squelched from any kind of circulation (such as the McCullough et al article up for 24 hours last week on the Lancet preprint server -- then mysteriously pulled down for, literally, no reason). And it is not just covid. Read this article from today in the Free Press about a similar article made to disappear because it was against the narrative -- in this case about gender. The article is solid -- but the truth doesn't matter...only the narrative. https://substack.com/inbox/post/134179542
I have lifetime subscriptions to NEJM and others -- I have stopped reading them because I have stopped believing there is truth in there. Marcia Angell pointed this out more than a decade ago. We all should have listened. And yes, it is sad...devastatingly so. We are now dependent on known-flawed papers that deliver the desired results rather than the true ones. And if there are papers that are correct, we will suspect those, too.
A disaster, the implications of which we have yet to begin to fully realize -- but if the covid debacle is a hint, we are in for massive increases in morbidity, mortality, and pharma spend (yes...related).