September 7, 2020
A Look At Mixed Methods Research
By Michael D. Shaw
In last week’s article, which covered some alternative voices on COVID-19, I said that randomized clinical trials have long been considered the gold standard in determining which drugs to bring to market, even though the sad legacy of failed and withdrawn pharmaceuticals should give one some pause. As former CDC director Tom Frieden noted in 2017:
“Despite their strengths, RCTs have substantial limitations. They can be very expensive to run. They can take many years to complete, and even then may not last long enough to assess the long-term effect of an intervention such as vaccine immunity, or to detect rare or long-term adverse effects. Findings from RCTs may not be valid beyond the study population–a trial that included a high-risk population in order to maximize the possibility of detecting an effect, for example, may not be relevant to a low-risk population. RCTs may not be practical for population-wide interventions and often aren’t relevant for urgent health issues such as infectious disease outbreaks, for which public health decisions must be made quickly.”
He goes on to cite the case of the nasal spray flu vaccine. RCTs suggested that this vaccine was superior to flu shots, at least for some populations. However, later results, from observational studies, showed that the spray was not effective, after all. To put it bluntly, the RCTs were an epic failure. Data from the 2015-2016 season showed that the spray only offered protection 3% of the time, as opposed to 63% with the shot.
Perhaps, the fixation with RCTs stems from the focus of the hard sciences to be, well, “hard.” That is, quantitative analysis of a situation is considered to be superior to qualitative observations. It was William Thomson, aka Lord Kelvin who famously said, in part “[W]hen you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”
Sometimes, though, we forget that the beginnings of nearly all scientific research stem from a simple qualitative observation, for which a hypothesis is formed, and is then tested by experiment. Professor John Creswell lays out the advantages and limitations of quantitative and qualitative research…
Quantitative: Advantages—suitable for large groups; efficient data analysis; people like numbers. Limitations—impersonal, dry; little understanding of context of participants; largely researcher driven.
Qualitative: Advantages—obtain detailed perspectives on fewer people and hear their voices; built from views of participants; people like stories. Limitations—Soft data; smaller groups; highly interpretive; minimizes researcher’s expertise.
As it happens, Creswell is a leading proponent and early adopter of Mixed Methods Research. The term is defined in one of his papers as:
“[A]n emergent methodology of research that advances the systematic integration, or ‘mixing,’ of quantitative and qualitative data within a single investigation or sustained program of inquiry. The basic premise of this methodology is that such integration permits a more complete and synergistic utilization of data than do separate quantitative and qualitative data collection and analysis.”
Mixed methods research originated in the social sciences and over the past few years has expanded into the health sciences. While the detailed techniques are beyond the scope of this article, a paper co-authored by Creswell lists some reasons for using mixed methods in health science…
1. Researchers may seek to view problems from multiple perspectives to enhance and enrich the meaning of a singular perspective. They also may want to contextualize information, to take a macro picture of a system (e.g., a hospital) and add in information about individuals (e.g., working at different levels in the hospital).
2. Merging of quantitative and qualitative data can develop a more complete understanding of a problem.
3. Quantitative and qualitative databases can build on one another. When a quantitative phase follows a qualitative phase, the intent of the investigator may be to develop a survey instrument, an intervention, or a program informed by qualitative findings. When the quantitative phase is followed by the qualitative phase, the intent may be to help determine the best participants with which to follow up or to explain the mechanism behind the quantitative results.
Further reasons for using mixed methods are available in this article, recommended by Creswell.
Imagine how much more we could learn about potential drug side effects, for example, if RCT participants were interviewed in a qualitative, open-ended sense, by a real person–rather than solely replying to a paint-by-numbers checklist as would be provided in a typical quantitative-only study.
Here’s a great qualitative question to be answered by the sponsors of the RCT (the pharmaceutical company): How is this drug any better than the current generic? Excluded answer: “Because it’s new and proprietary, and we can charge way more for this one.”