A Visual Ranking of Market Research Methods In Terms of Risk of Error and Bias
Not all research evidence is the same. It has long been acknowledged that not all research designs are equal in terms of the risk of error and bias. Some research methods provide better evidence than that provided by other methods. That is, the validity of the results of research varies due to the different methods used. The randomized-controlled trial (RCT) is considered the most reliable evidence in evidence-based practice and allows for the determining cause-and-effect. In the medical research world, the meta-analysis is a common statistical approach that collates the results of multiple studies of the same design (e.g. RCT) on a specific topic. Thus, the systematic review and meta-analysis of RCTs is considered the highest level of evidence in evidence-based decision making.
An evidence pyramid visually depicts the strength of different experimental designs (Figure 1). While there are many renderings of evidence-based pyramids, studies with the highest internal validity, characterized by the strength of experimental studies, quantitative analysis and stringent scientific methodology are at the top of the pyramid. These include meta-analyses, systematic reviews, and randomized controlled trials. Whereas at the base of the pyramid, the least reliable evidence come from ideas, opinions, anecdotes, and observations.
To date, there is no agreed upon hierarchy of evidence for consumer/market research, although we can learn from the foregoing medical model to begin to develop a hierarchy of evidence. Figure 2 shows our proposed hierarchy of evidence for common experimental approaches used in consumer and marketing research. It is worth noting that while every experimental design and/or approach has its strengths and limitations, Rule-developing experimentation (RDE), multivariate & A/B Testing, and conjoint analysis provide the basis for the highest level of evidence in consumer research decision making.
Let’s start with RDE. RDE is the underlying experimental design powering MindCart AI’s pattern analysis software, which overcomes multiple interlinked statistical problems that affect many other traditional consumer research approaches, thus leading to more reliable and targeted results in practice. RDE facilitates individual respondents’ models based on unique designs, thus allowing for pattern-based segmentation (not traditional demographic segmentation because who you are does not necessarily determine how you think). The approach also allows for the detection of any and all interactions between the elements (features) of the experiments, thus increasing the reliability. RDE has been utilized in many practical applications, such as for message optimization, early stage new product development, advertising, package and website optimization.
