This website uses cookies to store information on your computer. Some of these cookies are used for visitor analysis, others are essential to making our site function properly and improve the user experience. By using this site, you consent to the placement of these cookies. Click Accept to consent and dismiss this message or Deny to leave this website. Read our Privacy Statement for more.
Qual Power
Blog Home All Blogs
Search all posts for:   

 

View all (79) posts »

Practical Messiness Masked by the Qualitative and Quantitative Distinction

Posted By Aimée Caffrey , Tuesday, December 10, 2019
Updated: Monday, December 9, 2019

Practical Messiness Masked by the Qualitative and Quantitative Distinction

By: Aimée Caffrey 

This blog post discusses the practical messiness that can be masked by the qualitative/quantitative distinction and offers an approach for thinking about and dealing with that messiness.

Like many anthropologists, I have an abiding interest in the ways in which people construct and reproduce boundaries. During my doctoral work, my primary focus was on boundaries such as ethnicity, caste, and nationality. The professional path I have taken in more recent years has in part shifted my attention toward boundaries of another variety—the boundaries that demarcate scientific knowledge practices in industry, and toward a particular boundary with which the readers of this blog are already quite familiar—that between quantitative and qualitative. In my present role, I conduct and help support research that by most definitions would count as qualitative. At the same time, this work almost always feeds into, or follows on the heels of, research that by most definitions is quantitative. It might entail using IDIs, focus groups, or journaling exercises to better understand terminology or relevant dimensions of experience prior to writing a survey. At the other end of things, it might entail using these data collection formats in an effort to make sense of survey findings—when we have discovered the what but are uncertain of the why.

Working at this intersection instills a perhaps exaggerated awareness of, and sensitivity to, the risks of accepting the quant/qual boundary at face value. Like others of its type, this distinction is a productive shorthand for organizing and talking about a variety of practices; however, it can mask the messiness of reality. A very experienced industry researcher gestured toward this messiness on a recent L&E webinar when he remarked on the "under-powered quant" that can be at work when focus group moderators ask for a show of hands. Alternatively, consider that many of what are generally marketed as mobile ethnography or online qual tools often contain what we otherwise think of as quantitative question types (e.g., multiple choice). To offer another example, I regularly assist fellow researchers with the development of interview and focus group discussion guides, and often this assistance centers in part on rephrasing "how much" (i.e., quantitative) kinds of questions to help us make sure we are in fact collecting qualitative data.

These examples of the messiness relate to a tension between the method deployed and the data gathered. When we think of the boundary between qualitative and quantitative as pertaining to a (reporting) distinction between numbers and words, the lines are similarly blurred—we discover the use of stories and images to help explain the findings of quantitative analysis and the use of quantitative adjectives to convey insights from qualitative analysis. This isn't terribly surprising: If there is "terror in numbers," as Darrell Huff wrote in How to Lie with Statistics, the tensions and nuances at the very human heart of qualitative data can also induce discomfort. But, just as the pictures (e.g., graphs) we draw to quell the disquietude of quant can exaggerate the story that the numbers tell, so too can the words we use to describe our qualitative findings be misleading. What is more important than policing the qualitative/quantitative boundary? It is being watchful for what the messiness around that boundary might signal—that there is a misalignment somewhere among the objectives in mind, the method deployed, the data gathered, and ultimately, the claims that are made.

There may be justifiable and even good reasons to ask for a show of hands in a focus group—for example, as a quick "pulse check", or to help warm up participants at the start of the discussion. But whether we think of our work as quant or qual—and whether we are thinking of our questions, our methods, or our claims in making that determination—let's be deliberate and mindful about the implications of actively inviting that messiness into the picture.

Author Bio:

aimee caffreyAimée Caffrey is a cultural anthropologist and UX researcher. Since 2017, she has worked in the Advanced Analytics Group at Bain & Company, where she collaborates with consultants, developers, designers, and fellow researchers to help clients solve some of today’s most exciting business challenges. If you wish to get in touch, please email her at Aimee.Caffrey@Bain.com.

Tags:  QRCA Digest  qualitative  qualitative research  quantitative 

Permalink | Comments (4)
 

Comments on this post...

Ashish Maheshwari, consumer insight associates says...
Posted Wednesday, December 11, 2019
Hi Aimee, This is a significant point about underserving quant by a show of hands and after having spent some time in the industry, clients (and often times researchers too) still want to include such techniques in qualitative research. In a recent project on mobile devices, a client wanted a breakdown of usage across texting, voice, video, etc over 100%. Go beat that. In many cases, I find clients have an inherent comfort with numbers as they help contextualize that particular data (or they are simply trying to reconfirm the numbers they already have lol) Over time this practice has not stopped . Which brings the significance of moderation under light too. With rampant DIY projects clients claim to "moderate" on their own and often times the questions asked will make any respectable quallies skin crawl. Fact is that through qual we aim to get consumers to be themselves and not their rational selves, infact in a group setting a show of hands can also be due to bias, or because respondents have been cornered to take a stance (to help make analysis and interpretation easy for us) - this piece of data should then be processed in the context of what the overall data patterns suggest. As it becomes harder to get such requests to be withdrawn perhaps at the analysis stages one can make necessary adjustments.
Permalink to this Comment }

Ashish Maheshwari, consumer insight associates says...
Posted Wednesday, December 11, 2019
Hi Aimee, This is a significant point about underserving quant by a show of hands and after having spent some time in the industry, clients (and often times researchers too) still want to include such techniques in qualitative research. In a recent project on mobile devices, a client wanted a breakdown of usage across texting, voice, video, etc over 100%. Go beat that. In many cases, I find clients have an inherent comfort with numbers as they help contextualize that particular data (or they are simply trying to reconfirm the numbers they already have lol) Over time this practice has not stopped . Which brings the significance of moderation under light too. With rampant DIY projects clients claim to "moderate" on their own and often times the questions asked will make any respectable quallies skin crawl. Fact is that through qual we aim to get consumers to be themselves and not their rational selves, infact in a group setting a show of hands can also be due to bias, or because respondents have been cornered to take a stance (to help make analysis and interpretation easy for us) - this piece of data should then be processed in the context of what the overall data patterns suggest. As it becomes harder to get such requests to be withdrawn perhaps at the analysis stages one can make necessary adjustments.
Permalink to this Comment }

Janet Standen, Scoot Insights says...
Posted Tuesday, December 17, 2019
Great, thought provoking article Aimee - thanks for sharing your thinking. I also hate when I'm asked to invite respondents to "give the concept a quick score out of 10" based on appeal - which is fine to check the pulse of the room and start with positive reflections, BUT when I'm asked to provide a chart at the end of the qual summarizing the rankings, I really baulk. Often the strongest/most compelling/motivating concept will score low because of wording in the idea that isn't fully understood initially, but once tweaked/updated, it becomes the strongest idea. Rankings can be soooo misleading, so please all quallies, be strong and refuse to provide a summary chart of rankings in your qual report - even in the appendix!!!
Permalink to this Comment }

Kelly Heatly, Heatly Custom Research, LLC says...
Posted Thursday, December 19, 2019
Thanks for a great article, Aimee! It is definitely a challenge to explain the "why" in the right way, using words that tell the story and not getting bogged down in numbers. I'd love to chat with you more about this at the conference next month in Austin!
Permalink to this Comment }

Contact Us

QRCA
1000 Westgate Drive, Suite 252
St. Paul, MN 55114

phone:  651. 290. 7491
fax:  651. 290. 2266
info@qrca.org

Privacy Policy | Site Map | Sign Out
© 2020 Qualitative Research Consultants Association

This website is optimized for Firefox and Chrome. If you have difficulties using this site, see complete browser details.