Blog

Great insight! Is it reliable?

BY PAITRA RUSSELL

Often, qualitative researchers say our goal is to reach “data saturation” in our fieldwork, meaning roughly that we’re getting no new information about the specific topic we’re exploring (at least, not from the types of people we’re exploring it with). Achieving data saturation is meant to be an indicator that our findings are reliable: a thorough representation of the range of our participants’ perspectives about the topic—rather than a reflection of what the researcher remembers best or what the client hopes is true. But saturation is often identified after fieldwork is done—during the analysis phase of the work, based on the point at which no new themes or “codes” can be identified in the responses. So what happens if, while you’re coding data, you get to the very last interview or focus group, and a new theme comes up? Does that mean you didn’t reach saturation? And if so, does it mean you need to go back into the field and talk to more people about that topic so you know your results are reliable?

Let’s be frank: no one has time for that, let alone budget.

To avoid this scenario, one client said she wanted to track saturation in real time. She had done this in the past manually, noting responses on an enormous spreadsheet and ticking a box when a participant repeated an answer that had already been given. This enabled her to see the point at which she was no longer hearing new information in response to a question…which meant she had likely achieved saturation for that question. Great!

Effective, but definitely tedious. And there was no way to duplicate her work. With each new survey she tested, she rebuilt her spreadsheet and started from scratch.

At Philip Reese, we love tools. We had already built a proprietary electronic data capture (EDC) tool that we use in the field (yes, there are many EDC tools out there, but we wanted seamless integration with our existing design, analysis and reporting suite along with the benefit of full data security and control). So, we decided to try building a saturation tracker into our EDC. We could take over that task for our client—and anyone else using data saturation as an indicator of qualitative reliability.

And we did!

Our proprietary EDC has included saturation tracking capability for a couple of years now.

Not every qualitative project requires this level of rigor. If you’re doing four focus groups to know whether Direction A or Direction B is the more exciting approach for your message to a specific audience, you probably don’t need to track saturation to know the answer. But you might want to track saturation when you need to show evidence for your insights beyond a handful of verbatims. This can be because a particular internal client believes in numbers more so than words; or because the study may end up in front of a regulatory body, or to help justify a decision that’s unpopular with some stakeholders; or any number of reasons.

For our clients who need it, we include a reliability statement with our qualitative studies, along with all the documentation needed for researchers uninvolved with the study to repeat it. We don’t rely (haha) solely on saturation as an indicator of qualitative rigor. Achieving data saturation, however, goes a long way toward ensuring that qualitative results are trustworthy.

Achieving data saturation goes a long way toward ensuring that qualitative results are trustworthy.

Qualitative researchers often know intuitively that our results are true, but in an increasingly quant-dependent world, it can be valuable to be able to prove it in numerical language. Our tracker works well for claims research, message development, comprehension studies, and others. As far as we know, no other EDC tool provides this capability (but if you know of one, drop us a line. We’d love to check it out…because…we’re research geeks and we love tools).

Let's Talk

If you need comprehensive research and insights to achieve your goals or have questions about our services, we are here to help. Leave us a message below and we’ll respond back with lightening speed. You may also reach out to us by calling 888-445-6111.