"Members of an insight community are biased. Survey results aren’t really representative. ”
We hear this a lot. This statement often stems from the belief that by maintaining anonymity and surveying a general population sample, researchers will find the unbiased truth. (Spoiler: there's no such thing!)
However, times have changed and perceptions toward research sample should evolve as well.
- Participants are tired of anonymous surveys and response rates have plummeted. This is a trend that has been documented annually, with response rates now often dipping into the single digits.
- Technological advances have increased susceptibility of online panels to contamination by bots, gamers, and professional respondents.
The most recent Grit report also paints a sobering picture of the future of market research sample, with two of five researchers predicting sample quality will worsen over the next three years. Privacy worries, competing distractions, and the abundance of surveys top their list of concerns.
Given these trends, researchers should be questioning why a general population anonymous survey is still considered the unbiased gold standard. Why are we are still holding onto relics from the start of our industry in political research in which interviewers would spread themselves in all directions of a location and would knock on every fifth house to ensure ‘representativeness’? Times have changed. We are not conducting face-to-face interviews any longer, but we are still holding onto the idea that sample anonymity is superior.
Instead, imagine the benefits of conducting ongoing research via an insight community of your customers with detailed demographic profiles, recruited from known sources. Customers who are so engaged your research activities often yield response rates above 40%.
Of course, results from an insight community may not mirror results from other sample sources; however, different results from different sample sources is a dilemma researchers often face.
Which results can be trusted—the results from members whom you have recruited and have built robust profiles over time, or results from anonymous sample with unknown origins and historically low response rates?
Further, the bias concern often stems from researchers holding a perception that insight community members will be overly positive and not offer critical assessments. Alida’s research on research has shown the opposite; insight community members tend to be more critical and honest since they have a connection to your brand. Connecting consumers to brands they care about and sharing how the brand is using the information results in more authentic feedback.
Charted below is one example showing the results from a branded concept test conducted among insight community members and a general population sample. The results across the two sample groups are nearly identical, underscoring that insight community members are not any more likely to greenlight an idea than their gen pop counterparts.
Additionally, research conducted by my Alida peers in APAC determined that insight community members also give more in-depth feedback, averaging eight more words per open-ended response than their general population counterparts.
These results align with our own personal experiences. Ask yourself who gives you more thoughtful advice—a close friend or stranger on the street corner?
Charted below is one more example to emphasize the benefits of conducting research among engaged members recruited from known sources. The results were collected from a branded claim test among insight community members and an ad-hoc survey sent via a link in a brand newsletter. The results tell a dissimilar story—insight community members rate the claims as fair to moderately motivating, while the ad-hoc sample suggests most of the claims are highly motivating.
For the insight community members, the response rate was 33% and the researcher also has access to rich profiles from data collected over several years, including demographics, purchase data and purchase channels. Conversely, the ad-hoc sample garnered a response rate of <2% with demographic data collected limited to brands purchased and channels shopped. Which results would you trust and use to make a business decision?
Finally, Forrester conducted a Total Economic Impact report earlier this year to understand the benefits of conducting relationship-based customer intelligence via an insight community. They cited a number of benefits of the approach: known sample sources, agile and concise delivery, high response and completion rates, and robust open ends. They concluded that this approach allows researchers to deliver more value and higher ROI to an organization versus traditional ad-hoc research.
Maybe you’re still not convinced? Consider putting your perceptions of bias to the test and conduct parallel tests like those outlined above. This will allow you to develop an informed view and determine the best approach for delivering more value and higher ROI to your organization.
Want to read more on this topic? Take a look at this article which addresses marketing research from the respondent's perspective.
