The goal of an effective survey is to capture evaluations based on the voice of the consumer. In other words, allow the respondent to describe an experience in their own words. Three approaches are offered below to better capture member experiences in their own words.
Approach #1: Instead of Likes and Dislikes
Perhaps the most frequently asked open-ended questions are related to aspects liked or disliked about the experience. At the risk of overstatement, rarely does this vague request for detail provide much value. So, the first approach, usually used along with the typical set of ratings scales, is asking more engaging open-ended questions. For example, ask:
- “What past experiences does this remind you of?”
- “What comes to mind when you think about this experience?”
- “Describe the situation in which you had the experienced? What else was going one? Who was you with? Please provide whatever context you can to help us better understand the experience.”
- “What would be the typical setting in which you’d be using this (product/service)?”
- “Pretend you are going to describe this experience (e.g., product, concept, idea) to a friend. How would you describe it? Please include how you feel about the product, to help give your friend the best idea possible of the product you’re describing.”
Approach #2: Affect Scoring
This approach combines qualitative and quantitative measurements to produce a more informative and statistically sensitive evaluation of an experience. This approach allows members to use their own words to describe the experience (opposed to using a static set of performance attributes to be rated). After an experience, members are asked to list all positive and negative comments that come to mind regarding specific characteristics of the experience. This provides the “direction of affect”. Then, members provide an importance measure, “strength of affect” using a 3-point scale, applied to each comment.
Affect score = the difference between two sums (the sum of positive ratings – the sum of negative ratings)
The affect score does not consider the actual content of the comments. Rather, it is a very simple reflection of the affect direction (how many positive and negative comments are provided) and the strength (as per the 3-point ratings) associated with each.
The average affect score (averaged across members having the experience) serves as the basis for quantifying performance of the experience. The affect scoring does not make explicit use of the verbatim text. Using text analytic applications (e.g., word counts, Opinion Mining, Word Nets) can provide a deeper summary of word and phrase usage, to augment the score.
Approach #3: Think Aloud
In this approach, members are asked to recount in detail their experience. As an example of Think Aloud instructions:
- “We are very interested in learning about your experiences when shopping for _____. To learn more about this, we would like you to tell us about your last shopping experience for ____. We would like for you to think back to the shopping experience, and write everything you can remember about the experience, starting with entering the aisle or the part of the store where ____ is sold. To the best of your ability, please tell us what you were thinking and feeling, and tell us about the shopping process you followed that lead to your decision to buy the brand you bought.”
Think Aloud has a long history in cognitive psychology and neuroscience as a reliable way to capture a human’s thought process. This approach does elicit greater detail regarding the context and cognitive process used during the experience. The captured text can then be summarized and analyzed.
