When someone asks you how you are you say ‘great’, ‘okay’ or ‘not so good’. You don’t say ‘feeling 6 out of 7’. So why do we try to force a numeric answer in our surveys?
As a researcher, I've had lots of healthy debate with people about scales. Everyone has their favorites. There is a large body of research that shows humans do not comprehend scales in the way we often try to use them, and that shorter scales are the way to go.
From the administrative side of things we design questions we think we need in order to get detailed data, or based on what we’ve seen others do, what we were taught through academia, or what we’ve always done so we can “track” things.
But the reality is that the humans who answer our questions aren’t wired to answer questions this way. The questions we ask should correspond to the way we think.
Using short, verbally anchored scales yields more consistent data quality, reduces cultural bias, is more engaging to complete, and gives you permission to have some fun with your scales.
Why Short Scales?
Screen size bias
Our research on how people answer long, numeric scales shows there can be differences in how people use the scale based on the kind of device they are answering on. This does not happen with short, verbally anchored scales.
Perceived difference between larger number is subjective
There is a body of research that shows people find smaller numbers easy to understand and work with. People are good with very small numbers and then they tend to compress the difference between them as they grow larger than 4. It’s easy for our minds to figure out the difference between 1 and 2, but what is the difference between a 7, 8 or a 9 on a 10 point scale? Given this, it makes sense to use shorter scales, as the greater “sensitivity” we ascribe to larger scales is both illusory and burdensome to the people who answer our questions.
Long scales have higher cultural bias
Long scales are also more sensitive to cultural differences in how people use scales. Some cultures would never give something a “10” even if they were 100% satisfied, while others would only give 9’s and 10’s. Few give ratings below 4, even if they disliked the thing being rated.
You're going to "roll up" top 2 or 3 anyway
When we use ten point scales, or seven point scales, we never show the distribution. We reduce it to a top 3 or two 2 box score. Why? Because we don’t think in ten point scales. We think in simpler categories. With a three point you get what you need in the first position without needing a roll up.
Oh NPS. You're a tricky outlier
NPS or Net Promoter Score. It is an eleven point scale. Most companies today use NPS as an indicator of satisfaction throughout the customer journey. But all of the analysis collapses it into two numbers: promoters and detractors, which then gets simplified into a single number: the net number of promoters. Why? Because the differences between 9 and 10 are meaningless for the people who need to understand the results. Feels like the ship has sailed on trying to make NPS a tighter scale. Asking a corresponding open end will at least help you interpret more nuance in the results.
Use words, not numbers
Scales are better (more reliable and more valid) when each point is labeled, not just represented by a number. So a scale that is Great/Ok/Not great is much better than “on a scale of 1 to 10 where 1 is not great, and 10 is great…). If you think about how you respond when someone asks you how you are you say ‘great’, ‘okay’ or ‘not so good’. You don’t say ‘feeling 6 out of 7’.
Short scales are less tedious to balance
Scales must be balanced. So you can’t have...
Very satisfied/somewhat satisfied/neutral/dissatisfied
OR Great/good/ok/not great
If you spread out the scale, and balance it – Very good/good/neither good nor bad/poor/very poor – you can see it would start to get tedious for the person answering questions.
Plus what would you do with the analysis? The analysis would likely be top 2 box because you want to look at all the positives. But that just washes out all the results; it makes everything look more positive than it actually was.
With a 3 point scale the things that are great are great! And the things that are ok, are just ok. The goal is Great! So why not just ask the question and do the analysis that way?
