ARE YOU LOSING OUT ON VALUABLE CUSTOMER INTEL? - PART 2: How To Choose The Correct Type Of Survey Q
In my last blog post, I shared the tips I’d picked up about writing survey questions. That got me thinking about writing in general, its purpose and the myriad styles and approaches that different writers use.
Most writing…like journals, stories and even this blog…informs or provides information. However, surveys, like exams and some types of letters, ask for information, rather than give it.
When you create writing that informs, you can start at the beginning and go free form or start at the end and document the journey. However, when your writing asks a question, you can’t know what the end will be.
And if you start at the beginning you risk drifting off in the wrong direction…which is a longwinded way of saying when you approach writing a survey or any enquiring piece, you need to start in the middle.
START IN THE MIDDLE…
When I say ‘start in the middle’, I mean you need to start by thinking about how you want to analyse your data.
Do you need to find out how many of your respondents are male or female? Or do you need to know more than that…for example, how many of these female respondents work full-time?
Most likely you’ll want to know how your respondents feel. Do they think a product is poor or excellent? Do they agree or disagree with a particular statement?
Sometimes you may even want to channel your inner stats geek and find out whether there is any correlation between the different responses. For example, to what extent does the number of sales staff affect the total number of sales?
Where is the sweet spot…in terms of numbers of sales people? Or…to put it in slightly geekier terms…at what point is the correlation between sales and staff at the highest and most efficient point?
The consummate stats geek will probably get a twitchy urge to do some regression testing which would enable them to predict the behaviour of a dependent variable (e.g. “sales”) when an independent variable (let’s say “GDP”) fluctuates.
All this probably sounds a bit daunting, and probably, if you’re planning on conducting regression testing for the first time, you should probably get someone trained in research methodology to give you a helping hand.
But for those of you without a data scientist close by, I’m going to give you a leg up. I’ll start at the beginning and by the end of this journey you should have a fair idea of how to select your questions…maybe not like a virtuoso…but at least with insight and confidence.
Have patience, grasshopper…
TYPES OF QUESTIONS
The first thing to know is that there are two different categories of question, the open ended question and the closed ended question.
Open ended questions don’t give respondents a choice of answers; rather the respondent is invited to write an answer in their own words. These answers require individual analysis and coding and are especially useful for exploratory research.
Open-ended questions can also reveal unexpected ideas, which means that the extra analytical effort required is often rewarded.
Closed ended questions are questions that include a set of pre-determined responses or a scale.
They’re not standalone questions but a question/answer package, which is used when you have a fair idea of the likely answers. It is possible to use both styles in the one question as shown in Example 1 below.
In this blog I intend to deal with closed ended questions as these are the ones most used by online survey tools and the ones that’ll get you into trouble if you try to analyse the data in a way that the data doesn’t support.
CLOSEED-ENDED QUESTION TYPES
There are essentially 4 types of closed ended questions.
These ‘types’ package together a question/statement and a pre-determined set of answers also known as a ‘scale’. Each question type generates data, which can be analysed up to a certain level of complexity, with nominal data being the most basic and ratio data being the most sophisticated.
Just to make it interesting, these questions can be linked, layered and combined in any number of ways. But for now, let’s just jump in at the basic level of measurement – Nominal.
Nominal /Categorical Questions and Measurement
Nominal questions generate data that is the simplest to measure and compare. They ask a respondent to categorise themselves or their preference for certain products and services.
This question type includes dichotomous questions (questions that have only two possible answers e.g.: Yes/No or True/False).
The pre-determined answers to nominal questions have no particular order of importance (known as rank order) and are basically names, labels or identifiers.
The data collected from these types of questions support the most basic analysis and are really only sufficient for counting responses, allocating percentages and performing basic cross tabulations (comparing the data between two questions).
The following are 2 examples of nominal questions sets.
The data analysis on nominal questions might be simple, but writing the questions still requires the care mentioned in my previous blog. It’s worth re-iterating that precision, objectivity and knowing your audience are key.
Nominal questions may require a simple answer, but they can also cover sensitive topics and sensitive questions need care.
A question such as ‘What is your gender?’ may seem straightforward, but with our increasing awareness and understanding of LGBTQ identities the responses will need to be exhaustive and may need to be framed as follows:
‘Other’ could also be replaced by ‘Prefer not to answer’.
Knowing your audience means you know which response would be more sensitive and therefore easier for your respondent to answer.
Keep in mind that sensitive questions, unless they are absolutely essential, are best left out of surveys.
Ordinal Questions and Measurement
An ordinal question gives the respondent the chance to choose an answer from a ranked or ordered range of answers. It’s useful when you want to know how respondents feel about an issue or product.
Ordinal questions are used widely because they allow you to work out not only counts, percentages and cross tabulations, but also the magnitude of the respondents’ feelings.
Example 5 is an ordinal question/answer set and is in a common format known as the Likert Scale:
What defines an ordinal question is that the range of possible answers shows a relative order of magnitude, and only that...nothing more.
There is no measureable difference between ‘Agree’ and ‘Strongly Agree’ and the difference between Agree and Strongly disagree is not double the difference between Disagree and Strongly Disagree.
That means you can only calculate value comparisons and central tendency (using "Mode" or "Median") analysis, but unfortunately you’ll have to give "Mean" averaging a miss. Even so, this is still more information than the counts and percentages you get from Nominal measurement.
Adding a number next to each category makes the distance between the categories look measureable, but they’re not. It’s the equivalent of adding an image of a unicorn or an emoticon. Visually entertaining, but analytically deceptive as it suggests a sophistication that the data doesn’t have....
Example 6 is an example of this type of ordinal question, as the respondents is asked to rate their answers on a numerical, 1-5 slider scale.
This question uses a number range, but that doesn’t mean that the distance between the numbers are all of equal value.
Analysis of the data from these types of questions needs to stick to ordinal levels of measurement, such as comparisons and central tendency (that’s a summary measure that attempts to describe a whole set of data with a single value that represents the middle or centre of its distribution. Examples include Mean, Median and Mode).
Example 7 is another ordinal question that uses integers. Not as indicators of equal measures, but rather as a replacement for words. ‘2’ could represent ‘smidge of pain’ and ‘3’ a ‘wee bit more pain’.
Integers are used here simply because they’re easier for the respondents to process.
I like Example 7 because it makes it easy to understand another limitation of ordinal questions…which is that you really can’t compare one respondent with another.
One individual may answer “7” yet be in less pain than a respondent who answers “5”.
You cannot assume that a person that answers “8” is in twice as much pain as a person who answers “4”. All it tells us is that the respondent who chooses “8” is in more pain than if they had chosen “3”.
However, if this question is followed by the ordinal question in Example 8, below, then it becomes possible to compare the two levels and analyse the perceived effectiveness of medication on the respondent’s pain levels.
Another aspect of scaled responses to consider is how many answers you include in the scale. An odd number of possible answers allows for neutrality, whereas an even number forces the respondent to have an opinion (see Example 9, below).
In Example 9, the number of categories under ‘Quality’ means that ‘Good’ is the neutral option, although ‘Good’ may not seem very neutral.
In the ‘Importance’ categories there is no neutral option and the respondent is only allowed to see the topic as important or not important.
Interval Questions and Measurement
If your objective is to generate averaged data using mean calculations (remember…it’s a measure of central tendency), then you will need questions where the answer categories are not only ordered, but are part of a scale where the intervals between each answer are equal and can be measured. Data from measureable interval scales allow the ranking of items (in terms of importance) and also quantifying and comparing the magnitude between them.
The most cited example of an interval scale is the Fahrenheit temperature scale. The Fahrenheit temperature scale has no absolute zero (it’s possible to have a temperature of -10°F) and yet the intervals are of equal distance (the difference between 10°F and 20°F is the same as the difference between 85°F and 95°F...even if it doesn't "feel" the same).
This example is cited everywhere because it helps to explain the concept of interval scales, but it’s not easy to apply to general surveys.
Another example of scaled interval data would be the use of a manufactured scale such as dress or shoe size (Example 10) or to use a predetermined set of answer categories (Example 11).
Despite the fact that interval scales extend the power of analysis from percentages and value comparisons to standard deviation and Mean averaging, they’re not widely used. Some social sciences do use Mean averaging on data collected from ordinal scales (gasp!!) which can give the purist a bit of a palpitation, but it’s widely accepted as the social sciences often deal with immeasurable concepts.
The limitation of Interval data is that it lacks an absolute zero point so the data cannot be multiplied or divided and comparisons of magnitude aren’t possible (e.g. “A” is twice as large as “B”).
In practice, most surveys make do with data generated from ordinal scales because if you’re going to go to the effort of creating an interval scale then you may as well go the whole hog, supersize your measurement levels, create an answer scale with an absolute zero point…and generate Ratio data.
Ratio Questions and Measurement
Data from Ratio questions provides the highest level of measurement capability, letting you run complex correlations and regression testing analysis. This means that the respondent can answer with an absolute number…or they can answer on a scale with an absolute zero where zero means ‘an absence of’ or ‘none’.
Example 12 uses Example 11 in a very basic way to demonstrate the difference between an interval scale into a ratio scale.