Does the Order of Survey Questions Matter? You Bet!

Thanks to online survey tools, the cost of executing a survey has never been cheaper. Online surveys allow companies to ask respondents more questions, ask questions on multiple (though related) topics, and get their results faster and less expensively than was once possible with telephone surveys. But the ability to ask more questions on more topics has meant that the sequence of survey questions must be carefully taken into consideration. While the order of the questions in a survey has always mattered, they are even more crucial now.

Order Bias

When question order is not considered, several problems occur, most notably order bias. Imagine that a restaurant owner was conducting a customer satisfaction survey. With no prior survey background, he creates a short survey, with questions ordered like this:

  1. Please rate the temperature of your entrée.
  2. Please rate the taste of your food.
  3. Please rate the menu selection here.
  4. Please rate the courtesy of your server.
  5. Please rate the service you received.
  6. Please rate your overall experience at this restaurant.

What’s wrong with this line of questioning? Assuming they all have the same answer choices, ranging from “poor” to “excellent,” plenty! First, when there are several questions in sequence with the same rating scales, there’s a great chance a respondent will speed through the survey, providing truthful answers near the beginning of the survey, and less truthful answers further down. By placing the overall satisfaction at the end, the restaurateur is biasing the response to it. Hence, if the respondent had a positive experience to the temperature of his/her food, that might cause a halo effect, making him/her think the taste was also good, as well as the menu selection, etc. Halo effects can also be negative. That first question ends up setting the context in which the respondent views his satisfaction.

On the other hand, if the restaurateur shifted the order of the questionnaire as shown below, he will get more reliable answers:

  1. Please rate your overall experience at this restaurant.
  2. Please rate the menu selection here.
  3. Please rate the temperature of your entrée.
  4. Please rate the taste of your food.
  5. Please rate the service you received.
  6. Please rate the courtesy of your server.

Notice the difference? The restaurateur is starting with the overall satisfaction question, followed by satisfaction with the menu selection. Within the menu selection, the restaurateur asks specifically about temperature and taste of food. Then the restaurateur asks about the service, then specifically about the courtesy of the server. What this process does is to start with the respondent’s overall satisfaction. When a respondent offers an overall rating, he is then asked about a component (either the menu selection or the service) of overall satisfaction, so that the researcher can determine if a low overall satisfaction rating is brought on by a low satisfaction rating with either the menu or service, or both. This leads the respondent to speak truthfully of how each component contributed to his/her satisfaction.

Respondent Confusion/No Coherent Organization

Imagine you had developed a new product and wanted to gauge purchase intent for the new product. There’s a ton of stuff you want to know about: best price to charge, best way to promote the product, where respondents will go to buy it, etc. Many survey neophytes may commingle the pricing, promotion, and distribution questions. This is a mistake! The respondent will become confused and fatigued if there’s no clear organization for your survey. If you are asking a question about those three components, your questionnaire should have three sections. At the start of each section, you should indicate “this section asks you some questions about what you feel the ideal price for this product would be…” or “this section asks you about what features you would like and dislike in this product.” In this fashion, the respondent knows what the line of questioning is and doesn’t feel confused.

Tips for Ordering Survey Questions Effectively

These are just two examples. Essentially, if you want to order your questionnaire for maximum reliability, response, and clarity, remember to:

  1. Start with broad, general questions and move to narrow specific ones. If respondents haven’t formed a general opinion or point of view of your topic, you can start your questionnaire going from specific to general.
  2. As I mentioned in last week’s posts, sensitive questions should be asked late in the survey, after your previous questions have established rapport with the respondent.
  3. Unless the topic is highly sensitive, never start a questionnaire with an open-ended question.
  4. Save demographic and classification questions for the end of the questionnaire, unless you need to ask them in order to screen respondents for taking the survey.
  5. Use chronological sequences in questions when obtaining historical information from a respondent.
  6. Make sure all questions on a topic are complete before moving on to another topic and use transitory statements between the topics, like I described in the prior paragraph.

Much like designing survey questions, the order of the questioning is as much an art as it is a science. Taking time to organize your questions will reward you with results that are reliable and actionable.

Tags: , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: