Posts Tagged ‘survey incentives’

Survey Length Can Impact Findings

July 7, 2010

Last week, I talked about how it might be better to conduct a few short surveys in place of one longer survey. Whether or not the more frequent shorter surveys are better or feasible depends largely on your business problem, urgency, budget, and target respondents. In survey research, for the most part, shorter is almost always preferable to longer.

With more surveys being conducted online, respondent attention spans are very short and patience is in short supply. About 53% of respondents to online surveys say they will devote 10 minutes or less to a survey, according to InsightExpress back in September 2002. Dropout rates tend to increase as surveys get longer. Karen Paterson of Millward Brown found that after 10 minutes, each additional minute a survey takes lowers completion rates by 2%.

Moreover, the number of survey screens (that is, how many times a respondent clicks a “Next” or “Forward” arrow” on the Web survey) can greatly fatigue respondents, especially in business to business (B2B) research. Bill MacElroy demonstrated in the July/August 2000 issue of Quirk’s Marketing Research Review that the dropout rate of B2B respondents increases exponentially as the number of survey screens increases. According to MacElroy, the dropout rate is 7% for an online survey with 10 screens. With 15 screens, the dropout rate is 9%. But at 30 screens, the dropout rate is 30%, and at 45 screens, a whopping 73%!

The question that should enter all of our minds, then, is “what impact does the dropout rate have on both the integrity and findings of the survey?” Generally, respondents who terminate a survey are lumped together with the non-responders. Non-response error has always been a concern of the most dedicated researchers, but quite often is ignored in practice. However, with termination rates growing in the wake of online surveys, ignoring non-response error can cause misleading results.

Karl Irons, in the American Marketing Association’s November 2001 EXPLOR Forum pointed out that the longer the survey, the more inclined respondents who completed the survey were to check the top two boxes on a purchase intent survey:

Hence, when the survey took 14 minutes or more, nearly half of the respondents who completed the survey were likely to choose the top two boxes, indicating that they were most likely or definitely likely to buy, compared with just one-quarter of respondents when the survey was less than 6.5 minutes.

In addition, InsightExpress compared two surveys – a six-minute, 12-question survey and a 21-minute, 23-question survey – in Issue #11 of Today’s Insights. The completion rate of the shorter survey was 31.4%, but only 11% for the longer one. The demographics of the completing respondents weren’t dramatically different, but the results were markedly different: Just under 9% of the respondents in the shorter survey expressed intent to purchase, but almost 25% of those completing the longer survey did! Only four percent of those completing the shorter survey said the product concept appealed to them, compared to nearly 14% for those completing the longer survey!

Why is this? First, when a survey is long, the people who stay to complete it likely have some vested interest in the survey. If the survey is about premium chocolate, for example, a chocolate lover might stick it out through the duration. And someone like the chocoholic is more likely than the average respondent to purchase premium chocolate. Secondly, some respondents don’t want to terminate a survey, either because of the incentive offered or because they want to “be polite.” Hence, they might speed through the survey, just marking the top boxes. In either case, the researcher ends up with biased results.

So how do we rectify this? First and foremost, if you have to do a long survey, tell respondents upfront how long it is expected to take – both with dial-up and with high-speed broadband internet connections. Secondly, make sure there is an appropriate incentive for their participation. Also, make use of a progress bar to let respondents know how far along they are in the survey. Make the survey questions as short, as easy to understand, and as simple as possible. And always test the questionnaire before administration. Have someone else read certain questions, paraphrase them, and try to answer them. And of course, if you have the time and money to do a couple of shorter surveys instead, by all means do so.


Analysights is now on Facebook!

Analysights is now doing the social media thing! If you like Forecast Friday – or any of our other posts – then we want you to “Like” us on Facebook! By “Like-ing” us on Facebook, you’ll be informed every time a new blog post has been published, or when other information comes out. Check out our Facebook page!


A Typical-Length Survey or A Few Shorter Ones?

June 28, 2010

Most online surveys today take between 10 and 15 minutes, with a few going as long as 25 to 30 minutes. As marketing researchers, we have long pontificated that surveys should be a reasonable length, as longer ones tend to cause respondents to disengage in many ways: speeding through, skipping questions, even abandoning the survey. Most marketers realize this, and the 10-15 minute survey seems to be the norm. But I wonder how many marketing researchers – on both the client and supplier side – have ever considered the length of a survey from a strategic, rather than a tactical, point of view.

Sure, a typical-length survey is not super long, and is often cost effective for a client. After all, the client can survey several people about several topics in a relatively short time, for a set price, and can get results quickly. But sometimes I believe that instead of one 15-minute survey, some clients might benefit more by conducting two 7- or 8-minute, or three 5-minute surveys, stretched out over time. Marketing researchers on both sides will likely disagree with me here. After all, multiple shorter surveys can cost more to administer. However, I believe that – in the long-run – clients will derive value from the more frequent, shorter surveys that would offset their cost. Multiple, shorter surveys will benefit clients in the following ways:


As marketing research suppliers, it is our job to make sure we understand the client’s key business problem. Many times, clients have several problems that must be addressed. We need to help clients look at all of their business problems and prioritize them in the order of benefit that their resolution would bring. If we could get the client’s survey focused on the one or two problems whose resolution would result in the most positive difference, we can keep the survey short, with more targeted questions. As a result, the client doesn’t get bombarded with tons of data tables or reports with lots of recommendations and end up immobilized wondering which ones should be implemented first. On the contrary, the client will receive a few, very direct, insights about how to respond to these key problems.

Reduced Incentive Costs

Since surveys are shorter, respondents may be willing to do them for little or no incentive. This can save the client money.

Higher Response Rates

Surveys that are 10-15 minutes long generally get decent response rates. However, a survey that’s 3, 5, or 7 minutes long will likely get excellent response rates. Why? Because they’re more convenient, straight to the point, and can be knocked off quickly. As a result, respondents are less willing to put it off. Respondents are also less likely to terminate the survey, speed through it, or skip questions.

Increased Trust by Respondents

Because you didn’t waste their time with the first survey, respondents may be more inclined to participate in your subsequent surveys. If they took your 5-minute survey today, then you send them another 5-minute survey four to six weeks from now, they are likely to trust that this survey won’t take long either, and will likely respond to it. Of course, the key here is to space the surveys out. You don’t want to send all three at once!

More Reliable Data

As mentioned above, respondents are less likely to speed, terminate, or skip questions to a short survey than they are with a longer one. As a result, there will be less non-response error and more truthful responses in the data, and hence more trustworthy findings.

Ability to Act on Results Faster

Because the survey is short and to-the-point, and response rates are higher, the client can achieve the desired number of completed surveys sooner than if the survey were longer, so the survey doesn’t have to be in the field as long. And because the survey is short, the time the marketing research firm needs to tabulate and analyze the data is much shorter. Hence the client can start acting on the insights and implementing the recommendations much sooner.


What would happen if a client conducted a typical-length survey and found a theme emerging in open-ended questions or a trend in responses among a certain demographic group? The client may want to study that. But custom research is expensive. If the client did a typical-length survey, the budget may not be there to do another survey to investigate that newly discovered theme or trend. With a shorter survey, the cost may be somewhat lower, so funds might be left in the budget for another survey. In addition, if the client is scheduling subsequent shorter surveys, the learnings from the first survey can be used to shape questions for further investigation in those upcoming surveys.

The Shorter Survey May Be Enough

Several times, problems are interconnected, or generated by other problems. If research suppliers helped clients isolate their one or two biggest problems, and focused on those, the client might act on the insights and eliminate those problems. The resolution of those problems may also provide solutions to, or help extinguish, the lesser-priority problems. As a result, future surveys may not be needed. In that case, the research supplier did its job – solving the client’s problem in the shortest, most economical, and most effective manner possible.

Granted, many clients probably can’t do things this way. There are economies of scale in doing one longer survey as opposed to two or three shorter ones. Moreover, the client probably has several stakeholders, each of whom has a different opinion of which problem is most important. And each problem may have a different urgency to those stakeholders. This is why it is so important for the research supplier to get the client’s stakeholders and top management on board with this. As research suppliers, it is our job to inform and educate the client and its stakeholders on the research approach that maximizes the best interest of the client as a whole; and if that is not possible, work with those stakeholders to identify second-best solutions. But once the key issues – problems, budget, politics, and urgency – are on the table, research suppliers can work with the client to develop the shortest, most focused, most cost effective survey possible.