Posts Tagged ‘online surveys’

Are Mail Surveys Useless?

December 21, 2010

These days, most surveys are delivered online. Researchers – especially with the proliferation of consumer panels – can now program a survey, administer it to a relevant sample and get results within a few days, for a relatively low cost per complete. This is a far cry from the day when most surveys were conducted by mail. Mail surveys often needed to be planned out well in advance, had to be kept in the field for several weeks – if not a few months, – required incentives and reminders, and often generated low response rates. Needless to say, mail surveys were also quite costly and could not be changed once in the field.

Most marketing research professionals don’t even consider conducting a survey by mail anymore; most now view mail surveys as obsolete. While I certainly favor online and social media surveys more than mail surveys, I caution not to dismiss mail surveys out of hand. They still have some relevance and, depending on the business objective, may be a better choice than the popular online survey methods.

There are several reasons why you might still consider doing a survey by mail:

  1. Some people still aren’t online. What if you need to survey elderly persons? Or low-income households? Many persons in these groups do not have Internet access, so they cannot be reached online. Assuming they’re not homeless, virtually all of them live at a physical address with a mailbox.


  2. Advance permission is often needed to send e-mail surveys. Because of the Controlling the Assault of Non-Solicited Pornography and Marketing (CAN-SPAM) Act of 2003, marketers cannot send promotional e-mail to prospects without permission. While a survey is not promotional, consumers upset about receiving an unsolicited survey might still report it as SPAM, getting you into trouble. This is why most e-mail surveys make use of pre-recruited panels. Mail surveys don’t require such permission.


  3. Mailing lists can be obtained to conduct surveys. E-mail address lists cannot be sold. Quite often, you can rent mailing lists to send out surveys.


  4. Mail surveys these days may get a better-than-expected response rate. Response rates likely won’t be double-digit, but since few mail surveys are sent these days, those few that are have a better chance of catching the respondent’s attention. And since the respondent isn’t being bombarded with mail surveys, he or she may be more inclined to answer.


  5. Mail surveys offer greater perception of anonymity and confidentiality – and hence more truthful responses – than online surveys. Since surveys are administered online, it’s easy to tell who didn’t respond. When you send a respondent a reminder e-mail, the respondent knows his or her lack of response is known. This may lead him or her to feel that the answers he/she gives are also traceable back to him/her. As a result, he or she may be less-inclined to respond truthfully, let alone respond. Although tracking mechanisms have been placed on mail surveys, they’re not as easily discernable as they are for online surveys.

While online surveys appear to be the preferred survey method, there are still times when mail surveys are the better means of data collection. Sometimes, survey projects need to be multimodal in order to achieve a representative sample. Just because online surveys are faster and cheaper than mail surveys, you must consider the value of the insights each mode promises to bring to your business objective.

Insight Central Resumes Week of January 3, 2011!

In observance of the Christmas and New Years, Insight Central will resume the week of January 3, 2011.  We here at Analysights wish you and your family and friends a very Merry Christmas, and a Happy, Healthy, and Prosperous New Year! 


Be Sure to Follow us on Facebook and Twitter !

Thanks to all of you, Analysights now has more than 200 fans on Facebook … and we’d love more! If you like Forecast Friday – or any of our other posts – then we want you to “Like” us on Facebook! And if you like us that much, please also pass these posts on to your friends who like forecasting and invite them to “Like” Analysights! By “Like-ing” us on Facebook, you and they will be informed every time a new blog post has been published, or when new information comes out. Check out our Facebook page! You can also follow us on Twitter. Thanks for your help!

Survey Length Can Impact Findings

July 7, 2010

Last week, I talked about how it might be better to conduct a few short surveys in place of one longer survey. Whether or not the more frequent shorter surveys are better or feasible depends largely on your business problem, urgency, budget, and target respondents. In survey research, for the most part, shorter is almost always preferable to longer.

With more surveys being conducted online, respondent attention spans are very short and patience is in short supply. About 53% of respondents to online surveys say they will devote 10 minutes or less to a survey, according to InsightExpress back in September 2002. Dropout rates tend to increase as surveys get longer. Karen Paterson of Millward Brown found that after 10 minutes, each additional minute a survey takes lowers completion rates by 2%.

Moreover, the number of survey screens (that is, how many times a respondent clicks a “Next” or “Forward” arrow” on the Web survey) can greatly fatigue respondents, especially in business to business (B2B) research. Bill MacElroy demonstrated in the July/August 2000 issue of Quirk’s Marketing Research Review that the dropout rate of B2B respondents increases exponentially as the number of survey screens increases. According to MacElroy, the dropout rate is 7% for an online survey with 10 screens. With 15 screens, the dropout rate is 9%. But at 30 screens, the dropout rate is 30%, and at 45 screens, a whopping 73%!

The question that should enter all of our minds, then, is “what impact does the dropout rate have on both the integrity and findings of the survey?” Generally, respondents who terminate a survey are lumped together with the non-responders. Non-response error has always been a concern of the most dedicated researchers, but quite often is ignored in practice. However, with termination rates growing in the wake of online surveys, ignoring non-response error can cause misleading results.

Karl Irons, in the American Marketing Association’s November 2001 EXPLOR Forum pointed out that the longer the survey, the more inclined respondents who completed the survey were to check the top two boxes on a purchase intent survey:

Hence, when the survey took 14 minutes or more, nearly half of the respondents who completed the survey were likely to choose the top two boxes, indicating that they were most likely or definitely likely to buy, compared with just one-quarter of respondents when the survey was less than 6.5 minutes.

In addition, InsightExpress compared two surveys – a six-minute, 12-question survey and a 21-minute, 23-question survey – in Issue #11 of Today’s Insights. The completion rate of the shorter survey was 31.4%, but only 11% for the longer one. The demographics of the completing respondents weren’t dramatically different, but the results were markedly different: Just under 9% of the respondents in the shorter survey expressed intent to purchase, but almost 25% of those completing the longer survey did! Only four percent of those completing the shorter survey said the product concept appealed to them, compared to nearly 14% for those completing the longer survey!

Why is this? First, when a survey is long, the people who stay to complete it likely have some vested interest in the survey. If the survey is about premium chocolate, for example, a chocolate lover might stick it out through the duration. And someone like the chocoholic is more likely than the average respondent to purchase premium chocolate. Secondly, some respondents don’t want to terminate a survey, either because of the incentive offered or because they want to “be polite.” Hence, they might speed through the survey, just marking the top boxes. In either case, the researcher ends up with biased results.

So how do we rectify this? First and foremost, if you have to do a long survey, tell respondents upfront how long it is expected to take – both with dial-up and with high-speed broadband internet connections. Secondly, make sure there is an appropriate incentive for their participation. Also, make use of a progress bar to let respondents know how far along they are in the survey. Make the survey questions as short, as easy to understand, and as simple as possible. And always test the questionnaire before administration. Have someone else read certain questions, paraphrase them, and try to answer them. And of course, if you have the time and money to do a couple of shorter surveys instead, by all means do so.


Analysights is now on Facebook!

Analysights is now doing the social media thing! If you like Forecast Friday – or any of our other posts – then we want you to “Like” us on Facebook! By “Like-ing” us on Facebook, you’ll be informed every time a new blog post has been published, or when other information comes out. Check out our Facebook page!

A Typical-Length Survey or A Few Shorter Ones?

June 28, 2010

Most online surveys today take between 10 and 15 minutes, with a few going as long as 25 to 30 minutes. As marketing researchers, we have long pontificated that surveys should be a reasonable length, as longer ones tend to cause respondents to disengage in many ways: speeding through, skipping questions, even abandoning the survey. Most marketers realize this, and the 10-15 minute survey seems to be the norm. But I wonder how many marketing researchers – on both the client and supplier side – have ever considered the length of a survey from a strategic, rather than a tactical, point of view.

Sure, a typical-length survey is not super long, and is often cost effective for a client. After all, the client can survey several people about several topics in a relatively short time, for a set price, and can get results quickly. But sometimes I believe that instead of one 15-minute survey, some clients might benefit more by conducting two 7- or 8-minute, or three 5-minute surveys, stretched out over time. Marketing researchers on both sides will likely disagree with me here. After all, multiple shorter surveys can cost more to administer. However, I believe that – in the long-run – clients will derive value from the more frequent, shorter surveys that would offset their cost. Multiple, shorter surveys will benefit clients in the following ways:


As marketing research suppliers, it is our job to make sure we understand the client’s key business problem. Many times, clients have several problems that must be addressed. We need to help clients look at all of their business problems and prioritize them in the order of benefit that their resolution would bring. If we could get the client’s survey focused on the one or two problems whose resolution would result in the most positive difference, we can keep the survey short, with more targeted questions. As a result, the client doesn’t get bombarded with tons of data tables or reports with lots of recommendations and end up immobilized wondering which ones should be implemented first. On the contrary, the client will receive a few, very direct, insights about how to respond to these key problems.

Reduced Incentive Costs

Since surveys are shorter, respondents may be willing to do them for little or no incentive. This can save the client money.

Higher Response Rates

Surveys that are 10-15 minutes long generally get decent response rates. However, a survey that’s 3, 5, or 7 minutes long will likely get excellent response rates. Why? Because they’re more convenient, straight to the point, and can be knocked off quickly. As a result, respondents are less willing to put it off. Respondents are also less likely to terminate the survey, speed through it, or skip questions.

Increased Trust by Respondents

Because you didn’t waste their time with the first survey, respondents may be more inclined to participate in your subsequent surveys. If they took your 5-minute survey today, then you send them another 5-minute survey four to six weeks from now, they are likely to trust that this survey won’t take long either, and will likely respond to it. Of course, the key here is to space the surveys out. You don’t want to send all three at once!

More Reliable Data

As mentioned above, respondents are less likely to speed, terminate, or skip questions to a short survey than they are with a longer one. As a result, there will be less non-response error and more truthful responses in the data, and hence more trustworthy findings.

Ability to Act on Results Faster

Because the survey is short and to-the-point, and response rates are higher, the client can achieve the desired number of completed surveys sooner than if the survey were longer, so the survey doesn’t have to be in the field as long. And because the survey is short, the time the marketing research firm needs to tabulate and analyze the data is much shorter. Hence the client can start acting on the insights and implementing the recommendations much sooner.


What would happen if a client conducted a typical-length survey and found a theme emerging in open-ended questions or a trend in responses among a certain demographic group? The client may want to study that. But custom research is expensive. If the client did a typical-length survey, the budget may not be there to do another survey to investigate that newly discovered theme or trend. With a shorter survey, the cost may be somewhat lower, so funds might be left in the budget for another survey. In addition, if the client is scheduling subsequent shorter surveys, the learnings from the first survey can be used to shape questions for further investigation in those upcoming surveys.

The Shorter Survey May Be Enough

Several times, problems are interconnected, or generated by other problems. If research suppliers helped clients isolate their one or two biggest problems, and focused on those, the client might act on the insights and eliminate those problems. The resolution of those problems may also provide solutions to, or help extinguish, the lesser-priority problems. As a result, future surveys may not be needed. In that case, the research supplier did its job – solving the client’s problem in the shortest, most economical, and most effective manner possible.

Granted, many clients probably can’t do things this way. There are economies of scale in doing one longer survey as opposed to two or three shorter ones. Moreover, the client probably has several stakeholders, each of whom has a different opinion of which problem is most important. And each problem may have a different urgency to those stakeholders. This is why it is so important for the research supplier to get the client’s stakeholders and top management on board with this. As research suppliers, it is our job to inform and educate the client and its stakeholders on the research approach that maximizes the best interest of the client as a whole; and if that is not possible, work with those stakeholders to identify second-best solutions. But once the key issues – problems, budget, politics, and urgency – are on the table, research suppliers can work with the client to develop the shortest, most focused, most cost effective survey possible.

Free Online Survey Tools Can Yield Costly Useless Results if not Used Carefully

June 15, 2010

Thanks to online survey tools like Zoomerang, Surveymonkey, and SurveyPirate, the ability to conduct surveys has been greatly democratized. Small businesses, non-profits, and departments within larger firms can now conduct surveys that they would never have been able to do because of cost and lack of resources. Unfortunately, the greatest drawback of these free survey tools is the same as their greatest benefit: anyone can launch a survey. Launching an effective survey requires a clear definition of the business problem at hand; a carefully thought out discussion of the information needed to address the business problem, the audience of the survey, and how to reach it; determination of the sample size and how to select them; designing, testing, and implementing the questionnaire; and analyzing the results. Free online survey tools do not change this process.

Recently, a business owner from one of my networking groups sent me an online survey that he designed with one of these free tools. It was a questionnaire about children’s toys – which was the business he was in. He wasn’t sending me the survey to look at and give advice; he sent it to me as if I were a prospective customer. Unfortunately, I’m not married and don’t have kids; and all my nieces and nephews are past the age of toys. The survey was irrelevant to me. The toy purveyor needed to think about who his likely buyers were – and he should have good knowledge, based on his past sales, of who his typical buyers are. Then he could have purchased a list of people to whom he could send the survey. Even if that meant using a mail or phone survey, which could be costly, the owner could get more meaningful results. Imagine how many other irrelevant or uninterested recipients received the business owner’s survey. Most probably didn’t respond; but others might have responded untruthfully, giving the owner bogus results.

Also, the “toy-preneur’s” survey questions were poorly designed. A double-barreled question: “Does your child like educational or action toys?” What if a respondent’s child liked both educational and action toys? The owner should have asked two separate questions: “Does your child like educational toys?” and “Does your child like action toys?” Or he could have asked a multi-part question like, “Check the box next to each of the types of toys your child likes to play with,” followed with a list of the different types of toys.

The survey gets worse… questions like: “How much does your child’s happiness mean to you?” How many people are going to answer that question negatively? Hello? Another asking the respondent to rank-order various features of a toy for which there was no prototype pictured, and if that wasn’t bad enough, there were at least 9 items to rank? Most people can’t rank more than five items, especially not for an object they cannot visualize.

We also don’t know how the toy manufacturer selected his sample. My guess was that he sent it to everyone whose business card he collected. Hence, most of the people he was surveying were the wrong people. In addition to getting unacceptable results, another danger of these online survey tools is that people are more frequently bombarded with surveys that they stop participating in surveys altogether. Imagine if you were to receive five or more of these surveys in less than two weeks. How much time are you willing to give to answering these surveys? Then when a truly legitimate survey comes up, how likely are you to participate?

I think it’s great that most companies now have the ability to conduct surveys on the cheap. However, the savings can be greatly offset by the uselessness of the results if the survey is designed poorly or sent to the wrong sample. There is nothing wrong with reading up on how to do a survey and then executing it, as described, as long as the problem is well-defined, the relevant population is identified, and the sampling, execution, and analysis plans are in place. “Free” surveying isn’t good if it costs you money and time in rework and/or in faulty actions taken based on your findings.

Do you have trouble deciding whether you need to do a survey? Do you spend a lot of time trying to find out what you’re trying to learn from a survey? Or how many people to survey? Or the questions you need to ask? Or which people to survey? Let Analysights help. We have nearly 20 years of survey research experience and a strong background in data analysis. We can help you determine whether a survey is the best approach for your research needs, the best questions to ask to get the information you need, and help you understand what the findings mean. Feel free to call us at (847) 895-2565.

Beware of “Professional” Survey Respondents!

April 3, 2009

Thanks to the Internet, conducting surveys has never been easier.  Being able to use the Web to conduct marketing research has greatly reduced the cost and time involved and has democratized the process for many companies.

While online surveys have increased simplicity and cost-savings, they have also given rise to a dangerous breed of respondents – “Professional” survey-takers.   

A “professional” respondent is one who actively seeks out online surveys offering paid incentives – cash, rewards, or some other benefit – for completing the survey.  In fact, many blogs and online articles tell of different sites people can go to find paid online surveys.

If your company conducts online surveys, “professionals” can render your findings useless.  In order for your survey to provide accurate and useful results, the people surveyed must be representative of the population you are measuring and selected randomly (that is, everyone from the population has an equal chance of selection).

“Professionals” subvert the sampling principles of representativeness and randomness simply because they self-select to take the survey.  The survey tool does not know that they are not part of the population to be measured, nor their probability of selection.  What’s more, online surveys exclude persons from the population without Internet access.  This results in a survey bias double-whammy.

In addition, “professionals” may simply go through a survey for the sake of the incentive.  Hence they may speed through it, paying little or no attention to the questions, or they may give untruthful answers.  Now your survey results are both biased and wrong.

 Minimizing the impact of “Professionals”

There are some steps you can take to protect your survey from “professionals,” including:

  • Maintain complete control of your survey distribution.  If possible, use a professional online survey panel company, such as e-Rewards, Greenfield Online, or Harris Interactive.  There are lots of others, and all maintain tight screening processes for their survey participants and tight controls for distribution of your survey;
  • If an online survey panel is out of your budget, perhaps you can build your own controlled e-mail list (following CAN-SPAM laws, of course).  E-mailing your survey is less prone to bias than keeping it on a Web site for anyone to join.
  • Have adequate screening criteria in your survey.  If you can get respondents to sign in using a passcode and/or ask questions at the beginning, which terminate the survey for people whose responses indicate they are not representative of the population, you can reduce the number of “professionals”;
  • Put “speed bumps” into your survey.  An example would be to have a dummy question inside that simply says: “Select the 3rd radio bottom from the top.”  Put two or three bumps in your survey.  A respondent who answers two or more of those bump questions incorrectly is likely to be a speeder and the survey can be instructed to terminate;
  • Ask validation questions.  That is, ask a question one way and then later in the survey ask it in another form, and see if the responses are consistent.  If they’re not, then the respondent may be a “professional” or a speeder.

The Internet may have made marketing research easier, but it has also made it more susceptible to bias.  The tools to conduct marketing research have become much easier and more user-friendly, but that doesn’t change the principles of statistics and marketing research.  Online surveys, no matter how easily, fast, or cheaply they can be implemented, will waste time and money if those principles are violated.