Archive for May, 2009

Paid Surveys & Bad Respondents Redux

May 27, 2009

A while back, I wrote about how paid surveys are a good source of bad respondents and how just a few bad respondents can increase your likelihood of drawing incorrect conclusions from survey findings.  Yet, I still stumble upon blogs promoting and encouraging people to make money by taking paid surveys.

Remember when blood banks used to pay people to donate blood?  And how it led to alcoholics and junkies donating blood to get money to support their addictions?  Needless to say, their blood was worthless.  The same undesirability happens with paid surveys – companies engaging in paid surveys often end up with respondents whose objective is the money rather than the topic of the survey; and, just as needless to say, their responses are worthless.

There is nothing wrong with providing an incentive for a survey.  Often, an incentive is necessary to get more respondents to participate.  However, the respondent base and the incentives should be controlled.  As I described in my last post, How Much Damage do Bad Respondents do to Survey Results?, I suggest controls such as asking your sample vendor how it screens panelists, how it prevents respondents from getting the same survey multiple times through different panels, understanding how the sample vendor tracks survey-taking behavior, and how it prevents the same panelist from using multiple e-mail addresses to register as several separate panelists.

By the looks of this blog post, Paid Market Research – Is It Really a Way to Make Money?, which I’ve “tracked back,” it seems that this problem isn’t going away anytime soon.

How Much Damage do Bad Respondents do to Survey Results?

May 11, 2009

Minimizing both the number of bad respondents who take a survey and their impact on the survey results can seem as futile as Sisyphus pushing the rock up the mountain.  Bad respondents come in all flavors: professional respondents, speeders, retakers (people who take the same survey multiple times),  and outright frauds (people who aren’t who or what they claim to be).

Researchers have tried different approaches to these problems, including increasing sample size, eliminating one or two biggest types of bad respondents, or even ignoring the problem altogether.   Unfortunately, the first two approaches can actually cause more damage than doing nothing at all.  Let’s look at these three approaches more closely.

Approach 1: Increase the sample size

When concerned about accuracy, the common prescription among researchers is to increase the size of their sample.  Indeed, this approach reduces sampling error,  margin of error, and the impact of multicollinearity, and increases the confidence level in the results.   However, larger sample sizes are a double-edged sword.  Because a larger sample size reduces the standard error in the data, it also increases the t-value.  As a result, a small difference between two or more respondent groups can greatly increase the chance of committing a Type I error (rejecting a true null hypothesis).

Similarly,  if a sample has bad respondents, a larger sample size can actually exacerbate their impact on survey results.  After all, bad respondents are likely to respond to survey questions differently than legitimate respondents.  A larger sample size (even if every additional new respondent is good), will simply reduce the degree of these differences needed for statistical significance, and inflate the chance of drawing an erroneous conclusion from the survey findings.

Approach 2: Tackle the biggest offender

When faced with multiple problems, it is human nature to focus on eradicating the one or two worst problems.  While that might work in most situations, eliminating only one type of bad respondents can actually cause more problems.

Assume that a survey’s results include responses from both professional respondents and speeders.  Assume also that the survey has some ratings questions.  What if – compared to legitimate respondents – the former rates an item higher than average, and the latter lower than average?

By having both types of bad respondents in the survey, their overall impact on the mean may be negligible.  However, if you take out only one of them, the mean will become biased in favor of the type that was left alone, again exacerbating the impact of bad respondents.

Approach 3: Do Nothing

While doing nothing is preferable to the other two approaches, it has its own problems.  Return to the example of two types of bad responders.  While leaving both of them alone will keep the mean close to what it would be in the absence of both types, it will also inflate the variance of the data, resulting in an estimate of the mean that is untrustworthy.  Hence, removing one type of bad respondents causes biased results while doing nothing causes inefficient results, neither of which has pleasant outcomes. 

What to do about bad respondents

Bad respondents cannot be totally eliminated, but they can be minimized.  The best ways to go about this include:

  1. Ask how your sample vendor screens people wishing to join its panel;
  2. Find out how your vendor ensures that panelists who are on other panels are precluded from being sent the same survey;
  3. Determine how your vendor tracks the survey-taking behavior of its panelists, assesses the legitimacy of each, and purges itself of suspected bad respondents; and
  4. Determine how your vendor prevents a person with multiple e-mail addresses – if you’re doing online surveys – from trying to register each one as a separate panelist.        

8 Steps to Determining Market Size

May 1, 2009

Whether you’re an entrepreneur writing a business plan or an established firm looking to introduce a new product or service, you will encounter the need to estimate the size of the market/s that you plan to serve.  Market-sizing is an interesting and exciting branch of marketing research, but it can be almost as much an art as it is a science.  Today, I will walk you through the process of estimating market size, using the example of a financial planner looking to develop a practice in his community.

Step 1: Define your target market

This can never be stressed enough.  If you don’t know the type of customer you want to serve, you will waste a lot of time and money trying to get any customers.  Market-sizing is easier when you know the exact group you’re searching for.  Our financial planner has decided that his target market will be married couples with young children.

Step 2: Determine the needs of your target market and how they create demand for your product/service

Here you formulate a hypothesis.  Ask yourself the benefits your product or service offers your target customers.  What problem does your product help them solve?  Begin with a statement about why your target customers need your product.  Our financial planner’s statement might be: “Married couples with young children need my services because they must be prepared for college, as well as for unexpected emergencies such as disability and early death.”  This statement assumes, of course, that the financial planner sells financial products that address these needs; if the planner sells only financial plans, his statement will be different.  

Step 3: Identify the information you need to estimate the size of your market

Now that you have identified your target market and hypothesized about its demand for your product, what information do you need to develop your estimates?  Among other things, our financial planner would need to know:

  • The age distribution within the geographic area he serves;
  • The number of households with children in that area;
  • The distribution of family income in that area;
  • Home market values in the area;
  • Educational attainment and college enrollment rates for graduating high school students;
  • How many competitors, direct (other financial planners and insurance agents) and indirect (stock brokers, banks with financial planning services, etc.) are serving the market; and
  • What financial planning services people buy and how much they pay.

There are others, but this list is pretty comprehensive.

Step 4: Identify the sources you need to obtain that information

So where do you find information about your market?  These days, there is such a wealth of published statistics about almost every industry and market segment, that a combination of library and online research can fulfill most of your information needs.  In some cases, if you are looking for very specialized information, you may need to conduct your own primary research (surveys, focus groups, etc.) to get what you need. 

The U.S. Bureau of the Census provides comprehensive demographic statistics by metropolitan area, county, ZIP code, census tract, and state.  Information about population, age, income, educational attainment, presence of children, and home market value can easily be obtained at any of these levels, so the financial planner would be able to answer many of his questions.  In addition, the Census Bureau also produces County Business Patterns, which provides information about the activity of each industry by each of the same geographic levels listed earlier.  Hence, our financial planner can also obtain the number of financial planning establishments,  insurance agencies, and brokerage firms serving the area in which he hopes to establish his practice.

In addition, our financial planner may consult online data sources such as Dun & Bradstreet’s Million Dollar Database and ABI’s ReferenceUSA to identify specific financial planning firms and insurance agencies in his area and get estimates of their employment size and revenues.

The financial planner can also get lots of relevant information from trade associations, local chambers of commerce, Web sites of his existing competitors, and through primary research, such as surveys and interviews with experts.

Step 5: Collect the data

Now that you have identified your data sources, you need to extract the data.  The financial planner will scour all the sources he identified to pull out data that meets his information needs.  He will determine whether his data sources provide sufficient and useful data, or whether they provide insufficient or suspect data, at which point he may seek out additional sources to answer his questions.

Step 6: Analyze the data

Now that you have all the data, what does it mean?  What is it telling you?  Let’s say that the area our financial planner wants to serve has 200,000 households, of which 15% – or 30,000 – are two-parent households, with a median family income of $60,000 per year, a median age of 32, and an average household size of 4.  Immediately, the financial planner knows he is serving a young upscale market, and it’s very likely – without looking at the number of competition – that there will already be an above average number of financial planners trying to serve them.

The financial planner may also find from financial planning industry statistics that 60% of families in that age group carry life insurance, and that the average policy face value is $100,000.  Given the affluence of this area, the planner may reason that households in his target market have much greater assets and income to protect, so he may adjust his estimates of life insurance coverage for that area upward – to policies of maybe $250,000 or $500,000.  He’ll make similar estimates for any other financial products and services he offers.

Step 7: Derive your market estimate

Now that you’ve compiled and analyzed your data, you need to come up with an estimate of market size.  Our financial planner may – through all his data sources – come up with an average and standard deviation of the policy amounts of life, disability, and other policies aimed at his target market in that area.  He will then project that amount out by the number of households within that market to come up with an aggregate size of the financial planning market in that area.  From there, he will build in a margin of error, perhaps using a 95% confidence interval, to come up with a low estimate, a middle estimate (which would be the aggregate size he determined earlier), and a high estimate.

Step 8: Apply your estimate

Your market size estimate is useless if you do not apply it.  Once our financial planner derives his aggregate estimate, he will estimate how much of that market he can reasonably get based on his competition and the amount of money he can earn based on his commission structure.  This will feed his business plan projections.

In addition, the size and characteristics of this market will help our financial planner determine how best to market his services, whether by direct mail, giving presentations, networking, or other means.

Market-sizing can be a daunting, tedious task, but the value it adds to your planning and marketing efforts can make the time, money and effort invested in it more than worthwhile.