Posts Tagged ‘bias’

Forecast Friday Topic: Judgmental Bias in Forecasting

March 17, 2011

(Fortieth in a series)

Over the last several weeks, we have discussed many of the qualitative forecasting methods, approaches that rely heavily on judgment and less on analytical tools. Because judgmental forecasting techniques rely upon a person’s thought processes and experiences, they can be highly subjected to bias. Today, we will complete our coverage of judgmental forecasting methods with a discussion of some of the common biases they inspire.

Inconsistency and Conservatism

Two very opposite biases in judgmental forecasting are inconsistency and conservatism. Inconsistency occurs when decision-makers apply different decision criteria in similar situations. Sometimes memories fade; other times, a manager or decision-maker may overestimate the impact of some new or extraneous event that is occurring in the subsequent situation that makes it different from the previous; he/she could be influenced by his/her mood that day; or he/she just wants to try something new out of boredom. Inconsistency can have serious negative repercussions.

One way to overcome inconsistency is to have a set of formal decision rules, or “expert systems,” that set objective criteria for decision-making, which must be applied to each similar forecasting situation. These criteria would be the factors to measure, the weight each one gets, and the objective of the forecasting project. When formal decision rules are imposed and applied consistently, forecasts tend to improve. However, it is important to monitor your environment as your expert systems are applied, so that they can be changed as your market evolves. Otherwise, failing to change a process in light of strong new information or evidence is a new bias, conservatism.

Now, have I just contradicted myself? No. Learning must always be applied in any expert system. We live in a dynamic world, not a static one. However, most change to our environment, and hence our expert systems, doesn’t occur dramatically or immediately. Often, they occur gradually and more subtly. It’s important to apply your expert systems and practice them for time, monitoring anything else in the environment, as well as the quality of forecasts your expert systems are measuring. If the gap between your forecast and actual performance is growing consistently, then it might be time to revisit your criteria. Perhaps you assigned too much or too little weight to one or more factors; perhaps new technologies are being introduced in your industry.

Decision-makers walk a fine line between inconsistency and conservatism in judgmental forecasts. Trying to reduce one bias may inspire another.

Recency

Often, when there are shocks in the economy, or disasters, these recent events tend to dominate our thoughts about the future. We tend to believe these conditions are permanent, so we downplay or ignore relevant events from the past. So, to avoid recency bias, we must remember that business cycles exist, and that ups and downs don’t last forever. Moreover, we should still keep expert systems in place that force us to consider all factors relevant in forecasting the event of interest.

Optimism

I’m guilty of this bias! Actually, many people are. Our projections are often clouded by the future outcomes we desire. Sometimes, we feel compelled to provide rosy projections because of pressure by higher-up executives. Unfortunately, optimism in forecasting can be very dangerous, and its repercussions severe when it is discovered how different our forecasted vs. actual results are. Many a company’s stock price has plunged because of overly optimistic forecasts. The best ways to avoid optimism are to have a disinterested third party generate the forecasts; or have other individuals make their own independent forecasts.

************

These are just a sample of the biases common in judgmental forecasting methods. And as you’ve probably guessed, deciding which biases you’re able to live with and which you are not able to live with is also a subjective decision! In general, for your judgmental forecasts to be accurate, you must consistently guard against biases and have set procedures in place for decision-making, that include learning as you go along.

*************************************

Next Forecast Friday Topic: Combining Forecasts

For the last 10 months, I have introduced you to the various ways by which forecasts are generated and the strengths and limitations of each approach. Organizations frequently generate multiple forecasts based on different approaches, decision criteria, and different assumptions. Finding a way to combine these forecasts into a representative composite forecast for the organization, as well as evaluating each forecast is crucial to the learning process and, ultimately, the success of the organization. So, beginning with next week’s Forecast Friday post, we begin our final Forecast Friday mini-series on combining and evaluating forecasts.

Advertisements

Forecast Friday Topic: Multicollinearity – Correcting and Accepting it

July 22, 2010

(Fourteenth in a series)

In last week’s Forecast Friday post, we discussed how to detect multicollinearity in a regression model and how dropping a suspect variable or variables from the model can be one approach to reducing or eliminating multicollinearity. However, removing variables can cause other problems – particularly specification bias – if the suspect variable is indeed an important predictor. Today we will discuss two additional approaches to correcting multicollinearity – obtaining more data and transforming variables – and will discuss when it’s best to just accept the multicollinearity.

Obtaining More Data

Multicollinearity is really an issue with the sample, not the population. Sometimes, sampling produces a data set that might be too homogeneous. One way to remedy this would be to add more observations to the data set. Enlarging the sample will introduce more variation in the data series, which reduces the effect of sampling error and helps increase precision when estimating various properties of the data. Increased sample sizes can reduce either the presence or the impact of multicollinearity, or both. Obtaining more data is often the best way to remedy multicollinearity.

Obtaining more data does have problems, however. Sometimes, additional data just isn’t available. This is especially the case with time series data, which can be limited or otherwise finite. If you need to obtain that additional information through great effort, it can be costly and time consuming. Also, the additional data you add to your sample could be quite similar to your original data set, so there would be no benefit to enlarging your data set. The new data could even make problems worse!

Transforming Variables

Another way statisticians and modelers go about eliminating multicollinearity is through data transformation. This can be done in a number of ways.

Combine Some Variables

The most obvious way would be to find a way to combine some of the variables. After all, multicollinearity suggests that two or more independent variables are strongly correlated. Perhaps you can multiply two variables together and use the product of those two variables in place of them.

So, in our example of the donor history, we had the two variables “Average Contribution in Last 12 Months” and “Times Donated in Last 12 Months.” We can multiply them to create a composite variable, “Total Contributions in Last 12 Months,” and then use that new variable, along with the variable “Months Since Last Donation” to perform the regression. In fact, if we did that with our model, we end up with a model (not shown here) that has an R2=0.895, and this time the coefficient for “Months Since Last Donation” is significant, as is our “Total Contribution” variable. Our F statistic is a little over 72. Essentially, the R2 and F statistics are only slightly lower than in our original model, suggesting that the transformation was useful. However, looking at the correlation matrix, we still see a strong negative correlation between our two independent variables, suggesting that we still haven’t eliminated multicollinearity.

Centered Interaction Terms

Sometimes we can reduce multicollinearity by creating an interaction term between variables in question. In a model trying to predict performance on a test based on hours spent studying and hours of sleep, you might find that hours spent studying appears to be related with hours of sleep. So, you create a third independent variable, Sleep_Study_Interaction. You do this by computing the average value for both the hours of sleep and hours of studying variables. For each observation, you subtract each independent variable’s mean from its respective value for that observation. Once you’ve done that for each observation, multiply their differences together. This is your interaction term, Sleep_Study_Interaction. Run the regression now with the original two variables and the interaction term. When you subtract the means from the variables in question, you are in effect centering interaction term, which means you’re taking into account central tendency in your data.

Differencing Data

If you’re working with time series data, one way to reduce multicollinearity is to run your regression using differences. To do this, you take every variable – dependent and independent – and, beginning with the second observation – subtract the immediate prior observation’s values for those variables from the current observation. Now, instead of working with original data, you are working with the change in data from one period to the next. Differencing eliminates multicollinearity by removing the trend component of the time series. If all independent variables had followed more or less the same trend, they could end up highly correlated. Sometimes, however, trends can build on themselves for several periods, so multiple differencing may be required. In this case, subtracting the period before was taking a “first difference.” If we subtracted two periods before, it’s a “second difference,” and so on. Note also that with differencing, we lose the first observations in the data, depending on how many periods we have to difference, so if you have a small data set, differencing can reduce your degrees of freedom and increase your risk of making a Type I Error: concluding that an independent variable is not statistically significant when, in truth it is.

Other Transformations

Sometimes, it makes sense to take a look at a scatter plot of each independent variable’s values with that of the dependent variable to see if the relationship is fairly linear. If it is not, that’s a cue to transform an independent variable. If an independent variable appears to have a logarithmic relationship, you might substitute its natural log. Also, depending on the relationship, you can use other transformations: square root, square, negative reciprocal, etc.

Another consideration: if you’re predicting the impact of violent crime on a city’s median family income, instead of using the number of violent crimes committed in the city, you might instead divide it by the city’s population and come up with a per-capita figure. That will give more useful insights into the incidence of crime in the city.

Transforming data in these ways helps reduce multicollinearity by representing independent variables differently, so that they are less correlated with other independent variables.

Limits of Data Transformation

Transforming data has its own pitfalls. First, transforming data also transforms the model. A model that uses a per-capita crime figure for an independent variable has a very different interpretation than one using an aggregate crime figure. Also, interpretations of models and their results get more complicated as data is transformed. Ideally, models are supposed to be parsimonious – that is, they explain a great deal about the relationship as simply as possible. Typically, parsimony means as few independent variables as possible, but it also means as few transformations as possible. You also need to do more work. If you try to plug in new data to your resulting model for forecasting, you must remember to take the values for your data and transform them accordingly.

Living With Multicollinearity

Multicollinearity is par for the course when a model consists of two or more independent variables, so often the question isn’t whether multicollinearity exists, but rather how severe it is. Multicollinearity doesn’t bias your parameter estimates, but it inflates their variance, making them inefficient or untrustworthy. As you have seen from the remedies offered in this post, the cures can be worse than the disease. Correcting multicollinearity can also be an iterative process; the benefit of reducing multicollinearity may not justify the time and resources required to do so. Sometimes, any effort to reduce multicollinearity is futile. Generally, for the purposes of forecasting, it might be perfectly OK to disregard the multicollinearity. If, however, you’re using regression analysis to explain relationships, then you must try to reduce the multicollinearity.

A good approach is to run a couple of different models, some using variations of the remedies we’ve discussed here, and comparing their degree of multicollinearity with that of the original model. It is also important to compare the forecast accuracy of each. After all, if all you’re trying to do is forecast, then a model with slightly less multicollinearity but a higher degree of forecast error is probably not preferable to a more precise forecasting model with higher degrees of multicollinearity.

The Takeaways:

  1. Where you have multiple regression, you almost always have multicollinearity, especially in time series data.
  2. A correlation matrix is a good way to detect multicollinearity. Multicollinearity can be very serious if the correlation matrix shows that some of the independent variables are more highly correlated with each other than they are with the dependent variable.
  3. You should suspect multicollinearity if:
    1. You have a high R2 but low t-statistics;
    2. The sign for a coefficient is opposite of what is normally expected (a relationship that should be positive is negative, and vice-versa).
  4. Multicollinearity doesn’t bias parameter estimates, but makes them untrustworthy by enlarging their variance.
  5. There are several ways of remedying multicollinearity, with obtaining more data often being the best approach. Each remedy for multicollinearity contributes a new set of problems and limitations, so you must weigh the benefit of reduced multicollinearity on time and resources needed to do so, and the resulting impact on your forecast accuracy.

Next Forecast Friday Topic: Autocorrelation

These past two weeks, we discussed the problem of multicollinearity. Next week, we will discuss the problem of autocorrelation – the phenomenon that occurs when we violate the assumption that the error terms are not correlated with each other. We will discuss how to detect autocorrelation, discuss in greater depth the Durbin-Watson statistic’s use as a measure of the presence of autocorrelation, and how to correct for autocorrelation.

*************************

If you Like Our Posts, Then “Like” Us on Facebook and Twitter!

Analysights is now doing the social media thing! If you like Forecast Friday – or any of our other posts – then we want you to “Like” us on Facebook! By “Like-ing” us on Facebook, you’ll be informed every time a new blog post has been published, or when other information comes out. Check out our Facebook page! You can also follow us on Twitter.

Avoiding Biased Survey Questions

July 19, 2010

Adequate thought must be given to designing a questionnaire. Ask the wrong questions, or ask questions the wrong way, and you can end up with useless information; make the survey difficult or cumbersome, and respondents won’t participate; put the questions in the wrong order and you can end up with biased results. The most common problem with wording survey questions is bias. Biased questions are frequently asked in surveys administered by groups or organizations that are seeking to advance their political or social action agendas, or by certain departments or units within a corporation or organization likewise seeking to improve their political standing within the organization. Consider the questions below:

“Do you think the senseless war in Iraq that President Bush insisted on starting is going to result in thousands of unnecessary deaths?”

“Do you think the unprecedented trillion-dollar federal deficit the Democrats are creating with their out-of-control spending is going to have disastrous consequences for our nation?”

“Do you favor repeal of the death tax, so that many families won’t be unfairly burdened with hefty taxes at the time of their grief?”

Could these questions be more biased? Notice the adjectives in the questions, words like “senseless,” “unnecessary,” “unprecedented,” “out-of-control,” “disastrous,” “unfairly,” “burdened,” and “hefty.” All of them make it clear that a certain answer to each question is expected.

Look also at the descriptive words in some of the questions: “trillion-dollar,” “death” (as opposed to “estate”). You can see further manipulation. Worded the way they are, these questions stir up the emotions, which surveys are not supposed to do.

Removing the Bias

Can these questions be improved? Depending on the objectives of the survey, most definitely. In the first question, we might simply change the question to a multiple choice and ask:

What is your opinion regarding President Bush’s decision to send troops to Iraq?

Totally Sensible

Mostly Sensible

Somewhat Sensible

Not Sure

Somewhat Senseless

Mostly Senseless

Totally Senseless

 

Notice the difference? Here, the question is neutral. It also opens the survey taker to options that reflect the degree to which he/she feels about President Bush’s decision.

How about the second question? Perhaps we can try this:

In your opinion, how serious will the consequences of the federal budget deficit be for the nation?

Very Serious (5)

Serious (4)

Slightly Serious (3)

Not Very Serious (2)

Not at All Serious (1)

 

Here, we again neutralize the tone of the question and we let the respondent decide how severe the impact of the deficit will be. Notice also that we used an unbalanced scale, like we discussed last week. That’s because we would expect more respondents to select choices on the left hand side of the scale. This revised question focused on the seriousness of the deficit. We could also ask respondents about their perceptions of the size of deficit:

How do you feel about the size of the federal budget deficit?

Too Large (5)

Very Large (4)

Slightly Large (3)

Just Right (2)

Too Small (1)

 

Again, we use an unbalanced scale for this one. If we asked both the revised questions, we can gain great insights into the respondent’s perceptions of both the size and seriousness of the deficit. Ideally, we would ask the question about the deficit’s size before the question about its consequences.

These two revised questions should also point out another flaw with the original question: not only was it worded with bias, but it was also multipurpose or double-barreled. It was trying to fuse two thoughts about the deficit: it was too large and it was going to have serious consequences. These two revised questions will give us another advantage: we can now see how many people think the deficit is too large but do not see it as a serious threat. After all, we may agree something is excessive but we may not necessarily agree about the impact of that excess.

Now let’s look at the last question. Perhaps we can focus on the fairness of the estate tax:

What is your opinion regarding the fairness of the estate tax?

Absolutely Fair

Mostly Fair

Not Sure

Mostly Unfair

Absolutely Unfair

 

Of course, some respondents might not know what the estate tax is, so we need to describe it to them. Even in describing or defining something, we can open the door to bias, so we must choose our words carefully:

When a person dies, his or her heirs pay taxes on the amount of his/her estate that exceeds $1 million. This is known as the “estate” tax. What is your opinion regarding the fairness of such a tax?

Absolutely Fair

Mostly Fair

Not Sure

Mostly Unfair

Absolutely Unfair

 

This does a good job of describing the estate tax, but putting in the $1 million dollar figure can bias the results. If a respondent’s net worth is nowhere close to $1 million, he or she may consider the estate tax fair, just his or her heirs are unlikely to be affected by it. Perhaps the question can be worded this way:

When a person dies, a portion of his or her estate is subject to an “estate” tax. Would you say that such a tax is:

Absolutely Fair

Somewhat Fair

Not Sure

Somewhat Unfair

Absolutely Unfair

 

I think this example would be better, since it says “a portion” rather than a specific amount. While the $1 million example is more factual, it also adds in more normative considerations. By using “a portion,” we respondents won’t concentrate on the dollar amount of the estate, but on the fairness of the estate tax.

The adage “It’s not what you say but how you say it,” rings very true in questionnaire design. You must choose your words carefully in order to get the information you need to make well-informed business decisions.

*************************

If you Like Our Posts, Then “Like” Us on Facebook and Twitter!

Analysights is now doing the social media thing! If you like Forecast Friday – or any of our other posts – then we want you to “Like” us on Facebook! By “Like-ing” us on Facebook, you’ll be informed every time a new blog post has been published, or when other information comes out. Check out our Facebook page! You can also follow us on Twitter.

Radio Commercial Statistic: Another Example of Lies, Damn Lies, and then Statistics

May 10, 2010

Each morning, I awake to my favorite radio station, and the last few days, I’ve awakened to a commercial about a teaming up of Feeding America and the reality show Biggest Loser to support food banks.  While I think that’s a laudable joint venture, I have been somewhat puzzled by, if not leery of, a claim made in the commercial: that “49 million Americans struggled to put food on the table.”  Forty-nine million?  That’s one out of every six Americans! 

Lots of questions popped into my head: Where did this number come from?  How was it determined?  How did the study define “struggling?”  Why were the respondents struggling?  How did the researcher define the implied “enough food?”  What was the length of time these 49 million people went “struggling” for enough food?  And most importantly, what was the motive behind the study?

The Biggest Loser/Feeding America commercial is a good reminder of why we should never take numbers or statistics at face value.  Several things are fishy here.  Does “enough food” mean the standard daily calorie intake (which, incidentally, is another statistic)?  Or, given that two-thirds of Americans are either overweight or obese (another statistic I have trouble believing), is “enough food” defined as the average number of calories a person actually eats each day?

I also want to know how the people who conducted the study came up with 49 million people.  Surely they could not have surveyed so many people.  Most likely, they needed to survey a sample of people, and then make statistical estimations – extrapolations – based on the size of the population.  In order to do that, the sample needed to be selected randomly: that is, every American had to have an equal chance of being selected for the survey.  That’s the only way we could be sure the results are representative of the entire population.

Next, who and how many completed the survey?  The issue of hunger is political in nature, and hence is likely to be very polarizing.  Generally, people who respond to surveys based on such political issues have a vested interest in the subject matter.  This introduces sample bias.  Also, having an adequate sample size (neither too small nor too large) is important.  There’s no way to know if the study that came up with the “49 million” statistic accounted for these issues.

We also don’t know how long a time these 49 million had to struggle in order to be counted?  Was it just any one time during a certain year, or did it have to go for at least two consecutive weeks before it could be contacted?  We’re not told.

As you can see, the commercial’s claim of 49 million “struggling to put food on the table” just doesn’t jive with me.  Whenever you must rely on statistics, you must remember to:

  1. Consider the source of the statistic and its purpose in conducting the research;
  2. Ask how the sample was selected and the study executed, and how many responded;
  3. Understand the researcher’s definition of the variables being measured;
  4. Not look at just the survey’s margin of error, but also at the confidence level and the diversity within the population being sampled. 

The Feeding America/Biggest Loser team-up is great, but that radio claim is a sobering example of how statistics can mislead as well as inform.