Posts Tagged ‘Lucky Charms’

“Fat Tax” Experiment Results Must be Interpreted with Caution

June 1, 2010

The June 2010 issue of Men’s Health magazine displayed a brief on its “Nutrition Bulletin” page about an experiment researchers from the University of Buffalo did to test the effectiveness of a “fat tax” in combating obesity. According to the brief, researchers sent shoppers to a fake supermarket where unhealthy foods were assessed a 10% “fat tax”. The brief said the study found that shoppers were led to buy healthier items and that they carried away 6.5% fewer calories in their carts. The brief also claimed “other studies have reported similar effects in real-life situations.”

The brief reiterated something I already knew: the caution one must exercise when interpreting findings from an experiment. We must remember that experiments require subjects to be studied in a controlled environment. The more environmental factors we control for, the less realistic our findings. When designing an experiment, several details must be taken into account which, if ignored or done improperly, can greatly impact the results. There are a lot of questions we should ask about this “fat tax” test:

“Who funded the study?” This is perhaps the most important question. Whenever we hear a statistic, we must question the source. The study may have been funded by and carried out for a group, organization, or person with a vested financial or political interest in the outcome.

How were the research subjects selected?” The selection of participants – research subjects – is very important. How diverse is the subject pool? Where were they found? How many people were studied? Was their participation voluntary (as it should be in all experiments), and, if so, was there a way to measure if those who chose to participate are different in some way from those who chose not to? People who choose to participate in a survey or experiment can be fundamentally different from those who decline to participate. Participants who have a vested interest in the topic being measured can either consciously or unconsciously alter their behavior in a way that affects the experiment’s outcome. If participants are not selected randomly, the results of an experiment cannot be generalized to the population at-large. Furthermore, consumer behavior can be affected especially by cultural, ethnic, religious, socioeconomic, psychological, and sociological factors.

“What criteria were used to classify a food as ‘healthy’?” and “Who decided what was healthy or unhealthy?” are also very important considerations. Classifying food as “healthy” and “unhealthy” is subjective. Was the classification based on nutritional info, such as sugar, fat, sodium, and/or calorie content? Were all candy, cookies, sodas, pastries, cakes, potato chips, etc. classified as “unhealthy?” Were smaller-portion sizes given an exemption, but larger sizes imposed the tax? What about foods we generally think of as healthy, but can be heavily sweetened?

Take yogurt for example: plain yogurt is generally very healthy. A “cherry cheesecake” flavored yogurt is sweetened artificially. Cereal is another example: Cheerios is an unsweetened brand, but Lucky Charms is packed with sugar. In classifying yogurts and cereals, were all items in those categories classified as “healthy” or “unhealthy?” Or were some items within the same category assigned different classifications based on other criteria? Don’t laugh: here in Illinois – Land of the Jailed Governors – the state sales tax on candy is lower if the candy contains flour. Hence, a Milky Way bar is taxed at 2% while a Hershey bar is taxed at 9%! And chewing gum, which has fewer calories than candy is also taxed at 9%! Finally, the one who makes the decision about which foods are healthy or unhealthy is just as important as the person who funded the study, for the very same reason.

Other questions to ask include: “How long was the study conducted?” “Were the subjects required to do all their shopping at the fake store during the length of the experiment?” “In what ways were the subjects’ shopping cart items lower calorie?” Did the subjects have to use their own money in the store, or were they given money by the researchers?” Or, “Were the respondents given a set budget, and then told their carts would be examined after checkout?” “How did the researchers control for altered environments and behavior change?

As you can see, there’s a lot at play here. If participants know they are being monitored, they are likely to change their behavior. Also, people’s short-term behavior differs greatly from their long-term behavior. In addition, if subjects’ laboratory experiences are different from their daily uncontrolled experiences – e.g., they are given the money to shop, instead of using their own, or they are given an experimental shopping budget of $200, when their normal budget is $100 or $300 – their behaviors will be unrealistic. Finally, it’s important to see how the subjects reduced the calorie content of their purchases. Did they do it by buying fewer or no unhealthy items? Did they buy smaller sizes of unhealthy items? Or worse – did they substitute cheaper brands of healthier foods or get smaller sizes of them, so that they could still fit the higher-taxed healthier foods into their budgets? It is quite possible to reduce calories but still be eating junk.

I do not know how the University of Buffalo conducted the experiment nor am I saying they did it unprofessionally. I am merely saying that we must never accept statistics blindly and that we should challenge every fact that we are told, especially because the person giving us the information might have an ax to grind.