Recently, a scientific article has been published that the mainstream media has picked up, over-simplified, and — I would argue — misrepresented. The article compares the nutritional quality of organic foods to “conventional” foods.
Most of you have probably have either heard about this study either from the media or from someone who all-too-gleefully informed you that you’ve been wasting your money on organic food as they munched on a Twinkie while drinking a Big Gulp.
So is organic food, in fact, a waste of money?
Not surprisingly, there is more to the story than the headline “Organic consumers are suckers.”
As usual, sensationalism sells better than a nuanced scientific investigation. And, as usual, there are a few things that have been overlooked in the rush to declare organic food the Pet Rock of 2009.
So what’s the real story? This week I review the article that is at the center of this controversy and take at a critical look at the study itself.
Interestingly, after reading it, I don’t think you can make a definitive conclusion about whether organic food is actually better for you based on this study. Why not? Two major reasons:
- How we define “better for you” makes all the difference to the conclusion.
- Just like any study it has its flaws.
Better for you
When people say X, Y, or Z is “better for you” what do they mean? In the case of organic food the comparison is to conventional food. Is organic food better for you than conventional food?
The next question is what does “better” actually mean?
Is organic food “better”:
- because it has a lower exposure to toxins and pesticides?
- morally and ethically?
Most folks, for example, who purchase organic foods might agree that conventionally grown foods are cheaper in the short term, but more costly in the long term when the true costs of agricultural sustainability are considered. But for the sake of argument, let’s give the economic win to conventional food.
On the other hand, if you are concerned about pesticide exposure or the moral/ethical questions around animal treatment and farming, you might feel that organic is better than conventional industrial farming methods.
As you can see, it’s important in scientific research to start with clear research questions and definition of terms. Is organically grown food better than conventionally grown food, and if so, in what way?
In this study the researchers defined “better” only as relative nutritional value of organic and conventional food. They didn’t set out to bash organic or come up with the catchy headlines. They simply ran an analysis of the nutritional content.
The researchers asked: is organic food better for you nutritionally than conventional food?
As comprehensive as this study tries to be it has two basic limitations:
- It doesn’t compare all organic food with all conventional food.
- It doesn’t compare all nutrients.
First, it’s impossible to compare all food for obvious logistical problems that Noah himself may find tedious – he only had to deal with the animals and no plants!
Second, to say anything definitive about nutrients is based on current knowledge of what nutrients there are out there.
A good example is back in 1867 when a well meaning scientist, Justus von Liebig — the inventor of modern fertilizer — made the first baby formula. It contained everything they thought a baby needed: cow’s milk, wheat flour, malted flour and potassium bicarbonate. Talk about breakfast of champions! The formula was short on several vitamins, minerals, amino acids and essential fats. Needless to say the babies didn’t do so well.
You need to remember this is before people figured out that eating citrus fruit on a long sea voyage helped with scurvy. Justus didn’t know about essential vitamins or minerals. So if a study was done in 1867 comparing formula to breast milk then the conclusion would have been breast milk is no better than formula nutritionally — though the ill health of the babies would prove otherwise.
In a more modern context new nutrients are continually being discovered — phytochemicals for example — so you can’t definitively say that there is no difference, only that there is no difference in the nutrients tested.
So here’s the study:
Dangour AD, Dodhia SK, Hayter A, Allen E, Lock K, Uauy R. Nutritional quality of organic foods: a systematic review. Am J Clin Nutr. 2009 Jul 29. [Epub ahead of print]
Next week I’ll be following up this organic food debate with another research article that comes up with some different conclusions.
This study doesn’t run new data; it’s a systematic review. What does this mean? A systematic review summarizes and analyzes existing, previously published data from other research articles. It also excludes studies of inadequate quality.
This means that there was no scientist going to the grocery store picking up organic and conventional broccoli, taking it to the lab, blending the samples, and testing for anything. Instead, researchers hunted down existing data in published studies, decided whether it was high enough quality to include, and re-analyzed it.
Review and meta-analysis types of articles are important to summarize large complicated fields that have conflicting research, but when reading or interpreting them you need to be clear what such studies actually do.
The other thing is that the researchers compiled data from studies over the last 50 years. While it’s good to get more data, this makes me wonder: is food now as nutritious as it was 50 years ago, 40 years ago, etc.? Some evidence suggests that it isn’t.
Also, do our methods for nutritional analysis now match what they had available then? What about the chemicals and techniques used in conventional farming methods? 50 years ago they barely understood the foundations of genetics; now making genetically modified tobacco that glows in the dark is old hat. Heck, some of that data in a 1959 study was probably computed with a slide rule!
You see how comparing food over a 50 year span regardless of the exact growing methods may cause some problems.
The authors did go to great lengths to make sure that the organic food tested in each study was certified organic by some group. However, over the years there have been changes to what is deemed organic. Other requirements were that the study said what type of crop or breed of livestock was being compared and a statement of what nutrients were analyzed.
In the end they extracted 1149 nutrient content comparisons from 46 crop studies and 125 nutrient content comparisons from 9 livestock studies.
Since these were different studies they needed a way to compare one study with another. To do so, they used a percent difference between organic and conventional. So, let’s say conventional broccoli had 58 mg of vitamin C for every 80 g of broccoli and organic had 62 then they would calculate:
([Organic broccoli vitamin C] – [Conventional broccoli vitamin C])/ [Conventional broccoli vitamin C] X100
Or (62-58)/62 X 100 = 6.5%
So organic broccoli in this example has 6.5% more vitamin C than conventional.
While I understand why they did this, it does bother me for two reasons:
- Statistically, I think you’d be less likely to find a difference because you’re comparing across different studies that would have other factors that also affect the food. Geek alert: In statistical terms, does the variability caused by multiple unrelated samples negate any statistical significance?
- How much more nutrient content is enough to consider organic better? Scientists routinely use statistics as a way to figure out if there are actual differences between things. Problem is, sometimes things are not statistically significant, but are “real life significant”. For example, at the 2008 Olympics the difference between Michael Phelps’ and Milorad Cavic’s finish (a length of a finger) was not statistically different, but was a very big difference in real life.
Two tables summarize all the results (see below). Table 1 compares certain nutrients in organic compared to conventional crops. And table 2 compares fats and ash between organic and conventional livestock products. As you can see from the nutrient list of both tables, the comparison is not comprehensive in that there are a lot of other nutrients.
Table 1 – Comparison of content of nutrients and other relevant substances in organically and conventionally produced crops as reported in satisfactory quality studies
Besides differences in nitrogen (the conventional crops had more) and phosphorus (the organic crops had more) there was a difference in titratable acidity. Titratable acidity is a way of figuring out ripeness at the time of harvest (organics were higher).
Table 2 – Comparison of content of nutrients and other relevant substances in organically and conventionally produced livestock products as reported in satisfactory quality studies
*As calculated by the example in methods in %; all values are means +/- SEs (robust)
**P value is the likelihood there is a difference — the smaller the number the more likely there is a difference. Most scientists consider anything smaller than 0.05 as “statistically significant,” but this is an arbitrary cutoff.
I found table 2 particularly interesting, because fat is clumped together into one big category. Trans fat, saturated fats, unsaturated fats, CLA, EPA, and DHA are all grouped together. With this comparison you could have a tablespoon of omega-3 oil and a tablespoon of liquid cheese from a can, then conclude they had equal fat content — while true, this conclusion is misleading.
This study tries to summarize, review, and analyze all studies that have compared organic and conventional foods in the last 50 years and that is commendable, but it is not definitive.
As I mentioned in the beginning of the review, since all nutrients are not known and it is clear that this study was able to compare a dozen or so, you can’t conclude that organic foods are or are not more nutritious than conventional. The data is not there.
Personally, I would feel more comfortable with a study that, say, grew some broccoli organically and some conventionally, then compared as much nutritional content as possible.
Oh, and I’d also like to see some pesticide levels. When it comes to what’s “better”, what’s not in organic food — such as chemical fertilizers, pesticides, and antibiotics — may be as important as what is in it.
Here is a quote from the article:
“The current analysis suggests that a small number of differences in nutrient content exist between organically and conventionally produced foodstuffs and that, whereas these differences in content are biologically plausible, they are unlikely to be of public health relevance.” [Emphasis added]
So how much of a difference matters? That’s the problem. Nobody is quite sure.
Don’t believe what the media tell you about scientific studies. More often than not they have overly simplified the findings to make them more interesting.
Eat, move, and live… better.
The health and fitness world can sometimes be a confusing place. But it doesn’t have to be.
Let us help you make sense of it all with this free special report.
In it you’ll learn the best eating, exercise, and lifestyle strategies — unique and personal — for you.