Skip to main content

Why diet is hard to study

Testing a drug is hard enough. Testing a diet is harder. People eat thousands of different foods in shifting combinations, diseases such as heart disease and cancer take decades to appear, and nobody can be locked in a room and fed a fixed menu for thirty years. Nutrition science therefore leans on several kinds of evidence, each with its own blind spots.

Watching people over time

Much of what we know comes from large observational cohorts, which follow tens of thousands of people for years and record who falls ill. The Nurses' Health Study, begun in the United States in 1976, enrolled more than 120,000 nurses. To learn what participants eat, such studies usually rely on a food frequency questionnaire. A food frequency questionnaire asks how often a person ate each food on a long list over a past period, often the previous year. It is cheap but depends on memory, and few people can say with confidence how many servings of spinach they ate last spring.

What people forget

A clever tool exposed how large that problem is. A person drinks a dose of water in which hydrogen and oxygen are replaced by rare, harmless heavier versions. By tracking how fast each leaves the body in urine, researchers can calculate how much carbon dioxide the body made. The doubly labeled water method measures how much energy a person actually spends over one to three weeks, and in a person whose weight is stable, that equals what they eat.

The results were sobering. In a 1992 study, Steven Lichtman and colleagues examined people who said they could not lose weight despite eating very little. The "diet-resistant" volunteers ate far more than they reported, close to twice as much, and overstated their exercise. Their metabolism was normal. Later studies found that underreporting is common in the general population and grows with body weight.

Confounding

Even accurate records cannot settle cause. People who eat more vegetables also tend to smoke less, exercise more, earn more and see doctors more often. Healthy user bias is the tendency of people who follow one health habit to follow others too, which can make any single habit look more powerful than it is. Researchers adjust their statistics for factors they measured, but they cannot adjust for those they did not.

When trials overturn observations

The cleanest test is a randomized trial, and nutrition has a cautionary tale about why trials matter. Observational studies in the 1980s found that people who ate more fruit and vegetables rich in beta carotene had less lung cancer. In Finland, a trial of about 29,000 male smokers tested beta carotene pills directly. Published in 1994, it found that smokers given beta carotene got more lung cancer, not less. An American trial, CARET, was stopped early in 1996 for the same reason. Whatever protected vegetable eaters, it was not this one molecule in pill form.

Relative and absolute risk

Headlines usually report relative risk. In 2015 an expert group of the World Health Organization classified processed meat as a cause of bowel cancer, estimating that each 50 grams eaten daily raised the risk by about 18 percent. That sounds alarming. In absolute terms, it means moving a lifetime risk from roughly 5 in 100 people to roughly 6. A rise from 5 to 6 in 100 is an absolute increase of one percentage point but a relative increase of about 20 percent. Both numbers are true; the absolute one says more about a single person's choice.

Too many findings

In 2013 Jonathan Schoenfeld and John Ioannidis picked 50 ingredients at random from recipes in a classic cookbook and searched the medical literature. Of the 50 cookbook ingredients, 40 had been studied for links to cancer risk, and most of those studies reported either more or less cancer. Many of the effects were weak and would not hold up when studies were pooled. Small studies, many comparisons and a preference for publishing striking results all push in the same direction.

Newer tools

Researchers now combine methods. Mendelian randomization uses gene variants that affect a person's exposure as a natural experiment: because genes are shuffled at conception, people who carry a variant that raises cholesterol, for instance, are otherwise similar to those who do not. Trials also often measure surrogate endpoints, such as blood pressure or cholesterol, instead of heart attacks and deaths, because they are faster to change. Surrogates help, but they can mislead when a treatment moves the marker without changing the outcome that matters.

Worked example: coffee drinkers live longer

A headline announces that coffee drinkers live longer. A careful reader asks four questions. What kind of study? A large cohort, so it shows association, not proof of cause. How was coffee measured? By questionnaire, once, years ago. What might be confounding it? Very ill people often stop drinking coffee, and in older studies coffee drinkers smoked more, which would hide a benefit rather than create one. How big is the effect? About a 10 percent lower relative risk of death over the study, which in absolute terms is small. Many cohorts in different countries show the same pattern, which makes it more believable. The fair conclusion is that coffee is probably not harmful for most adults and may be mildly protective, not that it is a health drink.

Quiz

5 questions · Practice: scored, not saved

1. What does a food frequency questionnaire ask?
2. What does the doubly labeled water method measure?
3. What did Lichtman's 1992 study of "diet-resistant" volunteers find?
4. What is healthy user bias?
5. What did the Finnish beta carotene trial find in 1994?
6. A food raises a disease risk from 5 to 6 in 100 people. What is the relative increase?
7. What did Schoenfeld and Ioannidis find in 2013?
8. What does Mendelian randomization use as a natural experiment?

CivilSimian.com created by AxiomaticPanic, CivilSimian, Kalokagathia