Last week, we reported that some observers say statistics are often misused and misinterpreted in food research. Now, another new study raises questions about research that relies on subjective self-reports by participants of their own behaviour…
Experiments should never rely on participants’ own subjective
reports or recollections of their own conditions or behaviour.
“We found that while people generally know that fruits and vegetables are healthy, there may be a disconnect between what researchers and health care professionals consider to be a healthy and balanced diet, compared to what the public thinks is a healthy and balanced diet,” says study author Dr. Jessica Cheng at the Harvard T. H. Chan School of Public Health.
What they did
According to an abstract of the study findings: “Researchers evaluated the diets of 116 adults aged 35-58 years old in the greater Pittsburgh, PA, area who were trying to lose weight. Study participants met one-on-one with a dietitian to discuss their nutrition and then tracked everything they ate and drank every day for one year on the Fitbit app. They also weighed themselves daily and wore a Fitbit device to track their physical activity.”
So, the info on what they ate and day-to fluctuations in their weight were taken as correct and accurate by researchers, while info about their exercise activities was independently and objectively reported via FitBit devices. If the participants they could get away with little white lies about their diets during the study period, they were just kidding themselves.
The findings revealed…
What interested researchers more was how reports participants’ own reports at the beginning and end of the experiment compared with the researchers’ separate reports based on researchers’ interviews every two days during the experiment. The result was a pair of overall Healthy Eating Index (HEI) scores, at the beginning and end of the study based on the types of foods that participants reported eating.
Interpretation
Well… How closely did participants’ own estimates of their HEI scores (on a scale of 1 to 100) compare with reseachers’ calculations at the beginning and end of the study? A difference of 6 points or less between the researchers’ HEI score and the participant’s perceived score was considered ‘good agreement’. A higher difference in scores was considered poor, and twigged researchers that those participants had either lied abut their diets or misremembered them.
What did that ‘raw’ info show?
First: At the end of the study, about 1 in 4 participants’ scores had good agreement between their perceived diet score and the researcher-assessed score.
Second: The remaining 3 out of 4 participants’ scores had poor agreement, and most reported a perceived score that was higher than the HEI score assigned by researchers.
Third: The average participant-perceived score was 67.6, and the average researcher-calculated HEI score was 56.4.
Fourth: Participants’ own estimated over improvement in the healthiness of their diets over the entire 12-month experimental program averaged 18 percent.
Fifth: The resercher-calculated scores over the same period showed an average overall HEI score improvement of just 1 percent.
The takeaway
“People attempting to lose weight or health professionals who are helping people with weight loss or nutrition-related goals should be aware that there is likely more room for improvement in the diet than may be expected,” Cheng observes. She suggests providing concrete information on what areas of their diet can be improved and how to go about making healthy, sustainable nutrition changes.
“Future studies should examine the effects of helping people close the gap between their perceptions and objective diet quality measurements,” she said.
My take
In other words, the so-called stats in which participants’ own reports or recollections of their behaviour were factored could easily corrupt the results of such studies – in all areas of research, not just diet and health – suspicious or downright useless.
I think it’s crucial that researchers be aware of how this ‘little white white lie syndrome’ can effect their experimental outcomes. And we should all be aware of how off-base some new study findings could be as a result of bad experimental design and slushy statistics.
Muse on that…
~ Maggie J.