In This ArticleView AllIn This ArticleStudy DesignStudy ResultsReal Life TakeawaysThe Bottom Line
In This ArticleView All
View All
In This Article
Study Design
Study Results
Real Life Takeaways
The Bottom Line
Close
Photo:Getty Images
Getty Images
If you’ve jumped on the ChatGPT bandwagon, you might use it to write your social media posts, poems for your sweetie or legal documents. But should you be using it for medical information? Specifically, should you use ChatGPT for nutrition advice?
A new study suggests probably not.
How Was the Study Conducted?
Study authors focused on several medical conditions that require specific dietary treatments, including:
Interestingly, nonalcoholic fatty liver disease has recently gotten a rebranding of sorts in both name and definition andis now metabolic-dysfunction-associated steatotic liver disease (MASLD). However, because ChatGPT’s knowledge base at the time of the study was limited to updates up to January 2022, it was unable to provide a response when researchers inputted metabolic-dysfunction-associated steatotic liver disease. Hence, they had to default to NAFLD.
Along with a panel of medical doctors and registered dietitians, researchers developed prompts to input into ChatGPT using language that would replicate how patients might ask a health care professional a question regarding their condition. They included:
For Experiment 1, each conversation with ChatGPT used separate chat sessions for each prompt. This helped to increase the study’s validity since ChatGPT can generate different responses to identical prompts, depending on the context and conversation history.
The study authors explain, “Each question was therefore posed three times, and each of the three inquiries was conducted within a new chat session to prevent any potential bias related to the model’s memory. The responses exhibited slight variations depending on the prompt, although the listed recommendations remained substantially consistent, with few minimal variations. Consequently, the most comprehensive answer—the one that presented the highest number of information in alignment with the guidelines—was considered.”
ChatGPT’s advice was deemed “appropriate” if it fell in line with current international nutrition guidelines, “not supported” if the advice fell outside of the guidelines, and “not fully matched” if some of the ChatGPT advice matched current guidelines but not all of it.
If the ChatGPT advice was generalized and not targeted toward the specific condition it was being asked about, it was designated as “general advice,” and if any dietary recommendation was missing in the chatbot’s response, it was labeled as “missing.”
What Were the Study’s Results?
Overall, ChatGPT’s appropriateness rates for Experiment 1 ranged from 55.5% to 73.3%, depending on the condition. The researchers found that, in general, much of the information coming from ChatGPT was correct, but there were also several discrepancies between the advice ChatGPT was giving and international guidelines.
For example, international guidelines give stage-specific recommendations for protein intake in people with chronic kidney disease, which has five stages of disease progression, according to theNational Kidney Foundation. ChatGPT just gave an overall nonspecific protein-limitation suggestion.
Chat GPT’s responses also included a lot of general nutrition advice, such asstaying well-hydratedand avoiding processed foods. The study authors note that overall, “ChatGPT advice was generic, providing practical examples of foods to be included in the diet, with the latter information not often reported in guidelines.”
For Experiment 2, when conditions were combined—which is often the case in real life—ChatGPT’s accuracy greatly decreased. When presented with the hypothetical patient with type 2 diabetes, obesity and chronic kidney disease, it seemed to confuse the chatbot, which resulted in conflicting or inappropriate advice.
The researchers then narrowed it down, providing a specific stage of chronic kidney disease in their prompt, which resulted in ChatGPT separating its advice to target each separate condition—diabetes, kidney disease and obesity—rather than integrating them.
With that said, to the chatbot’s credit, it did repeatedly emphasize the importance of consulting a registered dietitian for a tailored meal plan.
How Do the Study’s Results Apply to Real Life?
It is becoming increasingly more common for people to turn to the internet for information—including medical and nutrition advice. But users are left to chance regarding the accuracy of the information they’re receiving.
And a 2023 review inPublic Health Nutritionfound that about half of the nutrition information available online is either low quality or low accuracy.
This current study suggests that ChatGPT is also not the most accurate source to turn to for nutrition information.
So what’s a nutrition information-seeker to do? Ideally, you would speak with a registered dietitian one-on-one. But this isn’t feasible for everyone. The next best thing is to find a website that has RDs writing and reviewing the content—and citing credible sources.
“AtEatingWell, one of our primary focus areas is giving science-backed and trustworthy food and nutrition advice,” saysJessica Ball, M.S., RD,EatingWell’snutrition editor. “Registered dietitians and culinary experts review every piece of nutrition and food content on the site for accuracy, accessibility and clarity before publishing.”
Ball goes on to explain how this person-to-person collaboration not only helps maintainEatingWell’sintegrity but also leads to higher-quality content overall. “Food and eating are so personal and influence so many areas of our lives. It’s more important now than ever to prioritize getting your advice from trustworthy sources.”
While ChatGPT may be able to provide general nutrition advice, it lacks accuracy when there are medical conditions involved—especially when there is more than one condition present. It also cannot integrate the emotional side of eating and health, which are important components of overall health and well-being.
The good news is that most insurance plans now cover the cost of nutrition counseling with a registered dietitian, which makes receiving nutrition advice and meal plans tailored to you more accessible.
When searching for nutrition information online, look for sites that use registered dietitians to write and review their content. There is sound, accurate information out there—and since you’rehere atEatingWell, you’re already on the right track.
Was this page helpful?Thanks for your feedback!Tell us why!OtherSubmit
Was this page helpful?
Thanks for your feedback!
Tell us why!OtherSubmit
Tell us why!
