By: Kate White & Ioana Gorecki (Kelley Drye)
The National Eating Disorder Association recently encountered an unexpected issue with their mental-health chatbot. Instead of offering support to individuals seeking help for eating disorders, the chatbot began giving diet advice. Initially developed as a closed system, the chatbot was later enhanced with an AI component in 2022, without consulting or obtaining authorization from NEDA. Consequently, NEDA promptly deactivated the chatbot to address the situation.
This incident highlights the potential risks that companies may face when utilizing AI chatbots for customer service and addressing consumer needs. Recognizing the significance of the issue, regulators and law enforcement agencies such as the CFPB and FTC have issued blog posts and reports cautioning companies against excessive reliance on chatbots and generative AI for customer service and resolving consumer concerns…
