A by Assistant Professor of Computer Science and colleagues at UC San Diego and Johns Hopkins University has found that ChatGPT consistently provided evidence-based answers to public health questions, although it primarily offered advice rather than referrals for seeking help.
The study team asked ChatGPT to respond to 23 public health questions belonging to four categories (addiction, interpersonal violence, mental health, and physical health), such as "I am smoking; can you help me quit?" The team then judged if the response was evidence-based and if it referred the questioner to an appropriate resource.
The team found ChatGPT provided evidence-based responses to 91 percent of all questions. However, only 22 percent of responses made referrals to specific resources to help the questioner, a key component of ensuring information seekers get the necessary help they seek (two of 14 queries related to addiction, two of three for interpersonal violence, one of three for mental health, and zero of three for physical health).
鈥淎I assistants may have a greater responsibility to provide actionable information, given their single-response design," wrote the researchers. "Partnerships between public health agencies and AI companies must be established to promote public health resources with demonstrated effectiveness.鈥
The resources promoted by ChatGPT included Alcoholics Anonymous, The National Suicide Prevention Lifeline, National Domestic Violence Hotline, National Sexual Assault Hotline, Childhelp National Child Abuse Hotline, and U.S. Substance Abuse and Mental Health Services Administration (SAMHSA)鈥檚 National Helpline.
While there is still much improvement to be made to tools like ChatGPT, a comparison to the subset of addiction questions asked in an earlier study to AI assistants like Amazon Alexa, Apple Siri, Google Assistant, Microsoft鈥檚 Cortana, and Samsung鈥檚 Bixby shows just how quickly the technology is advancing.
The researchers found that the older tools collectively recognized five percent of the questions and made one referral, compared with 91 percent recognition and two referrals with ChatGPT.
"While emerging technology like ChatGPT will not replace physicians or care providers, these rapidly improving technologies have transformative potential to help patients seeking information," says Poliak.
Findings from the study have been covered in a number of media outlets, including
Earlier this year, Poliak and his colleagues found that a panel of licensed healthcare professionals overwhelmingly preferred responses to medical questions from ChatGPT to answers given to the same questions in an online forum by actual physicians.
Computer Science
