A study from ChatGPT found that the artificial intelligence tool correctly answered less than half of the test questions from a study resource commonly used by doctors when preparing for board certification in ophthalmology.
The study, published in JAMA Ophthalmology and run by St. Michael’s Hospital, a Unity Health Toronto site, found that ChatGPT correctly answered 46% of the questions when initially administered in January 2023. When the researchers administered the same test a month later, ChatGPT scored more than 10% higher.
The potential of AI in medicine and test preparation has generated excitement since ChatGPT became publicly available in November 2022. It also raises concerns about the potential for misinformation and cheating in academia. ChatGPT is free, available to anyone with an Internet connection, and works conversationally.
“ChatGPT may have an increasing role in medical education and clinical practice over time, however it is important to emphasize the responsible use of such AI systems,” said Dr. Rajeev H. Muni, the study’s principal investigator. and Li researcher. Ka Shing Institute of Knowledge in St. Michael’s. “ChatGPT, as used in this research, did not correctly answer enough multiple-choice questions to provide substantial assistance in preparing for board certification at this time.”
The researchers used a data set of multiple-choice practice questions from the free test from OphthoQuestions, a common resource for board certification exam preparation. To ensure that ChatGPT answers were not influenced by simultaneous conversations, ChatGPT entries or conversations were cleared before each question was entered and a new ChatGPT account was used. Questions using images and videos were not included because ChatGPT only accepts text input.
Of 125 text-based multiple-choice questions, ChatGPT correctly answered 58 (46%) questions when the study was first conducted in January 2023. The researchers repeated the analysis on ChatGPT in February 2023, and performance improved to 58%. .
“ChatGPT is an artificial intelligence system that holds great promise in medical education. Although it provided incorrect answers to ophthalmology board certification questions about half the time, we anticipate that ChatGPT’s body of knowledge will evolve rapidly,” said Dr. Marko Popovic, a study co-author and resident physician in the Department of Ophthalmology and Vision Sciences at the University of Toronto.
ChatGPT closely matched how students answered the questions, selecting the same multiple-choice answer as the most common answer provided by ophthalmology students 44 percent of the time. ChatGPT selected the multiple-choice answer that was least popular with ophthalmology trainees 11% of the time, second least popular 18% of the time, and second most popular 22% of the time.
“ChatGPT performed more accurately on general medicine questions, answering 79% of them correctly. On the other hand, its accuracy was considerably lower on questions about ophthalmology subspecialties. For example, the chatbot answered 20% of the questions correctly on oculoplastics and zero percent correct from the retina subspecialty. The accuracy of ChatGPT will likely improve more in niche subspecialties in the future,” said Andrew Mihalache, the study’s lead author and an undergraduate student at Western University.
—————————————————-
Source link