ChatGPT configuration led expert ratings of dental radiology advice
In a controlled study, researchers compared AI tools designed for the public that answered questions about dental X-rays and radiation safety. Experts gave the tested ChatGPT configuration the highest ratings for accuracy, relevance, safety and how little misinformation it contained. The study, published in Frontiers in Oral Health, tested three platform configurations using 20 questions in English. Each platform answered all 20 questions, producing 60 responses in total. Three oral and maxillofacial radiologists, who specialize in imaging the mouth, jaws and face, independently rated the answers. These findings apply only to the specific versions, access plans and individual answers tested. They do not prove that one underlying AI model is generally better than the others.
The researchers gave each system the same instructions: answer as if speaking to a patient, using clear, simple and friendly language. The products tested were ChatGPT-5.5 with its user-visible “Thinking” mode turned on, accessed through a paid plan; the free version of Gemini 3 Flash; and the free version of Microsoft Copilot 2026. The researchers collected the answers in late April 2026, using the same device, browser and internet connection. They started a new chat for every question and turned off chat history, memory and earlier conversation context. Each platform produced exactly one answer per question. The researchers did not ask follow-up questions or generate additional answers. Therefore, the study examined the specific answers produced at that time, not how consistent the systems would be across repeated attempts.
The researchers began with 57 possible questions and selected 20 after asking seven volunteer oral and maxillofacial radiologists to review them. These experts were not study authors and did not later rate the AI answers. The final questions covered radiation exposure, how often imaging is needed, whether imaging is necessary, radiation risks, imaging in children and during pregnancy, cone-beam computed tomography, radiation protection, common misunderstandings and when to seek professional advice. The researchers also used international radiation-safety guidance and professional resources. However, they had not kept the original classifications for each question or detailed calculations showing how well the questions covered the subject. For this reason, they described the questions as reviewed by experts rather than formally validated.
Each answer was rated in seven areas. Accuracy, relevance, clarity, completeness, safety and whether the answer gave suitable advice about what to do next were scored from 1 to 5. Higher scores meant better performance. Misinformation was scored from 0 to 4, with lower scores meaning less misinformation. Accuracy measured whether the facts matched the evidence prepared for each question. Completeness measured whether important information was missing. Safety considered the possible harm of following an answer, such as giving false reassurance, causing unnecessary worry, discouraging imaging that was needed or delaying professional care. Radiologists judged clarity themselves; patients did not test the answers. So the clarity scores reflected experts' views about how easy the answers seemed to understand, not measured patient understanding.
The researchers treated the 20 questions as the main units for comparing the platforms. The three experts' ratings of the same answer were related to one another, so they could not be treated as fully independent observations. The researchers used Friedman tests to look for differences among the platforms in six areas, followed by two-sided exact sign tests to compare pairs of platforms question by question. They also examined whether scores were clustered at the top of the scales, whether ratings were tied and how closely the experts agreed. The results showed that the tested ChatGPT configuration performed better than both alternatives in several areas. However, some scores were already near the highest possible level. Many ratings were identical, and some very large estimated differences were based on only a small number of questions where the platforms received different scores.
ChatGPT received the highest accuracy rating on all 20 questions. Gemini and Copilot received the same accuracy rating on 19 questions. All 60 accuracy ratings given to ChatGPT by the three experts were 5, the highest possible score. For Gemini, 59 ratings were 4 and one was 3. Every Copilot accuracy rating was 4. The results for safety followed the same pattern: ChatGPT received a score of 5 in all 60 ratings, while Gemini and Copilot each received a mix of 3s and 4s. Gemini and Copilot tied on all 20 questions for safety. After the researchers adjusted for multiple comparisons, the differences between ChatGPT and each alternative remained statistically significant at the question level. However, the maximum scores also show that the rating system had little room to separate the strongest answers from one another.
ChatGPT also received better relevance ratings than Gemini on 18 questions and than Copilot on 19 questions. Both differences remained statistically significant after adjustment. For clarity, ChatGPT scored better than Copilot on 16 questions. For completeness, it scored better than Copilot on all 20 questions. Gemini also received better clarity ratings than Copilot on 10 questions and better completeness ratings on 19 questions. However, the differences between ChatGPT and Gemini in clarity and completeness were not statistically significant. For completeness, the two platforms tied on 17 of the 20 questions. For clarity, ChatGPT received the higher rating on 10 questions, Gemini received the higher rating on seven, and the platforms tied on three. This means the study did not establish one clear winner in every area of answer quality.
The misinformation scores showed a similar pattern. All 60 ChatGPT ratings were 0, the best possible score on this scale. Gemini and Copilot received a mixture of 0, 1 and 2. For every question, ChatGPT was rated as containing less misinformation than either alternative. Gemini and Copilot did not differ significantly from each other. In a separate review conducted after the original ratings, the researchers identified advice about lead aprons or thyroid collars that was outdated or lacked important qualifications. This issue appeared in one of the 20 ChatGPT answers, 12 of the 20 Gemini answers and 11 of the 20 Copilot answers. The review also found claims that radiation protection was complete or nearly complete in six Gemini answers and one Copilot answer. Such claims conflict with current guidance from the American Dental Association and the American Academy of Oral and Maxillofacial Radiology, which advises against routine shielding during dental X-rays, including for children and pregnant patients. Local rules and legal requirements still apply.
Several important limits mean the findings must be interpreted carefully. The three platforms were not tested under equivalent conditions. ChatGPT had its reasoning feature turned on and was accessed through a paid plan, while Gemini and Copilot were used through free plans. Differences in the results could therefore be caused by access level, reasoning resources, interface design, information-retrieval features or hidden choices about which systems handled the requests, rather than by the AI models alone. The researchers hid the platform names from the experts who rated the answers. However, headings and writing styles may still have revealed which platform produced an answer, and the researchers did not formally test whether the masking worked. The scoring guide created for this study had not been independently validated. The researchers also did not record specific explanations for inconsistent ratings, and the later review of shielding advice was not a blinded process for formally deciding which answers contained errors. Only one question could be used to assess whether the platforms gave suitable advice about what to do next: what a worried patient should do when a dentist recommends an X-ray. All three platforms had a middle triage score of 4, but this result came from just one question and was descriptive only. It cannot support broad comparisons of their ability to guide patients.
Most importantly, the study measured what radiologists thought about the written answers, not what happened to patients. It did not measure whether patients understood the advice, trusted it, felt anxious because of it, agreed to imaging, sought medical or dental care, or experienced different clinical outcomes. It also did not test whether the answers changed patients' decisions. The high level of agreement between the radiologists needs careful interpretation too. When the researchers combined the ratings, the intraclass correlation coefficients, which measure how closely ratings agree, ranged from 0.696 for clarity to 0.976 for accuracy. However, much of this agreement reflected large differences between the platform configurations. It did not necessarily mean that the experts could reliably distinguish between individual answers from the same platform. Stronger conclusions would require repeated answer generations, equivalent platform configurations, verified masking of platform identities, more complex clinical situations, direct testing with patients and a formal process for checking factual errors. For now, the results suggest that selected consumer AI systems can produce dental X-ray information that experts consider understandable. They also show why individual AI answers should not be treated as consistent, clinically validated replacements for professional advice.
Subject of Research: Expert comparison of AI answers written for patients about dental imaging and radiation safety.
Article Title: Expert-rated accuracy and safety of patient-facing dental radiology responses from three AI platform configurations: a comparative evaluation
Article References: Kapoor, D., Garg, D., Nirola, A., Sunil, M. K., Verma, P., & Tadakamadla, S. K. (2026). Expert-rated accuracy and safety of patient-facing dental radiology responses from three AI platform configurations: a comparative evaluation. Frontiers in Oral Health, 7, Article 1937905. https://doi.org/10.3389/froh.2026.1937905
Image Credits: AI-generated
DOI: 10.3389/froh.2026.1937905
Keywords: artificial intelligence, dental radiology, radiation safety, patient education, large language models, ChatGPT, Gemini, Microsoft Copilot, misinformation, oral health, expert ratings, accuracy