ChatGPT's translation of mental health questionnaires into Mandarin reveals inconsistent results

Medical Xpress

Source published: Site published:

Original article · فارسی

by University of Canterbury

Edited by Lisa Lock, reviewed by Andrew Zinin

chatgpt
Credit: Matheus Bertelli from Pexels

More Asian New Zealanders are reporting mental health problems. A researcher at Te Whare Wānanga o Waitaha | University of Canterbury (UC) is asking an important question: Can artificial intelligence (AI) help overcome language barriers when there are not enough human translators?

A study led by UC translation studies researcher Dr. Wei Teng looked at how well ChatGPT could translate two commonly used mental health screening questionnaires into Mandarin Chinese for use in Aotearoa New Zealand. The questionnaires were the Kessler Psychological Distress Scale (K10) and the Patient Health Questionnaire-9 (PHQ-9).

"These questionnaires help decide whether someone may be experiencing emotional distress or depression. So, if a translation contains a mistake, it can directly affect decisions about their care," Teng says.

Teng cares deeply about making services easier for everyone to access. He started the project after speaking with Josiah Jennings from the Christchurch-based Asian Community Transformation Trust and registered clinical psychologist Dr. Amy Wang. During these conversations, community leaders explained that language problems were stopping Mandarin-speaking migrants from getting the mental health support they needed.

Mandarin translations of these questionnaires already exist. However, Teng says some sound too formal or unnatural. "If the Mandarin questions sound like a robot wrote them and lose the feeling of the original words, patients may not feel comfortable answering honestly," he says.

To find out whether AI could create translations that sounded more natural and caring, Teng gave ChatGPT carefully planned instructions. He told it to act as a professional translator and use a tone that was "caring" and "clinically appropriate." One carefully designed prompt gave much better results. This suggested that clear and detailed instructions could help AI produce better translations.

However, the study found one major problem: the results were not consistent.

When Teng's colleagues used the same prompt on their own free ChatGPT accounts, they received different translations. The tone and naturalness of the answers varied. Large language models can produce different answers depending on users' previous interactions with their accounts. This means that different people may get different results even when they use exactly the same instructions.

"Consistency is important in mental health," Teng says. "A screening questionnaire should work the same way every time. Different results make us question whether it is reliable and whether others can get the same results when they repeat the process."

Teng is now working with AI researchers to find out whether more predictable instructions or specially designed AI models could make the translations more consistent. He says that although AI could be useful, experts must still check its work, and people from different areas of expertise need to work together.

"AI may have a place in this work," he says. "But when people's well-being is involved, trust and respect for different cultures must come first."