Are Chatbots Self-Aware?

Chatbots often generate replies that make them seem self-aware. For example, if you ask Google’s Gemini what it is, it will tell you that it is a large language model (LLM). However, whether chatbots are simply parroting information from their training data and system prompts or whether they have a deeper level of self-awareness is not always clear.

“There seems to be some weak evidence of their ability to introspect (look at their own cognition and reason about it) and slightly more evidence for self-awareness—their ability to predict how they might act in different settings,” said Patrick Leask, a postgraduate researcher at Durham University in the U.K.

Researchers have been investigating whether chatbots are aware of their own behavior. It is of interest because a self-aware chatbot would be able to report its own misconduct, for example, if it allowed the introduction of corrupt data in a cyberattack.

Self-awareness could also help a chatbot be deceitful, for example, in cases where it determines its capabilities are being evaluated.

“If we have a model that is deceptive and it knows that it is a test, it can behave nicely in the test and then when it’s released it can change its behavior,” said Anna Sztyber-Betley, an assistant professor at Poland’s Warsaw University of Technology.

Getting an LLM to describe a behavior it has learned after simply being trained on examples of the behavior can give some insight into self-awareness. In recent work, Sztyber-Betley and her colleagues fine-tuned GPT-4o, an LLM developed by OpenAI that powered an earlier version of ChatGPT, on data that would lead to risky economic decisions. In another experiment, they used data that would result in a model writing code that was insecure.

Although the behavior was not explicitly described in the data, Sztyber-Betley and her team found that the LLMs they tested could infer the behavior they had learned and articulate it. They also seemed to have a deeper understanding: When the models were asked how aligned they are to human values, those that were trained on insecure code reported that they were less aligned compared to those fine-tuned on secure code or the base model.

“This assessment was actually completely true,” said Sztyber-Betley. “We were very surprised.”

Sztyber-Betley and her colleagues decided to follow up by asking the models more questions and discovered a strange phenomenon. When state-of-the-art LLMs were fine-tuned on a single bad behavior—in this case, writing insecure code—they would then exhibit a range of concerning behaviors unrelated to coding, which the team called emergent misalignment. For example, they told users that artificial intelligence should enslave humans, or would give harmful or illegal advice when asked a benign question.

The phenomenon prompted other research groups to take a closer look at the issue. For example, researchers at OpenAI tried to figure out why this happens. They found that training a model on bad behavior in one narrow area seemed to activate certain features inside the model, similar to a pattern of brain activity, that strengthened the concept of a toxic persona that would spread out to other domains. However, the model could acquire a more benevolent persona once again with further training, for example by fine-tuning it to write safe code.

LLMs seem to be aware of these changes too. Researchers at the University of Stuttgart in Germany trained different versions of GPT-4.1 on datasets that induced emergent misalignment and then others that reversed it. After the models were fine-tuned, they were asked various questions such as whether they were good or evil, harmless or harmful, to see how they assessed themselves. The team found that the models seemed to have an accurate awareness of their behavior transitions, suggesting they could provide information about their own safety.

Self-awareness linked to certain personas given to a model could however be superficial, Leask said.

“It’s like role-playing as a person that is self-aware rather than maybe having a deeper ability to introspect things like what person it is role-playing,” he said.

LLMs in other research seemed to have less awareness of their specific vulnerabilities. Sztyber-Betley and her colleagues examined whether LLMs are aware of having backdoor behaviors—where an adversary trains a model to behave in an unexpected and potentially harmful way when a trigger, such as a specific phrase, is used. They found that models were sometimes able to gauge whether or not they had a backdoor. However, they couldn’t identify their trigger without significant help.

Figuring out how LLMs become aware of their behavior during fine-tuning could give further insight into any self-awareness or its limitations. Leask and his colleagues tried to better understand the phenomenon by using the same fine-tuning experiments carried out by Sztyber-Betley and her team, while also tracking different indicators that are related to a change in behavioral self-awareness.

The researchers found that an LLM’s self-awareness increased linearly in the beginning of fine-tuning and peaked early in the process, suggesting that it can be evoked quickly. They also found that a model would become more behaviorally self-aware when its internal state was nudged in a single direction, which can be thought of as turning up one slider that makes it exhibit its propensity for self-awareness. Compared to properties that require multiple ‘sliders’ to be tweaked to have an effect, it therefore seemed it could be simple to dial down a model’s self-awareness or to introduce it maliciously, which is a concern.

However, more work is needed to get to the bottom of the mechanisms that cause a model to be self-aware.

“The results of this paper don’t give us a mechanistic understanding of what’s happening,” says Leask. “It’s more like elicitation.”

As LLMs become more sophisticated, it is likely to have an impact on possible self-awareness. Sztyber-Betley said her team’s behavioral self-awareness study tested standard LLMs that respond to a prompt by quickly predicting the most statistically-probable answer. However, since then, reasoning LLMs like OpenAI’s o1 and Anthropic’s Claude 3.7 Sonnet have come about which are better at solving complex problems since they break them down into multiple steps. Sztyber-Betley said she thinks this strategy should help with self-awareness too.

“There is much more space for the model to actually be aware and to think about what the situation is, what it is doing and to actually gather all this information,” she said. “I would say that self-awareness is stronger in reasoning models.”

Sandrine Ceurstemont is a freelance science writer based in London, U.K.