Review maps how generative AI literacy is evidenced in language education

The researchers carefully matched each type of evidence to the claims it could support. Surveys and other self-report tools were treated as evidence of what people believed about their own AI literacy, including their reported knowledge, awareness, confidence, readiness, or ability. But these reports were not automatically treated as proof that people could actually perform the skills. Studies based on tasks could support claims about real use in specific situations when researchers observed learners or teachers writing prompts, revising work, using feedback, creating teaching materials, or making teaching decisions. Studies that used prompts designed by researchers or compared AI-generated answers were classed as indirect evidence unless they observed participants' own understanding, use, evaluation, or decisions. In total, 22 studies provided direct evidence for at least one of the three areas, while five provided related but indirect evidence without directly measuring any area. This distinction highlights a common problem: confidence, engagement, task results, or better writing cannot by themselves prove that someone has broad AI literacy.

Strategic application was the most frequently studied area, appearing in 21 of the 27 studies. Activities included writing and improving prompts, using AI feedback, revising English writing, creating lesson materials, planning lessons, and using generative AI in classroom activities. This area may be easier to study because researchers can directly observe how people use the tools. Conceptual understanding, however, involves knowing how generative AI works, what it can and cannot do, how it produces likely answers, why it may make up information, how uncertain its answers can be, what limits come from its training data, and how AI-generated content should be understood. These things often require people to explain their thinking or reflect on it, rather than simply answer a questionnaire. The review therefore describes differences in what researchers studied and how they measured it. It does not show that language teachers or learners lack conceptual knowledge.

Critical–ethical evaluation was directly studied in 20 articles, making it almost as common as strategic application. These studies examined checking information, using AI responsibly, academic honesty, who deserves credit for work, confirming AI-generated answers with other sources, recognizing bias, and deciding whether to accept, revise, credit, or reject generated content. However, the depth of these assessments varied. Some studies included checking and explaining decisions as part of writing or teaching tasks. Others recorded general awareness of responsible AI use without observing what people actually did. Stronger evidence came from actions that researchers could observe, such as comparing an AI answer with other information, identifying bias, explaining limitations, and giving reasons for using, changing, or rejecting an answer. These issues are especially important in language education because AI-generated content may be used as language examples, feedback, model texts, revision advice, assessment support, or teaching materials. Its use can affect a person's voice, whether language fits its purpose, whether content suits a culture, and how much control learners have over their own learning.

The researchers looked at both how many areas each study covered and how well those areas worked together. Fourteen studies directly covered all three areas, six covered two, two covered only one, and five included indirect evidence without directly measuring any area. Of the 20 studies that directly covered at least two areas, 18 showed meaningful connections between them, while two showed only limited connections. A study counted as meaningfully connected when the areas it covered were clearly linked in at least three of five parts: aims or objectives, task or teaching design, tools or sources of evidence, results, and interpretation. This did not prove that the areas caused changes in one another or that they formed a fully connected theoretical system. Fourteen studies clearly treated literacy, competence, or a related ability as a goal for teaching or assessment. The other 13 included related skills within broader activities such as writing, feedback, lesson planning, critical thinking, managing one's own learning, or professional development.

When reviewing the quality of the evidence, the researchers rated four studies as having low overall concern, 17 as having some concern, and six as having high overall concern. They focused on whether each study clearly defined what it was measuring, whether its evidence was suitable, and whether its measurements supported its claims. They also considered how relevant the study was and how clearly its methods were reported. Six studies received high-concern ratings for clearly defining what they measured, and seven received high-concern ratings for matching claims to measurements. All five studies that offered only indirect evidence about the three areas received high-concern ratings for claims about AI literacy in those areas because their measurements did not directly show the abilities being discussed. The reviewers made clear that this did not mean those studies lacked value overall. They did not assume that one research method or a larger sample was automatically better. Qualitative studies, case studies, studies with small samples, and self-report studies were not automatically marked down. In a test of how consistently researchers applied the coding rules, eight studies, or 29.6% of the full group, were independently coded by two reviewers. The overall Cohen's kappa score was 0.82 across the presence of each area, whether literacy was an explicit teaching goal, and how well the areas were connected. When reviewers disagreed, they discussed the issue, clarified the rules, and recoded studies when needed.

The review suggests that future research should clearly distinguish between what people think they can do, how they use AI in specific situations, the results they produce, and their overall ability. Researchers could test conceptual understanding by asking people to explain what generative AI can do, its limits, uncertainty, made-up answers, and when it is appropriate to use it. They could observe strategic application by looking at how people improve prompts, use feedback, revise writing, create materials, and plan lessons. They could assess critical–ethical evaluation by checking how people verify sources, identify bias, judge privacy risks, make decisions about authorship, and explain why they accept or reject generated content. The review suggests that think-aloud methods, screen recordings, learning analytics, and learners' reflections could help researchers connect the steps people take with their final work and self-reports. Long-term studies could also track changes over time. More research is needed across school levels, language backgrounds, geographical regions, and levels of access to resources because the studies found were concentrated in higher education, teacher education, and East Asian settings.

Within the limits of the search, generative AI literacy in English language education appears to involve several connected abilities that depend on the situation. However, the three-area framework is still a preliminary tool for analyzing research, not a fully tested final model. The main finding is not that one area matters more than another. Instead, different kinds of evidence support different kinds of claims: self-reports show how literate people believe they are, tasks show how they use AI in specific situations, and finished work shows the results of those tasks. To show that people have broad AI literacy across several areas, researchers must make sure that their definitions, measurements, evidence, and conclusions match. The review also draws a line between writing effective prompts and having a deeper understanding of AI. Good prompting can show skill in using AI, but more evidence is needed to show that people understand how generative systems work and can judge reliability, bias, authorship, teaching value, and the effects of using generated content in language learning and teaching.

Subject of Research: How generative AI literacy is defined, measured, and demonstrated in English language education

Article Title: A systematic research synthesis of generative AI literacy in English language education

Article References: Zhang, Y., Qiu, X., & Wang, J. (2026). A systematic research synthesis of generative AI literacy in English language education. Discover Artificial Intelligence, 6(1), Article 1074. https://doi.org/10.1007/s44163-026-02178-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02178-z

Keywords: generative AI literacy, English language education, systematic review, AI in education, critical evaluation, prompting, teacher education, higher education, construct validity, research synthesis, systematic, research