EAI · Research review of Zhang et al. (2026)

Does AI help students learn another language?

Zhang et al. report higher language-test scores among students who used AI. We checked their tables and several of the original studies. Table 2 adds up to 4,378 participants; the article reports 5,196. That leaves a difference of 818. We also found questions about which studies were included and how some results were calculated.

Hans Visser · 14 November 2026 · Updated 15 November 2026 · 9 pages

How much do three unusually large results change the average?

The authors combine 67 effect sizes from 48 studies. Their pooled estimate is Hedges’ g = 0.76. Remove three unusually large results and it falls to g = 0.57. The estimate still favours the AI groups, but those three results clearly affect its size.

The results vary widely (I² = 92.64%). Some students practised conversations with chatbots; others revised their writing using AI feedback. Those are different tasks. The average alone cannot tell us which kind of support helped which students.

Why do the participant totals differ by 818?

Table 2 lists the sample sizes for all 48 studies, on pages 9 to 12. Together they total 4,378 participants. The main text reports 5,196. The paper does not explain the difference.

4,378Sum of Table 2
5,196Reported in the main text

Different counting rules or an error in the table might explain the mismatch. The published figures do not tell us which. We would need the original group counts to check.

What do the original studies actually test?

In Bašić et al. (2023), students wrote in Croatian, their first language. Why was this included in a review of second-language learning? Zhang et al. classify Alenezi and Alenezi (2025) as a randomised experiment, although its authors describe it as quasi-experimental. Two of the three unusually large effects come from a study using Amazon Alexa. It is not yet clear whether that particular application meets the review's criteria for generative AI.

The calculations raise further questions. Using the published group means and standard deviations from Y. Wang (2025), we cannot reproduce the reported t-value as an independent-groups comparison. Zhang et al. also report a Q-test whose p-value does not match the stated degrees of freedom. Both need clarification. The PDF shows the calculations.

What can students still do without AI?

Only four of the 48 studies tested students again after a delay. Most measured performance during or shortly after the lessons. Higher scores at that point are meaningful, but they do not show whether students can still perform weeks later without AI, or use what they learned on another task.

The age group matters too. Thirty-nine of the 48 studies took place in higher education. Their results cannot simply be applied to younger pupils.

Read the review and check the figures

The nine-page PDF contains the page-by-page participant tally, our checks of the original studies and the statistical calculations. The sources are listed so you can follow each point. Reproducing the full meta-analysis would require all 67 effect sizes and the authors’ analysis code.

Back to research · All publications