Yu and his research colleagues didn’t interview any of the students and cannot say for certain that the students were using ChatGPT or any of its competitors, such as Claude or Gemini, to help them with their assignments. But the improvement in student writing following the introduction of ChatGPT does seem to be more than just a random coincidence.
Big upswings for international students
The unidentified university is a minority serving institution with a large number of Hispanic students who were raised speaking Spanish at home and a large number of international students who are non-native English speakers. And it was these students, whom the researchers classified as “linguistically disadvantaged,” who saw the biggest upswings in writing quality after the advent of ChatGPT. Students who entered college with weak writing skills, a metric that the university tracks, also saw outsized gains in their writing quality after ChatGPT. Meanwhile, stronger English speakers and those who entered college with stronger writing abilities saw smaller improvements in their writing quality. It’s unclear if they’re using ChatGPT less, or if the bot offers less dramatic improvement for a student who is already writing fairly well.
The gains for “linguistically disadvantaged” students were so strong after the fall of 2023 that the gap in writing quality between these students and stronger English speakers completely evaporated and sometimes reversed. In other words, the writing quality for students who didn’t speak English at home and those who entered college with weak writing skills was sometimes even stronger than that of students who were raised speaking English at home and those who entered college with stronger writing abilities.
Gains concentrated among high-income students
However, these gains in writing quality among the “linguistically disadvantaged” were concentrated among higher income students. The researchers were able to match students’ writing submissions with administrative data on students, including their family income, and they noticed that the writing of low-income students whose parents did not attend college didn’t improve quite so much. By contrast, the writing of high-income international students with college educated parents transformed markedly.
That’s a sign that low-income students weren’t using ChatGPT quite so much or not as effectively. Socioeconomic differences in how students benefit from technology aren’t uncommon. Previous studies of word processing software, for example, have found that higher income students tend to be more facile in taking advantage of editing features and see greater writing benefits from the ability to cut and paste and move text around.
Mark Warschauer is a professor of education at University of California, Irvine, and director of its Digital Learning Lab, where he studies the use of technology in education. Warschauer was not involved with this study and he said he suspects that the lopsided benefits for higher income students will be fleeting as low-income students become more acclimated and more facile with AI over time. “We often see with new technologies that high-income people get access first, but then it balances out. I believe that low-income people use cell phones and social media as much as high income people in the U.S.,” he said.
But he predicts that the substantial and larger improvements in writing for international students, far greater than for domestic students, will be “more important and durable.”
Of course, this improved writing quality doesn’t mean that these international students are actually learning to write better, but it does indicate that they’re adept at using technology to present ideas in well-written English.
The study’s researchers didn’t analyze the ideas, the quality of analysis or if the student submissions made any sense. And it’s unclear if the students fed the reading into the chatbot along with the professor’s question and simply copied and pasted the chatbot’s answer into the discussion board, or if students actually did the reading themselves, typed out some preliminary ideas and just asked the chatbot to polish their writing
In Yu’s own classes at Teachers College, he said he encourages students to use ChatGPT in their writing assignments as long as they acknowledge it and also submit transcripts of their conversations with the AI chatbot. In practice, he said, only a few students admit to using it.
He noticed that student writing in his classes had been improving until now. “This year has actually been horrible,” he said. More and more of his students have been submitting typical AI output that “seems reasonable but doesn’t make a lot of sense,” he said.
“It all comes down to motivation,” said Yu. “If they’re not motivated to learn, then students will only make a bad use of whatever the technology is.”
Contact staff writer Jill Barshay at 212-678-3595, jillbarshay.35 on Signal, or barshay@hechingerreport.org.