Subscribe to receive updates like this directly in your inbox with The 74 Newsletter.
This summer, I conducted a week-long artificial intelligence pilot program at the Chicago headquarters of Forging Opportunities for Refugees in America, a nonprofit focused on supporting refugees in their educational journeys. The group consisted of 22 newcomers between the ages of 11 and 17, most of whom were still developing their literacy skills and had experienced disrupted education due to their relocations from countries like Afghanistan, Burma, Malaysia, and Syria. Tragically, these are the students often overlooked by the educational system, with low expectations placed upon them.
On the second day of the program, a teen named Gemini expressed her frustration. She insisted that the AI tool was malfunctioning since it kept questioning her instead of providing straightforward answers. She firmly advocated for using ChatGPT instead.
What she didn’t initially realize was that I had programmed Gemini to engage her in a questioning format deliberately, encouraging her to reflect on her daily reading. Once she uncovered the instructions I had included in the prompt, her irritation gave way to comprehension. The tool wasn’t malfunctioning; it was prompting her to engage in critical thinking.
By the following day, Gemini scored a perfect 100% on a quiz based on the previous day's reading. She began to participate more actively in our lessons, taking the time to reflect and respond rather than rushing through the material. In our final session, she was one of the last students to submit her quiz, keeping her mother waiting—a rare occurrence, especially during summer break.
This experience underscored the potential of AI when it is designed to promote active thought instead of simply providing information.
During the week, I presented the students with a series of brief readings about cognitive processes, integrating AI in various ways to enhance their comprehension. On certain days, I asked the AI to summarize the readings, and on others, the AI facilitated a Socratic dialogue, prompting students to engage with the material more deeply. There were also sessions where they had to generate their questions first, allowing their natural curiosity to guide the learning process.
Subsequently, I quizzed them on their retention of the material, assessing their recall both immediately after our study sessions and again the next day.
The findings were intriguing. On the quizzes taken the same day, students' performance was relatively uniform, regardless of whether they'd engaged with the AI in a summary format or a questioning format. However, when assessed the following day, students who participated in the Socratic Q&A demonstrated greater retention, averaging 83% on their quizzes, compared to 67% for those who had simply relied on AI-generated summaries. Over two-thirds of the students indicated that the version of AI that prompted questions offered them a better learning experience than one that only summarized information.
As the week progressed, students’ confidence in using AI as a study aid grew significantly. However, when I compared their self-assessed skills with their actual retention rates, I found little correlation. Increased confidence in using AI did not translate into better recall; feeling like they had learned did not equate to actual learning.
This should raise concerns for anyone invested in education. AI has become ubiquitous, with students carrying these powerful tools in their pockets. School districts are projected to invest hundreds of millions of dollars in AI educational technology in the coming years, but many lack a framework for evaluating these investments. My pilot program revealed that engagement or satisfaction alone is insufficient metrics for assessing the effectiveness of AI tools. True success should be evaluated on students' retention of material once they no longer have access to the technology.
Additionally, students expressed a need for learning how to harness AI as a collaborative thinking partner, noting that they had received little guidance in this area. They reported that attempting to use AI for answers often resulted in difficulty accessing information tailored to their grade and comprehension levels. Consequently, the students who could benefit the most from personalized learning experiences often find these tools less effective, perpetuating the very inequities they aim to resolve.
This highlights a crucial need for AI literacy education in schools. While tech-savvy children from supportive homes may be learning to navigate AI effectively, many others are left to experiment with prompts and hope for favorable outcomes. Schools have a unique opportunity to bridge this gap and empower all students.
