Research indicates that certain AI models exhibit a greater tendency to stereotype individuals based on demographic characteristics compared to human subjects in a previous study. In an analysis of segregation, where a score of 2 indicates complete job niche confinement for each demographic, human participants achieved a score of 0.84. In stark contrast, the AI models scored approximately 65% higher, with OpenAI’s reasoning model, o3, reaching a score of 1.83, almost at the upper limit.
According to Ryan Liu, a Princeton University PhD candidate and one of the study’s authors, the propensity of large language models (LLMs) to generalize from limited data is a fundamental aspect of their design. He explains that these models often struggle with the “exploration-exploitation dilemma,” which is a psychological concept that describes the challenge of choosing between familiar successful strategies and new potentially better alternatives. It's akin to the decision-making process of selecting between a frequented restaurant and an untested one.
LLMs are typically trained on mathematical, coding, and scientific challenges that reward drawing conclusions from minimal examples. This tendency can lead these models to form hasty judgments, resulting in stereotyping. The study revealed that more advanced models, like OpenAI's o3 and DeepSeek’s R1, exhibited even more pronounced biases. Liu notes that quick generalizations in social contexts can lead to unfavorable outcomes, pointing out that both OpenAI and Anthropic did not provide comments regarding the findings.
The implications of this research are particularly significant as chatbots are increasingly incorporating enhanced memory and personalization capabilities. Angelina Wang, a computer scientist at Cornell University and not involved in the study, emphasizes that when chatbots leverage previous interactions, they may rely excessively on past behaviors, fostering biases. She adds that simply reducing the amount of memory stored isn't a viable solution, as users typically prefer chatbots that can recall their previous input. "We're still trying to determine the optimal balance between memory and overgeneralization," Wang remarks.
Notably, instructing models to prioritize fairness showed minimal impact on their behavior. Liu points out that these models either struggle to implement such principles effectively or that these ideals are overshadowed by their innate drive to optimize for achieving the highest rate of successful hires. Interestingly, incentivizing the models with bonuses for diverse hiring yielded a marked reduction in bias. Liu suggests that the key lies in crafting objectives that integrate positive social values to guide LLMs toward socially responsible behaviors.
