Transformers have become the cornerstone of every significant large language model (LLM) available today, marking a pivotal advancement in artificial intelligence over the past nine years. Justin Dangel, co-founder and CEO of the AI startup Subquadratic, emphasizes their importance, stating, “The entire AI industry is built on transformers. They represent one of the most critical innovations in computer science and have profoundly impacted the world.”
However, as time goes on, the limitations of transformers are becoming increasingly evident. Recent developments in LLMs, including new reasoning models and enhanced capabilities to handle extensive inputs simultaneously, do not serve as straightforward upgrades to the existing transformer framework. Instead, they often act as temporary solutions addressing inherent weaknesses.
This situation has prompted a growing community of scientists and engineers to ponder the future of LLMs. While these models are here to stay, their construction methods are highly subject to change. MIT Technology Review has referred to this forthcoming evolution of models as LLMs+, recognizing the significance of these advancements in AI.
In this dynamic landscape, a surge of startups is emerging, eager to explore the uncharted territories of this evolving technology. While some may inevitably fall short, these companies have the potential to innovate without the same level of risk that established leaders face.
The foundational issue lies in the properties of dense attention, the mechanism that allows transformers to encode the meaning of text into numerical data. This process involves extensive comparisons among all words (or tokens) within a given text through multiplication.
While dense attention excels in accurately capturing textual meanings, its computation demands escalate drastically as text length increases. For instance, processing a 10,000-word document could necessitate around 50 million multiplication operations, contributing to the substantial energy consumption of LLMs.
The financial implications are staggering. OpenAI is projected to allocate $50 billion for computing expenses this year alone, as reported by President Greg Brockman. The International Energy Agency has projected that data centers' electricity consumption may double by 2030.
Compounding these challenges is the fact that transformers are not particularly adept at managing substantial amounts of information simultaneously. Their method of processing words one at a time poses limitations on how expansive their context window can be. If LLMs are to undertake more complex tasks, they will need to process larger quantities of data—possibly entire libraries, comprehensive codebases, or outputs from other LLMs.
