The Future of AI Agents

The current AI landscape is dominated by transformers, and everyone seems convinced this architecture will define artificial intelligence for the foreseeable future. But history suggests otherwise. When a technology becomes subject to incremental refinements under intense competition, it often signals that a paradigm shift is approaching—even if that shift takes decades to fully materialize.

The Pattern of Technological Succession

The CNN Era and Its Decline

Convolutional Neural Networks (CNNs) once seemed unshakeable. From AlexNet’s breakthrough in 2012 to ResNet’s innovations in 2015, CNNs dominated computer vision. The incremental improvements—batch normalization, skip connections, attention mechanisms—seemed to promise endless progress.

But by 2017, the Vision Transformer (ViT) papers began showing that transformers could match or exceed CNN performance on image tasks. The writing was on the wall, even though many refused to read it.

Pre-Transformer Architectures

Before transformers, we had:

RNNs and LSTMs: Dominated sequence modeling until attention mechanisms made them obsolete
Encoder-Decoder Architectures: Powerful but limited by their sequential nature
Convolutional Sequence Models: Attempts to parallelize sequence processing

Each seemed revolutionary until the next paradigm emerged.

Signs That Transformer Dominance May Be Waning

Diminishing Returns on Scale

The returns on simply making transformers larger are showing clear signs of diminishing marginal utility:

Parameter Scaling: GPT-3 to GPT-4 required massive increases in parameters for modest performance gains
Compute Costs: Training costs are growing exponentially while improvements are becoming linear
Data Limitation*: High-quality training data is becoming scarce, forcing models to train on synthetic or lower-quality data