This new technology could blow away GPT-4 and everything like it

Poli and his team have created a unique fusion by integrating the groundbreaking research conducted by Stanford researcher Daniel Y. Fu and his team. They have successfully implemented convolutional filters to sequences of words. In addition to this, they have also incorporated the innovative work of scholar David Romero and his colleagues at the Vrije Universiteit Amsterdam. This incredible integration allows the program to dynamically adjust the filter size as needed. This adaptive capability significantly reduces the number of expensive parameters or weights that the program requires.

Poli and their team have created a unique combination of Stanford research and work from the Vrije Universiteit Amsterdam. They have successfully applied convolutional filters to word sequences while also allowing the program to adjust the filter size dynamically. This flexibility reduces the need for an excessive number of parameters, resulting in a more efficient program. This approach eliminates the need for attention and achieves similar perplexity and performance to larger models with fewer computational resources. The authors demonstrate the effectiveness of their program by testing it against various benchmarks that evaluate a language program's performance in AI tasks.

The key achievement of this mash-up is the ability to apply convolutions to text without constantly increasing the number of parameters to handle additional data. This "attention-free" approach has significant implications, as it allows convolutional architectures like Hyena to match the quality of attention-based models, such as GPT, while using fewer operations.

Hyena's capabilities were tested on The Pile, a vast collection of texts curated from reliable sources. The challenge was to generate the next word given a set of new sentences. Remarkably, the Hyena program achieved results comparable to OpenAI's original GPT model with 20% fewer computing operations. This convolution architecture is the first of its kind to match GPT quality with reduced operations.

The program was further evaluated using reasoning tasks known as SuperGLUE. In one task, given a sentence like "My body cast a shadow over the grass" and two possible causes, the program correctly identified "the sun was rising" as the appropriate output. Overall, the Hyena program achieved scores on par with or close to GPT, even with less training data.

Interestingly, the program's performance improved significantly when longer phrases were used as input. More words led to better results, and at higher token counts, such as 64,000, Hyena demonstrated a hundred-fold speed improvement over the attention-based approach. This breakthrough in eliminating the quadratic barrier represents a qualitative shift in computation difficulty for language programs.

Poli and the team emphasize that Hyena offers limitless context, enabling it to recall information from extensive texts or previous conversations, akin to how hyenas hunt over vast distances. The program's filter efficiently stretches over thousands of words, making context virtually unlimited. The authors envision new possibilities for deep learning, such as using entire textbooks as context, generating long-form music, or processing gigapixel-scale images.

In addition to words, Hyena's capabilities can also extend to diverse data modalities, including images, videos, and sounds. It is worth noting that the Hyena program described in the paper is smaller in size compared to GPT-4 or even GPT-3. While the largest Hyena version has 1.3 billion parameters, GPT-3 boasts 175 billion. However, if the efficiency demonstrated by Hyena translates to larger versions, it could become a prevalent paradigm in the same way attention has dominated the past decade.

In conclusion, Poli and the team suggest that simpler sub-quadratic designs like Hyena, guided by a set of fundamental principles and evaluated against interpretable benchmarks, could serve as the foundation for efficient large models.