ChatGPT, the new cutting-edge A.I. chatbot by OpenAI, needs no introduction–in the short time since its release, it has already taken over the world, completing homework assignments for curious students, threatening to make Google obsolete, and setting off a catalyst for the future of language technologies. In particular, its novelty is in its conversability and ability to remember past utterances as it talks. Its human-like abilities have left everyone with one question: just how is this possible? The answer comes down to one key technology: Transformers.
No, I’m not talking about Optimus Prime or Bumblebee. The “T” of ChatGPT, the Transformer, is a type of neural network architecture presented in 2017 by researchers at Google. It has truly lived up to its name, transforming the capabilities of natural language processing algorithms. In order for ChatGPT to better understand human language and follow through a conversation, Transformers enable it to pay attention with something called the attention mechanism.
Attention mechanisms are an integral part of Transformers that allow them to focus on and remember relevant words to understand context. Take for instance, when ChatGPT is asked, “What is a Transformer model?”. ChatGPT can understand that the question refers to the neural network rather than a fictional polymorphic automaton by attending to the word “model.” And attention allows for these connections to be made between words regardless of how far away they may be, making ChatGPT excellent at remembering previous information in a conversation when processing inputs.
In order to identify which words the model should be attending to, the attention mechanism computes a set of attention scores based on relevance for each word in the sequence. These scores indicate how much attention the model should pay to each word when generating the output. How to determine these weights for each element is learned over time by the Transformer from data. Through training on large amounts of text–300 billion words for ChatGPT, to be exact–the models learn statistical patterns and global dependencies between words and form representations of the text that capture semantic and syntactic meaning. Additionally, the structure of Transformers makes them more parallelizable and significantly faster, meaning they can learn with even more data, taking their abilities even further.
Like neural networks themselves, the effectiveness of attention is from its mimicry of the functions of the human brain and how we process language. Yoshua Bengio, a leading researcher in natural language processing, explains the value of Transformers, stating that “attention is one of the core ingredients” of the success of the Transformer architecture “which will become increasingly important as models become larger and researchers seek to model more complex phenomena.” He urges that researchers draw from neuroscience studies on consciousness and incorporate them into machine learning models.
As technology advances and becomes more integrated into our everyday lives, we can also learn from A.I. as it learns from us. Just like the title of the paper that the Transformer architecture was proposed, perhaps “Attention is All You Need.”
Note: this is a STEM editorial describing ChatGPT