Back to Blog

Getting Started with Transformer Models

January 15, 2024 8 min read Ngoc Luu
NLP Transformers Deep Learning

Introduction

The Transformer architecture, introduced in the landmark paper "Attention is All You Need" by Vaswani et al. (2017), has revolutionized the field of Natural Language Processing (NLP) and beyond. Unlike previous architectures that relied on recurrent or convolutional layers, Transformers are built entirely on attention mechanisms.

What Makes Transformers Special?

Transformers have several key advantages over traditional sequential models like RNNs and LSTMs:

  • Parallelization: Unlike RNNs, Transformers can process all tokens in a sequence simultaneously, making them much faster to train.
  • Long-range Dependencies: The self-attention mechanism allows the model to capture relationships between distant tokens effectively.
  • Scalability: Transformers scale well with data and model size, leading to powerful models like GPT-4 and BERT.

The Attention Mechanism

At the heart of the Transformer is the self-attention mechanism. The attention function can be described as mapping a query and a set of key-value pairs to an output. The scaled dot-product attention is computed as: 

Attention(Q, K, V) = softmax(QK^T / √d_k)V

Where:

  • Q (Query): What we're looking for
  • K (Key): What we're comparing against
  • V (Value): The actual information to aggregate
  • d_k: The dimension of the key vectors

Multi-Head Attention

Instead of performing a single attention function, Transformers use multi-head attention. This allows the model to attend to information from different representation subspaces at different positions: 

MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W^O
where head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)

Positional Encoding

Since Transformers don't have inherent sequential structure like RNNs, positional encodings are added to give the model information about token positions. The original paper uses sine and cosine functions: 

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Architecture Overview

The Transformer consists of two main components:

Encoder

The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers:

  1. Multi-head self-attention mechanism
  2. Position-wise fully connected feed-forward network

Decoder

The decoder is also composed of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer that performs multi-head attention over the encoder's output.

Applications in Modern NLP

Transformers have become the foundation for state-of-the-art models in various NLP tasks:

  • BERT: Bidirectional encoder for language understanding tasks
  • GPT series: Autoregressive decoder for text generation
  • T5: Text-to-text framework for unified task handling
  • Vision Transformers: Extending Transformers to computer vision

Getting Started: A Simple Example

Here's a minimal example using Hugging Face Transformers library: 

from transformers import AutoTokenizer, AutoModel

# Load pretrained model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

# Tokenize input text
text = "Transformers are powerful!"
inputs = tokenizer(text, return_tensors="pt")

# Get model outputs
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

print(last_hidden_states.shape)  # torch.Size([1, 5, 768])

Conclusion

Transformers have fundamentally changed how we approach sequence modeling in NLP and beyond. Their ability to capture long-range dependencies, parallelize computations, and scale effectively has made them the architecture of choice for most modern language models. As research continues, we're seeing Transformers being adapted to even more domains, from computer vision to protein folding prediction.

"Attention is all you need" - but understanding why and how it works is what makes you a better researcher!

Further Reading