Andrew Trice

A 2-minute introduction to the fundamental building block behind Large Language Models: Text Embeddings (This is the most helpful explanation you'll read online today. I promise.) The Internet is mainly text. For centuries, we've captured most of our knowledge using words, but there's one problem: Neural networks hate text. Judging by how good language models are today, this might not be obvious, but turning words into numbers is more complex than you think. Imagine a 4-word vocabulary: King, Queen, Prince, and Princess. The most straightforward approach to converting our vocabulary into numbers is to use consecutive values: • King → 1 • Queen → 2 • Prince → 3 • Princess → 4 Unfortunately, neural networks tend to see what's not there. Is a Princess four times as important as a King? Of course not, but the values say otherwise: Princess is "worth" 4 while a King is "worth" 1. We don't know how a neural network will interpret this, so we need a better representation. Instead of using numerical values, we can use vectors. We call this particular representation "one-hot encoding," where we use ones and zeros to differentiate each word: • King → [1, 0, 0, 0] • Queen → [0, 1, 0, 0] • Prince → [0, 0, 1, 0] • Princess → [0, 0, 0, 1] This encoding fixes the problem of a network misinterpreting ordinal values but introduces a new one: According to the Oxford English Dictionary, there are 171,476 words in use. We certainly don't want to deal with large vectors with mostly zeroes. Here is where the idea of "word embeddings" enters the picture. We know that the words King and Queen are related, just like Prince and Princess are. Word embeddings have a simple characteristic: related words should be close to each other, while words with different meanings should lie far away. The attached image is a two-dimensional chart where I placed the words from our vocabulary. Look at the image and something critical will become apparent: King and Queen are close to each other, just like the words Prince and Princess are. This encoding captures a crucial characteristic of our language: related concepts stay together! And this is just the beginning. Notice what happens when we move on the horizontal axis from left to right: we go from masculine (King and Prince) to feminine (Queen and Princess). Our embedding encodes the concept of "gender"! And if we move on the vertical axis, we go from a Prince to a King and from a Princess to a Queen. Our embedding also encodes the concept of "age"! We can derive the new vectors from the coordinates of our chart: • King → [3, 1] • Queen → [3, 2] • Prince → [1, 1] • Princess → [1, 2] The first component represents the concept of "age": King and Queen have a value of 3, indicating they are older than Prince and Princess with a value of 1. The second component represents the concept of "gender": King and Prince have a value of 1, indicating male, while Queen and Princess have a value of 2, indicating female. I used two dimensions for this example because we only have four words, but using more would allow us to represent other practical concepts besides gender and age. For instance, GPT3 uses 12,288 dimensions to encode their vocabulary. That's a lot! Text Embeddings are the backbone of some of the most impressive generative AI models we use today. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me @svpino so you don't miss what comes next.