Gaurav Mantri

Understand how Cosine similarity is used in LLMs Language models need to know how similar texts or words are. But how does it work? I will explain with the help of this single image 🔽 Models usually cannot work with textual data, so we need to convert words into numbers. This is mostly done with word embeddings. These are vector (numerical) representations of text. Note: The embeddings are usually huge, but for simplicity now I will use a vector that contains only X and Y coordinates for words. We have four words in this example: Apple, Banana, Cat and Dog. All of them are converted into vectors and visualized. As the name suggests Cosine similarity can tell how similar two vectors are. The range for cosine is usually between 0 and 1. 0 means no similarity, while 1 means exact match. To calculate Cosine similarity in the 2D space we can use the angle between the two points. Or we can use the formula from linear algebra where a⋅b is the dot product of a and b, and ∥a∥ and ∥b∥ are the magnitudes respectively. You can check the detailed calculation for Cat and Dog below. As you can see from the calculation, since both Cat and Dog are animals, they are pretty similar. But what about Apple and Dog? They are far from each other, and the angle is larger, so the Cosine similarity will be lower! What about huge embeddings? Look at the formula and think about it. The best thing about Cosine similarity is that it is not restricted to 2D vectors. You can do the same calculation for embeddings that contain hundreds of elements. Of course, we cannot visualize words the same way with bigger vectors, but the process is the same.