This is Modality Gap, the reason you can’t just use CLIP models to retrieve texts and images simultaneously and sort the results by score for multimodal retrieval. To understand this better, let’s see how this graph was created: We used the PokémonCards dataset, which contains 13K text-image pairs. These 26K data points were embedded using the Jina-CLIP-v1, and the embeddings were then mapped into 2D space via tSNE. The pairwise relationships are represented by white lines between the cyan and purple dots. Despite enforcing image-text alignment during training, it’s clear that the embeddings don’t form a single, coherent space but instead fall into two distinct subspaces. You can easily draw a boundary between these subspaces, which represents the modality gap between text data and image data.