POST 07 / 07VLM~7 min read
CLIP & the Birth of Vision-Language
Train an image encoder and a text encoder so that matching pairs land near each other. Suddenly you can classify any image with any caption, no fine-tuning. Type a query and watch images sort by similarity.
Nishanth Vanipenta2026.05.01Interactive · 1 viz
Contrastive learning, in one diagram
Take a batch of N image-caption pairs. Encode each image to a vector. Encode each caption to a vector. You have an N × N matrix of cosine similarities. The diagonal should be high (matching pairs); off-diagonal should be low. Loss: cross-entropy on rows + cross-entropy on columns.
- No labels — just web image-text pairs (LAION, ALIGN). Scale to billions.
- Image and text live in the same vector space after training.
- You can ask: "give me the image closest to
'a dog wearing sunglasses'" — and it works.
CLIP wasn't the first contrastive model, but it was the one that proved scale + alignment + simple loss → emergent zero-shot ability. Every modern VLM (LLaVA, GPT-4V, Gemini) descends from this idea.
Visualizer · text → image search
Type a description, watch images re-rank by similarity. Or click an image and find what's closest. Below: a synthetic embedding space where each item has tags; the "encoder" is just bag-of-tags overlap, but it captures the spirit.
VIZ_08 :: TEXT_IMAGE_RETRIEVAL—
From CLIP to VLMs
- CLIP (2021) — image ↔ text in shared space. Zero-shot classification, retrieval.
- BLIP / Flamingo — added language generation. Now you can ask the model questions.
- LLaVA / GPT-4V / Gemini — bolt a vision encoder onto a frozen LLM. The image becomes a sequence of tokens; the LLM "reads" them.
The trick that unlocked VLMs: treat image patches as tokens, then prepend them to the language model's input. The LLM doesn't need to be retrained from scratch — just a small projection layer that maps vision features into the LLM's embedding space. Surprisingly, this works.
Where it breaks
- Counting. "How many sheep?" — VLMs are notoriously bad. The encoder mostly captures presence/absence, not quantity.
- Spatial relationships. "left of" / "above" — fragile.
- Fine-grained text in images. Improving fast (GPT-4V, Gemini), but small print is still hit-or-miss.
- Out-of-distribution. CT scans, satellite imagery, schematics — needs domain-specific fine-tuning.
Sources & further reading