Two minds, one map
Teach one computer only with pictures. Teach another only with words. Never let them meet. This paper says that, if you teach them the right way, they end up with the same picture of the world inside their heads, just turned a different way round. Here is the story, one page at a time.
Everyone is looking at the same world
Think of two children. One only ever looks at photos. The other is blind and only ever hears stories. But the photos and the stories are about the same real world: the same cats, buses and snowstorms.
In 2024, some scientists noticed that big AI models seem to be like these two children growing up. The bigger and better they get, the more their insides look alike. They called this the Platonic Representation Hypothesis, after the old Greek idea that there is one true world behind everything we see. A hypothesis is a guess that hasn't been proved yet.
Inside, every model keeps a map
A model turns each thing it sees into a dot on a map. Things that are alike sit close together: the cat sits near the dog, the bus sits near the car. Scientists call the map an embedding. It has hundreds of directions, not two, but the idea is the same.
To check whether two maps match, you can do something simple. Pick a dot, find its 10 nearest neighbours on each map, and count how many are the same. That count, averaged over every dot, is the alignment score. A score of 1 means the neighbours are always identical. A score near 0 means the two maps have nothing in common.
Same map, just turned round
Here is the paper's main promise. The two maps come out the same shape, but one may be turned like a page on a table. Nothing is stretched or scrambled. One simple turn lines them up. Try it:
A guessing game called JEPA
The models in this paper learn by playing a guessing game. Show the model one thing, then ask it to guess where the matching thing will land on its map. It doesn't have to redraw the picture or rewrite the sentence. It only has to point at the right spot. That is what JEPA means: Joint-Embedding Predictive Architecture, or "guess the spot on the map".
The whole paper turns on one question: what counts as "the matching thing"? There are two kinds of game.
Two people describing one scene, or one sentence with holes in it
The paper proves that good views pull both models toward the world itself, so their maps meet. Bad views let each model fill its map with habits of its own kind of data, so the maps can drift apart. It also gives a limit on how far apart they can drift.
The surprise: "guess the next word", the way ChatGPT-style models learn, is a bad-view game. That would explain why huge language models only partly match picture models.
Small and well-taught beats big
Fewer examples. Same model, same text. Taught with good views, it matched the picture model as well as next-word training did, after seeing 42 times fewer examples (21.6 million against 900 million).
A small word model beat giants. Their text model has 109 million settings. On the 2024 paper's own test it scored 0.184 against the picture model DINOv2. Language models up to 72 billion settings, about 660 times bigger, scored at most 0.154.
Captions with almost no matching pairs. They built a captioner from three parts that were never trained on photo-and-caption pairs: two frozen picture models, and a language model trained on text alone. The only thing fitted using pairs is the one turn from page 3 that lines up the maps. It still writes sensible captions (0.838 on the CIDEr caption score, on COCO photos).
Matching with no pairs at all. Using another group's method, which lines up two maps from their shapes alone, their maps let a model sort CIFAR-10 photos into 10 classes correctly 86% of the time, above every result in that group's own paper.
What this does not mean
- "Proving" has conditions. The maths holds when its assumptions hold, for example that the training pairs really are two looks at the same moment. Real data is messier, which is why the measured scores are well below 1.
- Mostly checked by the authors. Every number above comes from their own runs: 345 training runs, about 5,500 hours of computer time on big graphics cards. They released the code and results so others can check them. That is what our study is for.
- Good views need help from the world. Two captions of one photo is a pairing that came from photos. The word model never sees the picture, but somebody had to know the two sentences described the same scene.
Our part: check it on a laptop
We can't redo 5,500 hours of big-computer training. We can run their own tests, rebuild their headline numbers from the files they shared, and measure the language-model scores on page 6 ourselves. If those hold, we can try a tiny version of the page-5 experiment overnight on a Mac: good views against bad views.