Show HN: A 155K-param transformer builds a map of a world it's never shown
A transformer model with only 155,000 parameters, trained on sequences of visited locations, can internally learn and build a geometric map of a 2D grid world it was never explicitly shown, demonstrating spontaneous topological learning.
Background
- The article presents a tiny transformer model (155K parameters) that, after training on sequences of agent movements in a grid world, develops an internal representation of that world's spatial layout — despite never being shown a map or coordinates.
- The key finding is that the model's hidden states can be "read out" to reconstruct the map, demonstrating that world models (internal representations of environment structure) can emerge in very small neural networks trained on simple sequential data.
- This is significant because it shows that even minimal architectures can learn structured internal models, challenging assumptions about how much data or model size is needed for such representations, and connecting to broader AI research on whether large language models build world models.