The Board Inside the Loop

A Weekend in the Game

Eli: Neel Nanda sits down to play Othello against a computer program called eOthello. He knows absolutely nothing about the game. He plays for thirty minutes, maybe an hour. Those games, he's said, were really valuable for building intuitions. But the game isn't what he's there to understand. He's there for a small AI called Othello-GPT.

Mara: The listener asked us whether chatbots actually understand anything, or whether they're just really good at guessing the next word. Today's angle is whether chatbots actually understand anything.

Eli: A model that only ever guessed the next Othello move turns out to be a sharp test of exactly that. We'll go inside it with Nanda, hear the case that prediction can never become meaning, and end where the line between the two is still being fought over.

Mara: And he moved fast. He did most of the project in one weekend, about twenty hours, and called it a "shockingly successful" experiment.

Eli: The model itself came from another research team. They trained it on randomly generated Othello games, made by picking legal moves at random. It never saw a board. Nobody told it the rules. All it got was lists of moves, and its one job was to predict the next one. In one later check, its top guess was a legal move in all of roughly 3,700 positions tested.

Mara: Every single one?

Eli: That's the puzzle Nanda walked into. Either the model had memorized patterns in move lists, or predicting the next move had pushed it to keep track of a board nobody ever showed it.

Mara: Chatbots run on the same bargain, just with text. A large language model is trained on enormous amounts of writing to predict what comes next. Strictly, it predicts tokens: common words whole, longer or rarer ones in pieces, plus punctuation and numbers. A cognitive scientist at Tufts describes the training as picking up very complex statistical relationships between words.

Eli: Graded by cross-entropy loss: a score that rewards high odds on the real next token.

Mara: Then comes generation. Given a prompt, the model estimates the odds for every possible next token, picks a likely one, tacks it on the end, and goes again. That loop is autoregressive generation. A whole reply is just that loop, run over and over. And it only ever writes forward. Quanta Magazine calls what comes out a "seemingly coherent reply."

Eli: "Seemingly" is the word critics underline twice.

Mara: And on paper, that loop is all there is. Othello-GPT ran on the same loop. Whatever's going on is happening inside the loop, not around it.

The Cable and the Room

Eli: In 2020, Emily M. Bender and Alexander Koller drew a hard line through language. On one side is form: anything observable, like marks on paper, pixels, bytes, the movements of speech. On the other is meaning: a relation between that form and something outside language, the intent a speaker uses it to get across. Their central thesis was that a model trained only to predict strings has, in principle, no way to learn that relation.

Mara: At most, a reflection of meaning, never meaning itself.

Eli: And they made it vivid. Two castaways on separate islands talk over an undersea telegraph cable. A hyperintelligent octopus taps the line, learns to predict one of them, and starts impersonating him. Small talk goes fine. Then the other castaway is being chased by a bear, has a couple of sticks, and asks what to do. The octopus has only ever seen word patterns. Never a bear. Never a stick. It can't help.

Mara: They offered it as an illustration, though, not a precise test.

Eli: The sharper point is about the person on the other end. If the castaway found the messages meaningful, they wrote, it was because the castaway could make sense of them, not because the messages made sense. Bender has kept pressing it: lining up which letters go with which other letters, she's said, doesn't get to the meaning. That argument has an older relative. In 1980, John Searle imagined a person locked in a room who knows no Chinese. Chinese characters are passed in. He consults an enormous rulebook that says which symbols to send back. From outside, the answers can't be told apart from a native speaker's.

Mara: That's the Chinese Room: following symbol rules flawlessly while grasping none of what they mean. In Searle's words, his inputs and outputs could match a native speaker's, "but I still understand nothing."

Eli: Swap the rulebook for a trained network, and the octopus is in that room.

Mara: Searle boiled it down to one line: syntax "is not by itself sufficient for, nor constitutive of, semantics." Rules about symbols aren't meaning.

Eli: That's partly why the fight stays deadlocked. Computer scientists tend to judge these systems by the tasks they can do. Philosophers like Searle judge them by conscious, embodied reference to the world. Mara, the Othello work took the question somewhere new: inside the model itself.

Listening to the Hidden Layers

Mara: Inside is exactly where the team behind Othello-GPT pointed it. They asked whether networks like this just memorize a collection of surface statistics, or rely on internal representations of the process that generates the sequences. In Othello, that process is a board. So they trained probes, small readouts that learn to decode one specific thing from the model's hidden activity; a linear probe is the simplest version. Their probes recovered the board state with high accuracy, and they reported evidence of an internal board representation the network had built on its own.

Eli: The very thing the octopus argument says form alone can't produce.

Mara: Except reading something out isn't the same as the model using it. Nanda warned that a probe could be extracting "vestigial features or a side effect" of some other computation. What's still unknown is simple: whether that board does any work at all.

Eli: Nanda wrote that a board made sense in there: literally the only thing Othello-GPT cares about is playing legal moves, and a board is valuable for that. Midway through, he noticed a possible explanation for the probe results. In the spirit of pursuing "big if true" hypotheses, and because by then he was late for work, he'd "just say YOLO" and train a linear probe. By his rough count, the board was taking up about 128 of the 512 dimensions in the model's residual stream, the central channel that carries information from layer to layer.

Mara: Spent on a board nobody gave it.

Eli: Researchers infer that minimizing prediction error on structured sequences can force a network to encode the hidden rules behind them. A readable board fits that. What nobody had shown yet is what happens if the board inside is changed mid-game while the move list stays exactly the same.

Steering the Board

Mara: That's the experiment that turns it. It's an intervention, sometimes called activation patching: editing the model's hidden numbers mid-run to see whether its next prediction changes. Researchers took Othello-GPT partway through a game, left the move history untouched, and changed the board it was holding inside. The predictions followed the edited board. A tutorial built around the experiment draws the line plainly: "That is what makes this a world model rather than a correlate."

Eli: A model trained on move lists alone.

Mara: Predicting the next move had pushed the network to build it.

Eli: Here's where it gets humbler. Melanie Mitchell and her collaborators tried to disprove Nanda and colleagues' claims and failed, which she wrote left her even more impressed. Then, in mid-2024, student researchers looked at how that board was built. Not one tidy model, they reported, but "many independent decision rules that are localized to small parts of the board." Call it a heuristic swarm: a pile of narrow rules of thumb rather than one clean algorithm. And she sees that pattern scaled up. Today's chatbots, she wrote, are much less brittle than older networks, but still brittle, possibly from the same reliance on bags of heuristics.

Mara: Which makes Othello a cleaner result than a general one. A board game with fixed rules is one setting. Open-ended human language is a much harder place to look for the same thing.

The Benchmark Fracture

Eli: Researchers have gone looking there. A team at Brown searched large language models for a plausibility vector, an internal direction that sorts events by how plausible they are. In models above two billion parameters, those vectors showed up. They could tell improbable from impossible with roughly 85 percent accuracy.

Mara: So somewhere inside, unlikely and impossible are kept apart.

Eli: Then a Yale-led study, out in 2026, found language models naturally converge on role-filler structure: slots for things like who acted and who was acted on, with specific words bound into them. The team could steer it, too. One edit moved "clever" from the subject to the object, so the model answered as if it had read "the doctor helped a clever lawyer." A coauthor said the work "only scratches the surface."

Mara: Now change the problems instead of the models. A counterfactual perturbation alters details that shouldn't matter, to see whether the reasoning survives. Apple's 2024 GSM-Symbolic benchmark did that to math word problems. Renaming variables or swapping numbers lowered accuracy across frontier models, the most advanced, cutting-edge systems. Adding one irrelevant clause could cut it by up to 65 percent.

Eli: That's been described as consistent with template matching, not reading the problem.

Mara: An MIT team found the same shape with arithmetic. Strong on familiar versions, severe drops on unfamiliar ones, like addition outside base ten. Zhaofeng Wu, one of the researchers, said the models "excel in familiar scenarios, almost like a well-worn path, but struggle when the terrain gets unfamiliar."

Eli: So the same models carry plausibility directions and symbolic slots, then stumble over a clause that changes nothing. Whether that inner machinery adds up to understanding is a separate question, Mara, and the two camps answer it in opposite ways.

Two Kinds of Comprehension

Mara: Experts disagree fiercely about whether it's even appropriate to say these models understand things about the world. One reading, held by Neel Nanda among others: to predict complex text well, models build internal world models and abstractions. Othello-GPT's board is the cleanest case of that.

Eli: The other reading belongs to Bender, Koller and Searle: models shuffle tokens by statistical pattern, with no real-world reference or intent behind them.

Mara: Part of the gap is two senses of the word. One analysis of the Chinese Room splits it. Functional understanding is doing the work: spotting relationships, inferring implications, using concepts appropriately, responding to context. Phenomenal understanding means a subjective experience comes with it. On that analysis, the room may show fluent language doesn't prove consciousness, but it doesn't show computation can't develop functional semantics.

Eli: The octopus still marks where the cost lands, though. A model built to produce plausible text can produce confident, false output, a hallucination: a citation, a statistic or a biography that looks right when the real one is unknown.

Mara: So, to the listener's question. Chatbots really are trained to guess the next token; that's the whole objective. But in Othello, guessing well pushed a model to build a board it steers by. Models capture a lot of structure about language and the world, and solve problems they haven't seen before. Whether that's understanding is still debated.

Eli: What would change the picture is plain enough. Whether boards like Othello's show up as robust world models in open-ended language. Whether text alone can ground meaning in the physical world without a body or other senses. And a test both camps would accept for telling comprehension from elaborate memorization, which doesn't exist yet.

Mara: Look up Autoregressive generation.

Eli: And stay with us on technology.

Mara: That's our story for today. I'm Mara.

Eli: And I'm Eli. Thanks for listening to Angle, and until next time.