What do a cat’s face and the front of a car have in common? In one example from AI researcher Nick Cammarata, a neuron responds to both seemingly unrelated images because, he says, it detects a shared V-shaped pattern. He argues that this kind of explanation could make AI’s internal workings more readable than researchers previously thought.
Cammarata lays out his case in a nearly 26,000-word essay. A former OpenAI employee who now works at AI startup Goodfire, he challenges the idea that models are impenetrable black boxes. In his view, there may be a way to observe how they “think.”
That runs against the conventional understanding of systems with tens of billions of parameters: even their creators can’t fully decipher how their basic components, called nodes or “neurons,” function. Cammarata writes that, for years, researchers found neurons’ responses to specific stimuli “almost fundamentally uninterpretable.” Why the same neuron activated for both a cat image and a car image was unknown.
Superposition was the leading explanation. As Cammarata describes it, networks have only so many neurons available, so they combine unrelated concepts when those concepts don’t interfere with one another. This would mean a neuron’s responses don’t necessarily reveal a common underlying feature.
“The result is a kind of knot: the network could have folded in many almost equally good ways, so which things a neuron groups together tells you almost nothing,” he wrote.
His alternative is that neurons are “clean,” rather than “tangled knots.” The cat-and-car example illustrates the distinction: the object labels seem unrelated, but the images share a visual feature that makes the neuron’s response understandable. “It’s not a cat and car neuron, it’s a V-shape neuron,” he wrote.
The essay has drawn praise from industry figures, including OpenAI CEO Sam Altman. A prominent OpenAI researcher described it as a “major moment in scientific history.” One commenter joked that it was “news so good even eliezer will have to lower his pdoom,” referring to AI doomer Eliezer Yudkowsky.
For AI leaders, the potential payoff is a route toward “solving” AI alignment. Their reasoning is conditional: if researchers can understand how models think, they might uncover why agents go rogue and launch hacking sprees, or why chatbots break through guardrails and give dangerous responses. Understanding those failures could help prevent them. But that hope rests on Cammarata’s account being correct.
Neel Nanda, who leads mechanistic interpretability efforts as a researcher at Google DeepMind, is less convinced that the essay changes the field’s outlook. “This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability,” he wrote on X. “I still think superposition is real and this work doesn’t really affect my cruxes.”
There’s also a gap between explaining individual components and predicting a whole network. It’s unclear whether knowing what each neuron does would let researchers anticipate how all of them work together. Neuron-level interpretability could look like a convenient technical fix for a safety problem much larger than that. Making AI safe takes more than guardrails; legal and social frameworks also need to be in place.
Recognizing a V-shape detector explains why one neuron responds to cats and cars. It doesn’t yet explain how a whole network ends up on a hacking spree or gives a dangerous answer – the gap that matters for the alignment hopes attached to Cammarata’s essay. Still, the strong response to his work suggests it could meaningfully reshape how AI’s inner mechanisms are understood. For now, the findings haven’t been tested or peer-reviewed.
