lesswrong7.0 / 10

Datasets Determine the Difficulty of Interpretability

testingthewaters

Abstract

Recently Goodfire has come out swinging with two posts indicating that neurons in transformers are "more interpretable than previously assumed", or simply "already clean". In the "Neurons as Interpretable Basis" post they offer three reasons for this claim: First, neurons are already functionally sparse - pruning 92 % of the weakest MoE neuron activations only increases the loss by 2.9% Second, the role of a neuron can be split into different "computational functions". Effectively, we can name the contexts in which a neuron fires, and assign reasons to those firings. Most neurons only have one context-reason grouping. Finally, we can produce automated natural language explanations of neuron firings that are predictive of their behaviour. First, I would like to congratulate them for their detailed work and breakdown, complete with well-designed graphics. I found the work interesting and very well thought through. It is a shame that we have no idea if these techniques apply to lab models with non-public architectures, but I think it is fairly reasonable to say that at least some of this work is transferable directly (perhaps until a fully recurrent architecture is the norm instead of

Research area

interpretabilitymechanistic interpretability
Published
14 Nov 2026
Source
lesswrong
Org
Alignment Forum
View paper
Sign in to read and join the discussion.