Unsupervised Feature Discovery via Simple Clustering
enricobottazzi
Abstract
This article was submitted as part of Neel Nanda's MATS Application Abstract . This article examines whether clustering can discover features. We define a feature as “a direction in activation space associated with a property that is both interpretable to humans and useful to the model”. Clustering is done on activations from Qwen2.5-7B-Instruct (layer 20) via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly split with k=2 until 2048 (2^11) leaves are obtained. Qualitative analysis shows interpretable clusters from the very first level (64 clusters) organized by lexical, syntactic, and semantic categories. Next, we run an "actual vs. decoy" distinguishability evaluation. An LLM judge can distinguish an actual cluster from a decoy cluster ≈97% of the time. The same judge can distinguish an actual SAE feature from a decoy one only ≈79% of the time. Lastly, we run an activation patching intervention by replacing the actual activation with a cluster centroid. Results show that, after patching, the model behaves as if the last token was the one correlated with the cluster centroid and continues th