How to Open Them Up – Part I

Abstract

TL;DR We suggest an approach to systematization of the mechanistic interpretability research field, which is tailored to our own research goals and tasks. We identified four main tasks we must solve in order to properly explore one chosen concept and its representations inside LLMs: finding the concept’s representation; establishing its causal role in an LLM’s behavior; establishing its necessity; steering the concept's representation in order to change an LLM’s behavior. In this post we explore approaches and methods applicable to task 1 – finding the concept. We analyzed the available literature and decided to focus on the following ones: linear probes; difference-in-means; sparse autoencoders (SAEs); PCA & clustering. Each has advantages and limitations, and we shared our own experiences and thoughts. In summary, linear probes and difference-in-means , which are somewhat similar, are fairly cheap in terms of computational resources needed, but they’ve been criticised for the lack of precision as well as for the lack of a mechanism that would highlight causal relationships between concepts / concepts and behaviors. To analyze causality, researchers suggested SAEs . They seem to b

Research area