VPD's Evaluation Adversary Finds Systematic Defects in the Released Decomposition
Tom Angsten
Abstract
TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and assigns labels that specify whether a component is needed for a given input. A core aspiration of VPD is robust mechanistic faithfulness - the property that switching off any set of unneeded components for a given input leaves the model's output approximately unchanged. The authors use an evaluation adversary to measure the degree of robustness, and report that the decomposition is not at all robust to very long adversarial searches. The authors list this as a limitation, but also argue that complete robustness is not necessarily the goal. Their discussion implies two conditions under which a lack of robustness at large step counts may not matter in practice: 1) the adversary could be finding defects in the decomposition that are tuned to particular inputs, rather than systematically carrying over to other inputs, or 2) the masks a practitioner might apply may be very different from adversarially found masks. To test the first condition, we ran a series of measurements on the paper's released decomposition of a four-layer, 67M-parameter language model and found that from 20 steps on