Inoculate or Reflect? Two training interventions under prompting, steering, and patching
Ayesha Imran
Abstract
Anthropic's recent paper, Verbalizable Representations Form a Global Workspace in Language Models , contains a small experiment near the end that we found more interesting than the main findings. The technique is called Counterfactual Reflection Training (CRT) . The model is fed a partial transcript in its context window, followed by an interruption with a question about what matters in that situation, and is trained only on its answer to that question. It is never trained on a corrected action in the original context. At test time, the interruption and reflection are removed, but the model's behavior still changes. This seemed quite similar to Inoculation Prompting (IP), introduced some time back by Tan et al. and Wichers et al. . With IP, an instruction that explicitly asks for an unwanted behavior is added during fine-tuning. The original training targets stay the same. The instruction is then removed at test time, which can stop the unwanted behavior from becoming the model's default. Both techniques change the context around a training signal rather than replacing the original response with a clean one. CRT asks the model to articulate a better principle after seeing the situa