Model's "emotions" noticeably change when they take risky actions
jain18ayush
Abstract
Executive summary We extend Anthropic's emotions vector work and observe strong evidence that a model’s internal representations of certain emotions change when it takes misaligned actions in two scenarios: reward hacking in code generation and specification gaming in a sales environment. We tested two approaches: a) measure drift in projection along emotion vectors across the trajectory, and b) ask the model to self-report its emotions. When the model takes some risky action, there is a substantial increase in the emotion signal for guilt, anxiety, hostility, and sadness. Steering these emotions causally affects the model's reward-hacking rate. Moreover, monitoring emotions can provide an early signal of whether the model will take a risky action several sentences later. Setting an emotion-resistant persona (by crafting a custom system prompt) makes the model less likely to take risky action. This work is from a focused sprint; we do not anticipate working on it for now but would like to share it with others since our results were interesting. Code: https://github.com/gutenbergpbc/emotion-vectors Introduction Agents run long, multi-step tasks with little human oversight; we need t