Gradient Routing Generalizes to Unrouted Hack Types
jacob_drori
Abstract
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand. Introduction Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is: Initialize a retain and a forget adapter. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much. I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here , with the best visual expla