Tracing Subliminal Learning Through Pretraining

Abstract

This post summarizes our paper , Tracing Subliminal Learning Through Pretraining, which was developed as part of EleutherAI’s SOAR program . TL;DR We study subliminal learning , the phenomenon in which a student model acquires a teacher’s preference through training data that appear semantically unrelated to that preference. Our work investigates subliminal learning through cross-model distillation , using the PolyPythia and DataDecide OLMo models to show that a teacher’s animal preference transfers to students with different weight initializations, pretraining data orders, training corpora, model sizes, and architectures. We examine two distillation settings: hard-label number distillation , the classic setting where the student trains on number sequences generated by the teacher, and soft-label on-policy distillation , where the student generates text and learns from the teacher’s next-token probability distributions on that text. Our main findings are: Shared initialization and pretraining conditions are not required. Cross-model transfer can survive differences in initialization, data order, pretraining corpus, model size, and model architecture between teacher and student. Tra

Research area