VisualJevBench: A Benchmark and Black-Box Attacks for Visual Typed Decision Models
Li Zeng, Yeqi Zeng, Hengkai Liang, Shiguo Wang, Meng Xie, Zeyu Ye, Jinheng Zhang, Xianlong Wang, Ziqi Zhou, Hangtao Zhang
Abstract
Typed decision models (TDMs) return structured decisions instead of generating free-text responses, enabling fast and cheap decisions in real-world applications. Recent support for image inputs extends Jev, a representative TDM, to visual decision-making. We refer to this setting as Visual Jev. However, red teaming of Visual Jev leaves two gaps: a systematic evaluation of direct image manipulations is lacking, and universal visual attacks remain underexplored. To address these gaps, we introduce VisualJevBench, a benchmark for evaluating the adversarial robustness of Visual Jev. It comprises 800 images and 3,200 questions across eight scenarios and four visual attack surfaces: typographic manipulation, visual evidence spoofing, target selection interference, and geometric transformation. We also introduce Adaptive Black-Box Parameter Search (ABPS), a lightweight framework that uses typed-output feedback to find effective image manipulations. ABPS supports both image-specific attacks and universal attacks that transfer to unseen images and tasks in new scenarios without further optimization. Extensive experiments on jev-1.14.0 show that image-specific attacks achieve an attack success rate (ASR) of up to 69.6%, while universal attacks achieve an average ASR of 33.15% on unseen images. Geometric transformations achieve an average ASR of 40.09% in cross-scenario transfer.