Publications

MobileProbe: Adaptive Security Evaluation of Mobile Agents against Trajectory-Aware Injection

NDSS 2027

Yingbin Jin, Ji'an Wang, HaoYang Chai, Hanjun Luo, Li Hu, Xinfeng Li, Huadi Zheng, PeiZhao Hu, Qingqing Ye, Haibo Hu.

Contribution: We introduced a trajectory-aware red-teaming framework that adaptively discovers attacker-controllable content surfaces, localizes vulnerable decision points along agent trajectories, and evolves context-aware environmental-injection payloads for mobile GUI agents. We also constructed a benchmark of 83 long-horizon attack instances across 12 mobile applications covering confidentiality, integrity, and availability threats.

CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding

ICML 2026

Hanjun Luo, Chiming Ni, Jiaheng Wen, Zhimu Huang, Yiran Wang, Bingduo Liao, Sylvia Chung, Yingbin Jin, Xinfeng Li, Wenyuan Xu, XiaoFeng Wang, Hanan Salam.

Contribution: We introduced a collaborative coding evaluation setup to quantify human-AI synergy end to end. Human-AI coding quality should be assessed through joint performance rather than isolated model generations.

AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters

ICML 2026

Hanjun Luo, Zhimu Huang, Sylvia Chung, Yiran Wang, Yingbin Jin, Jialin Li, Jiang Li, Xinfeng Li, Hanan Salam.

Contribution: We introduced a unified benchmark for evaluating humans and MLLMs as text-to-image prompters, with 360 expert-crafted tasks and an agentic evaluator that measures prompting proficiency beyond fixed-prompt T2I model evaluation. The results show that imitation-based prompting can outperform pure planning under complex constraints, suggesting image-augmented prompting as a stronger direction for future prompters.

PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction

ACL 2026 Findings

Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam.

Contribution: We designed a framework that infers and updates user preference signals during interaction. Preference adaptation should be modeled continuously during multi-turn human-agent collaboration.

AudioStealer: Extracting Audio Prompts via Shapley Value-Guided Query Search

ACL 2026 Findings

Yingbin Jin, Xingjian Du, Hanjun Luo, Zihao Wang, Haibo Hu, XiaoFeng Wang, Xinfeng Li.

Contribution: We introduced AudioStealer, a two-stage black-box prompt inversion attack for text-to-music models using embedding-guided candidate search and Shapley value attribution. The recovered prompts and regenerated audio expose concrete IP leakage risks in generative audio systems.

BiasIG: Benchmarking Multi-dimensional Social Biases in Text-to-Image Models

IJCNN 2026 Oral

Hanjun Luo, Zhimu Huang, Haoyu Huang, Ziye Deng, Ruizhe Chen, Xinfeng Li, Zuozhu Liu, Hanan Salam.

Contribution: We introduced BiasIG, a benchmark with 47,040 prompts and a four-dimensional taxonomy for diagnosing social bias in text-to-image models. Results across models and debiasing methods show that debiasing interventions can introduce unintended confounding effects on unrelated demographic groups.

AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models

ICLR 2026

Kai Li, Can Shen, Yile Liu, Jirui Han, Kelong Zheng, Xuechao Zou, Zhe Wang, Shun Zhang, Xingjian Du, Hanjun Luo, Yingbin Jin, Xinxin Xing, Ziyang Ma, Yue Liu, Yifan Zhang, Junfeng Fang, Kun Wang, Yibo Yan, Gelei Deng, Haoyang Li, Yiming Li, Xiaobin Zhuang, Tianlong Chen, Qingsong Wen, Tianwei Zhang, Yang Liu, Haibo Hu, Zhizheng Wu, Xiaolin Hu, Eng-Siong Chng, Wenyuan Xu, XiaoFeng Wang, Wei Dong, Xinfeng Li.

Contribution: We built a multifaceted benchmark and evaluation pipeline to measure reliability, safety, and robustness consistently for audio LLMs. Trustworthy audio LLM evaluation should jointly cover these dimensions instead of testing only one axis.

Agentauditor: Human-level Safety and Security Evaluation for LLM Agents

NeurIPS 2025

Hanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li, Guibin Zhang, Kun Wang, Tongliang Liu, Hanan Salam.

Contribution: We proposed AgentAuditor to test realistic agent behaviors across safety and security risk scenarios. Agent safety must be evaluated as an end-to-end system, not only by isolated model outputs.

DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognition

EMNLP 2025 Main

Hanjun Luo, Yingbin Jin, Yiran Wang, Xinfeng Li, Tong Shang, Xuecheng Liu, Ruizhe Chen, Kun Wang, Hanan Salam, Qingsong Wen, Zuozhu Liu.

Contribution: We built DynamicNER with new data construction and evaluation protocols tailored to LLM-based NER. NER evaluation for LLMs benefits from dynamic, multilingual, and fine-grained settings beyond static benchmarks.

A Comprehensive Survey in LLM(-agent) Full Stack Safety: Data, Training and Deployment

arXiv 2025

Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, Liang Lin, Zhihao Xu, Haolang Lu, Xinye Cao, Xinyun Zhou, Weifei Jin, Fanci Meng, Shicheng Xu, Junyuan Mao, Yu Wang, Hao Wu, Minghe Wang, Fan Zhang, Junfeng Fang, Wenjie Qu, Yue Liu, Chengwei Liu, Yifan Zhang, Qiankun Li, Chongye Guo, Yalan Qin, Zhaoxin Fan, Kai Wang, Yi Ding, Donghai Hong, Jiaming Ji, Yingxin Lai, Zitong Yu, Xinfeng Li, Yifan Jiang, Yanhui Li, Xinyu Deng, Junlin Wu, Dongxia Wang, Yihao Huang, Yufei Guo, Jen-tse Huang, Qiufeng Wang, Xiaolong Jin, Wenxuan Wang, Dongrui Liu, Yanwei Yue, Wenke Huang, Guancheng Wan, Heng Chang, Tianlin Li, Yi Yu, Chenghao Li, Jiawei Li, Lei Bai, Jie Zhang, Qing Guo, Jingyi Wang, Tianlong Chen, Joey Tianyi Zhou, Xiaojun Jia, Weisong Sun, Cong Wu, Jing Chen, Xuming Hu, Yiming Li, Xiao Wang, Ningyu Zhang, Luu Anh Tuan, Guowen Xu, Jiaheng Zhang, Tianwei Zhang, Xingjun Ma, Jindong Gu, Liang Pang, Xiang Wang, Bo An, Jun Sun, Mohit Bansal, Shirui Pan, Lingjuan Lyu, Yuval Elovici, Bhavya Kailkhura, Yaodong Yang, Hongwei Li, Wenyuan Xu, Yizhou Sun, Wei Wang, Qing Li, Ke Tang, Yu-Gang Jiang, Felix Juefei-Xu, Hui Xiong, Xiaofeng Wang, Dacheng Tao, Philip S Yu, Qingsong Wen, Yang Liu.

Contribution: We organized a full-stack taxonomy and synthesized mitigation strategies across the LLM/agent lifecycle. Safety risks propagate across the whole stack, so mitigation must align data, training, and deployment stages.

BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models

arXiv 2024

Hanjun Luo, Haoyu Huang, Ziye Deng, Xinfeng Li, Hewei Wang, Yingbin Jin, Yang Liu, Wenyuan Xu, Zuozhu Liu.

Contribution: We built BIGbench to provide a unified dataset and metric suite for multi-dimensional bias evaluation. Bias in text-to-image systems is multi-dimensional and needs unified metrics instead of single-axis checks.

FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models

ICML@DMLR 2024

Hanjun Luo, Ziye Deng, Ruizhe Chen, Zuozhu Liu.

Contribution: We developed FAIntbench with a holistic task design and more precise bias assessment criteria. Fine-grained bias auditing needs both semantic coverage and precise scoring protocols to avoid misleading conclusions.

VersusDebias: Universal Zero-shot Debiasing for Text-to-Image Models via SLM-based Prompt Engineering and Generative Adversary

arXiv 2024

Hanjun Luo, Ziye Deng, Haoyu Huang, Xuecheng Liu, Ruizhe Chen, Zuozhu Liu.

Contribution: We proposed VersusDebias, an SLM-guided prompting and adversarial strategy for practical debiasing. Zero-shot debiasing can be generalized with adaptive prompting without retraining diffusion backbones.

Uniap: Towards Universal Animal Perception in Vision via Few-shot Learning

AAAI 2024

Meiqi Sun, Zhonghan Zhao, Wenhao Chai, Hanjun Luo, Shidong Cao, Yanting Zhang, Jenq-Neng Hwang, Gaoang Wang.

Contribution: We implemented and validated a universal animal perception framework across diverse vision benchmarks. Few-shot transfer can unify multiple animal perception tasks under one scalable vision framework.